2023-01-24T03:20:50.925 INFO:root:teuthology version: 0.0.1.dev20+g5abecab 2023-01-24T03:20:50.926 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2023-01-23_22:48:43-fs-wip-yuri5-testing-2023-01-23-0928-quincy-distro-default-smithi --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} --archive /home/teuthworker/archive/yuriw-2023-01-23_22:48:43-fs-wip-yuri5-testing-2023-01-23-0928-quincy-distro-default-smithi/7134568 --owner scheduled_yuriw@teuthology --verbose -- /home/teuthworker/archive/yuriw-2023-01-23_22:48:43-fs-wip-yuri5-testing-2023-01-23-0928-quincy-distro-default-smithi/7134568/orig.config.yaml 2023-01-24T03:20:50.976 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2023-01-24T03:20:51.050 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-01-23_22:48:43-fs-wip-yuri5-testing-2023-01-23-0928-quincy-distro-default-smithi/7134568 branch: wip-yuri5-testing-2023-01-23-0928-quincy description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} email: null first_in_suite: false job_id: '7134568' kernel: client: sha1: distro kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2023-01-23_22:48:43-fs-wip-yuri5-testing-2023-01-23-0928-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.4' overrides: admin_socket: branch: wip-yuri5-testing-2023-01-23-0928-quincy ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd session_timeout: 300 conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds oft prefetch dirfrags: false mds op complaint time: 180 mds verify scatter: true mds_sleep_rank_change: 5000000.0 mon client directed command retry: 5 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds osd ms inject socket failures: 2500 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon client directed command retry: 5 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds ms inject socket failures: 2500 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - Replacing daemon mds sha1: 8387d427d91a301da19bd82e090b96e1348bfadb ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 8387d427d91a301da19bd82e090b96e1348bfadb kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri5-testing-2023-01-23-0928-quincy sha1: 8387d427d91a301da19bd82e090b96e1348bfadb owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 sha1: 8387d427d91a301da19bd82e090b96e1348bfadb sleep_before_teardown: 0 suite: fs suite_branch: wip-yuri5-testing-2023-01-23-0928-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_8387d427d91a301da19bd82e090b96e1348bfadb/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 8387d427d91a301da19bd82e090b96e1348bfadb targets: smithi112.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCi3HiWd6/sHDt8B4an31nK9OwwYDIAHLkfRs+K5B9seNKgOmdbng4qjJ/cHzHc8OxFIT3OwsNobN9VNcvxuLnq+MWWiBusVCPRnBnicqWunqzrKAHR6iTW5bg0dukS4kePZTedc9qbncPg0HbIfqzpP/acBMWwtLlXak3OrB8FxDC9et4N7RGOwhCtPHtpVRL5nA3VIWoRK4YRF7IH6LgefSSP/gdnL0qq7tL+Se9hNb/9ikcL6G/Xc6qRFfpcjhOcksMQZ8/yU4ageuzQrWDebSDakzasTBdpCDErWKU8FUIplyUpdC0S+6G1kvR8RoJEwY25Ii1X9eOLJSLOklWswMSxc5+3kRINqN/Ti1W8yjuxnIls3hJuPun774BeG0wMYnt75SdyMuReYAaNMcB6jG/drTl62uogQEMU0IwjcwEk29ev1It/rF1OTtg5Xvb+cfGzDb2uXI53IHplNj/OMNDVekuhBNuk8C3ojYV5ezaBBPgdPf/KPLWseCziVNM= smithi142.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDc9l7T/84L4XRGIvVuKHrzVvHDt6pX7G2I+pxnBYcmn5mgh5TUbkwQaXJbSjj7aI0rCfUBOGzHLYcuRciAv8FXCUWcazv7WD6lecaRdqHxxXjoDnz5caE0jXz8no6DqagbGfkjlfFVcXgsTueLc8Z5uABvFIZ95rPuiXYoWlhj/SmCDM9UUAcUfAyj9Vjcw2/yWpCTq3otEq0qx/OVXkjPD6Gd1cZ3PhUt8euiajxdNp1YdAdVYYuWn2FDYNzo2p8vlpG7Kk2iGS8D9/WEv2ascTMVvsiqphG9zG2U1cRQZLG6+rQTe6T8uUuK9AuoUqpCunGzY2UQmN8SRSUb/8AdqEliHi69qBDmUdCbH1NbCgG6lp9UbGGsnIhyiguVKIJkM6WNErjcUAP0H7LDG48pPRMFyPfsTyDrfO/t1WHWezBJz0T/Bpr6QLOA7iyQI8F5S3e85IN5phnLmQ8Hr9MQyQsMtzO8gVSPmJL67bNRgptQnJk5LgYbbMINPaI+oFM= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - ceph: null - kclient: null - mds_thrash: null - workunit: clients: all: - fs/snaps teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 5abecab36c9da4d70461d81781464cf5acaea292 timestamp: 2023-01-23_22:48:43 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.17735 2023-01-24T03:20:51.050 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_8387d427d91a301da19bd82e090b96e1348bfadb/qa; will attempt to use it 2023-01-24T03:20:51.051 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_8387d427d91a301da19bd82e090b96e1348bfadb/qa/tasks 2023-01-24T03:20:51.052 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-01-24T03:20:51.059 INFO:teuthology.task.internal:Checking packages... 2023-01-24T03:20:51.089 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '8387d427d91a301da19bd82e090b96e1348bfadb' 2023-01-24T03:20:51.089 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-01-24T03:20:51.089 INFO:teuthology.packaging:ref: None 2023-01-24T03:20:51.090 INFO:teuthology.packaging:tag: None 2023-01-24T03:20:51.090 INFO:teuthology.packaging:branch: wip-yuri5-testing-2023-01-23-0928-quincy 2023-01-24T03:20:51.090 INFO:teuthology.packaging:sha1: 8387d427d91a301da19bd82e090b96e1348bfadb 2023-01-24T03:20:51.091 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri5-testing-2023-01-23-0928-quincy 2023-01-24T03:20:51.243 INFO:teuthology.task.internal:Found packages for ceph version 17.2.5-701.g8387d427 2023-01-24T03:20:51.244 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-01-24T03:20:51.251 INFO:teuthology.task.internal:no buildpackages task found 2023-01-24T03:20:51.251 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-01-24T03:20:51.296 INFO:teuthology.task.internal:Saving configuration 2023-01-24T03:20:51.316 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-01-24T03:20:51.324 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-24T03:20:51.347 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi112.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-01-23_22:48:43-fs-wip-yuri5-testing-2023-01-23-0928-quincy-distro-default-smithi/7134568', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-01-24 03:15:58.882254', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBw6vu8T0kWm3rMDm1s64uBgEUKTaK9AT3NxtUEzIvQ7I5UCvgnn45zQfPOdm2ZZVd2YLm1wybkEzpj3CPXJUCg='} 2023-01-24T03:20:51.370 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi142.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-01-23_22:48:43-fs-wip-yuri5-testing-2023-01-23-0928-quincy-distro-default-smithi/7134568', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-01-24 03:15:58.880139', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHt4+dzmytKmwSWjRnubTVHEFD/2tIuSMNrcblxR8ntPWmJa43U8fWfkqAaNPY5NcxuxRYJCMz0S3G30HunpHD8='} 2023-01-24T03:20:51.371 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-01-24T03:20:51.379 INFO:teuthology.task.internal:roles: ubuntu@smithi112.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2023-01-24T03:20:51.379 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-01-24T03:20:51.379 INFO:teuthology.run_tasks:Running task console_log... 2023-01-24T03:20:51.464 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fdc9792a700>, signals=[15]) 2023-01-24T03:20:51.465 INFO:teuthology.run_tasks:Running task internal.connect... 2023-01-24T03:20:51.475 INFO:teuthology.task.internal:Opening connections... 2023-01-24T03:20:51.475 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi112.front.sepia.ceph.com 2023-01-24T03:20:51.477 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T03:20:51.545 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi142.front.sepia.ceph.com 2023-01-24T03:20:51.546 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T03:20:51.614 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-01-24T03:20:51.620 DEBUG:teuthology.orchestra.run.smithi112:> uname -m 2023-01-24T03:20:51.638 INFO:teuthology.orchestra.run.smithi112.stdout:x86_64 2023-01-24T03:20:51.638 DEBUG:teuthology.orchestra.run.smithi112:> cat /etc/os-release 2023-01-24T03:20:51.694 INFO:teuthology.orchestra.run.smithi112.stdout:NAME="Red Hat Enterprise Linux" 2023-01-24T03:20:51.695 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION="8.4 (Ootpa)" 2023-01-24T03:20:51.695 INFO:teuthology.orchestra.run.smithi112.stdout:ID="rhel" 2023-01-24T03:20:51.695 INFO:teuthology.orchestra.run.smithi112.stdout:ID_LIKE="fedora" 2023-01-24T03:20:51.695 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION_ID="8.4" 2023-01-24T03:20:51.696 INFO:teuthology.orchestra.run.smithi112.stdout:PLATFORM_ID="platform:el8" 2023-01-24T03:20:51.696 INFO:teuthology.orchestra.run.smithi112.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-01-24T03:20:51.696 INFO:teuthology.orchestra.run.smithi112.stdout:ANSI_COLOR="0;31" 2023-01-24T03:20:51.696 INFO:teuthology.orchestra.run.smithi112.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-01-24T03:20:51.697 INFO:teuthology.orchestra.run.smithi112.stdout:HOME_URL="https://www.redhat.com/" 2023-01-24T03:20:51.697 INFO:teuthology.orchestra.run.smithi112.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-01-24T03:20:51.697 INFO:teuthology.orchestra.run.smithi112.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-01-24T03:20:51.697 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:20:51.698 INFO:teuthology.orchestra.run.smithi112.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-01-24T03:20:51.698 INFO:teuthology.orchestra.run.smithi112.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-01-24T03:20:51.698 INFO:teuthology.orchestra.run.smithi112.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-01-24T03:20:51.698 INFO:teuthology.orchestra.run.smithi112.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-01-24T03:20:51.699 INFO:teuthology.lock.ops:Updating smithi112.front.sepia.ceph.com on lock server 2023-01-24T03:20:51.716 DEBUG:teuthology.orchestra.run.smithi142:> uname -m 2023-01-24T03:20:51.733 INFO:teuthology.orchestra.run.smithi142.stdout:x86_64 2023-01-24T03:20:51.733 DEBUG:teuthology.orchestra.run.smithi142:> cat /etc/os-release 2023-01-24T03:20:51.789 INFO:teuthology.orchestra.run.smithi142.stdout:NAME="Red Hat Enterprise Linux" 2023-01-24T03:20:51.789 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION="8.4 (Ootpa)" 2023-01-24T03:20:51.789 INFO:teuthology.orchestra.run.smithi142.stdout:ID="rhel" 2023-01-24T03:20:51.790 INFO:teuthology.orchestra.run.smithi142.stdout:ID_LIKE="fedora" 2023-01-24T03:20:51.790 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_ID="8.4" 2023-01-24T03:20:51.790 INFO:teuthology.orchestra.run.smithi142.stdout:PLATFORM_ID="platform:el8" 2023-01-24T03:20:51.790 INFO:teuthology.orchestra.run.smithi142.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-01-24T03:20:51.790 INFO:teuthology.orchestra.run.smithi142.stdout:ANSI_COLOR="0;31" 2023-01-24T03:20:51.791 INFO:teuthology.orchestra.run.smithi142.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-01-24T03:20:51.791 INFO:teuthology.orchestra.run.smithi142.stdout:HOME_URL="https://www.redhat.com/" 2023-01-24T03:20:51.791 INFO:teuthology.orchestra.run.smithi142.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-01-24T03:20:51.791 INFO:teuthology.orchestra.run.smithi142.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-01-24T03:20:51.792 INFO:teuthology.orchestra.run.smithi142.stdout: 2023-01-24T03:20:51.792 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-01-24T03:20:51.792 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-01-24T03:20:51.792 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-01-24T03:20:51.792 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-01-24T03:20:51.793 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server 2023-01-24T03:20:51.808 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-01-24T03:20:51.818 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-01-24T03:20:51.842 INFO:teuthology.task.internal:Checking for old test directory... 2023-01-24T03:20:51.842 DEBUG:teuthology.orchestra.run.smithi112:> test '!' -e /home/ubuntu/cephtest 2023-01-24T03:20:51.845 DEBUG:teuthology.orchestra.run.smithi142:> test '!' -e /home/ubuntu/cephtest 2023-01-24T03:20:51.860 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-01-24T03:20:51.870 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-01-24T03:20:51.870 DEBUG:teuthology.orchestra.run.smithi112:> test -z $(ls -A /var/lib/ceph) 2023-01-24T03:20:51.902 DEBUG:teuthology.orchestra.run.smithi142:> test -z $(ls -A /var/lib/ceph) 2023-01-24T03:20:51.915 INFO:teuthology.orchestra.run.smithi112.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-01-24T03:20:51.917 INFO:teuthology.orchestra.run.smithi142.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-01-24T03:20:51.918 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-01-24T03:20:51.968 INFO:teuthology.run_tasks:Running task kernel... 2023-01-24T03:20:51.984 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2023-01-24T03:20:51.985 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2023-01-24T03:20:51.985 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2023-01-24T03:20:51.985 DEBUG:teuthology.orchestra.run.smithi112:> test -f /run/.containerenv -o -f /.dockerenv 2023-01-24T03:20:52.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T03:20:52.001 DEBUG:teuthology.orchestra.run.smithi112:> uname -r 2023-01-24T03:20:52.056 INFO:teuthology.orchestra.run.smithi112.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-01-24T03:20:52.057 INFO:teuthology.task.kernel:Running kernel on smithi112: 4.18.0-305.25.1.el8_4.x86_64 2023-01-24T03:20:52.057 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum install -y kernel 2023-01-24T03:20:53.485 INFO:teuthology.orchestra.run.smithi112.stdout:Updating Subscription Management repositories. 2023-01-24T03:20:53.486 INFO:teuthology.orchestra.run.smithi112.stdout:Unable to read consumer identity 2023-01-24T03:20:54.856 INFO:teuthology.orchestra.run.smithi112.stdout:Copr repo for python3-asyncssh owned by ceph 33 kB/s | 3.5 kB 00:00 2023-01-24T03:20:55.712 INFO:teuthology.orchestra.run.smithi112.stdout:Extra Packages for Enterprise Linux 16 MB/s | 13 MB 00:00 2023-01-24T03:20:58.536 INFO:teuthology.orchestra.run.smithi112.stdout:lab-extras 4.4 MB/s | 24 kB 00:00 2023-01-24T03:20:59.167 INFO:teuthology.orchestra.run.smithi112.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-01-24T03:20:59.167 INFO:teuthology.orchestra.run.smithi112.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-01-24T03:20:59.168 INFO:teuthology.orchestra.run.smithi112.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-01-24T03:20:59.308 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-01-24T03:20:59.309 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2023-01-24T03:20:59.309 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-01-24T03:20:59.412 DEBUG:teuthology.orchestra.run.smithi112:> echo no | sudo yum reinstall kernel || true 2023-01-24T03:20:59.794 INFO:teuthology.orchestra.run.smithi112.stdout:Updating Subscription Management repositories. 2023-01-24T03:20:59.794 INFO:teuthology.orchestra.run.smithi112.stdout:Unable to read consumer identity 2023-01-24T03:21:00.071 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:00:02 ago on Tue 24 Jan 2023 03:20:58 AM UTC. 2023-01-24T03:21:00.198 INFO:teuthology.orchestra.run.smithi112.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-01-24T03:21:00.198 INFO:teuthology.orchestra.run.smithi112.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-01-24T03:21:00.198 INFO:teuthology.orchestra.run.smithi112.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-01-24T03:21:00.219 INFO:teuthology.orchestra.run.smithi112.stderr:Error: No packages marked for reinstall. 2023-01-24T03:21:00.268 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum reinstall -y kernel || true 2023-01-24T03:21:00.609 INFO:teuthology.orchestra.run.smithi112.stdout:Updating Subscription Management repositories. 2023-01-24T03:21:00.609 INFO:teuthology.orchestra.run.smithi112.stdout:Unable to read consumer identity 2023-01-24T03:21:00.881 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:00:02 ago on Tue 24 Jan 2023 03:20:58 AM UTC. 2023-01-24T03:21:01.005 INFO:teuthology.orchestra.run.smithi112.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-01-24T03:21:01.005 INFO:teuthology.orchestra.run.smithi112.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-01-24T03:21:01.005 INFO:teuthology.orchestra.run.smithi112.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-01-24T03:21:01.017 INFO:teuthology.orchestra.run.smithi112.stderr:Error: No packages marked for reinstall. 2023-01-24T03:21:01.072 DEBUG:teuthology.orchestra.run.smithi112:> rpm -q kernel --last | head -n 1 2023-01-24T03:21:01.194 INFO:teuthology.orchestra.run.smithi112.stdout:kernel-4.18.0-305.25.1.el8_4.x86_64 Wed 21 Dec 2022 01:05:24 PM UTC 2023-01-24T03:21:01.196 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-305.25.1.el8_4.x86_64 2023-01-24T03:21:01.196 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-01-24T03:21:01.196 DEBUG:teuthology.parallel:result is None 2023-01-24T03:21:01.197 INFO:teuthology.run_tasks:Running task internal.base... 2023-01-24T03:21:01.205 INFO:teuthology.task.internal:Creating test directory... 2023-01-24T03:21:01.206 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-01-24T03:21:01.208 DEBUG:teuthology.orchestra.run.smithi142:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-01-24T03:21:01.226 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-01-24T03:21:01.255 INFO:teuthology.run_tasks:Running task internal.archive... 2023-01-24T03:21:01.277 INFO:teuthology.task.internal:Creating archive directory... 2023-01-24T03:21:01.277 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-01-24T03:21:01.280 DEBUG:teuthology.orchestra.run.smithi142:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-01-24T03:21:01.315 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-01-24T03:21:01.326 INFO:teuthology.task.internal:Enabling coredump saving... 2023-01-24T03:21:01.327 DEBUG:teuthology.orchestra.run.smithi142:> test -f /run/.containerenv -o -f /.dockerenv 2023-01-24T03:21:01.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T03:21:01.366 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-01-24T03:21:01.369 DEBUG:teuthology.orchestra.run.smithi142:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-01-24T03:21:01.394 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-24T03:21:01.408 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-24T03:21:01.467 INFO:teuthology.orchestra.run.smithi142.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-24T03:21:01.483 INFO:teuthology.orchestra.run.smithi142.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-24T03:21:01.485 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-01-24T03:21:01.496 INFO:teuthology.task.internal:Configuring sudo... 2023-01-24T03:21:01.496 DEBUG:teuthology.orchestra.run.smithi112:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-01-24T03:21:01.499 DEBUG:teuthology.orchestra.run.smithi142:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-01-24T03:21:01.524 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-01-24T03:21:01.535 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-01-24T03:21:01.536 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-01-24T03:21:01.563 DEBUG:teuthology.orchestra.run.smithi142:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-01-24T03:21:01.579 DEBUG:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-01-24T03:21:01.652 DEBUG:teuthology.orchestra.run.smithi112:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-01-24T03:21:01.699 DEBUG:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-01-24T03:21:01.778 DEBUG:teuthology.orchestra.run.smithi112:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-01-24T03:21:01.818 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-01-24T03:21:01.818 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-01-24T03:21:01.881 DEBUG:teuthology.orchestra.run.smithi142:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-01-24T03:21:01.925 DEBUG:teuthology.orchestra.run.smithi142:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-01-24T03:21:01.967 DEBUG:teuthology.orchestra.run.smithi142:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-01-24T03:21:02.044 DEBUG:teuthology.orchestra.run.smithi142:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-01-24T03:21:02.089 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2023-01-24T03:21:02.090 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-01-24T03:21:02.154 DEBUG:teuthology.orchestra.run.smithi112:> sudo service rsyslog restart 2023-01-24T03:21:02.157 DEBUG:teuthology.orchestra.run.smithi142:> sudo service rsyslog restart 2023-01-24T03:21:02.201 INFO:teuthology.orchestra.run.smithi112.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-01-24T03:21:02.300 INFO:teuthology.orchestra.run.smithi142.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-01-24T03:21:02.721 INFO:teuthology.run_tasks:Running task internal.timer... 2023-01-24T03:21:02.733 INFO:teuthology.task.internal:Starting timer... 2023-01-24T03:21:02.733 INFO:teuthology.run_tasks:Running task pcp... 2023-01-24T03:21:02.783 INFO:teuthology.run_tasks:Running task selinux... 2023-01-24T03:21:02.804 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']} 2023-01-24T03:21:02.804 DEBUG:teuthology.orchestra.run.smithi112:> sudo service auditd rotate 2023-01-24T03:21:02.849 INFO:teuthology.orchestra.run.smithi112.stdout:Rotating logs: [ OK ] 2023-01-24T03:21:02.851 DEBUG:teuthology.orchestra.run.smithi142:> sudo service auditd rotate 2023-01-24T03:21:02.909 INFO:teuthology.orchestra.run.smithi142.stdout:Rotating logs: [ OK ] 2023-01-24T03:21:02.911 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-01-24T03:21:02.911 DEBUG:teuthology.orchestra.run.smithi112:> /usr/sbin/getenforce 2023-01-24T03:21:02.952 INFO:teuthology.orchestra.run.smithi112.stdout:Permissive 2023-01-24T03:21:02.953 DEBUG:teuthology.orchestra.run.smithi142:> /usr/sbin/getenforce 2023-01-24T03:21:02.982 INFO:teuthology.orchestra.run.smithi142.stdout:Permissive 2023-01-24T03:21:02.982 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi112.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi142.front.sepia.ceph.com': 'permissive'} 2023-01-24T03:21:02.983 DEBUG:teuthology.orchestra.run.smithi112:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2023-01-24T03:21:03.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T03:21:03.017 DEBUG:teuthology.orchestra.run.smithi142:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2023-01-24T03:21:03.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T03:21:03.048 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-01-24T03:21:03.049 DEBUG:teuthology.orchestra.run.smithi112:> sudo /usr/sbin/setenforce permissive 2023-01-24T03:21:03.082 DEBUG:teuthology.orchestra.run.smithi142:> sudo /usr/sbin/setenforce permissive 2023-01-24T03:21:03.133 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-01-24T03:21:03.142 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-01-24T03:21:03.147 INFO:teuthology.repo_utils:Fetching main from origin 2023-01-24T03:21:03.283 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-01-24T03:21:03.311 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-01-24T03:21:03.313 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi112.front.sepia.ceph.com,smithi142.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-01-24T03:27:28.948 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi112.front.sepia.ceph.com'), Remote(name='ubuntu@smithi142.front.sepia.ceph.com')] 2023-01-24T03:27:28.970 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-01-24T03:27:28.971 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T03:27:29.039 DEBUG:teuthology.orchestra.run.smithi112:> true 2023-01-24T03:27:29.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2023-01-24T03:27:29.104 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T03:27:29.172 DEBUG:teuthology.orchestra.run.smithi142:> true 2023-01-24T03:27:29.244 INFO:teuthology.run_tasks:Running task clock... 2023-01-24T03:27:29.254 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-01-24T03:27:29.254 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-01-24T03:27:29.255 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-01-24T03:27:29.258 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-01-24T03:27:29.258 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-01-24T03:27:29.285 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-01-24T03:27:29.299 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-01-24T03:27:29.315 INFO:teuthology.orchestra.run.smithi142.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-01-24T03:27:29.330 INFO:teuthology.orchestra.run.smithi142.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-01-24T03:27:29.331 INFO:teuthology.orchestra.run.smithi112.stderr:sudo: ntpd: command not found 2023-01-24T03:27:29.340 INFO:teuthology.orchestra.run.smithi112.stdout:506 Cannot talk to daemon 2023-01-24T03:27:29.353 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-01-24T03:27:29.367 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-01-24T03:27:29.370 INFO:teuthology.orchestra.run.smithi142.stderr:sudo: ntpd: command not found 2023-01-24T03:27:29.380 INFO:teuthology.orchestra.run.smithi142.stdout:506 Cannot talk to daemon 2023-01-24T03:27:29.394 INFO:teuthology.orchestra.run.smithi142.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-01-24T03:27:29.407 INFO:teuthology.orchestra.run.smithi142.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-01-24T03:27:29.433 INFO:teuthology.orchestra.run.smithi112.stderr:bash: ntpq: command not found 2023-01-24T03:27:29.437 INFO:teuthology.orchestra.run.smithi112.stdout:210 Number of sources = 4 2023-01-24T03:27:29.437 INFO:teuthology.orchestra.run.smithi112.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-01-24T03:27:29.437 INFO:teuthology.orchestra.run.smithi112.stdout:=============================================================================== 2023-01-24T03:27:29.437 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T03:27:29.437 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T03:27:29.438 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T03:27:29.438 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T03:27:29.470 INFO:teuthology.orchestra.run.smithi142.stderr:bash: ntpq: command not found 2023-01-24T03:27:29.473 INFO:teuthology.orchestra.run.smithi142.stdout:210 Number of sources = 4 2023-01-24T03:27:29.473 INFO:teuthology.orchestra.run.smithi142.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-01-24T03:27:29.473 INFO:teuthology.orchestra.run.smithi142.stdout:=============================================================================== 2023-01-24T03:27:29.475 INFO:teuthology.orchestra.run.smithi142.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T03:27:29.475 INFO:teuthology.orchestra.run.smithi142.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T03:27:29.475 INFO:teuthology.orchestra.run.smithi142.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T03:27:29.475 INFO:teuthology.orchestra.run.smithi142.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-24T03:27:29.476 INFO:teuthology.run_tasks:Running task install... 2023-01-24T03:27:29.485 DEBUG:teuthology.task.install:project ceph 2023-01-24T03:27:29.486 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8387d427d91a301da19bd82e090b96e1348bfadb'}} 2023-01-24T03:27:29.486 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '8387d427d91a301da19bd82e090b96e1348bfadb'} 2023-01-24T03:27:29.486 INFO:teuthology.task.install:Using flavor: default 2023-01-24T03:27:29.492 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-01-24T03:27:29.493 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-01-24T03:27:29.493 DEBUG:teuthology.orchestra.run.smithi112:> sudo lsb_release -is 2023-01-24T03:27:29.530 INFO:teuthology.orchestra.run.smithi112.stdout:RedHatEnterprise 2023-01-24T03:27:29.532 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2023-01-24T03:27:29.532 DEBUG:teuthology.orchestra.run.smithi142:> sudo lsb_release -is 2023-01-24T03:27:29.533 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8387d427d91a301da19bd82e090b96e1348bfadb', 'tag': None, 'wait_for_package': False} 2023-01-24T03:27:29.533 DEBUG:teuthology.orchestra.run.smithi112:> sudo lsb_release -is 2023-01-24T03:27:29.570 INFO:teuthology.orchestra.run.smithi142.stdout:RedHatEnterprise 2023-01-24T03:27:29.572 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2023-01-24T03:27:29.573 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8387d427d91a301da19bd82e090b96e1348bfadb', 'tag': None, 'wait_for_package': False} 2023-01-24T03:27:29.573 DEBUG:teuthology.orchestra.run.smithi142:> sudo lsb_release -is 2023-01-24T03:27:29.609 INFO:teuthology.orchestra.run.smithi112.stdout:RedHatEnterprise 2023-01-24T03:27:29.610 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2023-01-24T03:27:29.611 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=8387d427d91a301da19bd82e090b96e1348bfadb 2023-01-24T03:27:29.650 INFO:teuthology.orchestra.run.smithi142.stdout:RedHatEnterprise 2023-01-24T03:27:29.651 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2023-01-24T03:27:29.652 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=8387d427d91a301da19bd82e090b96e1348bfadb 2023-01-24T03:27:29.761 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-01-23-0928-quincy/8387d427d91a301da19bd82e090b96e1348bfadb/centos/8/flavors/default/ 2023-01-24T03:27:29.761 INFO:teuthology.task.install.rpm:Package version is 17.2.5-701.g8387d427 2023-01-24T03:27:29.813 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-01-23-0928-quincy/8387d427d91a301da19bd82e090b96e1348bfadb/centos/8/flavors/default/ 2023-01-24T03:27:29.814 INFO:teuthology.task.install.rpm:Package version is 17.2.5-701.g8387d427 2023-01-24T03:27:29.886 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-01-23-0928-quincy/8387d427d91a301da19bd82e090b96e1348bfadb/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-01-23-0928-quincy/8387d427d91a301da19bd82e090b96e1348bfadb/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-01-23-0928-quincy/8387d427d91a301da19bd82e090b96e1348bfadb/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-01-24T03:27:29.886 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-01-24T03:27:29.886 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-01-24T03:27:29.910 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2023-01-24T03:27:29.910 DEBUG:teuthology.orchestra.run.smithi112:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8387d427d91a301da19bd82e090b96e1348bfadb/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-01-24T03:27:29.930 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-01-23-0928-quincy/8387d427d91a301da19bd82e090b96e1348bfadb/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-01-23-0928-quincy/8387d427d91a301da19bd82e090b96e1348bfadb/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-01-23-0928-quincy/8387d427d91a301da19bd82e090b96e1348bfadb/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-01-24T03:27:29.930 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2023-01-24T03:27:29.930 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-01-24T03:27:29.954 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2023-01-24T03:27:29.955 DEBUG:teuthology.orchestra.run.smithi142:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8387d427d91a301da19bd82e090b96e1348bfadb/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-01-24T03:27:29.974 DEBUG:teuthology.orchestra.run.smithi112:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-01-24T03:27:30.020 DEBUG:teuthology.orchestra.run.smithi142:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-01-24T03:27:30.047 DEBUG:teuthology.orchestra.run.smithi112:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-01-24T03:27:30.093 DEBUG:teuthology.orchestra.run.smithi142:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-01-24T03:27:30.110 INFO:teuthology.orchestra.run.smithi112.stdout:check_obsoletes = 1 2023-01-24T03:27:30.111 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum clean all 2023-01-24T03:27:30.156 INFO:teuthology.orchestra.run.smithi142.stdout:check_obsoletes = 1 2023-01-24T03:27:30.157 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum clean all 2023-01-24T03:27:30.495 INFO:teuthology.orchestra.run.smithi112.stdout:Updating Subscription Management repositories. 2023-01-24T03:27:30.543 INFO:teuthology.orchestra.run.smithi142.stdout:Updating Subscription Management repositories. 2023-01-24T03:27:30.822 INFO:teuthology.orchestra.run.smithi112.stdout:44 files removed 2023-01-24T03:27:30.840 INFO:teuthology.orchestra.run.smithi142.stdout:44 files removed 2023-01-24T03:27:30.873 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2023-01-24T03:27:30.890 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2023-01-24T03:27:31.219 INFO:teuthology.orchestra.run.smithi112.stdout:Updating Subscription Management repositories. 2023-01-24T03:27:31.233 INFO:teuthology.orchestra.run.smithi142.stdout:Updating Subscription Management repositories. 2023-01-24T03:27:31.913 INFO:teuthology.orchestra.run.smithi112.stdout:ceph packages for x86_64 323 kB/s | 78 kB 00:00 2023-01-24T03:27:31.930 INFO:teuthology.orchestra.run.smithi142.stdout:ceph packages for x86_64 324 kB/s | 78 kB 00:00 2023-01-24T03:27:32.154 INFO:teuthology.orchestra.run.smithi112.stdout:ceph noarch packages 46 kB/s | 9.2 kB 00:00 2023-01-24T03:27:32.170 INFO:teuthology.orchestra.run.smithi142.stdout:ceph noarch packages 46 kB/s | 9.2 kB 00:00 2023-01-24T03:27:32.384 INFO:teuthology.orchestra.run.smithi112.stdout:ceph source packages 7.2 kB/s | 1.4 kB 00:00 2023-01-24T03:27:32.402 INFO:teuthology.orchestra.run.smithi142.stdout:ceph source packages 7.2 kB/s | 1.4 kB 00:00 2023-01-24T03:27:32.493 INFO:teuthology.orchestra.run.smithi112.stdout:Copr repo for python3-asyncssh owned by ceph 43 kB/s | 3.5 kB 00:00 2023-01-24T03:27:32.510 INFO:teuthology.orchestra.run.smithi142.stdout:Copr repo for python3-asyncssh owned by ceph 44 kB/s | 3.5 kB 00:00 2023-01-24T03:27:33.452 INFO:teuthology.orchestra.run.smithi112.stdout:Extra Packages for Enterprise Linux 14 MB/s | 13 MB 00:00 2023-01-24T03:27:33.481 INFO:teuthology.orchestra.run.smithi142.stdout:Extra Packages for Enterprise Linux 14 MB/s | 13 MB 00:00 2023-01-24T03:27:36.282 INFO:teuthology.orchestra.run.smithi112.stdout:lab-extras 2.6 MB/s | 24 kB 00:00 2023-01-24T03:27:36.283 INFO:teuthology.orchestra.run.smithi142.stdout:lab-extras 4.6 MB/s | 24 kB 00:00 2023-01-24T03:27:36.624 INFO:teuthology.orchestra.run.smithi112.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 122 MB/s | 33 MB 00:00 2023-01-24T03:27:36.654 INFO:teuthology.orchestra.run.smithi142.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 129 MB/s | 38 MB 00:00 2023-01-24T03:27:41.753 INFO:teuthology.orchestra.run.smithi112.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 132 MB/s | 38 MB 00:00 2023-01-24T03:27:42.096 INFO:teuthology.orchestra.run.smithi142.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 123 MB/s | 33 MB 00:00 2023-01-24T03:27:49.938 INFO:teuthology.orchestra.run.smithi142.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-01-24T03:27:49.939 INFO:teuthology.orchestra.run.smithi142.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-01-24T03:27:49.950 INFO:teuthology.orchestra.run.smithi142.stdout:Package NetworkManager-1:1.30.0-13.el8_4.x86_64 is already installed. 2023-01-24T03:27:49.950 INFO:teuthology.orchestra.run.smithi142.stdout:Package iproute-5.9.0-4.el8.x86_64 is already installed. 2023-01-24T03:27:49.951 INFO:teuthology.orchestra.run.smithi142.stdout:Package util-linux-2.32.1-27.el8.x86_64 is already installed. 2023-01-24T03:27:49.952 INFO:teuthology.orchestra.run.smithi142.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-01-24T03:27:49.954 INFO:teuthology.orchestra.run.smithi142.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-01-24T03:27:49.956 INFO:teuthology.orchestra.run.smithi142.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2023-01-24T03:27:49.957 INFO:teuthology.orchestra.run.smithi142.stdout:Package xfsprogs-5.0.0-8.el8.x86_64 is already installed. 2023-01-24T03:27:50.052 INFO:teuthology.orchestra.run.smithi112.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-01-24T03:27:50.053 INFO:teuthology.orchestra.run.smithi112.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-01-24T03:27:50.063 INFO:teuthology.orchestra.run.smithi112.stdout:Package NetworkManager-1:1.30.0-13.el8_4.x86_64 is already installed. 2023-01-24T03:27:50.064 INFO:teuthology.orchestra.run.smithi112.stdout:Package iproute-5.9.0-4.el8.x86_64 is already installed. 2023-01-24T03:27:50.065 INFO:teuthology.orchestra.run.smithi112.stdout:Package util-linux-2.32.1-27.el8.x86_64 is already installed. 2023-01-24T03:27:50.066 INFO:teuthology.orchestra.run.smithi112.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-01-24T03:27:50.068 INFO:teuthology.orchestra.run.smithi112.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-01-24T03:27:50.070 INFO:teuthology.orchestra.run.smithi112.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2023-01-24T03:27:50.071 INFO:teuthology.orchestra.run.smithi112.stdout:Package xfsprogs-5.0.0-8.el8.x86_64 is already installed. 2023-01-24T03:27:50.421 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2023-01-24T03:27:50.433 INFO:teuthology.orchestra.run.smithi142.stdout:====================================================================================================================== 2023-01-24T03:27:50.433 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repository Size 2023-01-24T03:27:50.433 INFO:teuthology.orchestra.run.smithi142.stdout:====================================================================================================================== 2023-01-24T03:27:50.434 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2023-01-24T03:27:50.434 INFO:teuthology.orchestra.run.smithi142.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-01-24T03:27:50.434 INFO:teuthology.orchestra.run.smithi142.stdout: ceph x86_64 2:17.2.5-701.g8387d427.el8 ceph 6.4 k 2023-01-24T03:27:50.434 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-base x86_64 2:17.2.5-701.g8387d427.el8 ceph 5.1 M 2023-01-24T03:27:50.434 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-fuse x86_64 2:17.2.5-701.g8387d427.el8 ceph 871 k 2023-01-24T03:27:50.435 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-immutable-object-cache x86_64 2:17.2.5-701.g8387d427.el8 ceph 162 k 2023-01-24T03:27:50.435 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr x86_64 2:17.2.5-701.g8387d427.el8 ceph 1.6 M 2023-01-24T03:27:50.435 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-cephadm noarch 2:17.2.5-701.g8387d427.el8 ceph-noarch 114 k 2023-01-24T03:27:50.435 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-dashboard noarch 2:17.2.5-701.g8387d427.el8 ceph-noarch 1.6 M 2023-01-24T03:27:50.435 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.5-701.g8387d427.el8 ceph-noarch 7.4 M 2023-01-24T03:27:50.436 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-rook noarch 2:17.2.5-701.g8387d427.el8 ceph-noarch 47 k 2023-01-24T03:27:50.436 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-radosgw x86_64 2:17.2.5-701.g8387d427.el8 ceph 17 M 2023-01-24T03:27:50.436 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-test x86_64 2:17.2.5-701.g8387d427.el8 ceph 52 M 2023-01-24T03:27:50.436 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-volume noarch 2:17.2.5-701.g8387d427.el8 ceph-noarch 263 k 2023-01-24T03:27:50.437 INFO:teuthology.orchestra.run.smithi142.stdout: cephadm noarch 2:17.2.5-701.g8387d427.el8 ceph-noarch 78 k 2023-01-24T03:27:50.437 INFO:teuthology.orchestra.run.smithi142.stdout: cephfs-mirror x86_64 2:17.2.5-701.g8387d427.el8 ceph 254 k 2023-01-24T03:27:50.437 INFO:teuthology.orchestra.run.smithi142.stdout: cephfs-top noarch 2:17.2.5-701.g8387d427.el8 ceph-noarch 15 k 2023-01-24T03:27:50.437 INFO:teuthology.orchestra.run.smithi142.stdout: elfutils-libelf-devel x86_64 0.182-3.el8 rhel-8-for-x86_64-baseos-rpms 59 k 2023-01-24T03:27:50.437 INFO:teuthology.orchestra.run.smithi142.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-01-24T03:27:50.438 INFO:teuthology.orchestra.run.smithi142.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-01-24T03:27:50.438 INFO:teuthology.orchestra.run.smithi142.stdout: libcephfs-devel x86_64 2:17.2.5-701.g8387d427.el8 ceph 25 k 2023-01-24T03:27:50.438 INFO:teuthology.orchestra.run.smithi142.stdout: libcephfs2 x86_64 2:17.2.5-701.g8387d427.el8 ceph 793 k 2023-01-24T03:27:50.438 INFO:teuthology.orchestra.run.smithi142.stdout: librados-devel x86_64 2:17.2.5-701.g8387d427.el8 ceph 130 k 2023-01-24T03:27:50.438 INFO:teuthology.orchestra.run.smithi142.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-01-24T03:27:50.439 INFO:teuthology.orchestra.run.smithi142.stdout: libuuid-devel x86_64 2.32.1-27.el8 rhel-8-for-x86_64-baseos-rpms 97 k 2023-01-24T03:27:50.439 INFO:teuthology.orchestra.run.smithi142.stdout: openssl-devel x86_64 1:1.1.1g-15.el8_3 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-01-24T03:27:50.439 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cephfs x86_64 2:17.2.5-701.g8387d427.el8 ceph 190 k 2023-01-24T03:27:50.439 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rados x86_64 2:17.2.5-701.g8387d427.el8 ceph 370 k 2023-01-24T03:27:50.439 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rbd x86_64 2:17.2.5-701.g8387d427.el8 ceph 358 k 2023-01-24T03:27:50.440 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rgw x86_64 2:17.2.5-701.g8387d427.el8 ceph 106 k 2023-01-24T03:27:50.440 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-fuse x86_64 2:17.2.5-701.g8387d427.el8 ceph 90 k 2023-01-24T03:27:50.440 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-mirror x86_64 2:17.2.5-701.g8387d427.el8 ceph 3.5 M 2023-01-24T03:27:50.441 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-nbd x86_64 2:17.2.5-701.g8387d427.el8 ceph 172 k 2023-01-24T03:27:50.441 INFO:teuthology.orchestra.run.smithi142.stdout: xfsprogs-devel x86_64 5.0.0-8.el8 rhel-8-for-x86_64-baseos-rpms 70 k 2023-01-24T03:27:50.441 INFO:teuthology.orchestra.run.smithi142.stdout:Upgrading: 2023-01-24T03:27:50.441 INFO:teuthology.orchestra.run.smithi142.stdout: librados2 x86_64 2:17.2.5-701.g8387d427.el8 ceph 3.5 M 2023-01-24T03:27:50.442 INFO:teuthology.orchestra.run.smithi142.stdout: librbd1 x86_64 2:17.2.5-701.g8387d427.el8 ceph 3.6 M 2023-01-24T03:27:50.442 INFO:teuthology.orchestra.run.smithi142.stdout:Installing dependencies: 2023-01-24T03:27:50.442 INFO:teuthology.orchestra.run.smithi142.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-01-24T03:27:50.442 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-common x86_64 2:17.2.5-701.g8387d427.el8 ceph 23 M 2023-01-24T03:27:50.442 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-grafana-dashboards noarch 2:17.2.5-701.g8387d427.el8 ceph-noarch 23 k 2023-01-24T03:27:50.443 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mds x86_64 2:17.2.5-701.g8387d427.el8 ceph 2.3 M 2023-01-24T03:27:50.443 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-modules-core noarch 2:17.2.5-701.g8387d427.el8 ceph-noarch 233 k 2023-01-24T03:27:50.443 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mon x86_64 2:17.2.5-701.g8387d427.el8 ceph 3.9 M 2023-01-24T03:27:50.443 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-osd x86_64 2:17.2.5-701.g8387d427.el8 ceph 17 M 2023-01-24T03:27:50.443 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-prometheus-alerts noarch 2:17.2.5-701.g8387d427.el8 ceph-noarch 14 k 2023-01-24T03:27:50.444 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-selinux x86_64 2:17.2.5-701.g8387d427.el8 ceph 24 k 2023-01-24T03:27:50.444 INFO:teuthology.orchestra.run.smithi142.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-01-24T03:27:50.444 INFO:teuthology.orchestra.run.smithi142.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-01-24T03:27:50.444 INFO:teuthology.orchestra.run.smithi142.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-01-24T03:27:50.444 INFO:teuthology.orchestra.run.smithi142.stdout: keyutils-libs-devel x86_64 1.5.10-6.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-01-24T03:27:50.445 INFO:teuthology.orchestra.run.smithi142.stdout: krb5-devel x86_64 1.18.2-8.3.el8_4 rhel-8-for-x86_64-baseos-rpms 559 k 2023-01-24T03:27:50.445 INFO:teuthology.orchestra.run.smithi142.stdout: libcephsqlite x86_64 2:17.2.5-701.g8387d427.el8 ceph 174 k 2023-01-24T03:27:50.445 INFO:teuthology.orchestra.run.smithi142.stdout: libcom_err-devel x86_64 1.45.6-1.el8 rhel-8-for-x86_64-baseos-rpms 38 k 2023-01-24T03:27:50.445 INFO:teuthology.orchestra.run.smithi142.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-01-24T03:27:50.445 INFO:teuthology.orchestra.run.smithi142.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-01-24T03:27:50.446 INFO:teuthology.orchestra.run.smithi142.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-01-24T03:27:50.446 INFO:teuthology.orchestra.run.smithi142.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-01-24T03:27:50.446 INFO:teuthology.orchestra.run.smithi142.stdout: libradosstriper1 x86_64 2:17.2.5-701.g8387d427.el8 ceph 506 k 2023-01-24T03:27:50.446 INFO:teuthology.orchestra.run.smithi142.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-01-24T03:27:50.446 INFO:teuthology.orchestra.run.smithi142.stdout: librgw2 x86_64 2:17.2.5-701.g8387d427.el8 ceph 5.5 M 2023-01-24T03:27:50.447 INFO:teuthology.orchestra.run.smithi142.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-01-24T03:27:50.447 INFO:teuthology.orchestra.run.smithi142.stdout: libsepol-devel x86_64 2.9-2.el8 rhel-8-for-x86_64-baseos-rpms 86 k 2023-01-24T03:27:50.447 INFO:teuthology.orchestra.run.smithi142.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-01-24T03:27:50.447 INFO:teuthology.orchestra.run.smithi142.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-01-24T03:27:50.448 INFO:teuthology.orchestra.run.smithi142.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-01-24T03:27:50.448 INFO:teuthology.orchestra.run.smithi142.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-01-24T03:27:50.448 INFO:teuthology.orchestra.run.smithi142.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-01-24T03:27:50.448 INFO:teuthology.orchestra.run.smithi142.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-01-24T03:27:50.449 INFO:teuthology.orchestra.run.smithi142.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-01-24T03:27:50.449 INFO:teuthology.orchestra.run.smithi142.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-01-24T03:27:50.449 INFO:teuthology.orchestra.run.smithi142.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-01-24T03:27:50.449 INFO:teuthology.orchestra.run.smithi142.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-01-24T03:27:50.449 INFO:teuthology.orchestra.run.smithi142.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-01-24T03:27:50.450 INFO:teuthology.orchestra.run.smithi142.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-01-24T03:27:50.450 INFO:teuthology.orchestra.run.smithi142.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-01-24T03:27:50.450 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-01-24T03:27:50.450 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ceph-argparse x86_64 2:17.2.5-701.g8387d427.el8 ceph 45 k 2023-01-24T03:27:50.450 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ceph-common x86_64 2:17.2.5-701.g8387d427.el8 ceph 108 k 2023-01-24T03:27:50.451 INFO:teuthology.orchestra.run.smithi142.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-01-24T03:27:50.451 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-01-24T03:27:50.451 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-01-24T03:27:50.451 INFO:teuthology.orchestra.run.smithi142.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-01-24T03:27:50.451 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-01-24T03:27:50.452 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-01-24T03:27:50.452 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-01-24T03:27:50.452 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-01-24T03:27:50.452 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-01-24T03:27:50.452 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-01-24T03:27:50.453 INFO:teuthology.orchestra.run.smithi142.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-01-24T03:27:50.453 INFO:teuthology.orchestra.run.smithi142.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-01-24T03:27:50.453 INFO:teuthology.orchestra.run.smithi142.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-01-24T03:27:50.453 INFO:teuthology.orchestra.run.smithi142.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-01-24T03:27:50.454 INFO:teuthology.orchestra.run.smithi142.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-01-24T03:27:50.454 INFO:teuthology.orchestra.run.smithi142.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-01-24T03:27:50.454 INFO:teuthology.orchestra.run.smithi142.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-01-24T03:27:50.454 INFO:teuthology.orchestra.run.smithi142.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-01-24T03:27:50.454 INFO:teuthology.orchestra.run.smithi142.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-01-24T03:27:50.455 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-01-24T03:27:50.455 INFO:teuthology.orchestra.run.smithi142.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-01-24T03:27:50.455 INFO:teuthology.orchestra.run.smithi142.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-01-24T03:27:50.455 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-01-24T03:27:50.455 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-01-24T03:27:50.455 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-01-24T03:27:50.456 INFO:teuthology.orchestra.run.smithi142.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-01-24T03:27:50.456 INFO:teuthology.orchestra.run.smithi142.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-01-24T03:27:50.456 INFO:teuthology.orchestra.run.smithi142.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-01-24T03:27:50.456 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-01-24T03:27:50.456 INFO:teuthology.orchestra.run.smithi142.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-01-24T03:27:50.457 INFO:teuthology.orchestra.run.smithi142.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-01-24T03:27:50.457 INFO:teuthology.orchestra.run.smithi142.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-01-24T03:27:50.457 INFO:teuthology.orchestra.run.smithi142.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-01-24T03:27:50.457 INFO:teuthology.orchestra.run.smithi142.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-01-24T03:27:50.457 INFO:teuthology.orchestra.run.smithi142.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2023-01-24T03:27:50.458 INFO:teuthology.orchestra.run.smithi142.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-01-24T03:27:50.458 INFO:teuthology.orchestra.run.smithi142.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-01-24T03:27:50.458 INFO:teuthology.orchestra.run.smithi142.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-01-24T03:27:50.458 INFO:teuthology.orchestra.run.smithi142.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-01-24T03:27:50.458 INFO:teuthology.orchestra.run.smithi142.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-01-24T03:27:50.459 INFO:teuthology.orchestra.run.smithi142.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-01-24T03:27:50.459 INFO:teuthology.orchestra.run.smithi142.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-01-24T03:27:50.459 INFO:teuthology.orchestra.run.smithi142.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-01-24T03:27:50.459 INFO:teuthology.orchestra.run.smithi142.stdout: zlib-devel x86_64 1.2.11-17.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2023-01-24T03:27:50.459 INFO:teuthology.orchestra.run.smithi142.stdout: 2023-01-24T03:27:50.459 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2023-01-24T03:27:50.460 INFO:teuthology.orchestra.run.smithi142.stdout:====================================================================================================================== 2023-01-24T03:27:50.460 INFO:teuthology.orchestra.run.smithi142.stdout:Install 118 Packages 2023-01-24T03:27:50.460 INFO:teuthology.orchestra.run.smithi142.stdout:Upgrade 2 Packages 2023-01-24T03:27:50.460 INFO:teuthology.orchestra.run.smithi142.stdout: 2023-01-24T03:27:50.461 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 202 M 2023-01-24T03:27:50.461 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2023-01-24T03:27:50.537 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2023-01-24T03:27:50.545 INFO:teuthology.orchestra.run.smithi142.stdout:(1/120): ceph-17.2.5-701.g8387d427.el8.x86_64.r 65 kB/s | 6.4 kB 00:00 2023-01-24T03:27:50.548 INFO:teuthology.orchestra.run.smithi112.stdout:====================================================================================================================== 2023-01-24T03:27:50.549 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2023-01-24T03:27:50.549 INFO:teuthology.orchestra.run.smithi112.stdout:====================================================================================================================== 2023-01-24T03:27:50.549 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2023-01-24T03:27:50.549 INFO:teuthology.orchestra.run.smithi112.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-01-24T03:27:50.549 INFO:teuthology.orchestra.run.smithi112.stdout: ceph x86_64 2:17.2.5-701.g8387d427.el8 ceph 6.4 k 2023-01-24T03:27:50.550 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-base x86_64 2:17.2.5-701.g8387d427.el8 ceph 5.1 M 2023-01-24T03:27:50.550 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-fuse x86_64 2:17.2.5-701.g8387d427.el8 ceph 871 k 2023-01-24T03:27:50.550 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-immutable-object-cache x86_64 2:17.2.5-701.g8387d427.el8 ceph 162 k 2023-01-24T03:27:50.550 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr x86_64 2:17.2.5-701.g8387d427.el8 ceph 1.6 M 2023-01-24T03:27:50.551 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-cephadm noarch 2:17.2.5-701.g8387d427.el8 ceph-noarch 114 k 2023-01-24T03:27:50.551 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-dashboard noarch 2:17.2.5-701.g8387d427.el8 ceph-noarch 1.6 M 2023-01-24T03:27:50.551 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.5-701.g8387d427.el8 ceph-noarch 7.4 M 2023-01-24T03:27:50.551 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-rook noarch 2:17.2.5-701.g8387d427.el8 ceph-noarch 47 k 2023-01-24T03:27:50.552 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-radosgw x86_64 2:17.2.5-701.g8387d427.el8 ceph 17 M 2023-01-24T03:27:50.552 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-test x86_64 2:17.2.5-701.g8387d427.el8 ceph 52 M 2023-01-24T03:27:50.552 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-volume noarch 2:17.2.5-701.g8387d427.el8 ceph-noarch 263 k 2023-01-24T03:27:50.552 INFO:teuthology.orchestra.run.smithi112.stdout: cephadm noarch 2:17.2.5-701.g8387d427.el8 ceph-noarch 78 k 2023-01-24T03:27:50.553 INFO:teuthology.orchestra.run.smithi112.stdout: cephfs-mirror x86_64 2:17.2.5-701.g8387d427.el8 ceph 254 k 2023-01-24T03:27:50.553 INFO:teuthology.orchestra.run.smithi112.stdout: cephfs-top noarch 2:17.2.5-701.g8387d427.el8 ceph-noarch 15 k 2023-01-24T03:27:50.553 INFO:teuthology.orchestra.run.smithi112.stdout: elfutils-libelf-devel x86_64 0.182-3.el8 rhel-8-for-x86_64-baseos-rpms 59 k 2023-01-24T03:27:50.553 INFO:teuthology.orchestra.run.smithi112.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-01-24T03:27:50.553 INFO:teuthology.orchestra.run.smithi112.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-01-24T03:27:50.554 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs-devel x86_64 2:17.2.5-701.g8387d427.el8 ceph 25 k 2023-01-24T03:27:50.554 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs2 x86_64 2:17.2.5-701.g8387d427.el8 ceph 793 k 2023-01-24T03:27:50.554 INFO:teuthology.orchestra.run.smithi112.stdout: librados-devel x86_64 2:17.2.5-701.g8387d427.el8 ceph 130 k 2023-01-24T03:27:50.554 INFO:teuthology.orchestra.run.smithi112.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-01-24T03:27:50.555 INFO:teuthology.orchestra.run.smithi112.stdout: libuuid-devel x86_64 2.32.1-27.el8 rhel-8-for-x86_64-baseos-rpms 97 k 2023-01-24T03:27:50.555 INFO:teuthology.orchestra.run.smithi112.stdout: openssl-devel x86_64 1:1.1.1g-15.el8_3 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-01-24T03:27:50.555 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cephfs x86_64 2:17.2.5-701.g8387d427.el8 ceph 190 k 2023-01-24T03:27:50.556 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rados x86_64 2:17.2.5-701.g8387d427.el8 ceph 370 k 2023-01-24T03:27:50.556 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rbd x86_64 2:17.2.5-701.g8387d427.el8 ceph 358 k 2023-01-24T03:27:50.556 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rgw x86_64 2:17.2.5-701.g8387d427.el8 ceph 106 k 2023-01-24T03:27:50.557 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-fuse x86_64 2:17.2.5-701.g8387d427.el8 ceph 90 k 2023-01-24T03:27:50.557 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-mirror x86_64 2:17.2.5-701.g8387d427.el8 ceph 3.5 M 2023-01-24T03:27:50.558 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-nbd x86_64 2:17.2.5-701.g8387d427.el8 ceph 172 k 2023-01-24T03:27:50.558 INFO:teuthology.orchestra.run.smithi112.stdout: xfsprogs-devel x86_64 5.0.0-8.el8 rhel-8-for-x86_64-baseos-rpms 70 k 2023-01-24T03:27:50.558 INFO:teuthology.orchestra.run.smithi112.stdout:Upgrading: 2023-01-24T03:27:50.559 INFO:teuthology.orchestra.run.smithi112.stdout: librados2 x86_64 2:17.2.5-701.g8387d427.el8 ceph 3.5 M 2023-01-24T03:27:50.559 INFO:teuthology.orchestra.run.smithi112.stdout: librbd1 x86_64 2:17.2.5-701.g8387d427.el8 ceph 3.6 M 2023-01-24T03:27:50.559 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2023-01-24T03:27:50.560 INFO:teuthology.orchestra.run.smithi112.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-01-24T03:27:50.560 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-common x86_64 2:17.2.5-701.g8387d427.el8 ceph 23 M 2023-01-24T03:27:50.561 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-grafana-dashboards noarch 2:17.2.5-701.g8387d427.el8 ceph-noarch 23 k 2023-01-24T03:27:50.561 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mds x86_64 2:17.2.5-701.g8387d427.el8 ceph 2.3 M 2023-01-24T03:27:50.561 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-modules-core noarch 2:17.2.5-701.g8387d427.el8 ceph-noarch 233 k 2023-01-24T03:27:50.562 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mon x86_64 2:17.2.5-701.g8387d427.el8 ceph 3.9 M 2023-01-24T03:27:50.562 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-osd x86_64 2:17.2.5-701.g8387d427.el8 ceph 17 M 2023-01-24T03:27:50.562 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-prometheus-alerts noarch 2:17.2.5-701.g8387d427.el8 ceph-noarch 14 k 2023-01-24T03:27:50.563 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-selinux x86_64 2:17.2.5-701.g8387d427.el8 ceph 24 k 2023-01-24T03:27:50.563 INFO:teuthology.orchestra.run.smithi112.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-01-24T03:27:50.563 INFO:teuthology.orchestra.run.smithi112.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-01-24T03:27:50.564 INFO:teuthology.orchestra.run.smithi112.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-01-24T03:27:50.564 INFO:teuthology.orchestra.run.smithi112.stdout: keyutils-libs-devel x86_64 1.5.10-6.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-01-24T03:27:50.564 INFO:teuthology.orchestra.run.smithi112.stdout: krb5-devel x86_64 1.18.2-8.3.el8_4 rhel-8-for-x86_64-baseos-rpms 559 k 2023-01-24T03:27:50.565 INFO:teuthology.orchestra.run.smithi112.stdout: libcephsqlite x86_64 2:17.2.5-701.g8387d427.el8 ceph 174 k 2023-01-24T03:27:50.565 INFO:teuthology.orchestra.run.smithi112.stdout: libcom_err-devel x86_64 1.45.6-1.el8 rhel-8-for-x86_64-baseos-rpms 38 k 2023-01-24T03:27:50.566 INFO:teuthology.orchestra.run.smithi112.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-01-24T03:27:50.566 INFO:teuthology.orchestra.run.smithi112.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-01-24T03:27:50.566 INFO:teuthology.orchestra.run.smithi112.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-01-24T03:27:50.567 INFO:teuthology.orchestra.run.smithi112.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-01-24T03:27:50.567 INFO:teuthology.orchestra.run.smithi112.stdout: libradosstriper1 x86_64 2:17.2.5-701.g8387d427.el8 ceph 506 k 2023-01-24T03:27:50.567 INFO:teuthology.orchestra.run.smithi112.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-01-24T03:27:50.568 INFO:teuthology.orchestra.run.smithi112.stdout: librgw2 x86_64 2:17.2.5-701.g8387d427.el8 ceph 5.5 M 2023-01-24T03:27:50.568 INFO:teuthology.orchestra.run.smithi112.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-01-24T03:27:50.568 INFO:teuthology.orchestra.run.smithi112.stdout: libsepol-devel x86_64 2.9-2.el8 rhel-8-for-x86_64-baseos-rpms 86 k 2023-01-24T03:27:50.569 INFO:teuthology.orchestra.run.smithi112.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-01-24T03:27:50.569 INFO:teuthology.orchestra.run.smithi112.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-01-24T03:27:50.569 INFO:teuthology.orchestra.run.smithi112.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-01-24T03:27:50.570 INFO:teuthology.orchestra.run.smithi112.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-01-24T03:27:50.570 INFO:teuthology.orchestra.run.smithi112.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-01-24T03:27:50.570 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-01-24T03:27:50.571 INFO:teuthology.orchestra.run.smithi112.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-01-24T03:27:50.571 INFO:teuthology.orchestra.run.smithi112.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-01-24T03:27:50.572 INFO:teuthology.orchestra.run.smithi112.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-01-24T03:27:50.572 INFO:teuthology.orchestra.run.smithi112.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-01-24T03:27:50.572 INFO:teuthology.orchestra.run.smithi112.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-01-24T03:27:50.573 INFO:teuthology.orchestra.run.smithi112.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-01-24T03:27:50.573 INFO:teuthology.orchestra.run.smithi112.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-01-24T03:27:50.573 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-01-24T03:27:50.574 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-argparse x86_64 2:17.2.5-701.g8387d427.el8 ceph 45 k 2023-01-24T03:27:50.574 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-common x86_64 2:17.2.5-701.g8387d427.el8 ceph 108 k 2023-01-24T03:27:50.574 INFO:teuthology.orchestra.run.smithi112.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-01-24T03:27:50.575 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-01-24T03:27:50.575 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-01-24T03:27:50.575 INFO:teuthology.orchestra.run.smithi112.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-01-24T03:27:50.576 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-01-24T03:27:50.576 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-01-24T03:27:50.577 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-01-24T03:27:50.577 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-01-24T03:27:50.577 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-01-24T03:27:50.578 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-01-24T03:27:50.578 INFO:teuthology.orchestra.run.smithi112.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-01-24T03:27:50.578 INFO:teuthology.orchestra.run.smithi112.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-01-24T03:27:50.579 INFO:teuthology.orchestra.run.smithi112.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-01-24T03:27:50.579 INFO:teuthology.orchestra.run.smithi112.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-01-24T03:27:50.579 INFO:teuthology.orchestra.run.smithi112.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-01-24T03:27:50.580 INFO:teuthology.orchestra.run.smithi112.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-01-24T03:27:50.580 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-01-24T03:27:50.580 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-01-24T03:27:50.581 INFO:teuthology.orchestra.run.smithi112.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-01-24T03:27:50.581 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-01-24T03:27:50.581 INFO:teuthology.orchestra.run.smithi112.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-01-24T03:27:50.582 INFO:teuthology.orchestra.run.smithi112.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-01-24T03:27:50.582 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-01-24T03:27:50.582 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-01-24T03:27:50.583 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-01-24T03:27:50.583 INFO:teuthology.orchestra.run.smithi112.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-01-24T03:27:50.584 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-01-24T03:27:50.584 INFO:teuthology.orchestra.run.smithi112.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-01-24T03:27:50.584 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-01-24T03:27:50.585 INFO:teuthology.orchestra.run.smithi112.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-01-24T03:27:50.585 INFO:teuthology.orchestra.run.smithi112.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-01-24T03:27:50.585 INFO:teuthology.orchestra.run.smithi112.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-01-24T03:27:50.586 INFO:teuthology.orchestra.run.smithi112.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-01-24T03:27:50.586 INFO:teuthology.orchestra.run.smithi112.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-01-24T03:27:50.586 INFO:teuthology.orchestra.run.smithi112.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2023-01-24T03:27:50.587 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-01-24T03:27:50.587 INFO:teuthology.orchestra.run.smithi112.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-01-24T03:27:50.587 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-01-24T03:27:50.588 INFO:teuthology.orchestra.run.smithi112.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-01-24T03:27:50.588 INFO:teuthology.orchestra.run.smithi112.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-01-24T03:27:50.588 INFO:teuthology.orchestra.run.smithi112.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-01-24T03:27:50.589 INFO:teuthology.orchestra.run.smithi112.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-01-24T03:27:50.589 INFO:teuthology.orchestra.run.smithi112.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-01-24T03:27:50.589 INFO:teuthology.orchestra.run.smithi112.stdout: zlib-devel x86_64 1.2.11-17.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2023-01-24T03:27:50.590 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:27:50.590 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2023-01-24T03:27:50.590 INFO:teuthology.orchestra.run.smithi112.stdout:====================================================================================================================== 2023-01-24T03:27:50.591 INFO:teuthology.orchestra.run.smithi112.stdout:Install 118 Packages 2023-01-24T03:27:50.591 INFO:teuthology.orchestra.run.smithi112.stdout:Upgrade 2 Packages 2023-01-24T03:27:50.591 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:27:50.592 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 202 M 2023-01-24T03:27:50.593 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2023-01-24T03:27:50.663 INFO:teuthology.orchestra.run.smithi112.stdout:(1/120): ceph-17.2.5-701.g8387d427.el8.x86_64.r 64 kB/s | 6.4 kB 00:00 2023-01-24T03:27:50.729 INFO:teuthology.orchestra.run.smithi142.stdout:(2/120): ceph-fuse-17.2.5-701.g8387d427.el8.x86 4.6 MB/s | 871 kB 00:00 2023-01-24T03:27:50.754 INFO:teuthology.orchestra.run.smithi142.stdout:(3/120): ceph-immutable-object-cache-17.2.5-701 6.4 MB/s | 162 kB 00:00 2023-01-24T03:27:50.910 INFO:teuthology.orchestra.run.smithi112.stdout:(2/120): ceph-fuse-17.2.5-701.g8387d427.el8.x86 3.4 MB/s | 871 kB 00:00 2023-01-24T03:27:50.954 INFO:teuthology.orchestra.run.smithi112.stdout:(3/120): ceph-immutable-object-cache-17.2.5-701 3.6 MB/s | 162 kB 00:00 2023-01-24T03:27:51.249 INFO:teuthology.orchestra.run.smithi142.stdout:(4/120): ceph-mds-17.2.5-701.g8387d427.el8.x86_ 4.6 MB/s | 2.3 MB 00:00 2023-01-24T03:27:51.442 INFO:teuthology.orchestra.run.smithi142.stdout:(5/120): ceph-base-17.2.5-701.g8387d427.el8.x86 5.2 MB/s | 5.1 MB 00:00 2023-01-24T03:27:51.559 INFO:teuthology.orchestra.run.smithi112.stdout:(4/120): ceph-mds-17.2.5-701.g8387d427.el8.x86_ 3.8 MB/s | 2.3 MB 00:00 2023-01-24T03:27:51.567 INFO:teuthology.orchestra.run.smithi142.stdout:(6/120): ceph-mgr-17.2.5-701.g8387d427.el8.x86_ 4.9 MB/s | 1.6 MB 00:00 2023-01-24T03:27:51.981 INFO:teuthology.orchestra.run.smithi112.stdout:(5/120): ceph-mgr-17.2.5-701.g8387d427.el8.x86_ 3.7 MB/s | 1.6 MB 00:00 2023-01-24T03:27:52.095 INFO:teuthology.orchestra.run.smithi112.stdout:(6/120): ceph-base-17.2.5-701.g8387d427.el8.x86 3.4 MB/s | 5.1 MB 00:01 2023-01-24T03:27:52.130 INFO:teuthology.orchestra.run.smithi142.stdout:(7/120): ceph-mon-17.2.5-701.g8387d427.el8.x86_ 5.7 MB/s | 3.9 MB 00:00 2023-01-24T03:27:53.306 INFO:teuthology.orchestra.run.smithi112.stdout:(7/120): ceph-mon-17.2.5-701.g8387d427.el8.x86_ 3.0 MB/s | 3.9 MB 00:01 2023-01-24T03:27:53.716 INFO:teuthology.orchestra.run.smithi142.stdout:(8/120): ceph-common-17.2.5-701.g8387d427.el8.x 7.1 MB/s | 23 MB 00:03 2023-01-24T03:27:53.774 INFO:teuthology.orchestra.run.smithi142.stdout:(9/120): ceph-selinux-17.2.5-701.g8387d427.el8. 421 kB/s | 24 kB 00:00 2023-01-24T03:27:55.199 INFO:teuthology.orchestra.run.smithi142.stdout:(10/120): ceph-osd-17.2.5-701.g8387d427.el8.x86 4.6 MB/s | 17 MB 00:03 2023-01-24T03:27:55.272 INFO:teuthology.orchestra.run.smithi142.stdout:(11/120): cephfs-mirror-17.2.5-701.g8387d427.el 3.4 MB/s | 254 kB 00:00 2023-01-24T03:27:55.406 INFO:teuthology.orchestra.run.smithi142.stdout:(12/120): ceph-radosgw-17.2.5-701.g8387d427.el8 5.1 MB/s | 17 MB 00:03 2023-01-24T03:27:55.410 INFO:teuthology.orchestra.run.smithi142.stdout:(13/120): libcephfs-devel-17.2.5-701.g8387d427. 185 kB/s | 25 kB 00:00 2023-01-24T03:27:55.457 INFO:teuthology.orchestra.run.smithi142.stdout:(14/120): libcephsqlite-17.2.5-701.g8387d427.el 3.4 MB/s | 174 kB 00:00 2023-01-24T03:27:55.498 INFO:teuthology.orchestra.run.smithi142.stdout:(15/120): librados-devel-17.2.5-701.g8387d427.e 3.1 MB/s | 130 kB 00:00 2023-01-24T03:27:55.519 INFO:teuthology.orchestra.run.smithi112.stdout:(8/120): ceph-osd-17.2.5-701.g8387d427.el8.x86_ 4.9 MB/s | 17 MB 00:03 2023-01-24T03:27:55.523 INFO:teuthology.orchestra.run.smithi142.stdout:(16/120): libcephfs2-17.2.5-701.g8387d427.el8.x 6.6 MB/s | 793 kB 00:00 2023-01-24T03:27:55.569 INFO:teuthology.orchestra.run.smithi112.stdout:(9/120): ceph-selinux-17.2.5-701.g8387d427.el8. 491 kB/s | 24 kB 00:00 2023-01-24T03:27:55.589 INFO:teuthology.orchestra.run.smithi142.stdout:(17/120): libradosstriper1-17.2.5-701.g8387d427 5.4 MB/s | 506 kB 00:00 2023-01-24T03:27:55.651 INFO:teuthology.orchestra.run.smithi142.stdout:(18/120): python3-ceph-argparse-17.2.5-701.g838 726 kB/s | 45 kB 00:00 2023-01-24T03:27:55.715 INFO:teuthology.orchestra.run.smithi142.stdout:(19/120): python3-ceph-common-17.2.5-701.g8387d 1.7 MB/s | 108 kB 00:00 2023-01-24T03:27:55.781 INFO:teuthology.orchestra.run.smithi142.stdout:(20/120): python3-cephfs-17.2.5-701.g8387d427.e 2.8 MB/s | 190 kB 00:00 2023-01-24T03:27:55.856 INFO:teuthology.orchestra.run.smithi142.stdout:(21/120): python3-rados-17.2.5-701.g8387d427.el 4.9 MB/s | 370 kB 00:00 2023-01-24T03:27:55.932 INFO:teuthology.orchestra.run.smithi142.stdout:(22/120): python3-rbd-17.2.5-701.g8387d427.el8. 4.6 MB/s | 358 kB 00:00 2023-01-24T03:27:55.998 INFO:teuthology.orchestra.run.smithi142.stdout:(23/120): python3-rgw-17.2.5-701.g8387d427.el8. 1.6 MB/s | 106 kB 00:00 2023-01-24T03:27:56.062 INFO:teuthology.orchestra.run.smithi142.stdout:(24/120): rbd-fuse-17.2.5-701.g8387d427.el8.x86 1.4 MB/s | 90 kB 00:00 2023-01-24T03:27:56.635 INFO:teuthology.orchestra.run.smithi142.stdout:(25/120): librgw2-17.2.5-701.g8387d427.el8.x86_ 5.0 MB/s | 5.5 MB 00:01 2023-01-24T03:27:56.707 INFO:teuthology.orchestra.run.smithi142.stdout:(26/120): rbd-nbd-17.2.5-701.g8387d427.el8.x86_ 2.3 MB/s | 172 kB 00:00 2023-01-24T03:27:56.726 INFO:teuthology.orchestra.run.smithi142.stdout:(27/120): rbd-mirror-17.2.5-701.g8387d427.el8.x 5.2 MB/s | 3.5 MB 00:00 2023-01-24T03:27:56.766 INFO:teuthology.orchestra.run.smithi142.stdout:(28/120): ceph-grafana-dashboards-17.2.5-701.g8 393 kB/s | 23 kB 00:00 2023-01-24T03:27:56.781 INFO:teuthology.orchestra.run.smithi142.stdout:(29/120): ceph-mgr-cephadm-17.2.5-701.g8387d427 2.0 MB/s | 114 kB 00:00 2023-01-24T03:27:57.133 INFO:teuthology.orchestra.run.smithi142.stdout:(30/120): ceph-mgr-dashboard-17.2.5-701.g8387d4 4.5 MB/s | 1.6 MB 00:00 2023-01-24T03:27:57.207 INFO:teuthology.orchestra.run.smithi142.stdout:(31/120): ceph-mgr-modules-core-17.2.5-701.g838 3.1 MB/s | 233 kB 00:00 2023-01-24T03:27:57.273 INFO:teuthology.orchestra.run.smithi142.stdout:(32/120): ceph-mgr-rook-17.2.5-701.g8387d427.el 712 kB/s | 47 kB 00:00 2023-01-24T03:27:57.331 INFO:teuthology.orchestra.run.smithi142.stdout:(33/120): ceph-prometheus-alerts-17.2.5-701.g83 253 kB/s | 14 kB 00:00 2023-01-24T03:27:57.382 INFO:teuthology.orchestra.run.smithi142.stdout:(34/120): ceph-volume-17.2.5-701.g8387d427.el8. 5.0 MB/s | 263 kB 00:00 2023-01-24T03:27:57.436 INFO:teuthology.orchestra.run.smithi142.stdout:(35/120): cephadm-17.2.5-701.g8387d427.el8.noar 1.4 MB/s | 78 kB 00:00 2023-01-24T03:27:57.460 INFO:teuthology.orchestra.run.smithi112.stdout:(10/120): ceph-common-17.2.5-701.g8387d427.el8. 3.4 MB/s | 23 MB 00:06 2023-01-24T03:27:57.492 INFO:teuthology.orchestra.run.smithi142.stdout:(36/120): cephfs-top-17.2.5-701.g8387d427.el8.n 277 kB/s | 15 kB 00:00 2023-01-24T03:27:57.584 INFO:teuthology.orchestra.run.smithi112.stdout:(11/120): cephfs-mirror-17.2.5-701.g8387d427.el 2.0 MB/s | 254 kB 00:00 2023-01-24T03:27:57.642 INFO:teuthology.orchestra.run.smithi112.stdout:(12/120): libcephfs-devel-17.2.5-701.g8387d427. 430 kB/s | 25 kB 00:00 2023-01-24T03:27:57.676 INFO:teuthology.orchestra.run.smithi142.stdout:(37/120): gperftools-libs-2.7-9.el8.x86_64.rpm 1.6 MB/s | 306 kB 00:00 2023-01-24T03:27:57.705 INFO:teuthology.orchestra.run.smithi142.stdout:(38/120): liboath-2.6.2-3.el8.x86_64.rpm 2.0 MB/s | 59 kB 00:00 2023-01-24T03:27:57.735 INFO:teuthology.orchestra.run.smithi142.stdout:(39/120): libunwind-1.3.1-3.el8.x86_64.rpm 2.5 MB/s | 75 kB 00:00 2023-01-24T03:27:57.793 INFO:teuthology.orchestra.run.smithi142.stdout:(40/120): python3-asyncssh-2.7.0-2.el8.noarch.r 7.9 MB/s | 476 kB 00:00 2023-01-24T03:27:57.822 INFO:teuthology.orchestra.run.smithi142.stdout:(41/120): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 1.5 MB/s | 44 kB 00:00 2023-01-24T03:27:57.858 INFO:teuthology.orchestra.run.smithi142.stdout:(42/120): python3-beautifulsoup4-4.6.3-2.el8.1. 5.0 MB/s | 185 kB 00:00 2023-01-24T03:27:57.887 INFO:teuthology.orchestra.run.smithi142.stdout:(43/120): python3-cachetools-3.1.1-4.el8.noarch 1.1 MB/s | 33 kB 00:00 2023-01-24T03:27:57.898 INFO:teuthology.orchestra.run.smithi112.stdout:(13/120): libcephfs2-17.2.5-701.g8387d427.el8.x 3.0 MB/s | 793 kB 00:00 2023-01-24T03:27:57.915 INFO:teuthology.orchestra.run.smithi142.stdout:(44/120): python3-certifi-2018.10.15-7.el8.noar 581 kB/s | 16 kB 00:00 2023-01-24T03:27:57.945 INFO:teuthology.orchestra.run.smithi142.stdout:(45/120): python3-cheroot-8.5.2-1.el8.noarch.rp 5.6 MB/s | 173 kB 00:00 2023-01-24T03:27:57.962 INFO:teuthology.orchestra.run.smithi112.stdout:(14/120): libcephsqlite-17.2.5-701.g8387d427.el 2.7 MB/s | 174 kB 00:00 2023-01-24T03:27:58.008 INFO:teuthology.orchestra.run.smithi142.stdout:(46/120): ceph-mgr-diskprediction-local-17.2.5- 6.0 MB/s | 7.4 MB 00:01 2023-01-24T03:27:58.009 INFO:teuthology.orchestra.run.smithi142.stdout:(47/120): python3-cherrypy-18.4.0-1.el8.noarch. 5.8 MB/s | 384 kB 00:00 2023-01-24T03:27:58.015 INFO:teuthology.orchestra.run.smithi112.stdout:(15/120): librados-devel-17.2.5-701.g8387d427.e 2.4 MB/s | 130 kB 00:00 2023-01-24T03:27:58.038 INFO:teuthology.orchestra.run.smithi142.stdout:(48/120): python3-jaraco-6.2-6.el8.noarch.rpm 396 kB/s | 11 kB 00:00 2023-01-24T03:27:58.067 INFO:teuthology.orchestra.run.smithi142.stdout:(49/120): python3-jaraco-functools-2.0-4.el8.no 617 kB/s | 18 kB 00:00 2023-01-24T03:27:58.166 INFO:teuthology.orchestra.run.smithi142.stdout:(50/120): python3-google-auth-1.1.1-10.el8.noar 515 kB/s | 81 kB 00:00 2023-01-24T03:27:58.178 INFO:teuthology.orchestra.run.smithi112.stdout:(16/120): libradosstriper1-17.2.5-701.g8387d427 3.0 MB/s | 506 kB 00:00 2023-01-24T03:27:58.197 INFO:teuthology.orchestra.run.smithi142.stdout:(51/120): python3-kubernetes-11.0.0-6.el8.noarc 8.9 MB/s | 1.2 MB 00:00 2023-01-24T03:27:58.199 INFO:teuthology.orchestra.run.smithi142.stdout:(52/120): python3-logutils-0.3.5-11.el8.noarch. 1.4 MB/s | 49 kB 00:00 2023-01-24T03:27:58.226 INFO:teuthology.orchestra.run.smithi142.stdout:(53/120): python3-more-itertools-7.2.0-3.el8.no 2.0 MB/s | 59 kB 00:00 2023-01-24T03:27:58.228 INFO:teuthology.orchestra.run.smithi142.stdout:(54/120): python3-natsort-7.1.1-2.el8.noarch.rp 2.0 MB/s | 60 kB 00:00 2023-01-24T03:27:58.257 INFO:teuthology.orchestra.run.smithi142.stdout:(55/120): python3-portend-2.6-1.el8.noarch.rpm 541 kB/s | 16 kB 00:00 2023-01-24T03:27:58.284 INFO:teuthology.orchestra.run.smithi142.stdout:(56/120): python3-pecan-1.3.2-9.el8.noarch.rpm 4.8 MB/s | 283 kB 00:00 2023-01-24T03:27:58.287 INFO:teuthology.orchestra.run.smithi142.stdout:(57/120): python3-repoze-lru-0.7-6.el8.noarch.r 1.1 MB/s | 34 kB 00:00 2023-01-24T03:27:58.314 INFO:teuthology.orchestra.run.smithi142.stdout:(58/120): python3-routes-2.4.1-12.el8.noarch.rp 6.3 MB/s | 196 kB 00:00 2023-01-24T03:27:58.316 INFO:teuthology.orchestra.run.smithi142.stdout:(59/120): python3-rsa-4.8-1.el8.noarch.rpm 2.1 MB/s | 61 kB 00:00 2023-01-24T03:27:58.343 INFO:teuthology.orchestra.run.smithi142.stdout:(60/120): python3-simplegeneric-0.8.1-17.el8.no 658 kB/s | 19 kB 00:00 2023-01-24T03:27:58.343 INFO:teuthology.orchestra.run.smithi142.stdout:(61/120): python3-singledispatch-3.4.0.3-18.el8 864 kB/s | 24 kB 00:00 2023-01-24T03:27:58.371 INFO:teuthology.orchestra.run.smithi142.stdout:(62/120): python3-trustme-0.6.0-4.el8.noarch.rp 1.0 MB/s | 27 kB 00:00 2023-01-24T03:27:58.372 INFO:teuthology.orchestra.run.smithi142.stdout:(63/120): python3-tempora-1.14.1-5.el8.noarch.r 1.0 MB/s | 29 kB 00:00 2023-01-24T03:27:58.396 INFO:teuthology.orchestra.run.smithi112.stdout:(17/120): ceph-radosgw-17.2.5-701.g8387d427.el8 3.3 MB/s | 17 MB 00:05 2023-01-24T03:27:58.404 INFO:teuthology.orchestra.run.smithi142.stdout:(64/120): python3-webob-1.8.5-1.el8.1.noarch.rp 7.7 MB/s | 251 kB 00:00 2023-01-24T03:27:58.442 INFO:teuthology.orchestra.run.smithi112.stdout:(18/120): python3-ceph-argparse-17.2.5-701.g838 977 kB/s | 45 kB 00:00 2023-01-24T03:27:58.456 INFO:teuthology.orchestra.run.smithi142.stdout:(65/120): python3-websocket-client-0.56.0-5.el8 1.2 MB/s | 61 kB 00:00 2023-01-24T03:27:58.482 INFO:teuthology.orchestra.run.smithi142.stdout:(66/120): python3-waitress-1.2.1-2.el8.1.noarch 2.1 MB/s | 241 kB 00:00 2023-01-24T03:27:58.485 INFO:teuthology.orchestra.run.smithi142.stdout:(67/120): python3-webtest-2.0.33-1.el8.noarch.r 2.9 MB/s | 85 kB 00:00 2023-01-24T03:27:58.488 INFO:teuthology.orchestra.run.smithi112.stdout:(19/120): python3-ceph-common-17.2.5-701.g8387d 2.3 MB/s | 108 kB 00:00 2023-01-24T03:27:58.510 INFO:teuthology.orchestra.run.smithi142.stdout:(68/120): python3-zc-lockfile-2.0-2.el8.noarch. 835 kB/s | 23 kB 00:00 2023-01-24T03:27:58.513 INFO:teuthology.orchestra.run.smithi142.stdout:(69/120): xmlstarlet-1.6.1-20.el8.x86_64.rpm 20 MB/s | 70 kB 00:00 2023-01-24T03:27:58.538 INFO:teuthology.orchestra.run.smithi112.stdout:(20/120): python3-cephfs-17.2.5-701.g8387d427.e 3.8 MB/s | 190 kB 00:00 2023-01-24T03:27:58.544 INFO:teuthology.orchestra.run.smithi142.stdout:(70/120): libaio-devel-0.3.112-1.el8.x86_64.rpm 619 kB/s | 19 kB 00:00 2023-01-24T03:27:58.558 INFO:teuthology.orchestra.run.smithi142.stdout:(71/120): python3-jwt-1.6.1-2.el8.noarch.rpm 3.0 MB/s | 43 kB 00:00 2023-01-24T03:27:58.573 INFO:teuthology.orchestra.run.smithi142.stdout:(72/120): python3-oauthlib-2.1.0-1.el8.noarch.r 10 MB/s | 155 kB 00:00 2023-01-24T03:27:58.588 INFO:teuthology.orchestra.run.smithi142.stdout:(73/120): pcre2-utf32-10.32-2.el8.x86_64.rpm 15 MB/s | 220 kB 00:00 2023-01-24T03:27:58.601 INFO:teuthology.orchestra.run.smithi142.stdout:(74/120): keyutils-libs-devel-1.5.10-6.el8.x86_ 3.7 MB/s | 48 kB 00:00 2023-01-24T03:27:58.615 INFO:teuthology.orchestra.run.smithi142.stdout:(75/120): pcre2-utf16-10.32-2.el8.x86_64.rpm 16 MB/s | 229 kB 00:00 2023-01-24T03:27:58.628 INFO:teuthology.orchestra.run.smithi142.stdout:(76/120): python3-requests-oauthlib-1.0.0-1.el8 3.2 MB/s | 43 kB 00:00 2023-01-24T03:27:58.633 INFO:teuthology.orchestra.run.smithi142.stdout:(77/120): thrift-0.13.0-2.el8.x86_64.rpm 12 MB/s | 1.7 MB 00:00 2023-01-24T03:27:58.635 INFO:teuthology.orchestra.run.smithi112.stdout:(21/120): python3-rados-17.2.5-701.g8387d427.el 3.7 MB/s | 370 kB 00:00 2023-01-24T03:27:58.661 INFO:teuthology.orchestra.run.smithi142.stdout:(78/120): libverto-devel-0.3.0-5.el8.x86_64.rpm 656 kB/s | 18 kB 00:00 2023-01-24T03:27:58.671 INFO:teuthology.orchestra.run.smithi142.stdout:(79/120): openssl-devel-1.1.1g-15.el8_3.x86_64. 54 MB/s | 2.3 MB 00:00 2023-01-24T03:27:58.679 INFO:teuthology.orchestra.run.smithi142.stdout:(80/120): pcre2-devel-10.32-2.el8.x86_64.rpm 33 MB/s | 605 kB 00:00 2023-01-24T03:27:58.686 INFO:teuthology.orchestra.run.smithi142.stdout:(81/120): fuse-2.9.7-12.el8.x86_64.rpm 5.7 MB/s | 83 kB 00:00 2023-01-24T03:27:58.695 INFO:teuthology.orchestra.run.smithi142.stdout:(82/120): elfutils-libelf-devel-0.182-3.el8.x86 3.7 MB/s | 59 kB 00:00 2023-01-24T03:27:58.702 INFO:teuthology.orchestra.run.smithi142.stdout:(83/120): libquadmath-8.4.1-1.el8.x86_64.rpm 10 MB/s | 169 kB 00:00 2023-01-24T03:27:58.710 INFO:teuthology.orchestra.run.smithi142.stdout:(84/120): libcom_err-devel-1.45.6-1.el8.x86_64. 2.5 MB/s | 38 kB 00:00 2023-01-24T03:27:58.719 INFO:teuthology.orchestra.run.smithi142.stdout:(85/120): libsepol-devel-2.9-2.el8.x86_64.rpm 5.0 MB/s | 86 kB 00:00 2023-01-24T03:27:58.723 INFO:teuthology.orchestra.run.smithi142.stdout:(86/120): zlib-devel-1.2.11-17.el8.x86_64.rpm 4.3 MB/s | 58 kB 00:00 2023-01-24T03:27:58.731 INFO:teuthology.orchestra.run.smithi142.stdout:(87/120): librabbitmq-0.9.0-3.el8.x86_64.rpm 3.7 MB/s | 47 kB 00:00 2023-01-24T03:27:58.733 INFO:teuthology.orchestra.run.smithi112.stdout:(22/120): python3-rbd-17.2.5-701.g8387d427.el8. 3.6 MB/s | 358 kB 00:00 2023-01-24T03:27:58.741 INFO:teuthology.orchestra.run.smithi142.stdout:(88/120): krb5-devel-1.18.2-8.3.el8_4.x86_64.rp 30 MB/s | 559 kB 00:00 2023-01-24T03:27:58.750 INFO:teuthology.orchestra.run.smithi142.stdout:(89/120): libgfortran-8.4.1-1.el8.x86_64.rpm 33 MB/s | 639 kB 00:00 2023-01-24T03:27:58.754 INFO:teuthology.orchestra.run.smithi142.stdout:(90/120): xfsprogs-devel-5.0.0-8.el8.x86_64.rpm 5.3 MB/s | 70 kB 00:00 2023-01-24T03:27:58.765 INFO:teuthology.orchestra.run.smithi142.stdout:(91/120): libselinux-devel-2.9-5.el8.x86_64.rpm 14 MB/s | 200 kB 00:00 2023-01-24T03:27:58.768 INFO:teuthology.orchestra.run.smithi142.stdout:(92/120): libuuid-devel-2.32.1-27.el8.x86_64.rp 6.9 MB/s | 97 kB 00:00 2023-01-24T03:27:58.780 INFO:teuthology.orchestra.run.smithi112.stdout:(23/120): python3-rgw-17.2.5-701.g8387d427.el8. 2.2 MB/s | 106 kB 00:00 2023-01-24T03:27:58.782 INFO:teuthology.orchestra.run.smithi142.stdout:(93/120): python3-prettytable-0.7.2-14.el8.noar 3.2 MB/s | 44 kB 00:00 2023-01-24T03:27:58.795 INFO:teuthology.orchestra.run.smithi142.stdout:(94/120): python3-jsonpatch-1.21-2.el8.noarch.r 2.1 MB/s | 27 kB 00:00 2023-01-24T03:27:58.809 INFO:teuthology.orchestra.run.smithi142.stdout:(95/120): jq-1.5-12.el8.x86_64.rpm 12 MB/s | 161 kB 00:00 2023-01-24T03:27:58.825 INFO:teuthology.orchestra.run.smithi142.stdout:(96/120): flex-2.6.1-9.el8.x86_64.rpm 20 MB/s | 320 kB 00:00 2023-01-24T03:27:58.826 INFO:teuthology.orchestra.run.smithi112.stdout:(24/120): rbd-fuse-17.2.5-701.g8387d427.el8.x86 1.9 MB/s | 90 kB 00:00 2023-01-24T03:27:58.843 INFO:teuthology.orchestra.run.smithi142.stdout:(97/120): python3-werkzeug-0.12.2-4.el8.noarch. 25 MB/s | 457 kB 00:00 2023-01-24T03:27:58.918 INFO:teuthology.orchestra.run.smithi142.stdout:(98/120): python3-babel-2.5.1-5.el8.noarch.rpm 64 MB/s | 4.8 MB 00:00 2023-01-24T03:27:59.032 INFO:teuthology.orchestra.run.smithi142.stdout:(99/120): python3-scipy-1.0.0-20.module+el8.1.0 55 MB/s | 15 MB 00:00 2023-01-24T03:27:59.033 INFO:teuthology.orchestra.run.smithi142.stdout:(100/120): python3-markupsafe-0.23-19.el8.x86_6 337 kB/s | 39 kB 00:00 2023-01-24T03:27:59.049 INFO:teuthology.orchestra.run.smithi142.stdout:(101/120): lttng-ust-2.8.1-11.el8.x86_64.rpm 15 MB/s | 259 kB 00:00 2023-01-24T03:27:59.055 INFO:teuthology.orchestra.run.smithi142.stdout:(102/120): bison-3.0.4-10.el8.x86_64.rpm 29 MB/s | 688 kB 00:00 2023-01-24T03:27:59.066 INFO:teuthology.orchestra.run.smithi142.stdout:(103/120): python3-numpy-f2py-1.14.3-9.el8.x86_ 14 MB/s | 225 kB 00:00 2023-01-24T03:27:59.074 INFO:teuthology.orchestra.run.smithi142.stdout:(104/120): python3-pyasn1-0.3.7-6.el8.noarch.rp 6.5 MB/s | 126 kB 00:00 2023-01-24T03:27:59.090 INFO:teuthology.orchestra.run.smithi142.stdout:(105/120): oniguruma-6.8.2-2.el8.x86_64.rpm 11 MB/s | 187 kB 00:00 2023-01-24T03:27:59.122 INFO:teuthology.orchestra.run.smithi142.stdout:(106/120): python3-numpy-1.14.3-9.el8.x86_64.rp 67 MB/s | 3.7 MB 00:00 2023-01-24T03:27:59.128 INFO:teuthology.orchestra.run.smithi142.stdout:(107/120): python3-jinja2-2.10.1-2.el8_0.noarch 14 MB/s | 538 kB 00:00 2023-01-24T03:27:59.138 INFO:teuthology.orchestra.run.smithi142.stdout:(108/120): socat-1.7.3.3-2.el8.x86_64.rpm 19 MB/s | 302 kB 00:00 2023-01-24T03:27:59.143 INFO:teuthology.orchestra.run.smithi142.stdout:(109/120): python3-mako-1.0.6-13.el8.noarch.rpm 10 MB/s | 157 kB 00:00 2023-01-24T03:27:59.151 INFO:teuthology.orchestra.run.smithi142.stdout:(110/120): python3-pyasn1-modules-0.3.7-6.el8.n 7.9 MB/s | 110 kB 00:00 2023-01-24T03:27:59.159 INFO:teuthology.orchestra.run.smithi142.stdout:(111/120): librdkafka-0.11.4-1.el8.x86_64.rpm 22 MB/s | 353 kB 00:00 2023-01-24T03:27:59.164 INFO:teuthology.orchestra.run.smithi142.stdout:(112/120): python3-jsonpointer-1.10-11.el8.noar 1.5 MB/s | 20 kB 00:00 2023-01-24T03:27:59.177 INFO:teuthology.orchestra.run.smithi142.stdout:(113/120): python3-pytz-2017.2-9.el8.noarch.rpm 4.1 MB/s | 54 kB 00:00 2023-01-24T03:27:59.182 INFO:teuthology.orchestra.run.smithi142.stdout:(114/120): libtool-2.4.6-25.el8.x86_64.rpm 30 MB/s | 709 kB 00:00 2023-01-24T03:27:59.339 INFO:teuthology.orchestra.run.smithi112.stdout:(25/120): librgw2-17.2.5-701.g8387d427.el8.x86_ 4.7 MB/s | 5.5 MB 00:01 2023-01-24T03:27:59.374 INFO:teuthology.orchestra.run.smithi112.stdout:(26/120): rbd-nbd-17.2.5-701.g8387d427.el8.x86_ 4.9 MB/s | 172 kB 00:00 2023-01-24T03:27:59.403 INFO:teuthology.orchestra.run.smithi112.stdout:(27/120): rbd-mirror-17.2.5-701.g8387d427.el8.x 6.0 MB/s | 3.5 MB 00:00 2023-01-24T03:27:59.403 INFO:teuthology.orchestra.run.smithi112.stdout:(28/120): ceph-grafana-dashboards-17.2.5-701.g8 781 kB/s | 23 kB 00:00 2023-01-24T03:27:59.427 INFO:teuthology.orchestra.run.smithi112.stdout:(29/120): ceph-mgr-cephadm-17.2.5-701.g8387d427 4.7 MB/s | 114 kB 00:00 2023-01-24T03:27:59.532 INFO:teuthology.orchestra.run.smithi142.stdout:(115/120): ceph-test-17.2.5-701.g8387d427.el8.x 9.1 MB/s | 52 MB 00:05 2023-01-24T03:27:59.570 INFO:teuthology.orchestra.run.smithi142.stdout:(116/120): automake-1.16.1-7.el8.noarch.rpm 18 MB/s | 713 kB 00:00 2023-01-24T03:27:59.610 INFO:teuthology.orchestra.run.smithi142.stdout:(117/120): openblas-0.3.12-1.el8.x86_64.rpm 11 MB/s | 4.6 MB 00:00 2023-01-24T03:27:59.622 INFO:teuthology.orchestra.run.smithi142.stdout:(118/120): openblas-threads-0.3.12-1.el8.x86_64 11 MB/s | 4.7 MB 00:00 2023-01-24T03:27:59.631 INFO:teuthology.orchestra.run.smithi112.stdout:(30/120): ceph-mgr-dashboard-17.2.5-701.g8387d4 7.3 MB/s | 1.6 MB 00:00 2023-01-24T03:27:59.665 INFO:teuthology.orchestra.run.smithi112.stdout:(31/120): ceph-mgr-modules-core-17.2.5-701.g838 6.7 MB/s | 233 kB 00:00 2023-01-24T03:27:59.694 INFO:teuthology.orchestra.run.smithi112.stdout:(32/120): ceph-mgr-rook-17.2.5-701.g8387d427.el 1.6 MB/s | 47 kB 00:00 2023-01-24T03:27:59.723 INFO:teuthology.orchestra.run.smithi112.stdout:(33/120): ceph-prometheus-alerts-17.2.5-701.g83 488 kB/s | 14 kB 00:00 2023-01-24T03:27:59.800 INFO:teuthology.orchestra.run.smithi112.stdout:(34/120): ceph-volume-17.2.5-701.g8387d427.el8. 3.4 MB/s | 263 kB 00:00 2023-01-24T03:27:59.844 INFO:teuthology.orchestra.run.smithi112.stdout:(35/120): cephadm-17.2.5-701.g8387d427.el8.noar 1.7 MB/s | 78 kB 00:00 2023-01-24T03:27:59.893 INFO:teuthology.orchestra.run.smithi112.stdout:(36/120): cephfs-top-17.2.5-701.g8387d427.el8.n 316 kB/s | 15 kB 00:00 2023-01-24T03:28:00.058 INFO:teuthology.orchestra.run.smithi112.stdout:(37/120): gperftools-libs-2.7-9.el8.x86_64.rpm 1.8 MB/s | 306 kB 00:00 2023-01-24T03:28:00.087 INFO:teuthology.orchestra.run.smithi112.stdout:(38/120): liboath-2.6.2-3.el8.x86_64.rpm 2.0 MB/s | 59 kB 00:00 2023-01-24T03:28:00.115 INFO:teuthology.orchestra.run.smithi112.stdout:(39/120): libunwind-1.3.1-3.el8.x86_64.rpm 2.6 MB/s | 75 kB 00:00 2023-01-24T03:28:00.172 INFO:teuthology.orchestra.run.smithi112.stdout:(40/120): python3-asyncssh-2.7.0-2.el8.noarch.r 8.2 MB/s | 476 kB 00:00 2023-01-24T03:28:00.178 INFO:teuthology.orchestra.run.smithi142.stdout:(119/120): librados2-17.2.5-701.g8387d427.el8.x 5.7 MB/s | 3.5 MB 00:00 2023-01-24T03:28:00.200 INFO:teuthology.orchestra.run.smithi112.stdout:(41/120): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 1.5 MB/s | 44 kB 00:00 2023-01-24T03:28:00.227 INFO:teuthology.orchestra.run.smithi142.stdout:(120/120): librbd1-17.2.5-701.g8387d427.el8.x86 5.8 MB/s | 3.6 MB 00:00 2023-01-24T03:28:00.229 INFO:teuthology.orchestra.run.smithi112.stdout:(42/120): python3-beautifulsoup4-4.6.3-2.el8.1. 6.2 MB/s | 185 kB 00:00 2023-01-24T03:28:00.230 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2023-01-24T03:28:00.231 INFO:teuthology.orchestra.run.smithi142.stdout:Total 21 MB/s | 202 MB 00:09 2023-01-24T03:28:00.257 INFO:teuthology.orchestra.run.smithi112.stdout:(43/120): python3-cachetools-3.1.1-4.el8.noarch 1.2 MB/s | 33 kB 00:00 2023-01-24T03:28:00.277 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2023-01-24T03:28:00.284 INFO:teuthology.orchestra.run.smithi112.stdout:(44/120): python3-certifi-2018.10.15-7.el8.noar 591 kB/s | 16 kB 00:00 2023-01-24T03:28:00.314 INFO:teuthology.orchestra.run.smithi112.stdout:(45/120): python3-cheroot-8.5.2-1.el8.noarch.rp 5.8 MB/s | 173 kB 00:00 2023-01-24T03:28:00.370 INFO:teuthology.orchestra.run.smithi112.stdout:(46/120): python3-cherrypy-18.4.0-1.el8.noarch. 6.7 MB/s | 384 kB 00:00 2023-01-24T03:28:00.391 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2023-01-24T03:28:00.391 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2023-01-24T03:28:00.416 INFO:teuthology.orchestra.run.smithi112.stdout:(47/120): ceph-mgr-diskprediction-local-17.2.5- 7.5 MB/s | 7.4 MB 00:00 2023-01-24T03:28:00.417 INFO:teuthology.orchestra.run.smithi112.stdout:(48/120): python3-google-auth-1.1.1-10.el8.noar 1.7 MB/s | 81 kB 00:00 2023-01-24T03:28:00.446 INFO:teuthology.orchestra.run.smithi112.stdout:(49/120): python3-jaraco-functools-2.0-4.el8.no 624 kB/s | 18 kB 00:00 2023-01-24T03:28:00.471 INFO:teuthology.orchestra.run.smithi112.stdout:(50/120): python3-jaraco-6.2-6.el8.noarch.rpm 202 kB/s | 11 kB 00:00 2023-01-24T03:28:00.526 INFO:teuthology.orchestra.run.smithi112.stdout:(51/120): python3-logutils-0.3.5-11.el8.noarch. 894 kB/s | 49 kB 00:00 2023-01-24T03:28:00.554 INFO:teuthology.orchestra.run.smithi112.stdout:(52/120): python3-more-itertools-7.2.0-3.el8.no 2.1 MB/s | 59 kB 00:00 2023-01-24T03:28:00.558 INFO:teuthology.orchestra.run.smithi112.stdout:(53/120): python3-kubernetes-11.0.0-6.el8.noarc 10 MB/s | 1.2 MB 00:00 2023-01-24T03:28:00.584 INFO:teuthology.orchestra.run.smithi112.stdout:(54/120): python3-natsort-7.1.1-2.el8.noarch.rp 2.0 MB/s | 60 kB 00:00 2023-01-24T03:28:00.589 INFO:teuthology.orchestra.run.smithi112.stdout:(55/120): python3-pecan-1.3.2-9.el8.noarch.rpm 9.1 MB/s | 283 kB 00:00 2023-01-24T03:28:00.612 INFO:teuthology.orchestra.run.smithi112.stdout:(56/120): python3-portend-2.6-1.el8.noarch.rpm 554 kB/s | 16 kB 00:00 2023-01-24T03:28:00.616 INFO:teuthology.orchestra.run.smithi112.stdout:(57/120): python3-repoze-lru-0.7-6.el8.noarch.r 1.2 MB/s | 34 kB 00:00 2023-01-24T03:28:00.644 INFO:teuthology.orchestra.run.smithi112.stdout:(58/120): python3-rsa-4.8-1.el8.noarch.rpm 2.2 MB/s | 61 kB 00:00 2023-01-24T03:28:00.668 INFO:teuthology.orchestra.run.smithi112.stdout:(59/120): python3-routes-2.4.1-12.el8.noarch.rp 3.5 MB/s | 196 kB 00:00 2023-01-24T03:28:00.672 INFO:teuthology.orchestra.run.smithi112.stdout:(60/120): python3-simplegeneric-0.8.1-17.el8.no 670 kB/s | 19 kB 00:00 2023-01-24T03:28:00.696 INFO:teuthology.orchestra.run.smithi112.stdout:(61/120): python3-singledispatch-3.4.0.3-18.el8 858 kB/s | 24 kB 00:00 2023-01-24T03:28:00.700 INFO:teuthology.orchestra.run.smithi112.stdout:(62/120): python3-tempora-1.14.1-5.el8.noarch.r 1.0 MB/s | 29 kB 00:00 2023-01-24T03:28:00.724 INFO:teuthology.orchestra.run.smithi112.stdout:(63/120): python3-trustme-0.6.0-4.el8.noarch.rp 983 kB/s | 27 kB 00:00 2023-01-24T03:28:00.731 INFO:teuthology.orchestra.run.smithi112.stdout:(64/120): python3-waitress-1.2.1-2.el8.1.noarch 7.9 MB/s | 241 kB 00:00 2023-01-24T03:28:00.755 INFO:teuthology.orchestra.run.smithi112.stdout:(65/120): python3-webob-1.8.5-1.el8.1.noarch.rp 8.1 MB/s | 251 kB 00:00 2023-01-24T03:28:00.759 INFO:teuthology.orchestra.run.smithi112.stdout:(66/120): python3-websocket-client-0.56.0-5.el8 2.1 MB/s | 61 kB 00:00 2023-01-24T03:28:00.783 INFO:teuthology.orchestra.run.smithi112.stdout:(67/120): python3-webtest-2.0.33-1.el8.noarch.r 2.9 MB/s | 85 kB 00:00 2023-01-24T03:28:00.786 INFO:teuthology.orchestra.run.smithi112.stdout:(68/120): python3-zc-lockfile-2.0-2.el8.noarch. 836 kB/s | 23 kB 00:00 2023-01-24T03:28:00.790 INFO:teuthology.orchestra.run.smithi112.stdout:(69/120): xmlstarlet-1.6.1-20.el8.x86_64.rpm 20 MB/s | 70 kB 00:00 2023-01-24T03:28:01.054 INFO:teuthology.orchestra.run.smithi112.stdout:(70/120): python3-scipy-1.0.0-20.module+el8.1.0 55 MB/s | 15 MB 00:00 2023-01-24T03:28:01.059 INFO:teuthology.orchestra.run.smithi112.stdout:(71/120): thrift-0.13.0-2.el8.x86_64.rpm 6.3 MB/s | 1.7 MB 00:00 2023-01-24T03:28:01.074 INFO:teuthology.orchestra.run.smithi112.stdout:(72/120): python3-prettytable-0.7.2-14.el8.noar 2.2 MB/s | 44 kB 00:00 2023-01-24T03:28:01.090 INFO:teuthology.orchestra.run.smithi112.stdout:(73/120): python3-jsonpatch-1.21-2.el8.noarch.r 867 kB/s | 27 kB 00:00 2023-01-24T03:28:01.092 INFO:teuthology.orchestra.run.smithi112.stdout:(74/120): jq-1.5-12.el8.x86_64.rpm 8.7 MB/s | 161 kB 00:00 2023-01-24T03:28:01.108 INFO:teuthology.orchestra.run.smithi112.stdout:(75/120): flex-2.6.1-9.el8.x86_64.rpm 17 MB/s | 320 kB 00:00 2023-01-24T03:28:01.113 INFO:teuthology.orchestra.run.smithi112.stdout:(76/120): python3-werkzeug-0.12.2-4.el8.noarch. 22 MB/s | 457 kB 00:00 2023-01-24T03:28:01.128 INFO:teuthology.orchestra.run.smithi112.stdout:(77/120): python3-markupsafe-0.23-19.el8.x86_64 2.6 MB/s | 39 kB 00:00 2023-01-24T03:28:01.150 INFO:teuthology.orchestra.run.smithi112.stdout:(78/120): bison-3.0.4-10.el8.x86_64.rpm 30 MB/s | 688 kB 00:00 2023-01-24T03:28:01.167 INFO:teuthology.orchestra.run.smithi112.stdout:(79/120): lttng-ust-2.8.1-11.el8.x86_64.rpm 15 MB/s | 259 kB 00:00 2023-01-24T03:28:01.181 INFO:teuthology.orchestra.run.smithi112.stdout:(80/120): python3-babel-2.5.1-5.el8.noarch.rpm 66 MB/s | 4.8 MB 00:00 2023-01-24T03:28:01.183 INFO:teuthology.orchestra.run.smithi112.stdout:(81/120): python3-numpy-f2py-1.14.3-9.el8.x86_6 14 MB/s | 225 kB 00:00 2023-01-24T03:28:01.195 INFO:teuthology.orchestra.run.smithi112.stdout:(82/120): python3-pyasn1-0.3.7-6.el8.noarch.rpm 8.8 MB/s | 126 kB 00:00 2023-01-24T03:28:01.211 INFO:teuthology.orchestra.run.smithi112.stdout:(83/120): oniguruma-6.8.2-2.el8.x86_64.rpm 12 MB/s | 187 kB 00:00 2023-01-24T03:28:01.593 INFO:teuthology.orchestra.run.smithi112.stdout:(84/120): ceph-test-17.2.5-701.g8387d427.el8.x8 8.7 MB/s | 52 MB 00:06 2023-01-24T03:28:01.602 INFO:teuthology.orchestra.run.smithi112.stdout:(85/120): python3-jinja2-2.10.1-2.el8_0.noarch. 1.3 MB/s | 538 kB 00:00 2023-01-24T03:28:01.633 INFO:teuthology.orchestra.run.smithi112.stdout:(86/120): python3-numpy-1.14.3-9.el8.x86_64.rpm 8.3 MB/s | 3.7 MB 00:00 2023-01-24T03:28:01.635 INFO:teuthology.orchestra.run.smithi112.stdout:(87/120): python3-mako-1.0.6-13.el8.noarch.rpm 5.6 MB/s | 157 kB 00:00 2023-01-24T03:28:01.638 INFO:teuthology.orchestra.run.smithi112.stdout:(88/120): socat-1.7.3.3-2.el8.x86_64.rpm 6.6 MB/s | 302 kB 00:00 2023-01-24T03:28:01.650 INFO:teuthology.orchestra.run.smithi112.stdout:(89/120): python3-pyasn1-modules-0.3.7-6.el8.no 6.3 MB/s | 110 kB 00:00 2023-01-24T03:28:01.654 INFO:teuthology.orchestra.run.smithi112.stdout:(90/120): python3-jsonpointer-1.10-11.el8.noarc 1.3 MB/s | 20 kB 00:00 2023-01-24T03:28:01.655 INFO:teuthology.orchestra.run.smithi112.stdout:(91/120): librdkafka-0.11.4-1.el8.x86_64.rpm 18 MB/s | 353 kB 00:00 2023-01-24T03:28:01.668 INFO:teuthology.orchestra.run.smithi112.stdout:(92/120): python3-pytz-2017.2-9.el8.noarch.rpm 3.8 MB/s | 54 kB 00:00 2023-01-24T03:28:01.676 INFO:teuthology.orchestra.run.smithi112.stdout:(93/120): libtool-2.4.6-25.el8.x86_64.rpm 27 MB/s | 709 kB 00:00 2023-01-24T03:28:01.702 INFO:teuthology.orchestra.run.smithi112.stdout:(94/120): automake-1.16.1-7.el8.noarch.rpm 27 MB/s | 713 kB 00:00 2023-01-24T03:28:01.716 INFO:teuthology.orchestra.run.smithi112.stdout:(95/120): libaio-devel-0.3.112-1.el8.x86_64.rpm 1.3 MB/s | 19 kB 00:00 2023-01-24T03:28:01.729 INFO:teuthology.orchestra.run.smithi112.stdout:(96/120): python3-jwt-1.6.1-2.el8.noarch.rpm 3.3 MB/s | 43 kB 00:00 2023-01-24T03:28:01.747 INFO:teuthology.orchestra.run.smithi112.stdout:(97/120): openblas-threads-0.3.12-1.el8.x86_64. 51 MB/s | 4.7 MB 00:00 2023-01-24T03:28:01.750 INFO:teuthology.orchestra.run.smithi112.stdout:(98/120): python3-oauthlib-2.1.0-1.el8.noarch.r 7.3 MB/s | 155 kB 00:00 2023-01-24T03:28:01.765 INFO:teuthology.orchestra.run.smithi112.stdout:(99/120): openblas-0.3.12-1.el8.x86_64.rpm 47 MB/s | 4.6 MB 00:00 2023-01-24T03:28:01.766 INFO:teuthology.orchestra.run.smithi112.stdout:(100/120): keyutils-libs-devel-1.5.10-6.el8.x86 3.0 MB/s | 48 kB 00:00 2023-01-24T03:28:01.768 INFO:teuthology.orchestra.run.smithi112.stdout:(101/120): pcre2-utf32-10.32-2.el8.x86_64.rpm 10 MB/s | 220 kB 00:00 2023-01-24T03:28:01.778 INFO:teuthology.orchestra.run.smithi112.stdout:(102/120): python3-requests-oauthlib-1.0.0-1.el 3.4 MB/s | 43 kB 00:00 2023-01-24T03:28:01.782 INFO:teuthology.orchestra.run.smithi112.stdout:(103/120): pcre2-utf16-10.32-2.el8.x86_64.rpm 13 MB/s | 229 kB 00:00 2023-01-24T03:28:01.794 INFO:teuthology.orchestra.run.smithi112.stdout:(104/120): libverto-devel-0.3.0-5.el8.x86_64.rp 1.1 MB/s | 18 kB 00:00 2023-01-24T03:28:01.808 INFO:teuthology.orchestra.run.smithi112.stdout:(105/120): pcre2-devel-10.32-2.el8.x86_64.rpm 23 MB/s | 605 kB 00:00 2023-01-24T03:28:01.815 INFO:teuthology.orchestra.run.smithi112.stdout:(106/120): openssl-devel-1.1.1g-15.el8_3.x86_64 50 MB/s | 2.3 MB 00:00 2023-01-24T03:28:01.815 INFO:teuthology.orchestra.run.smithi112.stdout:(107/120): fuse-2.9.7-12.el8.x86_64.rpm 3.8 MB/s | 83 kB 00:00 2023-01-24T03:28:01.821 INFO:teuthology.orchestra.run.smithi112.stdout:(108/120): elfutils-libelf-devel-0.182-3.el8.x8 4.3 MB/s | 59 kB 00:00 2023-01-24T03:28:01.831 INFO:teuthology.orchestra.run.smithi112.stdout:(109/120): libcom_err-devel-1.45.6-1.el8.x86_64 2.5 MB/s | 38 kB 00:00 2023-01-24T03:28:01.831 INFO:teuthology.orchestra.run.smithi112.stdout:(110/120): libquadmath-8.4.1-1.el8.x86_64.rpm 10 MB/s | 169 kB 00:00 2023-01-24T03:28:01.836 INFO:teuthology.orchestra.run.smithi112.stdout:(111/120): libsepol-devel-2.9-2.el8.x86_64.rpm 5.6 MB/s | 86 kB 00:00 2023-01-24T03:28:01.844 INFO:teuthology.orchestra.run.smithi112.stdout:(112/120): zlib-devel-1.2.11-17.el8.x86_64.rpm 4.3 MB/s | 58 kB 00:00 2023-01-24T03:28:01.847 INFO:teuthology.orchestra.run.smithi112.stdout:(113/120): librabbitmq-0.9.0-3.el8.x86_64.rpm 3.0 MB/s | 47 kB 00:00 2023-01-24T03:28:01.856 INFO:teuthology.orchestra.run.smithi112.stdout:(114/120): krb5-devel-1.18.2-8.3.el8_4.x86_64.r 29 MB/s | 559 kB 00:00 2023-01-24T03:28:01.861 INFO:teuthology.orchestra.run.smithi112.stdout:(115/120): xfsprogs-devel-5.0.0-8.el8.x86_64.rp 4.9 MB/s | 70 kB 00:00 2023-01-24T03:28:01.864 INFO:teuthology.orchestra.run.smithi112.stdout:(116/120): libgfortran-8.4.1-1.el8.x86_64.rpm 31 MB/s | 639 kB 00:00 2023-01-24T03:28:01.871 INFO:teuthology.orchestra.run.smithi112.stdout:(117/120): libselinux-devel-2.9-5.el8.x86_64.rp 13 MB/s | 200 kB 00:00 2023-01-24T03:28:01.875 INFO:teuthology.orchestra.run.smithi112.stdout:(118/120): libuuid-devel-2.32.1-27.el8.x86_64.r 7.3 MB/s | 97 kB 00:00 2023-01-24T03:28:02.075 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2023-01-24T03:28:02.080 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2023-01-24T03:28:02.152 INFO:teuthology.orchestra.run.smithi112.stdout:(119/120): librbd1-17.2.5-701.g8387d427.el8.x86 13 MB/s | 3.6 MB 00:00 2023-01-24T03:28:02.257 INFO:teuthology.orchestra.run.smithi112.stdout:(120/120): librados2-17.2.5-701.g8387d427.el8.x 8.8 MB/s | 3.5 MB 00:00 2023-01-24T03:28:02.260 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2023-01-24T03:28:02.260 INFO:teuthology.orchestra.run.smithi112.stdout:Total 17 MB/s | 202 MB 00:11 2023-01-24T03:28:02.307 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2023-01-24T03:28:02.418 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2023-01-24T03:28:02.418 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2023-01-24T03:28:04.178 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2023-01-24T03:28:04.182 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2023-01-24T03:28:06.917 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2023-01-24T03:28:07.168 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/122 2023-01-24T03:28:07.404 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 2/122 2023-01-24T03:28:07.536 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 2/122 2023-01-24T03:28:07.779 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/122 2023-01-24T03:28:08.150 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/122 2023-01-24T03:28:08.404 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : librados2-2:17.2.5-701.g8387d427.el8.x86_64 4/122 2023-01-24T03:28:08.779 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2023-01-24T03:28:08.783 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librados2-2:17.2.5-701.g8387d427.el8.x86_64 4/122 2023-01-24T03:28:08.974 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/122 2023-01-24T03:28:09.012 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : librbd1-2:17.2.5-701.g8387d427.el8.x86_64 5/122 2023-01-24T03:28:09.140 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librbd1-2:17.2.5-701.g8387d427.el8.x86_64 5/122 2023-01-24T03:28:09.214 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 2/122 2023-01-24T03:28:09.345 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libcephfs2-2:17.2.5-701.g8387d427.el8.x86_64 6/122 2023-01-24T03:28:09.427 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: libcephfs2-2:17.2.5-701.g8387d427.el8.x86_64 6/122 2023-01-24T03:28:09.477 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 3/122 2023-01-24T03:28:09.603 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 3/122 2023-01-24T03:28:09.642 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-rados-2:17.2.5-701.g8387d427.el8.x86_64 7/122 2023-01-24T03:28:09.786 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/122 2023-01-24T03:28:10.193 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/122 2023-01-24T03:28:10.436 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : librados2-2:17.2.5-701.g8387d427.el8.x86_64 5/122 2023-01-24T03:28:10.452 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/122 2023-01-24T03:28:10.832 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados2-2:17.2.5-701.g8387d427.el8.x86_64 5/122 2023-01-24T03:28:11.044 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : librbd1-2:17.2.5-701.g8387d427.el8.x86_64 6/122 2023-01-24T03:28:11.159 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/122 2023-01-24T03:28:11.182 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librbd1-2:17.2.5-701.g8387d427.el8.x86_64 6/122 2023-01-24T03:28:11.320 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : librgw2-2:17.2.5-701.g8387d427.el8.x86_64 10/122 2023-01-24T03:28:11.360 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephfs2-2:17.2.5-701.g8387d427.el8.x86_64 7/122 2023-01-24T03:28:11.418 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librgw2-2:17.2.5-701.g8387d427.el8.x86_64 10/122 2023-01-24T03:28:11.464 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libcephfs2-2:17.2.5-701.g8387d427.el8.x86_64 7/122 2023-01-24T03:28:11.666 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/122 2023-01-24T03:28:11.832 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rados-2:17.2.5-701.g8387d427.el8.x86_64 8/122 2023-01-24T03:28:11.869 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/122 2023-01-24T03:28:12.070 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libcephsqlite-2:17.2.5-701.g8387d427.el8.x86_64 13/122 2023-01-24T03:28:12.216 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: libcephsqlite-2:17.2.5-701.g8387d427.el8.x86_64 13/122 2023-01-24T03:28:12.370 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libradosstriper1-2:17.2.5-701.g8387d427.el8.x86_ 14/122 2023-01-24T03:28:12.473 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: libradosstriper1-2:17.2.5-701.g8387d427.el8.x86_ 14/122 2023-01-24T03:28:12.484 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/122 2023-01-24T03:28:12.642 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 15/122 2023-01-24T03:28:12.643 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librgw2-2:17.2.5-701.g8387d427.el8.x86_64 10/122 2023-01-24T03:28:12.744 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librgw2-2:17.2.5-701.g8387d427.el8.x86_64 10/122 2023-01-24T03:28:12.920 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 16/122 2023-01-24T03:28:12.964 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/122 2023-01-24T03:28:13.154 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/122 2023-01-24T03:28:13.160 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 17/122 2023-01-24T03:28:13.309 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : zlib-devel-1.2.11-17.el8.x86_64 18/122 2023-01-24T03:28:13.343 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephsqlite-2:17.2.5-701.g8387d427.el8.x86_64 13/122 2023-01-24T03:28:13.451 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libcephsqlite-2:17.2.5-701.g8387d427.el8.x86_64 13/122 2023-01-24T03:28:13.495 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 19/122 2023-01-24T03:28:13.629 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 19/122 2023-01-24T03:28:13.652 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libradosstriper1-2:17.2.5-701.g8387d427.el8.x86_ 14/122 2023-01-24T03:28:13.721 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libradosstriper1-2:17.2.5-701.g8387d427.el8.x86_ 14/122 2023-01-24T03:28:13.820 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 20/122 2023-01-24T03:28:13.872 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : zlib-devel-1.2.11-17.el8.x86_64 15/122 2023-01-24T03:28:14.068 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 16/122 2023-01-24T03:28:14.197 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 16/122 2023-01-24T03:28:14.350 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 20/122 2023-01-24T03:28:14.520 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 21/122 2023-01-24T03:28:14.735 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 17/122 2023-01-24T03:28:15.046 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 21/122 2023-01-24T03:28:15.219 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : openblas-0.3.12-1.el8.x86_64 22/122 2023-01-24T03:28:15.774 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 17/122 2023-01-24T03:28:15.951 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 18/122 2023-01-24T03:28:16.163 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 22/122 2023-01-24T03:28:16.448 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 23/122 2023-01-24T03:28:16.472 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 18/122 2023-01-24T03:28:16.634 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : openblas-0.3.12-1.el8.x86_64 19/122 2023-01-24T03:28:16.712 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 24/122 2023-01-24T03:28:16.963 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/122 2023-01-24T03:28:17.126 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-ceph-common-2:17.2.5-701.g8387d427.el8.x 26/122 2023-01-24T03:28:17.280 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-ceph-argparse-2:17.2.5-701.g8387d427.el8 27/122 2023-01-24T03:28:17.390 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 19/122 2023-01-24T03:28:17.536 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cephfs-2:17.2.5-701.g8387d427.el8.x86_64 28/122 2023-01-24T03:28:17.663 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 20/122 2023-01-24T03:28:17.758 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 29/122 2023-01-24T03:28:17.833 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 21/122 2023-01-24T03:28:18.047 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 22/122 2023-01-24T03:28:18.052 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 30/122 2023-01-24T03:28:18.299 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/122 2023-01-24T03:28:18.311 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 31/122 2023-01-24T03:28:18.620 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 24/122 2023-01-24T03:28:18.871 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/122 2023-01-24T03:28:19.033 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-ceph-common-2:17.2.5-701.g8387d427.el8.x 26/122 2023-01-24T03:28:19.161 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-ceph-argparse-2:17.2.5-701.g8387d427.el8 27/122 2023-01-24T03:28:19.434 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cephfs-2:17.2.5-701.g8387d427.el8.x86_64 28/122 2023-01-24T03:28:20.357 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 31/122 2023-01-24T03:28:20.483 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 29/122 2023-01-24T03:28:20.716 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 32/122 2023-01-24T03:28:20.926 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 30/122 2023-01-24T03:28:21.219 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 31/122 2023-01-24T03:28:21.491 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 32/122 2023-01-24T03:28:21.802 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 33/122 2023-01-24T03:28:21.936 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/122 2023-01-24T03:28:22.009 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 34/122 2023-01-24T03:28:22.281 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 34/122 2023-01-24T03:28:22.547 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 35/122 2023-01-24T03:28:22.798 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-rsa-4.8-1.el8.noarch 36/122 2023-01-24T03:28:23.021 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 37/122 2023-01-24T03:28:23.228 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-rgw-2:17.2.5-701.g8387d427.el8.x86_64 38/122 2023-01-24T03:28:23.460 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-rbd-2:17.2.5-701.g8387d427.el8.x86_64 39/122 2023-01-24T03:28:23.752 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : librados-devel-2:17.2.5-701.g8387d427.el8.x86_64 40/122 2023-01-24T03:28:23.980 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : automake-1.16.1-7.el8.noarch 41/122 2023-01-24T03:28:24.131 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 34/122 2023-01-24T03:28:24.159 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 42/122 2023-01-24T03:28:24.398 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 43/122 2023-01-24T03:28:24.447 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 35/122 2023-01-24T03:28:24.627 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 44/122 2023-01-24T03:28:24.699 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rsa-4.8-1.el8.noarch 36/122 2023-01-24T03:28:24.793 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 45/122 2023-01-24T03:28:24.878 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 45/122 2023-01-24T03:28:24.911 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 37/122 2023-01-24T03:28:25.043 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : jq-1.5-12.el8.x86_64 46/122 2023-01-24T03:28:25.143 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rgw-2:17.2.5-701.g8387d427.el8.x86_64 38/122 2023-01-24T03:28:25.215 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 47/122 2023-01-24T03:28:25.341 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rbd-2:17.2.5-701.g8387d427.el8.x86_64 39/122 2023-01-24T03:28:25.392 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libuuid-devel-2.32.1-27.el8.x86_64 48/122 2023-01-24T03:28:25.562 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librados-devel-2:17.2.5-701.g8387d427.el8.x86_64 40/122 2023-01-24T03:28:25.573 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libsepol-devel-2.9-2.el8.x86_64 49/122 2023-01-24T03:28:25.741 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libuuid-devel-2.32.1-27.el8.x86_64 41/122 2023-01-24T03:28:25.750 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libcom_err-devel-1.45.6-1.el8.x86_64 50/122 2023-01-24T03:28:25.913 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libsepol-devel-2.9-2.el8.x86_64 42/122 2023-01-24T03:28:25.937 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : fuse-2.9.7-12.el8.x86_64 51/122 2023-01-24T03:28:26.077 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcom_err-devel-1.45.6-1.el8.x86_64 43/122 2023-01-24T03:28:26.081 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 52/122 2023-01-24T03:28:26.219 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : fuse-2.9.7-12.el8.x86_64 44/122 2023-01-24T03:28:26.304 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 53/122 2023-01-24T03:28:26.388 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 45/122 2023-01-24T03:28:26.455 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 54/122 2023-01-24T03:28:26.558 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 46/122 2023-01-24T03:28:26.762 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 47/122 2023-01-24T03:28:27.080 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 48/122 2023-01-24T03:28:27.095 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/122 2023-01-24T03:28:27.355 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/122 2023-01-24T03:28:27.385 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 49/122 2023-01-24T03:28:27.630 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/122 2023-01-24T03:28:27.669 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 50/122 2023-01-24T03:28:27.928 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : krb5-devel-1.18.2-8.3.el8_4.x86_64 58/122 2023-01-24T03:28:27.994 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : krb5-devel-1.18.2-8.3.el8_4.x86_64 51/122 2023-01-24T03:28:28.118 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 59/122 2023-01-24T03:28:28.182 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : automake-1.16.1-7.el8.noarch 52/122 2023-01-24T03:28:28.299 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 60/122 2023-01-24T03:28:28.382 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 53/122 2023-01-24T03:28:28.573 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 61/122 2023-01-24T03:28:28.599 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 54/122 2023-01-24T03:28:28.809 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 55/122 2023-01-24T03:28:28.818 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 62/122 2023-01-24T03:28:28.966 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 56/122 2023-01-24T03:28:28.999 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 63/122 2023-01-24T03:28:29.028 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 56/122 2023-01-24T03:28:29.186 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 64/122 2023-01-24T03:28:29.241 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : jq-1.5-12.el8.x86_64 57/122 2023-01-24T03:28:29.398 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 65/122 2023-01-24T03:28:29.435 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 58/122 2023-01-24T03:28:29.578 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 66/122 2023-01-24T03:28:29.625 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 59/122 2023-01-24T03:28:29.805 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 60/122 2023-01-24T03:28:29.807 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 67/122 2023-01-24T03:28:30.057 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 68/122 2023-01-24T03:28:30.062 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 61/122 2023-01-24T03:28:30.246 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 69/122 2023-01-24T03:28:30.267 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 62/122 2023-01-24T03:28:30.416 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 70/122 2023-01-24T03:28:30.465 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 63/122 2023-01-24T03:28:30.609 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 64/122 2023-01-24T03:28:30.628 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 71/122 2023-01-24T03:28:30.771 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 65/122 2023-01-24T03:28:30.882 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 72/122 2023-01-24T03:28:31.003 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 66/122 2023-01-24T03:28:31.116 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 73/122 2023-01-24T03:28:31.222 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 67/122 2023-01-24T03:28:31.431 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-portend-2.6-1.el8.noarch 74/122 2023-01-24T03:28:31.486 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 68/122 2023-01-24T03:28:31.694 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 69/122 2023-01-24T03:28:31.708 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 75/122 2023-01-24T03:28:31.882 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 70/122 2023-01-24T03:28:31.928 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 76/122 2023-01-24T03:28:32.122 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 71/122 2023-01-24T03:28:32.191 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 77/122 2023-01-24T03:28:32.364 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 72/122 2023-01-24T03:28:32.623 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 73/122 2023-01-24T03:28:32.926 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-portend-2.6-1.el8.noarch 74/122 2023-01-24T03:28:33.183 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 75/122 2023-01-24T03:28:33.341 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 78/122 2023-01-24T03:28:33.435 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 76/122 2023-01-24T03:28:33.672 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 79/122 2023-01-24T03:28:33.706 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 77/122 2023-01-24T03:28:33.943 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 80/122 2023-01-24T03:28:34.286 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 81/122 2023-01-24T03:28:34.559 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 82/122 2023-01-24T03:28:34.900 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 78/122 2023-01-24T03:28:34.925 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 83/122 2023-01-24T03:28:35.173 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 84/122 2023-01-24T03:28:35.214 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 79/122 2023-01-24T03:28:35.375 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 85/122 2023-01-24T03:28:35.460 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 80/122 2023-01-24T03:28:35.609 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 86/122 2023-01-24T03:28:35.803 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 81/122 2023-01-24T03:28:36.150 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 82/122 2023-01-24T03:28:36.523 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 83/122 2023-01-24T03:28:36.806 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 84/122 2023-01-24T03:28:37.007 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 85/122 2023-01-24T03:28:37.232 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 86/122 2023-01-24T03:28:40.343 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-common-2:17.2.5-701.g8387d427.el8.x86_64 87/122 2023-01-24T03:28:40.621 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-common-2:17.2.5-701.g8387d427.el8.x86_64 87/122 2023-01-24T03:28:41.190 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-common-2:17.2.5-701.g8387d427.el8.x86_64 87/122 2023-01-24T03:28:41.527 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-base-2:17.2.5-701.g8387d427.el8.x86_64 88/122 2023-01-24T03:28:41.615 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-base-2:17.2.5-701.g8387d427.el8.x86_64 88/122 2023-01-24T03:28:41.765 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-selinux-2:17.2.5-701.g8387d427.el8.x86_64 89/122 2023-01-24T03:28:41.885 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-common-2:17.2.5-701.g8387d427.el8.x86_64 87/122 2023-01-24T03:28:42.143 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-common-2:17.2.5-701.g8387d427.el8.x86_64 87/122 2023-01-24T03:28:42.712 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-common-2:17.2.5-701.g8387d427.el8.x86_64 87/122 2023-01-24T03:28:43.009 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-base-2:17.2.5-701.g8387d427.el8.x86_64 88/122 2023-01-24T03:28:43.094 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-base-2:17.2.5-701.g8387d427.el8.x86_64 88/122 2023-01-24T03:28:43.206 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-selinux-2:17.2.5-701.g8387d427.el8.x86_64 89/122 2023-01-24T03:28:59.603 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-selinux-2:17.2.5-701.g8387d427.el8.x86_64 89/122 2023-01-24T03:28:59.604 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /sys 2023-01-24T03:28:59.604 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /proc 2023-01-24T03:28:59.604 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /dev 2023-01-24T03:28:59.604 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /run 2023-01-24T03:28:59.604 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /mnt 2023-01-24T03:28:59.605 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /var/tmp 2023-01-24T03:28:59.605 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /home 2023-01-24T03:28:59.605 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /tmp 2023-01-24T03:28:59.605 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /dev 2023-01-24T03:28:59.605 INFO:teuthology.orchestra.run.smithi142.stdout: 2023-01-24T03:28:59.963 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-volume-2:17.2.5-701.g8387d427.el8.noarch 90/122 2023-01-24T03:29:01.074 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-selinux-2:17.2.5-701.g8387d427.el8.x86_64 89/122 2023-01-24T03:29:01.075 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /sys 2023-01-24T03:29:01.075 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /proc 2023-01-24T03:29:01.075 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /dev 2023-01-24T03:29:01.075 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /run 2023-01-24T03:29:01.076 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /mnt 2023-01-24T03:29:01.076 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /var/tmp 2023-01-24T03:29:01.076 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /home 2023-01-24T03:29:01.076 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /tmp 2023-01-24T03:29:01.076 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /dev 2023-01-24T03:29:01.077 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:29:01.433 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-volume-2:17.2.5-701.g8387d427.el8.noarch 90/122 2023-01-24T03:29:01.479 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-volume-2:17.2.5-701.g8387d427.el8.noarch 90/122 2023-01-24T03:29:01.640 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-osd-2:17.2.5-701.g8387d427.el8.x86_64 91/122 2023-01-24T03:29:01.865 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-osd-2:17.2.5-701.g8387d427.el8.x86_64 91/122 2023-01-24T03:29:02.141 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mds-2:17.2.5-701.g8387d427.el8.x86_64 92/122 2023-01-24T03:29:02.431 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mds-2:17.2.5-701.g8387d427.el8.x86_64 92/122 2023-01-24T03:29:02.620 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mon-2:17.2.5-701.g8387d427.el8.x86_64 93/122 2023-01-24T03:29:02.636 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mon-2:17.2.5-701.g8387d427.el8.x86_64 93/122 2023-01-24T03:29:02.950 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-volume-2:17.2.5-701.g8387d427.el8.noarch 90/122 2023-01-24T03:29:03.157 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-osd-2:17.2.5-701.g8387d427.el8.x86_64 91/122 2023-01-24T03:29:03.382 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-osd-2:17.2.5-701.g8387d427.el8.x86_64 91/122 2023-01-24T03:29:03.562 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mds-2:17.2.5-701.g8387d427.el8.x86_64 92/122 2023-01-24T03:29:03.948 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mds-2:17.2.5-701.g8387d427.el8.x86_64 92/122 2023-01-24T03:29:04.112 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mon-2:17.2.5-701.g8387d427.el8.x86_64 93/122 2023-01-24T03:29:04.128 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mon-2:17.2.5-701.g8387d427.el8.x86_64 93/122 2023-01-24T03:29:05.151 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: cephadm-2:17.2.5-701.g8387d427.el8.noarch 94/122 2023-01-24T03:29:05.344 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : cephadm-2:17.2.5-701.g8387d427.el8.noarch 94/122 2023-01-24T03:29:05.526 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-prometheus-alerts-2:17.2.5-701.g8387d427.el 95/122 2023-01-24T03:29:05.728 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-grafana-dashboards-2:17.2.5-701.g8387d427.e 96/122 2023-01-24T03:29:05.903 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-cephadm-2:17.2.5-701.g8387d427.el8.noar 97/122 2023-01-24T03:29:06.108 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.5-701.g8387d427.el8.noar 97/122 2023-01-24T03:29:06.330 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-dashboard-2:17.2.5-701.g8387d427.el8.no 98/122 2023-01-24T03:29:06.565 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: cephadm-2:17.2.5-701.g8387d427.el8.noarch 94/122 2023-01-24T03:29:06.735 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : cephadm-2:17.2.5-701.g8387d427.el8.noarch 94/122 2023-01-24T03:29:06.901 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-prometheus-alerts-2:17.2.5-701.g8387d427.el 95/122 2023-01-24T03:29:07.069 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-grafana-dashboards-2:17.2.5-701.g8387d427.e 96/122 2023-01-24T03:29:07.261 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-cephadm-2:17.2.5-701.g8387d427.el8.noar 97/122 2023-01-24T03:29:07.289 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.5-701.g8387d427.el8.no 98/122 2023-01-24T03:29:07.461 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.5-701.g8387 99/122 2023-01-24T03:29:07.468 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.5-701.g8387d427.el8.noar 97/122 2023-01-24T03:29:07.556 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.5-701.g8387 99/122 2023-01-24T03:29:07.714 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-dashboard-2:17.2.5-701.g8387d427.el8.no 98/122 2023-01-24T03:29:07.935 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-modules-core-2:17.2.5-701.g8387d427.el8 100/122 2023-01-24T03:29:08.097 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-2:17.2.5-701.g8387d427.el8.x86_64 101/122 2023-01-24T03:29:08.132 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mgr-2:17.2.5-701.g8387d427.el8.x86_64 101/122 2023-01-24T03:29:08.336 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-rook-2:17.2.5-701.g8387d427.el8.noarch 102/122 2023-01-24T03:29:08.346 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.5-701.g8387d427.el8.noarch 102/122 2023-01-24T03:29:08.677 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.5-701.g8387d427.el8.no 98/122 2023-01-24T03:29:08.878 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.5-701.g8387 99/122 2023-01-24T03:29:08.975 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.5-701.g8387 99/122 2023-01-24T03:29:09.369 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-modules-core-2:17.2.5-701.g8387d427.el8 100/122 2023-01-24T03:29:09.547 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-2:17.2.5-701.g8387d427.el8.x86_64 101/122 2023-01-24T03:29:09.584 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-2:17.2.5-701.g8387d427.el8.x86_64 101/122 2023-01-24T03:29:09.769 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-rook-2:17.2.5-701.g8387d427.el8.noarch 102/122 2023-01-24T03:29:09.780 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.5-701.g8387d427.el8.noarch 102/122 2023-01-24T03:29:09.907 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-2:17.2.5-701.g8387d427.el8.x86_64 103/122 2023-01-24T03:29:10.164 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-radosgw-2:17.2.5-701.g8387d427.el8.x86_64 104/122 2023-01-24T03:29:10.242 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-radosgw-2:17.2.5-701.g8387d427.el8.x86_64 104/122 2023-01-24T03:29:10.427 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-immutable-object-cache-2:17.2.5-701.g8387d4 105/122 2023-01-24T03:29:10.469 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.5-701.g8387d4 105/122 2023-01-24T03:29:10.619 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : cephfs-mirror-2:17.2.5-701.g8387d427.el8.x86_64 106/122 2023-01-24T03:29:10.964 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: cephfs-mirror-2:17.2.5-701.g8387d427.el8.x86_64 106/122 2023-01-24T03:29:11.144 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : rbd-mirror-2:17.2.5-701.g8387d427.el8.x86_64 107/122 2023-01-24T03:29:11.345 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-2:17.2.5-701.g8387d427.el8.x86_64 103/122 2023-01-24T03:29:11.564 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-radosgw-2:17.2.5-701.g8387d427.el8.x86_64 104/122 2023-01-24T03:29:11.655 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-radosgw-2:17.2.5-701.g8387d427.el8.x86_64 104/122 2023-01-24T03:29:11.811 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-immutable-object-cache-2:17.2.5-701.g8387d4 105/122 2023-01-24T03:29:11.853 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.5-701.g8387d4 105/122 2023-01-24T03:29:12.002 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : cephfs-mirror-2:17.2.5-701.g8387d427.el8.x86_64 106/122 2023-01-24T03:29:12.348 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: cephfs-mirror-2:17.2.5-701.g8387d427.el8.x86_64 106/122 2023-01-24T03:29:12.519 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-mirror-2:17.2.5-701.g8387d427.el8.x86_64 107/122 2023-01-24T03:29:16.081 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: rbd-mirror-2:17.2.5-701.g8387d427.el8.x86_64 107/122 2023-01-24T03:29:17.018 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-test-2:17.2.5-701.g8387d427.el8.x86_64 108/122 2023-01-24T03:29:17.401 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : openssl-devel-1:1.1.1g-15.el8_3.x86_64 109/122 2023-01-24T03:29:17.467 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: rbd-mirror-2:17.2.5-701.g8387d427.el8.x86_64 107/122 2023-01-24T03:29:17.619 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-fuse-2:17.2.5-701.g8387d427.el8.x86_64 110/122 2023-01-24T03:29:17.796 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-test-2:17.2.5-701.g8387d427.el8.x86_64 108/122 2023-01-24T03:29:17.841 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : xfsprogs-devel-5.0.0-8.el8.x86_64 111/122 2023-01-24T03:29:17.968 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libtool-2.4.6-25.el8.x86_64 109/122 2023-01-24T03:29:18.034 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libtool-2.4.6-25.el8.x86_64 112/122 2023-01-24T03:29:18.045 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 112/122 2023-01-24T03:29:18.239 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libcephfs-devel-2:17.2.5-701.g8387d427.el8.x86_6 113/122 2023-01-24T03:29:18.389 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : elfutils-libelf-devel-0.182-3.el8.x86_64 114/122 2023-01-24T03:29:18.568 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : cephfs-top-2:17.2.5-701.g8387d427.el8.noarch 115/122 2023-01-24T03:29:18.658 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 109/122 2023-01-24T03:29:18.740 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : rbd-fuse-2:17.2.5-701.g8387d427.el8.x86_64 116/122 2023-01-24T03:29:19.015 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : rbd-nbd-2:17.2.5-701.g8387d427.el8.x86_64 117/122 2023-01-24T03:29:19.021 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : openssl-devel-1:1.1.1g-15.el8_3.x86_64 110/122 2023-01-24T03:29:19.217 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : bison-3.0.4-10.el8.x86_64 118/122 2023-01-24T03:29:19.236 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-fuse-2:17.2.5-701.g8387d427.el8.x86_64 111/122 2023-01-24T03:29:19.274 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 118/122 2023-01-24T03:29:19.405 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : xfsprogs-devel-5.0.0-8.el8.x86_64 112/122 2023-01-24T03:29:19.492 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : flex-2.6.1-9.el8.x86_64 119/122 2023-01-24T03:29:19.501 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 119/122 2023-01-24T03:29:19.556 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephfs-devel-2:17.2.5-701.g8387d427.el8.x86_6 113/122 2023-01-24T03:29:19.620 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 120/122 2023-01-24T03:29:19.620 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-01-24T03:29:19.714 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : elfutils-libelf-devel-0.182-3.el8.x86_64 114/122 2023-01-24T03:29:19.846 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-01-24T03:29:19.846 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 122/122 2023-01-24T03:29:19.868 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : cephfs-top-2:17.2.5-701.g8387d427.el8.noarch 115/122 2023-01-24T03:29:20.085 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-fuse-2:17.2.5-701.g8387d427.el8.x86_64 116/122 2023-01-24T03:29:20.236 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-nbd-2:17.2.5-701.g8387d427.el8.x86_64 117/122 2023-01-24T03:29:20.390 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 122/122 2023-01-24T03:29:20.391 INFO:teuthology.orchestra.run.smithi142.stdout:[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping 2023-01-24T03:29:20.391 INFO:teuthology.orchestra.run.smithi142.stdout:All rules containing unresolvable specifiers will be skipped. 2023-01-24T03:29:20.391 INFO:teuthology.orchestra.run.smithi142.stdout: 2023-01-24T03:29:20.454 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 118/122 2023-01-24T03:29:20.617 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : bison-3.0.4-10.el8.x86_64 119/122 2023-01-24T03:29:20.673 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 119/122 2023-01-24T03:29:20.723 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-2:17.2.5-701.g8387d427.el8.x86_64 1/122 2023-01-24T03:29:20.723 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-base-2:17.2.5-701.g8387d427.el8.x86_64 2/122 2023-01-24T03:29:20.724 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-common-2:17.2.5-701.g8387d427.el8.x86_64 3/122 2023-01-24T03:29:20.724 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-fuse-2:17.2.5-701.g8387d427.el8.x86_64 4/122 2023-01-24T03:29:20.724 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-immutable-object-cache-2:17.2.5-701.g8387d4 5/122 2023-01-24T03:29:20.724 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mds-2:17.2.5-701.g8387d427.el8.x86_64 6/122 2023-01-24T03:29:20.724 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-2:17.2.5-701.g8387d427.el8.x86_64 7/122 2023-01-24T03:29:20.725 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mon-2:17.2.5-701.g8387d427.el8.x86_64 8/122 2023-01-24T03:29:20.725 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-osd-2:17.2.5-701.g8387d427.el8.x86_64 9/122 2023-01-24T03:29:20.725 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-radosgw-2:17.2.5-701.g8387d427.el8.x86_64 10/122 2023-01-24T03:29:20.725 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-selinux-2:17.2.5-701.g8387d427.el8.x86_64 11/122 2023-01-24T03:29:20.725 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-test-2:17.2.5-701.g8387d427.el8.x86_64 12/122 2023-01-24T03:29:20.726 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : cephfs-mirror-2:17.2.5-701.g8387d427.el8.x86_64 13/122 2023-01-24T03:29:20.726 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libcephfs-devel-2:17.2.5-701.g8387d427.el8.x86_6 14/122 2023-01-24T03:29:20.726 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libcephfs2-2:17.2.5-701.g8387d427.el8.x86_64 15/122 2023-01-24T03:29:20.726 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libcephsqlite-2:17.2.5-701.g8387d427.el8.x86_64 16/122 2023-01-24T03:29:20.726 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librados-devel-2:17.2.5-701.g8387d427.el8.x86_64 17/122 2023-01-24T03:29:20.727 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libradosstriper1-2:17.2.5-701.g8387d427.el8.x86_ 18/122 2023-01-24T03:29:20.727 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librgw2-2:17.2.5-701.g8387d427.el8.x86_64 19/122 2023-01-24T03:29:20.727 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-ceph-argparse-2:17.2.5-701.g8387d427.el8 20/122 2023-01-24T03:29:20.727 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-ceph-common-2:17.2.5-701.g8387d427.el8.x 21/122 2023-01-24T03:29:20.727 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cephfs-2:17.2.5-701.g8387d427.el8.x86_64 22/122 2023-01-24T03:29:20.727 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-rados-2:17.2.5-701.g8387d427.el8.x86_64 23/122 2023-01-24T03:29:20.728 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-rbd-2:17.2.5-701.g8387d427.el8.x86_64 24/122 2023-01-24T03:29:20.728 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-rgw-2:17.2.5-701.g8387d427.el8.x86_64 25/122 2023-01-24T03:29:20.736 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : rbd-fuse-2:17.2.5-701.g8387d427.el8.x86_64 26/122 2023-01-24T03:29:20.736 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : rbd-mirror-2:17.2.5-701.g8387d427.el8.x86_64 27/122 2023-01-24T03:29:20.737 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : rbd-nbd-2:17.2.5-701.g8387d427.el8.x86_64 28/122 2023-01-24T03:29:20.737 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-grafana-dashboards-2:17.2.5-701.g8387d427.e 29/122 2023-01-24T03:29:20.737 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-cephadm-2:17.2.5-701.g8387d427.el8.noar 30/122 2023-01-24T03:29:20.737 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-dashboard-2:17.2.5-701.g8387d427.el8.no 31/122 2023-01-24T03:29:20.737 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.5-701.g8387 32/122 2023-01-24T03:29:20.738 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-modules-core-2:17.2.5-701.g8387d427.el8 33/122 2023-01-24T03:29:20.738 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-rook-2:17.2.5-701.g8387d427.el8.noarch 34/122 2023-01-24T03:29:20.738 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-prometheus-alerts-2:17.2.5-701.g8387d427.el 35/122 2023-01-24T03:29:20.738 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-volume-2:17.2.5-701.g8387d427.el8.noarch 36/122 2023-01-24T03:29:20.738 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : cephadm-2:17.2.5-701.g8387d427.el8.noarch 37/122 2023-01-24T03:29:20.739 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : cephfs-top-2:17.2.5-701.g8387d427.el8.noarch 38/122 2023-01-24T03:29:20.739 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/122 2023-01-24T03:29:20.739 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/122 2023-01-24T03:29:20.739 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/122 2023-01-24T03:29:20.739 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/122 2023-01-24T03:29:20.740 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/122 2023-01-24T03:29:20.740 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/122 2023-01-24T03:29:20.740 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/122 2023-01-24T03:29:20.740 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/122 2023-01-24T03:29:20.740 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/122 2023-01-24T03:29:20.741 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/122 2023-01-24T03:29:20.741 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/122 2023-01-24T03:29:20.741 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/122 2023-01-24T03:29:20.741 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/122 2023-01-24T03:29:20.741 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/122 2023-01-24T03:29:20.742 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/122 2023-01-24T03:29:20.742 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/122 2023-01-24T03:29:20.742 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/122 2023-01-24T03:29:20.742 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/122 2023-01-24T03:29:20.742 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/122 2023-01-24T03:29:20.743 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/122 2023-01-24T03:29:20.743 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/122 2023-01-24T03:29:20.743 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/122 2023-01-24T03:29:20.743 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/122 2023-01-24T03:29:20.743 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/122 2023-01-24T03:29:20.743 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/122 2023-01-24T03:29:20.744 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/122 2023-01-24T03:29:20.744 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 65/122 2023-01-24T03:29:20.744 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/122 2023-01-24T03:29:20.744 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/122 2023-01-24T03:29:20.744 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/122 2023-01-24T03:29:20.745 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/122 2023-01-24T03:29:20.745 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/122 2023-01-24T03:29:20.745 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/122 2023-01-24T03:29:20.745 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 72/122 2023-01-24T03:29:20.745 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 73/122 2023-01-24T03:29:20.746 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 74/122 2023-01-24T03:29:20.746 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/122 2023-01-24T03:29:20.746 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 76/122 2023-01-24T03:29:20.746 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 77/122 2023-01-24T03:29:20.746 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/122 2023-01-24T03:29:20.747 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : openssl-devel-1:1.1.1g-15.el8_3.x86_64 79/122 2023-01-24T03:29:20.747 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 80/122 2023-01-24T03:29:20.747 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 81/122 2023-01-24T03:29:20.747 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 82/122 2023-01-24T03:29:20.747 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : elfutils-libelf-devel-0.182-3.el8.x86_64 83/122 2023-01-24T03:29:20.748 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 84/122 2023-01-24T03:29:20.748 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libcom_err-devel-1.45.6-1.el8.x86_64 85/122 2023-01-24T03:29:20.748 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libsepol-devel-2.9-2.el8.x86_64 86/122 2023-01-24T03:29:20.748 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : zlib-devel-1.2.11-17.el8.x86_64 87/122 2023-01-24T03:29:20.748 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 88/122 2023-01-24T03:29:20.749 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : krb5-devel-1.18.2-8.3.el8_4.x86_64 89/122 2023-01-24T03:29:20.749 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 90/122 2023-01-24T03:29:20.749 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : xfsprogs-devel-5.0.0-8.el8.x86_64 91/122 2023-01-24T03:29:20.749 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 92/122 2023-01-24T03:29:20.749 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libuuid-devel-2.32.1-27.el8.x86_64 93/122 2023-01-24T03:29:20.750 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 94/122 2023-01-24T03:29:20.750 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 95/122 2023-01-24T03:29:20.750 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 96/122 2023-01-24T03:29:20.750 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : jq-1.5-12.el8.x86_64 97/122 2023-01-24T03:29:20.750 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : flex-2.6.1-9.el8.x86_64 98/122 2023-01-24T03:29:20.750 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 99/122 2023-01-24T03:29:20.752 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 100/122 2023-01-24T03:29:20.752 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 101/122 2023-01-24T03:29:20.752 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : bison-3.0.4-10.el8.x86_64 102/122 2023-01-24T03:29:20.752 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 103/122 2023-01-24T03:29:20.752 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 104/122 2023-01-24T03:29:20.753 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 105/122 2023-01-24T03:29:20.753 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 106/122 2023-01-24T03:29:20.753 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/122 2023-01-24T03:29:20.753 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 108/122 2023-01-24T03:29:20.753 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 109/122 2023-01-24T03:29:20.754 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 110/122 2023-01-24T03:29:20.754 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 111/122 2023-01-24T03:29:20.754 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 112/122 2023-01-24T03:29:20.754 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 113/122 2023-01-24T03:29:20.754 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 114/122 2023-01-24T03:29:20.754 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 115/122 2023-01-24T03:29:20.755 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 116/122 2023-01-24T03:29:20.755 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 117/122 2023-01-24T03:29:20.755 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : automake-1.16.1-7.el8.noarch 118/122 2023-01-24T03:29:20.755 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librados2-2:17.2.5-701.g8387d427.el8.x86_64 119/122 2023-01-24T03:29:20.755 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 120/122 2023-01-24T03:29:20.756 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librbd1-2:17.2.5-701.g8387d427.el8.x86_64 121/122 2023-01-24T03:29:20.876 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : flex-2.6.1-9.el8.x86_64 120/122 2023-01-24T03:29:20.885 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 120/122 2023-01-24T03:29:20.886 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-01-24T03:29:21.087 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-01-24T03:29:21.087 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 122/122 2023-01-24T03:29:21.629 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 122/122 2023-01-24T03:29:21.629 INFO:teuthology.orchestra.run.smithi112.stdout:[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping 2023-01-24T03:29:21.630 INFO:teuthology.orchestra.run.smithi112.stdout:All rules containing unresolvable specifiers will be skipped. 2023-01-24T03:29:21.630 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:29:21.991 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-2:17.2.5-701.g8387d427.el8.x86_64 1/122 2023-01-24T03:29:21.991 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-base-2:17.2.5-701.g8387d427.el8.x86_64 2/122 2023-01-24T03:29:21.992 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-common-2:17.2.5-701.g8387d427.el8.x86_64 3/122 2023-01-24T03:29:21.992 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-fuse-2:17.2.5-701.g8387d427.el8.x86_64 4/122 2023-01-24T03:29:21.992 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-immutable-object-cache-2:17.2.5-701.g8387d4 5/122 2023-01-24T03:29:21.992 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mds-2:17.2.5-701.g8387d427.el8.x86_64 6/122 2023-01-24T03:29:21.992 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-2:17.2.5-701.g8387d427.el8.x86_64 7/122 2023-01-24T03:29:21.993 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mon-2:17.2.5-701.g8387d427.el8.x86_64 8/122 2023-01-24T03:29:21.993 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-osd-2:17.2.5-701.g8387d427.el8.x86_64 9/122 2023-01-24T03:29:21.993 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-radosgw-2:17.2.5-701.g8387d427.el8.x86_64 10/122 2023-01-24T03:29:21.993 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-selinux-2:17.2.5-701.g8387d427.el8.x86_64 11/122 2023-01-24T03:29:21.993 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-test-2:17.2.5-701.g8387d427.el8.x86_64 12/122 2023-01-24T03:29:21.994 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : cephfs-mirror-2:17.2.5-701.g8387d427.el8.x86_64 13/122 2023-01-24T03:29:21.994 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephfs-devel-2:17.2.5-701.g8387d427.el8.x86_6 14/122 2023-01-24T03:29:21.994 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephfs2-2:17.2.5-701.g8387d427.el8.x86_64 15/122 2023-01-24T03:29:21.994 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephsqlite-2:17.2.5-701.g8387d427.el8.x86_64 16/122 2023-01-24T03:29:21.994 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados-devel-2:17.2.5-701.g8387d427.el8.x86_64 17/122 2023-01-24T03:29:21.994 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libradosstriper1-2:17.2.5-701.g8387d427.el8.x86_ 18/122 2023-01-24T03:29:21.995 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librgw2-2:17.2.5-701.g8387d427.el8.x86_64 19/122 2023-01-24T03:29:21.995 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-ceph-argparse-2:17.2.5-701.g8387d427.el8 20/122 2023-01-24T03:29:21.995 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-ceph-common-2:17.2.5-701.g8387d427.el8.x 21/122 2023-01-24T03:29:21.995 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cephfs-2:17.2.5-701.g8387d427.el8.x86_64 22/122 2023-01-24T03:29:21.995 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rados-2:17.2.5-701.g8387d427.el8.x86_64 23/122 2023-01-24T03:29:21.996 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rbd-2:17.2.5-701.g8387d427.el8.x86_64 24/122 2023-01-24T03:29:21.996 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rgw-2:17.2.5-701.g8387d427.el8.x86_64 25/122 2023-01-24T03:29:21.996 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-fuse-2:17.2.5-701.g8387d427.el8.x86_64 26/122 2023-01-24T03:29:21.996 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-mirror-2:17.2.5-701.g8387d427.el8.x86_64 27/122 2023-01-24T03:29:21.996 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-nbd-2:17.2.5-701.g8387d427.el8.x86_64 28/122 2023-01-24T03:29:21.997 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-grafana-dashboards-2:17.2.5-701.g8387d427.e 29/122 2023-01-24T03:29:21.997 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-cephadm-2:17.2.5-701.g8387d427.el8.noar 30/122 2023-01-24T03:29:21.997 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-dashboard-2:17.2.5-701.g8387d427.el8.no 31/122 2023-01-24T03:29:21.997 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.5-701.g8387 32/122 2023-01-24T03:29:21.997 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-modules-core-2:17.2.5-701.g8387d427.el8 33/122 2023-01-24T03:29:21.998 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-rook-2:17.2.5-701.g8387d427.el8.noarch 34/122 2023-01-24T03:29:21.998 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-prometheus-alerts-2:17.2.5-701.g8387d427.el 35/122 2023-01-24T03:29:21.998 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-volume-2:17.2.5-701.g8387d427.el8.noarch 36/122 2023-01-24T03:29:21.998 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : cephadm-2:17.2.5-701.g8387d427.el8.noarch 37/122 2023-01-24T03:29:21.998 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : cephfs-top-2:17.2.5-701.g8387d427.el8.noarch 38/122 2023-01-24T03:29:21.998 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/122 2023-01-24T03:29:21.999 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/122 2023-01-24T03:29:21.999 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/122 2023-01-24T03:29:21.999 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/122 2023-01-24T03:29:21.999 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/122 2023-01-24T03:29:21.999 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/122 2023-01-24T03:29:22.000 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/122 2023-01-24T03:29:22.000 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/122 2023-01-24T03:29:22.000 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/122 2023-01-24T03:29:22.000 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/122 2023-01-24T03:29:22.000 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/122 2023-01-24T03:29:22.001 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/122 2023-01-24T03:29:22.001 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/122 2023-01-24T03:29:22.001 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/122 2023-01-24T03:29:22.001 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/122 2023-01-24T03:29:22.001 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/122 2023-01-24T03:29:22.002 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/122 2023-01-24T03:29:22.002 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/122 2023-01-24T03:29:22.002 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/122 2023-01-24T03:29:22.002 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/122 2023-01-24T03:29:22.002 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/122 2023-01-24T03:29:22.002 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/122 2023-01-24T03:29:22.003 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/122 2023-01-24T03:29:22.003 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/122 2023-01-24T03:29:22.003 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/122 2023-01-24T03:29:22.003 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/122 2023-01-24T03:29:22.003 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 65/122 2023-01-24T03:29:22.004 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/122 2023-01-24T03:29:22.004 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/122 2023-01-24T03:29:22.004 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/122 2023-01-24T03:29:22.004 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/122 2023-01-24T03:29:22.004 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/122 2023-01-24T03:29:22.005 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/122 2023-01-24T03:29:22.005 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 72/122 2023-01-24T03:29:22.005 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 73/122 2023-01-24T03:29:22.005 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 74/122 2023-01-24T03:29:22.005 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : jq-1.5-12.el8.x86_64 75/122 2023-01-24T03:29:22.006 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : flex-2.6.1-9.el8.x86_64 76/122 2023-01-24T03:29:22.006 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 77/122 2023-01-24T03:29:22.006 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 78/122 2023-01-24T03:29:22.006 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 79/122 2023-01-24T03:29:22.006 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : bison-3.0.4-10.el8.x86_64 80/122 2023-01-24T03:29:22.007 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 81/122 2023-01-24T03:29:22.007 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 82/122 2023-01-24T03:29:22.007 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 83/122 2023-01-24T03:29:22.007 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 84/122 2023-01-24T03:29:22.007 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 85/122 2023-01-24T03:29:22.007 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 86/122 2023-01-24T03:29:22.008 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 87/122 2023-01-24T03:29:22.008 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 88/122 2023-01-24T03:29:22.008 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 89/122 2023-01-24T03:29:22.008 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 90/122 2023-01-24T03:29:22.008 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 91/122 2023-01-24T03:29:22.009 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 92/122 2023-01-24T03:29:22.009 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 93/122 2023-01-24T03:29:22.009 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 94/122 2023-01-24T03:29:22.009 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 95/122 2023-01-24T03:29:22.009 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : automake-1.16.1-7.el8.noarch 96/122 2023-01-24T03:29:22.009 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 97/122 2023-01-24T03:29:22.010 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 98/122 2023-01-24T03:29:22.010 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 99/122 2023-01-24T03:29:22.010 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 100/122 2023-01-24T03:29:22.010 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 101/122 2023-01-24T03:29:22.010 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 102/122 2023-01-24T03:29:22.011 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 103/122 2023-01-24T03:29:22.011 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openssl-devel-1:1.1.1g-15.el8_3.x86_64 104/122 2023-01-24T03:29:22.011 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 105/122 2023-01-24T03:29:22.011 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 106/122 2023-01-24T03:29:22.011 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 107/122 2023-01-24T03:29:22.012 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : elfutils-libelf-devel-0.182-3.el8.x86_64 108/122 2023-01-24T03:29:22.012 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 109/122 2023-01-24T03:29:22.012 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcom_err-devel-1.45.6-1.el8.x86_64 110/122 2023-01-24T03:29:22.012 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libsepol-devel-2.9-2.el8.x86_64 111/122 2023-01-24T03:29:22.012 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : zlib-devel-1.2.11-17.el8.x86_64 112/122 2023-01-24T03:29:22.013 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 113/122 2023-01-24T03:29:22.013 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : krb5-devel-1.18.2-8.3.el8_4.x86_64 114/122 2023-01-24T03:29:22.013 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 115/122 2023-01-24T03:29:22.013 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : xfsprogs-devel-5.0.0-8.el8.x86_64 116/122 2023-01-24T03:29:22.013 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 117/122 2023-01-24T03:29:22.014 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libuuid-devel-2.32.1-27.el8.x86_64 118/122 2023-01-24T03:29:22.014 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados2-2:17.2.5-701.g8387d427.el8.x86_64 119/122 2023-01-24T03:29:22.014 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 120/122 2023-01-24T03:29:22.014 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librbd1-2:17.2.5-701.g8387d427.el8.x86_64 121/122 2023-01-24T03:29:22.918 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 122/122 2023-01-24T03:29:22.918 INFO:teuthology.orchestra.run.smithi142.stdout:Last metadata expiration check: 0:01:41 ago on Tue 24 Jan 2023 03:27:41 AM UTC. 2023-01-24T03:29:24.281 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 122/122 2023-01-24T03:29:24.282 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:43 ago on Tue 24 Jan 2023 03:27:41 AM UTC. 2023-01-24T03:29:25.142 INFO:teuthology.orchestra.run.smithi142.stdout:Installed products updated. 2023-01-24T03:29:25.282 INFO:teuthology.orchestra.run.smithi142.stdout: 2023-01-24T03:29:25.282 INFO:teuthology.orchestra.run.smithi142.stdout:Upgraded: 2023-01-24T03:29:25.282 INFO:teuthology.orchestra.run.smithi142.stdout: librados2-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:25.282 INFO:teuthology.orchestra.run.smithi142.stdout: librbd1-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:25.283 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2023-01-24T03:29:25.283 INFO:teuthology.orchestra.run.smithi142.stdout: automake-1.16.1-7.el8.noarch 2023-01-24T03:29:25.283 INFO:teuthology.orchestra.run.smithi142.stdout: bison-3.0.4-10.el8.x86_64 2023-01-24T03:29:25.283 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:25.283 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-base-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:25.284 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-common-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:25.284 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-fuse-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:25.284 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-grafana-dashboards-2:17.2.5-701.g8387d427.el8.noarch 2023-01-24T03:29:25.284 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-immutable-object-cache-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:25.284 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mds-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:25.285 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:25.285 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-cephadm-2:17.2.5-701.g8387d427.el8.noarch 2023-01-24T03:29:25.285 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-dashboard-2:17.2.5-701.g8387d427.el8.noarch 2023-01-24T03:29:25.285 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-diskprediction-local-2:17.2.5-701.g8387d427.el8.noarch 2023-01-24T03:29:25.285 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-modules-core-2:17.2.5-701.g8387d427.el8.noarch 2023-01-24T03:29:25.286 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-rook-2:17.2.5-701.g8387d427.el8.noarch 2023-01-24T03:29:25.286 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mon-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:25.286 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-osd-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:25.286 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-prometheus-alerts-2:17.2.5-701.g8387d427.el8.noarch 2023-01-24T03:29:25.286 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-radosgw-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:25.287 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-selinux-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:25.287 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-test-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:25.287 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-volume-2:17.2.5-701.g8387d427.el8.noarch 2023-01-24T03:29:25.287 INFO:teuthology.orchestra.run.smithi142.stdout: cephadm-2:17.2.5-701.g8387d427.el8.noarch 2023-01-24T03:29:25.287 INFO:teuthology.orchestra.run.smithi142.stdout: cephfs-mirror-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:25.287 INFO:teuthology.orchestra.run.smithi142.stdout: cephfs-top-2:17.2.5-701.g8387d427.el8.noarch 2023-01-24T03:29:25.288 INFO:teuthology.orchestra.run.smithi142.stdout: elfutils-libelf-devel-0.182-3.el8.x86_64 2023-01-24T03:29:25.288 INFO:teuthology.orchestra.run.smithi142.stdout: flex-2.6.1-9.el8.x86_64 2023-01-24T03:29:25.288 INFO:teuthology.orchestra.run.smithi142.stdout: fuse-2.9.7-12.el8.x86_64 2023-01-24T03:29:25.288 INFO:teuthology.orchestra.run.smithi142.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-01-24T03:29:25.288 INFO:teuthology.orchestra.run.smithi142.stdout: jq-1.5-12.el8.x86_64 2023-01-24T03:29:25.289 INFO:teuthology.orchestra.run.smithi142.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2023-01-24T03:29:25.289 INFO:teuthology.orchestra.run.smithi142.stdout: krb5-devel-1.18.2-8.3.el8_4.x86_64 2023-01-24T03:29:25.289 INFO:teuthology.orchestra.run.smithi142.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-01-24T03:29:25.289 INFO:teuthology.orchestra.run.smithi142.stdout: libcephfs-devel-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:25.289 INFO:teuthology.orchestra.run.smithi142.stdout: libcephfs2-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:25.290 INFO:teuthology.orchestra.run.smithi142.stdout: libcephsqlite-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:25.290 INFO:teuthology.orchestra.run.smithi142.stdout: libcom_err-devel-1.45.6-1.el8.x86_64 2023-01-24T03:29:25.290 INFO:teuthology.orchestra.run.smithi142.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-01-24T03:29:25.290 INFO:teuthology.orchestra.run.smithi142.stdout: liboath-2.6.2-3.el8.x86_64 2023-01-24T03:29:25.290 INFO:teuthology.orchestra.run.smithi142.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-01-24T03:29:25.291 INFO:teuthology.orchestra.run.smithi142.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-01-24T03:29:25.291 INFO:teuthology.orchestra.run.smithi142.stdout: librados-devel-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:25.291 INFO:teuthology.orchestra.run.smithi142.stdout: libradosstriper1-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:25.291 INFO:teuthology.orchestra.run.smithi142.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-01-24T03:29:25.291 INFO:teuthology.orchestra.run.smithi142.stdout: librgw2-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:25.291 INFO:teuthology.orchestra.run.smithi142.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-01-24T03:29:25.292 INFO:teuthology.orchestra.run.smithi142.stdout: libsepol-devel-2.9-2.el8.x86_64 2023-01-24T03:29:25.292 INFO:teuthology.orchestra.run.smithi142.stdout: libtool-2.4.6-25.el8.x86_64 2023-01-24T03:29:25.292 INFO:teuthology.orchestra.run.smithi142.stdout: libunwind-1.3.1-3.el8.x86_64 2023-01-24T03:29:25.292 INFO:teuthology.orchestra.run.smithi142.stdout: libuuid-devel-2.32.1-27.el8.x86_64 2023-01-24T03:29:25.292 INFO:teuthology.orchestra.run.smithi142.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-01-24T03:29:25.293 INFO:teuthology.orchestra.run.smithi142.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-01-24T03:29:25.293 INFO:teuthology.orchestra.run.smithi142.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-01-24T03:29:25.293 INFO:teuthology.orchestra.run.smithi142.stdout: openblas-0.3.12-1.el8.x86_64 2023-01-24T03:29:25.293 INFO:teuthology.orchestra.run.smithi142.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-01-24T03:29:25.293 INFO:teuthology.orchestra.run.smithi142.stdout: openssl-devel-1:1.1.1g-15.el8_3.x86_64 2023-01-24T03:29:25.294 INFO:teuthology.orchestra.run.smithi142.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-01-24T03:29:25.294 INFO:teuthology.orchestra.run.smithi142.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-01-24T03:29:25.294 INFO:teuthology.orchestra.run.smithi142.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-01-24T03:29:25.294 INFO:teuthology.orchestra.run.smithi142.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-01-24T03:29:25.294 INFO:teuthology.orchestra.run.smithi142.stdout: python3-babel-2.5.1-5.el8.noarch 2023-01-24T03:29:25.295 INFO:teuthology.orchestra.run.smithi142.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-01-24T03:29:25.295 INFO:teuthology.orchestra.run.smithi142.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-01-24T03:29:25.295 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-01-24T03:29:25.295 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ceph-argparse-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:25.295 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ceph-common-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:25.295 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cephfs-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:25.296 INFO:teuthology.orchestra.run.smithi142.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-01-24T03:29:25.296 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-01-24T03:29:25.296 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-01-24T03:29:25.296 INFO:teuthology.orchestra.run.smithi142.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-01-24T03:29:25.296 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-6.2-6.el8.noarch 2023-01-24T03:29:25.297 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-01-24T03:29:25.297 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-01-24T03:29:25.297 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-01-24T03:29:25.297 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-01-24T03:29:25.297 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-01-24T03:29:25.298 INFO:teuthology.orchestra.run.smithi142.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-01-24T03:29:25.298 INFO:teuthology.orchestra.run.smithi142.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-01-24T03:29:25.298 INFO:teuthology.orchestra.run.smithi142.stdout: python3-mako-1.0.6-13.el8.noarch 2023-01-24T03:29:25.298 INFO:teuthology.orchestra.run.smithi142.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-01-24T03:29:25.298 INFO:teuthology.orchestra.run.smithi142.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-01-24T03:29:25.299 INFO:teuthology.orchestra.run.smithi142.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-01-24T03:29:25.299 INFO:teuthology.orchestra.run.smithi142.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-01-24T03:29:25.299 INFO:teuthology.orchestra.run.smithi142.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-01-24T03:29:25.299 INFO:teuthology.orchestra.run.smithi142.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-01-24T03:29:25.299 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-01-24T03:29:25.300 INFO:teuthology.orchestra.run.smithi142.stdout: python3-portend-2.6-1.el8.noarch 2023-01-24T03:29:25.300 INFO:teuthology.orchestra.run.smithi142.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-01-24T03:29:25.300 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-01-24T03:29:25.300 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-01-24T03:29:25.300 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pytz-2017.2-9.el8.noarch 2023-01-24T03:29:25.301 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rados-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:25.301 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rbd-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:25.301 INFO:teuthology.orchestra.run.smithi142.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-01-24T03:29:25.301 INFO:teuthology.orchestra.run.smithi142.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-01-24T03:29:25.301 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rgw-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:25.301 INFO:teuthology.orchestra.run.smithi142.stdout: python3-routes-2.4.1-12.el8.noarch 2023-01-24T03:29:25.302 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rsa-4.8-1.el8.noarch 2023-01-24T03:29:25.302 INFO:teuthology.orchestra.run.smithi142.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-01-24T03:29:25.302 INFO:teuthology.orchestra.run.smithi142.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-01-24T03:29:25.302 INFO:teuthology.orchestra.run.smithi142.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-01-24T03:29:25.303 INFO:teuthology.orchestra.run.smithi142.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-01-24T03:29:25.303 INFO:teuthology.orchestra.run.smithi142.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-01-24T03:29:25.303 INFO:teuthology.orchestra.run.smithi142.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2023-01-24T03:29:25.303 INFO:teuthology.orchestra.run.smithi142.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-01-24T03:29:25.303 INFO:teuthology.orchestra.run.smithi142.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-01-24T03:29:25.304 INFO:teuthology.orchestra.run.smithi142.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-01-24T03:29:25.304 INFO:teuthology.orchestra.run.smithi142.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-01-24T03:29:25.304 INFO:teuthology.orchestra.run.smithi142.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-01-24T03:29:25.304 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-fuse-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:25.304 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-mirror-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:25.304 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-nbd-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:25.305 INFO:teuthology.orchestra.run.smithi142.stdout: socat-1.7.3.3-2.el8.x86_64 2023-01-24T03:29:25.305 INFO:teuthology.orchestra.run.smithi142.stdout: thrift-0.13.0-2.el8.x86_64 2023-01-24T03:29:25.305 INFO:teuthology.orchestra.run.smithi142.stdout: xfsprogs-devel-5.0.0-8.el8.x86_64 2023-01-24T03:29:25.305 INFO:teuthology.orchestra.run.smithi142.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-01-24T03:29:25.306 INFO:teuthology.orchestra.run.smithi142.stdout: zlib-devel-1.2.11-17.el8.x86_64 2023-01-24T03:29:25.306 INFO:teuthology.orchestra.run.smithi142.stdout: 2023-01-24T03:29:25.306 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2023-01-24T03:29:25.619 DEBUG:teuthology.parallel:result is None 2023-01-24T03:29:26.577 INFO:teuthology.orchestra.run.smithi112.stdout:Installed products updated. 2023-01-24T03:29:26.716 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:29:26.716 INFO:teuthology.orchestra.run.smithi112.stdout:Upgraded: 2023-01-24T03:29:26.716 INFO:teuthology.orchestra.run.smithi112.stdout: librados2-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:26.717 INFO:teuthology.orchestra.run.smithi112.stdout: librbd1-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:26.717 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2023-01-24T03:29:26.717 INFO:teuthology.orchestra.run.smithi112.stdout: automake-1.16.1-7.el8.noarch 2023-01-24T03:29:26.717 INFO:teuthology.orchestra.run.smithi112.stdout: bison-3.0.4-10.el8.x86_64 2023-01-24T03:29:26.718 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:26.718 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-base-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:26.718 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-common-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:26.718 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-fuse-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:26.718 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-grafana-dashboards-2:17.2.5-701.g8387d427.el8.noarch 2023-01-24T03:29:26.718 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-immutable-object-cache-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:26.719 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mds-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:26.719 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:26.719 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-cephadm-2:17.2.5-701.g8387d427.el8.noarch 2023-01-24T03:29:26.719 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-dashboard-2:17.2.5-701.g8387d427.el8.noarch 2023-01-24T03:29:26.719 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-diskprediction-local-2:17.2.5-701.g8387d427.el8.noarch 2023-01-24T03:29:26.720 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-modules-core-2:17.2.5-701.g8387d427.el8.noarch 2023-01-24T03:29:26.720 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-rook-2:17.2.5-701.g8387d427.el8.noarch 2023-01-24T03:29:26.720 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mon-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:26.720 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-osd-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:26.721 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-prometheus-alerts-2:17.2.5-701.g8387d427.el8.noarch 2023-01-24T03:29:26.721 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-radosgw-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:26.721 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-selinux-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:26.721 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-test-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:26.721 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-volume-2:17.2.5-701.g8387d427.el8.noarch 2023-01-24T03:29:26.722 INFO:teuthology.orchestra.run.smithi112.stdout: cephadm-2:17.2.5-701.g8387d427.el8.noarch 2023-01-24T03:29:26.722 INFO:teuthology.orchestra.run.smithi112.stdout: cephfs-mirror-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:26.722 INFO:teuthology.orchestra.run.smithi112.stdout: cephfs-top-2:17.2.5-701.g8387d427.el8.noarch 2023-01-24T03:29:26.722 INFO:teuthology.orchestra.run.smithi112.stdout: elfutils-libelf-devel-0.182-3.el8.x86_64 2023-01-24T03:29:26.722 INFO:teuthology.orchestra.run.smithi112.stdout: flex-2.6.1-9.el8.x86_64 2023-01-24T03:29:26.723 INFO:teuthology.orchestra.run.smithi112.stdout: fuse-2.9.7-12.el8.x86_64 2023-01-24T03:29:26.723 INFO:teuthology.orchestra.run.smithi112.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-01-24T03:29:26.723 INFO:teuthology.orchestra.run.smithi112.stdout: jq-1.5-12.el8.x86_64 2023-01-24T03:29:26.723 INFO:teuthology.orchestra.run.smithi112.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2023-01-24T03:29:26.723 INFO:teuthology.orchestra.run.smithi112.stdout: krb5-devel-1.18.2-8.3.el8_4.x86_64 2023-01-24T03:29:26.724 INFO:teuthology.orchestra.run.smithi112.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-01-24T03:29:26.724 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs-devel-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:26.724 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs2-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:26.724 INFO:teuthology.orchestra.run.smithi112.stdout: libcephsqlite-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:26.724 INFO:teuthology.orchestra.run.smithi112.stdout: libcom_err-devel-1.45.6-1.el8.x86_64 2023-01-24T03:29:26.725 INFO:teuthology.orchestra.run.smithi112.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-01-24T03:29:26.725 INFO:teuthology.orchestra.run.smithi112.stdout: liboath-2.6.2-3.el8.x86_64 2023-01-24T03:29:26.725 INFO:teuthology.orchestra.run.smithi112.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-01-24T03:29:26.725 INFO:teuthology.orchestra.run.smithi112.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-01-24T03:29:26.725 INFO:teuthology.orchestra.run.smithi112.stdout: librados-devel-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:26.726 INFO:teuthology.orchestra.run.smithi112.stdout: libradosstriper1-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:26.726 INFO:teuthology.orchestra.run.smithi112.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-01-24T03:29:26.726 INFO:teuthology.orchestra.run.smithi112.stdout: librgw2-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:26.726 INFO:teuthology.orchestra.run.smithi112.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-01-24T03:29:26.726 INFO:teuthology.orchestra.run.smithi112.stdout: libsepol-devel-2.9-2.el8.x86_64 2023-01-24T03:29:26.727 INFO:teuthology.orchestra.run.smithi112.stdout: libtool-2.4.6-25.el8.x86_64 2023-01-24T03:29:26.727 INFO:teuthology.orchestra.run.smithi112.stdout: libunwind-1.3.1-3.el8.x86_64 2023-01-24T03:29:26.727 INFO:teuthology.orchestra.run.smithi112.stdout: libuuid-devel-2.32.1-27.el8.x86_64 2023-01-24T03:29:26.727 INFO:teuthology.orchestra.run.smithi112.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-01-24T03:29:26.727 INFO:teuthology.orchestra.run.smithi112.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-01-24T03:29:26.728 INFO:teuthology.orchestra.run.smithi112.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-01-24T03:29:26.728 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-0.3.12-1.el8.x86_64 2023-01-24T03:29:26.728 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-01-24T03:29:26.728 INFO:teuthology.orchestra.run.smithi112.stdout: openssl-devel-1:1.1.1g-15.el8_3.x86_64 2023-01-24T03:29:26.728 INFO:teuthology.orchestra.run.smithi112.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-01-24T03:29:26.729 INFO:teuthology.orchestra.run.smithi112.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-01-24T03:29:26.729 INFO:teuthology.orchestra.run.smithi112.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-01-24T03:29:26.729 INFO:teuthology.orchestra.run.smithi112.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-01-24T03:29:26.729 INFO:teuthology.orchestra.run.smithi112.stdout: python3-babel-2.5.1-5.el8.noarch 2023-01-24T03:29:26.729 INFO:teuthology.orchestra.run.smithi112.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-01-24T03:29:26.730 INFO:teuthology.orchestra.run.smithi112.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-01-24T03:29:26.730 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-01-24T03:29:26.730 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-argparse-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:26.730 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-common-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:26.730 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cephfs-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:26.731 INFO:teuthology.orchestra.run.smithi112.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-01-24T03:29:26.731 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-01-24T03:29:26.731 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-01-24T03:29:26.731 INFO:teuthology.orchestra.run.smithi112.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-01-24T03:29:26.731 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-6.2-6.el8.noarch 2023-01-24T03:29:26.732 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-01-24T03:29:26.732 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-01-24T03:29:26.732 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-01-24T03:29:26.732 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-01-24T03:29:26.733 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-01-24T03:29:26.733 INFO:teuthology.orchestra.run.smithi112.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-01-24T03:29:26.733 INFO:teuthology.orchestra.run.smithi112.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-01-24T03:29:26.733 INFO:teuthology.orchestra.run.smithi112.stdout: python3-mako-1.0.6-13.el8.noarch 2023-01-24T03:29:26.733 INFO:teuthology.orchestra.run.smithi112.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-01-24T03:29:26.734 INFO:teuthology.orchestra.run.smithi112.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-01-24T03:29:26.734 INFO:teuthology.orchestra.run.smithi112.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-01-24T03:29:26.734 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-01-24T03:29:26.734 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-01-24T03:29:26.734 INFO:teuthology.orchestra.run.smithi112.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-01-24T03:29:26.734 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-01-24T03:29:26.735 INFO:teuthology.orchestra.run.smithi112.stdout: python3-portend-2.6-1.el8.noarch 2023-01-24T03:29:26.735 INFO:teuthology.orchestra.run.smithi112.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-01-24T03:29:26.735 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-01-24T03:29:26.735 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-01-24T03:29:26.735 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pytz-2017.2-9.el8.noarch 2023-01-24T03:29:26.736 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rados-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:26.736 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rbd-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:26.736 INFO:teuthology.orchestra.run.smithi112.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-01-24T03:29:26.736 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-01-24T03:29:26.737 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rgw-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:26.737 INFO:teuthology.orchestra.run.smithi112.stdout: python3-routes-2.4.1-12.el8.noarch 2023-01-24T03:29:26.737 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rsa-4.8-1.el8.noarch 2023-01-24T03:29:26.737 INFO:teuthology.orchestra.run.smithi112.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-01-24T03:29:26.737 INFO:teuthology.orchestra.run.smithi112.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-01-24T03:29:26.738 INFO:teuthology.orchestra.run.smithi112.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-01-24T03:29:26.738 INFO:teuthology.orchestra.run.smithi112.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-01-24T03:29:26.738 INFO:teuthology.orchestra.run.smithi112.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-01-24T03:29:26.738 INFO:teuthology.orchestra.run.smithi112.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2023-01-24T03:29:26.738 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-01-24T03:29:26.738 INFO:teuthology.orchestra.run.smithi112.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-01-24T03:29:26.739 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-01-24T03:29:26.739 INFO:teuthology.orchestra.run.smithi112.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-01-24T03:29:26.739 INFO:teuthology.orchestra.run.smithi112.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-01-24T03:29:26.739 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-fuse-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:26.740 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-mirror-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:26.740 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-nbd-2:17.2.5-701.g8387d427.el8.x86_64 2023-01-24T03:29:26.740 INFO:teuthology.orchestra.run.smithi112.stdout: socat-1.7.3.3-2.el8.x86_64 2023-01-24T03:29:26.740 INFO:teuthology.orchestra.run.smithi112.stdout: thrift-0.13.0-2.el8.x86_64 2023-01-24T03:29:26.740 INFO:teuthology.orchestra.run.smithi112.stdout: xfsprogs-devel-5.0.0-8.el8.x86_64 2023-01-24T03:29:26.740 INFO:teuthology.orchestra.run.smithi112.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-01-24T03:29:26.741 INFO:teuthology.orchestra.run.smithi112.stdout: zlib-devel-1.2.11-17.el8.x86_64 2023-01-24T03:29:26.741 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:29:26.742 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2023-01-24T03:29:26.982 DEBUG:teuthology.parallel:result is None 2023-01-24T03:29:26.982 DEBUG:teuthology.orchestra.run.smithi112:> sudo lsb_release -is 2023-01-24T03:29:27.032 INFO:teuthology.orchestra.run.smithi112.stdout:RedHatEnterprise 2023-01-24T03:29:27.033 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2023-01-24T03:29:27.033 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=8387d427d91a301da19bd82e090b96e1348bfadb 2023-01-24T03:29:27.320 DEBUG:teuthology.orchestra.run.smithi112:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-01-24T03:29:27.339 INFO:teuthology.orchestra.run.smithi112.stdout:17.2.5-701.g8387d427.el8 2023-01-24T03:29:27.339 INFO:teuthology.packaging:The installed version of ceph is 17.2.5-701.g8387d427.el8 2023-01-24T03:29:27.340 INFO:teuthology.task.install:The correct ceph version 17.2.5-701.g8387d427 is installed. 2023-01-24T03:29:27.340 DEBUG:teuthology.orchestra.run.smithi142:> sudo lsb_release -is 2023-01-24T03:29:27.399 INFO:teuthology.orchestra.run.smithi142.stdout:RedHatEnterprise 2023-01-24T03:29:27.401 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2023-01-24T03:29:27.401 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=8387d427d91a301da19bd82e090b96e1348bfadb 2023-01-24T03:29:27.678 DEBUG:teuthology.orchestra.run.smithi142:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-01-24T03:29:27.701 INFO:teuthology.orchestra.run.smithi142.stdout:17.2.5-701.g8387d427.el8 2023-01-24T03:29:27.701 INFO:teuthology.packaging:The installed version of ceph is 17.2.5-701.g8387d427.el8 2023-01-24T03:29:27.702 INFO:teuthology.task.install:The correct ceph version 17.2.5-701.g8387d427 is installed. 2023-01-24T03:29:27.702 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-01-24T03:29:27.702 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-01-24T03:29:27.703 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-01-24T03:29:27.727 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2023-01-24T03:29:27.728 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-01-24T03:29:27.769 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-01-24T03:29:27.769 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-01-24T03:29:27.770 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/daemon-helper 2023-01-24T03:29:27.793 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-01-24T03:29:27.856 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2023-01-24T03:29:27.857 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/usr/bin/daemon-helper 2023-01-24T03:29:27.884 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-01-24T03:29:27.950 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-01-24T03:29:27.951 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-01-24T03:29:27.951 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/adjust-ulimits 2023-01-24T03:29:27.976 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-01-24T03:29:28.038 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2023-01-24T03:29:28.039 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/usr/bin/adjust-ulimits 2023-01-24T03:29:28.066 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-01-24T03:29:28.133 INFO:teuthology.run_tasks:Running task ceph... 2023-01-24T03:29:28.211 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-01-24T03:29:28.212 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 777 /var/log/ceph 2023-01-24T03:29:28.215 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 777 /var/log/ceph 2023-01-24T03:29:28.241 INFO:tasks.ceph:Disabling ceph logrotate... 2023-01-24T03:29:28.241 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /etc/logrotate.d/ceph 2023-01-24T03:29:28.278 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f -- /etc/logrotate.d/ceph 2023-01-24T03:29:28.305 INFO:tasks.ceph:Creating extra log directories... 2023-01-24T03:29:28.306 DEBUG:teuthology.orchestra.run.smithi112:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-01-24T03:29:28.340 DEBUG:teuthology.orchestra.run.smithi142:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-01-24T03:29:28.376 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-01-24T03:29:28.377 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-01-24T03:29:28.377 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-01-24T03:29:28.378 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-01-24T03:29:28.378 DEBUG:teuthology.orchestra.run.smithi112:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-01-24T03:29:28.415 DEBUG:teuthology.orchestra.run.smithi112:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-01-24T03:29:28.529 DEBUG:teuthology.orchestra.run.smithi112:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-01-24T03:29:28.553 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2023-01-24T03:29:28.553 DEBUG:teuthology.orchestra.run.smithi142:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-01-24T03:29:28.569 DEBUG:teuthology.orchestra.run.smithi142:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-01-24T03:29:28.685 DEBUG:teuthology.orchestra.run.smithi142:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-01-24T03:29:28.717 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-01-24T03:29:28.717 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'Replacing daemon mds'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-01-24T03:29:28.718 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-01-23_22:48:43-fs-wip-yuri5-testing-2023-01-23-0928-quincy-distro-default-smithi/7134568', 'branch': 'wip-yuri5-testing-2023-01-23-0928-quincy', 'description': 'fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}}', 'email': None, 'first_in_suite': False, 'job_id': '7134568', 'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'yuriw-2023-01-23_22:48:43-fs-wip-yuri5-testing-2023-01-23-0928-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'rhel', 'os_version': '8.4', 'overrides': {'admin_socket': {'branch': 'wip-yuri5-testing-2023-01-23-0928-quincy'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'Replacing daemon mds'], 'sha1': '8387d427d91a301da19bd82e090b96e1348bfadb'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8387d427d91a301da19bd82e090b96e1348bfadb'}}, 'kclient': {'syntax': 'v1'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri5-testing-2023-01-23-0928-quincy', 'sha1': '8387d427d91a301da19bd82e090b96e1348bfadb'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 71, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'sha1': '8387d427d91a301da19bd82e090b96e1348bfadb', 'sleep_before_teardown': 0, 'suite': 'fs', 'suite_branch': 'wip-yuri5-testing-2023-01-23-0928-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_8387d427d91a301da19bd82e090b96e1348bfadb/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '8387d427d91a301da19bd82e090b96e1348bfadb', 'targets': {'smithi112.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCi3HiWd6/sHDt8B4an31nK9OwwYDIAHLkfRs+K5B9seNKgOmdbng4qjJ/cHzHc8OxFIT3OwsNobN9VNcvxuLnq+MWWiBusVCPRnBnicqWunqzrKAHR6iTW5bg0dukS4kePZTedc9qbncPg0HbIfqzpP/acBMWwtLlXak3OrB8FxDC9et4N7RGOwhCtPHtpVRL5nA3VIWoRK4YRF7IH6LgefSSP/gdnL0qq7tL+Se9hNb/9ikcL6G/Xc6qRFfpcjhOcksMQZ8/yU4ageuzQrWDebSDakzasTBdpCDErWKU8FUIplyUpdC0S+6G1kvR8RoJEwY25Ii1X9eOLJSLOklWswMSxc5+3kRINqN/Ti1W8yjuxnIls3hJuPun774BeG0wMYnt75SdyMuReYAaNMcB6jG/drTl62uogQEMU0IwjcwEk29ev1It/rF1OTtg5Xvb+cfGzDb2uXI53IHplNj/OMNDVekuhBNuk8C3ojYV5ezaBBPgdPf/KPLWseCziVNM=', 'smithi142.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDc9l7T/84L4XRGIvVuKHrzVvHDt6pX7G2I+pxnBYcmn5mgh5TUbkwQaXJbSjj7aI0rCfUBOGzHLYcuRciAv8FXCUWcazv7WD6lecaRdqHxxXjoDnz5caE0jXz8no6DqagbGfkjlfFVcXgsTueLc8Z5uABvFIZ95rPuiXYoWlhj/SmCDM9UUAcUfAyj9Vjcw2/yWpCTq3otEq0qx/OVXkjPD6Gd1cZ3PhUt8euiajxdNp1YdAdVYYuWn2FDYNzo2p8vlpG7Kk2iGS8D9/WEv2ascTMVvsiqphG9zG2U1cRQZLG6+rQTe6T8uUuK9AuoUqpCunGzY2UQmN8SRSUb/8AdqEliHi69qBDmUdCbH1NbCgG6lp9UbGGsnIhyiguVKIJkM6WNErjcUAP0H7LDG48pPRMFyPfsTyDrfO/t1WHWezBJz0T/Bpr6QLOA7iyQI8F5S3e85IN5phnLmQ8Hr9MQyQsMtzO8gVSPmJL67bNRgptQnJk5LgYbbMINPaI+oFM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '8387d427d91a301da19bd82e090b96e1348bfadb'}}, {'ceph': None}, {'kclient': None}, {'mds_thrash': None}, {'workunit': {'clients': {'all': ['fs/snaps']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '5abecab36c9da4d70461d81781464cf5acaea292', 'timestamp': '2023-01-23_22:48:43', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.17735'} 2023-01-24T03:29:28.719 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-01-24T03:29:28.721 DEBUG:teuthology.orchestra.run.smithi142:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-01-24T03:29:28.766 DEBUG:teuthology.orchestra.run.smithi112:> sudo install -d -m0777 -- /var/run/ceph 2023-01-24T03:29:28.776 DEBUG:teuthology.orchestra.run.smithi142:> sudo install -d -m0777 -- /var/run/ceph 2023-01-24T03:29:28.830 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-01-24T03:29:28.830 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/scratch_devs of=/dev/stdout 2023-01-24T03:29:28.850 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-01-24T03:29:28.851 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_1 2023-01-24T03:29:28.905 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-01-24T03:29:28.905 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T03:29:28.906 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 6h/6d Inode: 201348 Links: 1 2023-01-24T03:29:28.906 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T03:29:28.906 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T03:29:28.906 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2023-01-24 03:29:21.547015287 +0000 2023-01-24T03:29:28.907 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2023-01-24 03:26:56.859378817 +0000 2023-01-24T03:29:28.907 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2023-01-24 03:26:56.859378817 +0000 2023-01-24T03:29:28.907 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: - 2023-01-24T03:29:28.908 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-01-24T03:29:28.967 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2023-01-24T03:29:28.967 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2023-01-24T03:29:28.968 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000252966 s, 2.0 MB/s 2023-01-24T03:29:28.969 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-01-24T03:29:29.024 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_2 2023-01-24T03:29:29.079 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-01-24T03:29:29.079 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T03:29:29.080 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 6h/6d Inode: 203230 Links: 1 2023-01-24T03:29:29.080 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T03:29:29.080 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T03:29:29.080 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2023-01-24 03:29:21.548015260 +0000 2023-01-24T03:29:29.080 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2023-01-24 03:26:57.318366165 +0000 2023-01-24T03:29:29.081 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2023-01-24 03:26:57.318366165 +0000 2023-01-24T03:29:29.081 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: - 2023-01-24T03:29:29.081 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-01-24T03:29:29.141 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2023-01-24T03:29:29.141 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2023-01-24T03:29:29.142 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000219699 s, 2.3 MB/s 2023-01-24T03:29:29.143 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-01-24T03:29:29.197 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_3 2023-01-24T03:29:29.252 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-01-24T03:29:29.252 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T03:29:29.253 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 6h/6d Inode: 205198 Links: 1 2023-01-24T03:29:29.253 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T03:29:29.253 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T03:29:29.253 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2023-01-24 03:29:21.548015260 +0000 2023-01-24T03:29:29.253 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2023-01-24 03:26:57.800352879 +0000 2023-01-24T03:29:29.254 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2023-01-24 03:26:57.800352879 +0000 2023-01-24T03:29:29.254 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: - 2023-01-24T03:29:29.254 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-01-24T03:29:29.314 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2023-01-24T03:29:29.314 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2023-01-24T03:29:29.315 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000219225 s, 2.3 MB/s 2023-01-24T03:29:29.316 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-01-24T03:29:29.370 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_4 2023-01-24T03:29:29.425 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-01-24T03:29:29.425 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T03:29:29.426 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 6h/6d Inode: 198167 Links: 1 2023-01-24T03:29:29.426 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T03:29:29.426 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T03:29:29.426 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2023-01-24 03:29:21.548015260 +0000 2023-01-24T03:29:29.427 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2023-01-24 03:26:58.302339041 +0000 2023-01-24T03:29:29.427 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2023-01-24 03:26:58.302339041 +0000 2023-01-24T03:29:29.427 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: - 2023-01-24T03:29:29.427 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-01-24T03:29:29.487 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2023-01-24T03:29:29.488 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2023-01-24T03:29:29.488 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.00018567 s, 2.8 MB/s 2023-01-24T03:29:29.489 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-01-24T03:29:29.543 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-01-24T03:29:29.544 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2023-01-24T03:29:29.544 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/scratch_devs of=/dev/stdout 2023-01-24T03:29:29.561 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-01-24T03:29:29.562 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_1 2023-01-24T03:29:29.619 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-01-24T03:29:29.619 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T03:29:29.619 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 6h/6d Inode: 198561 Links: 1 2023-01-24T03:29:29.619 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T03:29:29.620 INFO:teuthology.orchestra.run.smithi142.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T03:29:29.620 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2023-01-24 03:29:20.309479875 +0000 2023-01-24T03:29:29.620 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2023-01-24 03:26:59.564590516 +0000 2023-01-24T03:29:29.620 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2023-01-24 03:26:59.564590516 +0000 2023-01-24T03:29:29.621 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: - 2023-01-24T03:29:29.621 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-01-24T03:29:29.684 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2023-01-24T03:29:29.685 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2023-01-24T03:29:29.685 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000341592 s, 1.5 MB/s 2023-01-24T03:29:29.686 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-01-24T03:29:29.743 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_2 2023-01-24T03:29:29.799 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-01-24T03:29:29.799 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T03:29:29.799 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 6h/6d Inode: 198647 Links: 1 2023-01-24T03:29:29.800 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T03:29:29.800 INFO:teuthology.orchestra.run.smithi142.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T03:29:29.800 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2023-01-24 03:29:20.310479857 +0000 2023-01-24T03:29:29.800 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2023-01-24 03:27:00.055581493 +0000 2023-01-24T03:29:29.800 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2023-01-24 03:27:00.055581493 +0000 2023-01-24T03:29:29.801 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: - 2023-01-24T03:29:29.801 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-01-24T03:29:29.863 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2023-01-24T03:29:29.863 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2023-01-24T03:29:29.863 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000236963 s, 2.2 MB/s 2023-01-24T03:29:29.864 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-01-24T03:29:29.921 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_3 2023-01-24T03:29:29.976 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-01-24T03:29:29.977 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T03:29:29.977 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 6h/6d Inode: 205946 Links: 1 2023-01-24T03:29:29.977 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T03:29:29.977 INFO:teuthology.orchestra.run.smithi142.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T03:29:29.977 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2023-01-24 03:29:20.310479857 +0000 2023-01-24T03:29:29.978 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2023-01-24 03:27:00.562572176 +0000 2023-01-24T03:29:29.978 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2023-01-24 03:27:00.562572176 +0000 2023-01-24T03:29:29.978 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: - 2023-01-24T03:29:29.978 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-01-24T03:29:30.041 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2023-01-24T03:29:30.041 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2023-01-24T03:29:30.041 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000224985 s, 2.3 MB/s 2023-01-24T03:29:30.042 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-01-24T03:29:30.097 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_4 2023-01-24T03:29:30.151 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-01-24T03:29:30.152 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T03:29:30.152 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 6h/6d Inode: 203077 Links: 1 2023-01-24T03:29:30.152 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T03:29:30.152 INFO:teuthology.orchestra.run.smithi142.stdout:Context: system_u:object_r:device_t:s0 2023-01-24T03:29:30.153 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2023-01-24 03:29:20.310479857 +0000 2023-01-24T03:29:30.153 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2023-01-24 03:27:01.048563245 +0000 2023-01-24T03:29:30.153 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2023-01-24 03:27:01.048563245 +0000 2023-01-24T03:29:30.153 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: - 2023-01-24T03:29:30.153 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-01-24T03:29:30.214 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2023-01-24T03:29:30.215 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2023-01-24T03:29:30.215 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000189518 s, 2.7 MB/s 2023-01-24T03:29:30.216 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-01-24T03:29:30.271 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-01-24T03:29:30.271 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi112.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi142.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-01-24T03:29:30.271 INFO:tasks.ceph:Generating config... 2023-01-24T03:29:30.273 INFO:tasks.ceph:[client] client mount timeout = 600 2023-01-24T03:29:30.273 INFO:tasks.ceph:[client] debug client = 20 2023-01-24T03:29:30.274 INFO:tasks.ceph:[client] debug ms = 1 2023-01-24T03:29:30.274 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-01-24T03:29:30.274 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-01-24T03:29:30.274 INFO:tasks.ceph:[global] ms die on skipped message = False 2023-01-24T03:29:30.275 INFO:tasks.ceph:[mds] debug mds = 20 2023-01-24T03:29:30.275 INFO:tasks.ceph:[mds] debug ms = 1 2023-01-24T03:29:30.275 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2023-01-24T03:29:30.275 INFO:tasks.ceph:[mds] mds bal merge size = 5 2023-01-24T03:29:30.276 INFO:tasks.ceph:[mds] mds bal split bits = 3 2023-01-24T03:29:30.276 INFO:tasks.ceph:[mds] mds bal split size = 100 2023-01-24T03:29:30.276 INFO:tasks.ceph:[mds] mds debug frag = True 2023-01-24T03:29:30.277 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-01-24T03:29:30.277 INFO:tasks.ceph:[mds] mds oft prefetch dirfrags = False 2023-01-24T03:29:30.277 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-01-24T03:29:30.277 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-01-24T03:29:30.278 INFO:tasks.ceph:[mds] mds_sleep_rank_change = 5000000.0 2023-01-24T03:29:30.278 INFO:tasks.ceph:[mds] mon client directed command retry = 5 2023-01-24T03:29:30.278 INFO:tasks.ceph:[mds] ms inject delay max = 1 2023-01-24T03:29:30.278 INFO:tasks.ceph:[mds] ms inject delay probability = 0.005 2023-01-24T03:29:30.279 INFO:tasks.ceph:[mds] ms inject delay type = client mds osd 2023-01-24T03:29:30.279 INFO:tasks.ceph:[mds] ms inject socket failures = 2500 2023-01-24T03:29:30.279 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-01-24T03:29:30.279 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-01-24T03:29:30.280 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-01-24T03:29:30.280 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-01-24T03:29:30.280 INFO:tasks.ceph:[mgr] debug ms = 1 2023-01-24T03:29:30.280 INFO:tasks.ceph:[mon] debug mon = 20 2023-01-24T03:29:30.281 INFO:tasks.ceph:[mon] debug ms = 1 2023-01-24T03:29:30.281 INFO:tasks.ceph:[mon] debug paxos = 20 2023-01-24T03:29:30.281 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-01-24T03:29:30.281 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-01-24T03:29:30.282 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-01-24T03:29:30.282 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-01-24T03:29:30.282 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-01-24T03:29:30.282 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-01-24T03:29:30.283 INFO:tasks.ceph:[osd] debug ms = 1 2023-01-24T03:29:30.283 INFO:tasks.ceph:[osd] debug osd = 20 2023-01-24T03:29:30.283 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-01-24T03:29:30.283 INFO:tasks.ceph:[osd] mon client directed command retry = 5 2023-01-24T03:29:30.284 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-01-24T03:29:30.284 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-01-24T03:29:30.284 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-01-24T03:29:30.285 INFO:tasks.ceph:[osd] ms inject delay max = 1 2023-01-24T03:29:30.285 INFO:tasks.ceph:[osd] ms inject delay probability = 0.005 2023-01-24T03:29:30.285 INFO:tasks.ceph:[osd] ms inject delay type = client mds 2023-01-24T03:29:30.285 INFO:tasks.ceph:[osd] ms inject socket failures = 2500 2023-01-24T03:29:30.286 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-01-24T03:29:30.286 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2023-01-24T03:29:30.286 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-01-24T03:29:30.286 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-01-24T03:29:30.287 INFO:tasks.ceph:Setting up mon.a... 2023-01-24T03:29:30.287 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-01-24T03:29:30.343 INFO:teuthology.orchestra.run.smithi112.stdout:creating /etc/ceph/ceph.keyring 2023-01-24T03:29:30.346 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-01-24T03:29:30.385 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-01-24T03:29:30.449 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.112'), ('mon.b', '172.21.15.142'), ('mon.c', '[v2:172.21.15.142:3301,v1:172.21.15.142:6790]')] 2023-01-24T03:29:30.450 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.112,172.21.15.142,[v2:172.21.15.142:3301,v1:172.21.15.142:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': False, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-01-24T03:29:30.451 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-01-24T03:29:30.451 DEBUG:teuthology.orchestra.run.smithi112:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-01-24T03:29:30.504 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.112 --add b 172.21.15.142 --addv c '[v2:172.21.15.142:3301,v1:172.21.15.142:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-01-24T03:29:30.586 INFO:teuthology.orchestra.run.smithi112.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-01-24T03:29:30.586 INFO:teuthology.orchestra.run.smithi112.stdout:monmaptool: generated fsid 054b9ac9-b7b0-40e5-b932-6c8a955c5a72 2023-01-24T03:29:30.586 INFO:teuthology.orchestra.run.smithi112.stdout:setting min_mon_release = octopus 2023-01-24T03:29:30.586 INFO:teuthology.orchestra.run.smithi112.stdout:epoch 0 2023-01-24T03:29:30.587 INFO:teuthology.orchestra.run.smithi112.stdout:fsid 054b9ac9-b7b0-40e5-b932-6c8a955c5a72 2023-01-24T03:29:30.587 INFO:teuthology.orchestra.run.smithi112.stdout:last_changed 2023-01-24T03:29:30.584849+0000 2023-01-24T03:29:30.587 INFO:teuthology.orchestra.run.smithi112.stdout:created 2023-01-24T03:29:30.584849+0000 2023-01-24T03:29:30.587 INFO:teuthology.orchestra.run.smithi112.stdout:min_mon_release 15 (octopus) 2023-01-24T03:29:30.587 INFO:teuthology.orchestra.run.smithi112.stdout:election_strategy: 1 2023-01-24T03:29:30.588 INFO:teuthology.orchestra.run.smithi112.stdout:0: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.a 2023-01-24T03:29:30.588 INFO:teuthology.orchestra.run.smithi112.stdout:1: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.b 2023-01-24T03:29:30.588 INFO:teuthology.orchestra.run.smithi112.stdout:2: [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] mon.c 2023-01-24T03:29:30.588 INFO:teuthology.orchestra.run.smithi112.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-01-24T03:29:30.590 DEBUG:teuthology.orchestra.run.smithi112:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-01-24T03:29:30.643 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 054b9ac9-b7b0-40e5-b932-6c8a955c5a72... 2023-01-24T03:29:30.645 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-01-24T03:29:30.686 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-01-24T03:29:30.724 INFO:teuthology.orchestra.run.smithi112.stdout:[global] 2023-01-24T03:29:30.724 INFO:teuthology.orchestra.run.smithi112.stdout: chdir = "" 2023-01-24T03:29:30.724 INFO:teuthology.orchestra.run.smithi112.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-01-24T03:29:30.725 INFO:teuthology.orchestra.run.smithi112.stdout: auth supported = cephx 2023-01-24T03:29:30.725 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:29:30.725 INFO:teuthology.orchestra.run.smithi112.stdout: filestore xattr use omap = true 2023-01-24T03:29:30.725 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:29:30.725 INFO:teuthology.orchestra.run.smithi112.stdout: mon clock drift allowed = 1.000 2023-01-24T03:29:30.726 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:29:30.726 INFO:teuthology.orchestra.run.smithi112.stdout: osd crush chooseleaf type = 0 2023-01-24T03:29:30.726 INFO:teuthology.orchestra.run.smithi112.stdout: auth debug = true 2023-01-24T03:29:30.726 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:29:30.726 INFO:teuthology.orchestra.run.smithi112.stdout: ms die on old message = true 2023-01-24T03:29:30.727 INFO:teuthology.orchestra.run.smithi112.stdout: ms die on bug = true 2023-01-24T03:29:30.727 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:29:30.727 INFO:teuthology.orchestra.run.smithi112.stdout: mon max pg per osd = 10000 # >= luminous 2023-01-24T03:29:30.727 INFO:teuthology.orchestra.run.smithi112.stdout: mon pg warn max object skew = 0 2023-01-24T03:29:30.727 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:29:30.727 INFO:teuthology.orchestra.run.smithi112.stdout: # disable pg_autoscaler by default for new pools 2023-01-24T03:29:30.728 INFO:teuthology.orchestra.run.smithi112.stdout: osd_pool_default_pg_autoscale_mode = off 2023-01-24T03:29:30.728 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:29:30.728 INFO:teuthology.orchestra.run.smithi112.stdout: osd pool default size = 2 2023-01-24T03:29:30.729 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:29:30.729 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd allow primary affinity = true 2023-01-24T03:29:30.729 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd allow pg remap = true 2023-01-24T03:29:30.729 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on legacy crush tunables = false 2023-01-24T03:29:30.729 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on crush straw calc version zero = false 2023-01-24T03:29:30.730 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on no sortbitwise = false 2023-01-24T03:29:30.730 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on osd down out interval zero = false 2023-01-24T03:29:30.730 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on too few osds = false 2023-01-24T03:29:30.730 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-01-24T03:29:30.730 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_pool_no_redundancy = false 2023-01-24T03:29:30.731 INFO:teuthology.orchestra.run.smithi112.stdout: mon_allow_pool_size_one = true 2023-01-24T03:29:30.731 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:29:30.731 INFO:teuthology.orchestra.run.smithi112.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-01-24T03:29:30.731 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:29:30.731 INFO:teuthology.orchestra.run.smithi112.stdout: osd default data pool replay window = 5 2023-01-24T03:29:30.732 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:29:30.732 INFO:teuthology.orchestra.run.smithi112.stdout: mon allow pool delete = true 2023-01-24T03:29:30.732 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:29:30.732 INFO:teuthology.orchestra.run.smithi112.stdout: mon cluster log file level = debug 2023-01-24T03:29:30.732 INFO:teuthology.orchestra.run.smithi112.stdout: debug asserts on shutdown = true 2023-01-24T03:29:30.733 INFO:teuthology.orchestra.run.smithi112.stdout: mon health detail to clog = false 2023-01-24T03:29:30.733 INFO:teuthology.orchestra.run.smithi112.stdout: mon host = "172.21.15.112,172.21.15.142,[v2:172.21.15.142:3301,v1:172.21.15.142:6790]" 2023-01-24T03:29:30.733 INFO:teuthology.orchestra.run.smithi112.stdout: ms die on skipped message = False 2023-01-24T03:29:30.734 INFO:teuthology.orchestra.run.smithi112.stdout: fsid = 054b9ac9-b7b0-40e5-b932-6c8a955c5a72 2023-01-24T03:29:30.734 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:29:30.734 INFO:teuthology.orchestra.run.smithi112.stdout:[osd] 2023-01-24T03:29:30.735 INFO:teuthology.orchestra.run.smithi112.stdout: osd journal size = 100 2023-01-24T03:29:30.735 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:29:30.735 INFO:teuthology.orchestra.run.smithi112.stdout: osd scrub load threshold = 5.0 2023-01-24T03:29:30.736 INFO:teuthology.orchestra.run.smithi112.stdout: osd scrub max interval = 600 2023-01-24T03:29:30.736 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:29:30.736 INFO:teuthology.orchestra.run.smithi112.stdout: osd recover clone overlap = true 2023-01-24T03:29:30.737 INFO:teuthology.orchestra.run.smithi112.stdout: osd recovery max chunk = 1048576 2023-01-24T03:29:30.737 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:29:30.737 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug shutdown = true 2023-01-24T03:29:30.737 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug op order = true 2023-01-24T03:29:30.738 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug verify stray on activate = true 2023-01-24T03:29:30.738 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:29:30.738 INFO:teuthology.orchestra.run.smithi112.stdout: osd open classes on start = true 2023-01-24T03:29:30.738 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug pg log writeout = true 2023-01-24T03:29:30.739 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:29:30.739 INFO:teuthology.orchestra.run.smithi112.stdout: osd deep scrub update digest min age = 30 2023-01-24T03:29:30.739 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:29:30.740 INFO:teuthology.orchestra.run.smithi112.stdout: osd map max advance = 10 2023-01-24T03:29:30.740 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:29:30.740 INFO:teuthology.orchestra.run.smithi112.stdout: journal zero on create = true 2023-01-24T03:29:30.740 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:29:30.741 INFO:teuthology.orchestra.run.smithi112.stdout: filestore ondisk finisher threads = 3 2023-01-24T03:29:30.741 INFO:teuthology.orchestra.run.smithi112.stdout: filestore apply finisher threads = 3 2023-01-24T03:29:30.741 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:29:30.741 INFO:teuthology.orchestra.run.smithi112.stdout: bdev debug aio = true 2023-01-24T03:29:30.741 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug misdirected ops = true 2023-01-24T03:29:30.742 INFO:teuthology.orchestra.run.smithi112.stdout: bluestore block size = 96636764160 2023-01-24T03:29:30.742 INFO:teuthology.orchestra.run.smithi112.stdout: bluestore compression mode = aggressive 2023-01-24T03:29:30.742 INFO:teuthology.orchestra.run.smithi112.stdout: bluestore fsck on mount = True 2023-01-24T03:29:30.742 INFO:teuthology.orchestra.run.smithi112.stdout: debug bluefs = 20 2023-01-24T03:29:30.742 INFO:teuthology.orchestra.run.smithi112.stdout: debug bluestore = 20 2023-01-24T03:29:30.743 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2023-01-24T03:29:30.743 INFO:teuthology.orchestra.run.smithi112.stdout: debug osd = 20 2023-01-24T03:29:30.743 INFO:teuthology.orchestra.run.smithi112.stdout: debug rocksdb = 10 2023-01-24T03:29:30.743 INFO:teuthology.orchestra.run.smithi112.stdout: mon client directed command retry = 5 2023-01-24T03:29:30.743 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd backfillfull_ratio = 0.85 2023-01-24T03:29:30.744 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd full ratio = 0.9 2023-01-24T03:29:30.744 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd nearfull ratio = 0.8 2023-01-24T03:29:30.744 INFO:teuthology.orchestra.run.smithi112.stdout: ms inject delay max = 1 2023-01-24T03:29:30.744 INFO:teuthology.orchestra.run.smithi112.stdout: ms inject delay probability = 0.005 2023-01-24T03:29:30.744 INFO:teuthology.orchestra.run.smithi112.stdout: ms inject delay type = client mds 2023-01-24T03:29:30.745 INFO:teuthology.orchestra.run.smithi112.stdout: ms inject socket failures = 2500 2023-01-24T03:29:30.745 INFO:teuthology.orchestra.run.smithi112.stdout: osd failsafe full ratio = 0.95 2023-01-24T03:29:30.745 INFO:teuthology.orchestra.run.smithi112.stdout: osd max markdown count = 1000 2023-01-24T03:29:30.745 INFO:teuthology.orchestra.run.smithi112.stdout: osd objectstore = bluestore 2023-01-24T03:29:30.745 INFO:teuthology.orchestra.run.smithi112.stdout: osd op complaint time = 180 2023-01-24T03:29:30.746 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:29:30.746 INFO:teuthology.orchestra.run.smithi112.stdout:[mgr] 2023-01-24T03:29:30.746 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2023-01-24T03:29:30.746 INFO:teuthology.orchestra.run.smithi112.stdout: debug mgr = 20 2023-01-24T03:29:30.746 INFO:teuthology.orchestra.run.smithi112.stdout: debug mon = 20 2023-01-24T03:29:30.747 INFO:teuthology.orchestra.run.smithi112.stdout: debug auth = 20 2023-01-24T03:29:30.747 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min pgs per osd = 4 2023-01-24T03:29:30.747 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min bytes per osd = 10 2023-01-24T03:29:30.747 INFO:teuthology.orchestra.run.smithi112.stdout: mgr/telemetry/nag = false 2023-01-24T03:29:30.747 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:29:30.748 INFO:teuthology.orchestra.run.smithi112.stdout:[mon] 2023-01-24T03:29:30.748 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2023-01-24T03:29:30.748 INFO:teuthology.orchestra.run.smithi112.stdout: debug mon = 20 2023-01-24T03:29:30.748 INFO:teuthology.orchestra.run.smithi112.stdout: debug paxos = 20 2023-01-24T03:29:30.748 INFO:teuthology.orchestra.run.smithi112.stdout: debug auth = 20 2023-01-24T03:29:30.749 INFO:teuthology.orchestra.run.smithi112.stdout: mon data avail warn = 5 2023-01-24T03:29:30.749 INFO:teuthology.orchestra.run.smithi112.stdout: mon mgr mkfs grace = 240 2023-01-24T03:29:30.749 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min pgs per osd = 4 2023-01-24T03:29:30.749 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd reporter subtree level = osd 2023-01-24T03:29:30.749 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd prime pg temp = true 2023-01-24T03:29:30.750 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min bytes per osd = 10 2023-01-24T03:29:30.750 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:29:30.750 INFO:teuthology.orchestra.run.smithi112.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-01-24T03:29:30.750 INFO:teuthology.orchestra.run.smithi112.stdout: auth mon ticket ttl = 660 # 11m 2023-01-24T03:29:30.750 INFO:teuthology.orchestra.run.smithi112.stdout: auth service ticket ttl = 240 # 4m 2023-01-24T03:29:30.751 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:29:30.751 INFO:teuthology.orchestra.run.smithi112.stdout: # don't complain about insecure global_id in the test suite 2023-01-24T03:29:30.751 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-01-24T03:29:30.751 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-01-24T03:29:30.751 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:29:30.752 INFO:teuthology.orchestra.run.smithi112.stdout: # 1m isn't quite enough 2023-01-24T03:29:30.752 INFO:teuthology.orchestra.run.smithi112.stdout: mon_down_mkfs_grace = 2m 2023-01-24T03:29:30.752 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:29:30.752 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_filestore_osds = false 2023-01-24T03:29:30.752 INFO:teuthology.orchestra.run.smithi112.stdout: mon op complaint time = 120 2023-01-24T03:29:30.753 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:29:30.753 INFO:teuthology.orchestra.run.smithi112.stdout:[client] 2023-01-24T03:29:30.753 INFO:teuthology.orchestra.run.smithi112.stdout: rgw cache enabled = true 2023-01-24T03:29:30.753 INFO:teuthology.orchestra.run.smithi112.stdout: rgw enable ops log = true 2023-01-24T03:29:30.753 INFO:teuthology.orchestra.run.smithi112.stdout: rgw enable usage log = true 2023-01-24T03:29:30.754 INFO:teuthology.orchestra.run.smithi112.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-01-24T03:29:30.754 INFO:teuthology.orchestra.run.smithi112.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-01-24T03:29:30.754 INFO:teuthology.orchestra.run.smithi112.stdout: client mount timeout = 600 2023-01-24T03:29:30.754 INFO:teuthology.orchestra.run.smithi112.stdout: debug client = 20 2023-01-24T03:29:30.754 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2023-01-24T03:29:30.755 INFO:teuthology.orchestra.run.smithi112.stdout: rados mon op timeout = 900 2023-01-24T03:29:30.755 INFO:teuthology.orchestra.run.smithi112.stdout: rados osd op timeout = 900 2023-01-24T03:29:30.755 INFO:teuthology.orchestra.run.smithi112.stdout:[mon.a] 2023-01-24T03:29:30.755 INFO:teuthology.orchestra.run.smithi112.stdout:[mon.b] 2023-01-24T03:29:30.755 INFO:teuthology.orchestra.run.smithi112.stdout:[mon.c] 2023-01-24T03:29:30.756 INFO:teuthology.orchestra.run.smithi112.stdout:[mds.a] 2023-01-24T03:29:30.756 INFO:teuthology.orchestra.run.smithi112.stdout:[mds.c] 2023-01-24T03:29:30.756 INFO:teuthology.orchestra.run.smithi112.stdout:[mds.e] 2023-01-24T03:29:30.756 INFO:teuthology.orchestra.run.smithi112.stdout:[mds.b] 2023-01-24T03:29:30.756 INFO:teuthology.orchestra.run.smithi112.stdout:[mds.d] 2023-01-24T03:29:30.757 INFO:teuthology.orchestra.run.smithi112.stdout:[mds.f] 2023-01-24T03:29:30.757 INFO:teuthology.orchestra.run.smithi112.stdout:[mds] 2023-01-24T03:29:30.757 INFO:teuthology.orchestra.run.smithi112.stdout: debug mds = 20 2023-01-24T03:29:30.757 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2023-01-24T03:29:30.757 INFO:teuthology.orchestra.run.smithi112.stdout: mds bal fragment size max = 10000 2023-01-24T03:29:30.757 INFO:teuthology.orchestra.run.smithi112.stdout: mds bal merge size = 5 2023-01-24T03:29:30.758 INFO:teuthology.orchestra.run.smithi112.stdout: mds bal split bits = 3 2023-01-24T03:29:30.758 INFO:teuthology.orchestra.run.smithi112.stdout: mds bal split size = 100 2023-01-24T03:29:30.758 INFO:teuthology.orchestra.run.smithi112.stdout: mds debug frag = True 2023-01-24T03:29:30.758 INFO:teuthology.orchestra.run.smithi112.stdout: mds debug scatterstat = True 2023-01-24T03:29:30.758 INFO:teuthology.orchestra.run.smithi112.stdout: mds oft prefetch dirfrags = False 2023-01-24T03:29:30.759 INFO:teuthology.orchestra.run.smithi112.stdout: mds op complaint time = 180 2023-01-24T03:29:30.759 INFO:teuthology.orchestra.run.smithi112.stdout: mds verify scatter = True 2023-01-24T03:29:30.759 INFO:teuthology.orchestra.run.smithi112.stdout: mds_sleep_rank_change = 5000000.0 2023-01-24T03:29:30.759 INFO:teuthology.orchestra.run.smithi112.stdout: mon client directed command retry = 5 2023-01-24T03:29:30.759 INFO:teuthology.orchestra.run.smithi112.stdout: ms inject delay max = 1 2023-01-24T03:29:30.760 INFO:teuthology.orchestra.run.smithi112.stdout: ms inject delay probability = 0.005 2023-01-24T03:29:30.760 INFO:teuthology.orchestra.run.smithi112.stdout: ms inject delay type = client mds osd 2023-01-24T03:29:30.760 INFO:teuthology.orchestra.run.smithi112.stdout: ms inject socket failures = 2500 2023-01-24T03:29:30.760 INFO:teuthology.orchestra.run.smithi112.stdout: osd op complaint time = 180 2023-01-24T03:29:30.760 INFO:teuthology.orchestra.run.smithi112.stdout: rados mon op timeout = 900 2023-01-24T03:29:30.761 INFO:teuthology.orchestra.run.smithi112.stdout: rados osd op timeout = 900 2023-01-24T03:29:30.763 INFO:teuthology.orchestra.run.smithi142.stdout:[global] 2023-01-24T03:29:30.764 INFO:teuthology.orchestra.run.smithi142.stdout: chdir = "" 2023-01-24T03:29:30.764 INFO:teuthology.orchestra.run.smithi142.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-01-24T03:29:30.764 INFO:teuthology.orchestra.run.smithi142.stdout: auth supported = cephx 2023-01-24T03:29:30.764 INFO:teuthology.orchestra.run.smithi142.stdout: 2023-01-24T03:29:30.765 INFO:teuthology.orchestra.run.smithi142.stdout: filestore xattr use omap = true 2023-01-24T03:29:30.765 INFO:teuthology.orchestra.run.smithi142.stdout: 2023-01-24T03:29:30.765 INFO:teuthology.orchestra.run.smithi142.stdout: mon clock drift allowed = 1.000 2023-01-24T03:29:30.766 INFO:teuthology.orchestra.run.smithi142.stdout: 2023-01-24T03:29:30.766 INFO:teuthology.orchestra.run.smithi142.stdout: osd crush chooseleaf type = 0 2023-01-24T03:29:30.766 INFO:teuthology.orchestra.run.smithi142.stdout: auth debug = true 2023-01-24T03:29:30.766 INFO:teuthology.orchestra.run.smithi142.stdout: 2023-01-24T03:29:30.767 INFO:teuthology.orchestra.run.smithi142.stdout: ms die on old message = true 2023-01-24T03:29:30.767 INFO:teuthology.orchestra.run.smithi142.stdout: ms die on bug = true 2023-01-24T03:29:30.767 INFO:teuthology.orchestra.run.smithi142.stdout: 2023-01-24T03:29:30.768 INFO:teuthology.orchestra.run.smithi142.stdout: mon max pg per osd = 10000 # >= luminous 2023-01-24T03:29:30.768 INFO:teuthology.orchestra.run.smithi142.stdout: mon pg warn max object skew = 0 2023-01-24T03:29:30.768 INFO:teuthology.orchestra.run.smithi142.stdout: 2023-01-24T03:29:30.768 INFO:teuthology.orchestra.run.smithi142.stdout: # disable pg_autoscaler by default for new pools 2023-01-24T03:29:30.769 INFO:teuthology.orchestra.run.smithi142.stdout: osd_pool_default_pg_autoscale_mode = off 2023-01-24T03:29:30.769 INFO:teuthology.orchestra.run.smithi142.stdout: 2023-01-24T03:29:30.769 INFO:teuthology.orchestra.run.smithi142.stdout: osd pool default size = 2 2023-01-24T03:29:30.769 INFO:teuthology.orchestra.run.smithi142.stdout: 2023-01-24T03:29:30.770 INFO:teuthology.orchestra.run.smithi142.stdout: mon osd allow primary affinity = true 2023-01-24T03:29:30.770 INFO:teuthology.orchestra.run.smithi142.stdout: mon osd allow pg remap = true 2023-01-24T03:29:30.770 INFO:teuthology.orchestra.run.smithi142.stdout: mon warn on legacy crush tunables = false 2023-01-24T03:29:30.770 INFO:teuthology.orchestra.run.smithi142.stdout: mon warn on crush straw calc version zero = false 2023-01-24T03:29:30.770 INFO:teuthology.orchestra.run.smithi142.stdout: mon warn on no sortbitwise = false 2023-01-24T03:29:30.771 INFO:teuthology.orchestra.run.smithi142.stdout: mon warn on osd down out interval zero = false 2023-01-24T03:29:30.771 INFO:teuthology.orchestra.run.smithi142.stdout: mon warn on too few osds = false 2023-01-24T03:29:30.771 INFO:teuthology.orchestra.run.smithi142.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-01-24T03:29:30.771 INFO:teuthology.orchestra.run.smithi142.stdout: mon_warn_on_pool_no_redundancy = false 2023-01-24T03:29:30.771 INFO:teuthology.orchestra.run.smithi142.stdout: mon_allow_pool_size_one = true 2023-01-24T03:29:30.772 INFO:teuthology.orchestra.run.smithi142.stdout: 2023-01-24T03:29:30.772 INFO:teuthology.orchestra.run.smithi142.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-01-24T03:29:30.772 INFO:teuthology.orchestra.run.smithi142.stdout: 2023-01-24T03:29:30.772 INFO:teuthology.orchestra.run.smithi142.stdout: osd default data pool replay window = 5 2023-01-24T03:29:30.772 INFO:teuthology.orchestra.run.smithi142.stdout: 2023-01-24T03:29:30.773 INFO:teuthology.orchestra.run.smithi142.stdout: mon allow pool delete = true 2023-01-24T03:29:30.773 INFO:teuthology.orchestra.run.smithi142.stdout: 2023-01-24T03:29:30.773 INFO:teuthology.orchestra.run.smithi142.stdout: mon cluster log file level = debug 2023-01-24T03:29:30.773 INFO:teuthology.orchestra.run.smithi142.stdout: debug asserts on shutdown = true 2023-01-24T03:29:30.773 INFO:teuthology.orchestra.run.smithi142.stdout: mon health detail to clog = false 2023-01-24T03:29:30.774 INFO:teuthology.orchestra.run.smithi142.stdout: mon host = "172.21.15.112,172.21.15.142,[v2:172.21.15.142:3301,v1:172.21.15.142:6790]" 2023-01-24T03:29:30.774 INFO:teuthology.orchestra.run.smithi142.stdout: ms die on skipped message = False 2023-01-24T03:29:30.774 INFO:teuthology.orchestra.run.smithi142.stdout: fsid = 054b9ac9-b7b0-40e5-b932-6c8a955c5a72 2023-01-24T03:29:30.774 INFO:teuthology.orchestra.run.smithi142.stdout: 2023-01-24T03:29:30.774 INFO:teuthology.orchestra.run.smithi142.stdout:[osd] 2023-01-24T03:29:30.775 INFO:teuthology.orchestra.run.smithi142.stdout: osd journal size = 100 2023-01-24T03:29:30.775 INFO:teuthology.orchestra.run.smithi142.stdout: 2023-01-24T03:29:30.775 INFO:teuthology.orchestra.run.smithi142.stdout: osd scrub load threshold = 5.0 2023-01-24T03:29:30.775 INFO:teuthology.orchestra.run.smithi142.stdout: osd scrub max interval = 600 2023-01-24T03:29:30.775 INFO:teuthology.orchestra.run.smithi142.stdout: 2023-01-24T03:29:30.776 INFO:teuthology.orchestra.run.smithi142.stdout: osd recover clone overlap = true 2023-01-24T03:29:30.776 INFO:teuthology.orchestra.run.smithi142.stdout: osd recovery max chunk = 1048576 2023-01-24T03:29:30.776 INFO:teuthology.orchestra.run.smithi142.stdout: 2023-01-24T03:29:30.776 INFO:teuthology.orchestra.run.smithi142.stdout: osd debug shutdown = true 2023-01-24T03:29:30.776 INFO:teuthology.orchestra.run.smithi142.stdout: osd debug op order = true 2023-01-24T03:29:30.777 INFO:teuthology.orchestra.run.smithi142.stdout: osd debug verify stray on activate = true 2023-01-24T03:29:30.777 INFO:teuthology.orchestra.run.smithi142.stdout: 2023-01-24T03:29:30.777 INFO:teuthology.orchestra.run.smithi142.stdout: osd open classes on start = true 2023-01-24T03:29:30.777 INFO:teuthology.orchestra.run.smithi142.stdout: osd debug pg log writeout = true 2023-01-24T03:29:30.777 INFO:teuthology.orchestra.run.smithi142.stdout: 2023-01-24T03:29:30.778 INFO:teuthology.orchestra.run.smithi142.stdout: osd deep scrub update digest min age = 30 2023-01-24T03:29:30.778 INFO:teuthology.orchestra.run.smithi142.stdout: 2023-01-24T03:29:30.778 INFO:teuthology.orchestra.run.smithi142.stdout: osd map max advance = 10 2023-01-24T03:29:30.778 INFO:teuthology.orchestra.run.smithi142.stdout: 2023-01-24T03:29:30.778 INFO:teuthology.orchestra.run.smithi142.stdout: journal zero on create = true 2023-01-24T03:29:30.779 INFO:teuthology.orchestra.run.smithi142.stdout: 2023-01-24T03:29:30.779 INFO:teuthology.orchestra.run.smithi142.stdout: filestore ondisk finisher threads = 3 2023-01-24T03:29:30.779 INFO:teuthology.orchestra.run.smithi142.stdout: filestore apply finisher threads = 3 2023-01-24T03:29:30.779 INFO:teuthology.orchestra.run.smithi142.stdout: 2023-01-24T03:29:30.779 INFO:teuthology.orchestra.run.smithi142.stdout: bdev debug aio = true 2023-01-24T03:29:30.780 INFO:teuthology.orchestra.run.smithi142.stdout: osd debug misdirected ops = true 2023-01-24T03:29:30.780 INFO:teuthology.orchestra.run.smithi142.stdout: bluestore block size = 96636764160 2023-01-24T03:29:30.780 INFO:teuthology.orchestra.run.smithi142.stdout: bluestore compression mode = aggressive 2023-01-24T03:29:30.780 INFO:teuthology.orchestra.run.smithi142.stdout: bluestore fsck on mount = True 2023-01-24T03:29:30.780 INFO:teuthology.orchestra.run.smithi142.stdout: debug bluefs = 20 2023-01-24T03:29:30.781 INFO:teuthology.orchestra.run.smithi142.stdout: debug bluestore = 20 2023-01-24T03:29:30.781 INFO:teuthology.orchestra.run.smithi142.stdout: debug ms = 1 2023-01-24T03:29:30.781 INFO:teuthology.orchestra.run.smithi142.stdout: debug osd = 20 2023-01-24T03:29:30.781 INFO:teuthology.orchestra.run.smithi142.stdout: debug rocksdb = 10 2023-01-24T03:29:30.781 INFO:teuthology.orchestra.run.smithi142.stdout: mon client directed command retry = 5 2023-01-24T03:29:30.782 INFO:teuthology.orchestra.run.smithi142.stdout: mon osd backfillfull_ratio = 0.85 2023-01-24T03:29:30.782 INFO:teuthology.orchestra.run.smithi142.stdout: mon osd full ratio = 0.9 2023-01-24T03:29:30.782 INFO:teuthology.orchestra.run.smithi142.stdout: mon osd nearfull ratio = 0.8 2023-01-24T03:29:30.782 INFO:teuthology.orchestra.run.smithi142.stdout: ms inject delay max = 1 2023-01-24T03:29:30.782 INFO:teuthology.orchestra.run.smithi142.stdout: ms inject delay probability = 0.005 2023-01-24T03:29:30.783 INFO:teuthology.orchestra.run.smithi142.stdout: ms inject delay type = client mds 2023-01-24T03:29:30.783 INFO:teuthology.orchestra.run.smithi142.stdout: ms inject socket failures = 2500 2023-01-24T03:29:30.783 INFO:teuthology.orchestra.run.smithi142.stdout: osd failsafe full ratio = 0.95 2023-01-24T03:29:30.783 INFO:teuthology.orchestra.run.smithi142.stdout: osd max markdown count = 1000 2023-01-24T03:29:30.783 INFO:teuthology.orchestra.run.smithi142.stdout: osd objectstore = bluestore 2023-01-24T03:29:30.784 INFO:teuthology.orchestra.run.smithi142.stdout: osd op complaint time = 180 2023-01-24T03:29:30.784 INFO:teuthology.orchestra.run.smithi142.stdout: 2023-01-24T03:29:30.784 INFO:teuthology.orchestra.run.smithi142.stdout:[mgr] 2023-01-24T03:29:30.784 INFO:teuthology.orchestra.run.smithi142.stdout: debug ms = 1 2023-01-24T03:29:30.784 INFO:teuthology.orchestra.run.smithi142.stdout: debug mgr = 20 2023-01-24T03:29:30.785 INFO:teuthology.orchestra.run.smithi142.stdout: debug mon = 20 2023-01-24T03:29:30.785 INFO:teuthology.orchestra.run.smithi142.stdout: debug auth = 20 2023-01-24T03:29:30.785 INFO:teuthology.orchestra.run.smithi142.stdout: mon reweight min pgs per osd = 4 2023-01-24T03:29:30.785 INFO:teuthology.orchestra.run.smithi142.stdout: mon reweight min bytes per osd = 10 2023-01-24T03:29:30.785 INFO:teuthology.orchestra.run.smithi142.stdout: mgr/telemetry/nag = false 2023-01-24T03:29:30.786 INFO:teuthology.orchestra.run.smithi142.stdout: 2023-01-24T03:29:30.786 INFO:teuthology.orchestra.run.smithi142.stdout:[mon] 2023-01-24T03:29:30.786 INFO:teuthology.orchestra.run.smithi142.stdout: debug ms = 1 2023-01-24T03:29:30.786 INFO:teuthology.orchestra.run.smithi142.stdout: debug mon = 20 2023-01-24T03:29:30.786 INFO:teuthology.orchestra.run.smithi142.stdout: debug paxos = 20 2023-01-24T03:29:30.787 INFO:teuthology.orchestra.run.smithi142.stdout: debug auth = 20 2023-01-24T03:29:30.787 INFO:teuthology.orchestra.run.smithi142.stdout: mon data avail warn = 5 2023-01-24T03:29:30.787 INFO:teuthology.orchestra.run.smithi142.stdout: mon mgr mkfs grace = 240 2023-01-24T03:29:30.787 INFO:teuthology.orchestra.run.smithi142.stdout: mon reweight min pgs per osd = 4 2023-01-24T03:29:30.787 INFO:teuthology.orchestra.run.smithi142.stdout: mon osd reporter subtree level = osd 2023-01-24T03:29:30.788 INFO:teuthology.orchestra.run.smithi142.stdout: mon osd prime pg temp = true 2023-01-24T03:29:30.788 INFO:teuthology.orchestra.run.smithi142.stdout: mon reweight min bytes per osd = 10 2023-01-24T03:29:30.788 INFO:teuthology.orchestra.run.smithi142.stdout: 2023-01-24T03:29:30.788 INFO:teuthology.orchestra.run.smithi142.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-01-24T03:29:30.788 INFO:teuthology.orchestra.run.smithi142.stdout: auth mon ticket ttl = 660 # 11m 2023-01-24T03:29:30.789 INFO:teuthology.orchestra.run.smithi142.stdout: auth service ticket ttl = 240 # 4m 2023-01-24T03:29:30.789 INFO:teuthology.orchestra.run.smithi142.stdout: 2023-01-24T03:29:30.789 INFO:teuthology.orchestra.run.smithi142.stdout: # don't complain about insecure global_id in the test suite 2023-01-24T03:29:30.789 INFO:teuthology.orchestra.run.smithi142.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-01-24T03:29:30.789 INFO:teuthology.orchestra.run.smithi142.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-01-24T03:29:30.790 INFO:teuthology.orchestra.run.smithi142.stdout: 2023-01-24T03:29:30.790 INFO:teuthology.orchestra.run.smithi142.stdout: # 1m isn't quite enough 2023-01-24T03:29:30.790 INFO:teuthology.orchestra.run.smithi142.stdout: mon_down_mkfs_grace = 2m 2023-01-24T03:29:30.790 INFO:teuthology.orchestra.run.smithi142.stdout: 2023-01-24T03:29:30.790 INFO:teuthology.orchestra.run.smithi142.stdout: mon_warn_on_filestore_osds = false 2023-01-24T03:29:30.791 INFO:teuthology.orchestra.run.smithi142.stdout: mon op complaint time = 120 2023-01-24T03:29:30.791 INFO:teuthology.orchestra.run.smithi142.stdout: 2023-01-24T03:29:30.791 INFO:teuthology.orchestra.run.smithi142.stdout:[client] 2023-01-24T03:29:30.791 INFO:teuthology.orchestra.run.smithi142.stdout: rgw cache enabled = true 2023-01-24T03:29:30.791 INFO:teuthology.orchestra.run.smithi142.stdout: rgw enable ops log = true 2023-01-24T03:29:30.792 INFO:teuthology.orchestra.run.smithi142.stdout: rgw enable usage log = true 2023-01-24T03:29:30.792 INFO:teuthology.orchestra.run.smithi142.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-01-24T03:29:30.792 INFO:teuthology.orchestra.run.smithi142.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-01-24T03:29:30.792 INFO:teuthology.orchestra.run.smithi142.stdout: client mount timeout = 600 2023-01-24T03:29:30.792 INFO:teuthology.orchestra.run.smithi142.stdout: debug client = 20 2023-01-24T03:29:30.793 INFO:teuthology.orchestra.run.smithi142.stdout: debug ms = 1 2023-01-24T03:29:30.793 INFO:teuthology.orchestra.run.smithi142.stdout: rados mon op timeout = 900 2023-01-24T03:29:30.793 INFO:teuthology.orchestra.run.smithi142.stdout: rados osd op timeout = 900 2023-01-24T03:29:30.793 INFO:teuthology.orchestra.run.smithi142.stdout:[mon.a] 2023-01-24T03:29:30.793 INFO:teuthology.orchestra.run.smithi142.stdout:[mon.b] 2023-01-24T03:29:30.794 INFO:teuthology.orchestra.run.smithi142.stdout:[mon.c] 2023-01-24T03:29:30.794 INFO:teuthology.orchestra.run.smithi142.stdout:[mds.a] 2023-01-24T03:29:30.794 INFO:teuthology.orchestra.run.smithi142.stdout:[mds.c] 2023-01-24T03:29:30.794 INFO:teuthology.orchestra.run.smithi142.stdout:[mds.e] 2023-01-24T03:29:30.794 INFO:teuthology.orchestra.run.smithi142.stdout:[mds.b] 2023-01-24T03:29:30.795 INFO:teuthology.orchestra.run.smithi142.stdout:[mds.d] 2023-01-24T03:29:30.795 INFO:teuthology.orchestra.run.smithi142.stdout:[mds.f] 2023-01-24T03:29:30.795 INFO:teuthology.orchestra.run.smithi142.stdout:[mds] 2023-01-24T03:29:30.795 INFO:teuthology.orchestra.run.smithi142.stdout: debug mds = 20 2023-01-24T03:29:30.795 INFO:teuthology.orchestra.run.smithi142.stdout: debug ms = 1 2023-01-24T03:29:30.795 INFO:teuthology.orchestra.run.smithi142.stdout: mds bal fragment size max = 10000 2023-01-24T03:29:30.796 INFO:teuthology.orchestra.run.smithi142.stdout: mds bal merge size = 5 2023-01-24T03:29:30.796 INFO:teuthology.orchestra.run.smithi142.stdout: mds bal split bits = 3 2023-01-24T03:29:30.796 INFO:teuthology.orchestra.run.smithi142.stdout: mds bal split size = 100 2023-01-24T03:29:30.796 INFO:teuthology.orchestra.run.smithi142.stdout: mds debug frag = True 2023-01-24T03:29:30.796 INFO:teuthology.orchestra.run.smithi142.stdout: mds debug scatterstat = True 2023-01-24T03:29:30.797 INFO:teuthology.orchestra.run.smithi142.stdout: mds oft prefetch dirfrags = False 2023-01-24T03:29:30.797 INFO:teuthology.orchestra.run.smithi142.stdout: mds op complaint time = 180 2023-01-24T03:29:30.797 INFO:teuthology.orchestra.run.smithi142.stdout: mds verify scatter = True 2023-01-24T03:29:30.797 INFO:teuthology.orchestra.run.smithi142.stdout: mds_sleep_rank_change = 5000000.0 2023-01-24T03:29:30.797 INFO:teuthology.orchestra.run.smithi142.stdout: mon client directed command retry = 5 2023-01-24T03:29:30.798 INFO:teuthology.orchestra.run.smithi142.stdout: ms inject delay max = 1 2023-01-24T03:29:30.798 INFO:teuthology.orchestra.run.smithi142.stdout: ms inject delay probability = 0.005 2023-01-24T03:29:30.798 INFO:teuthology.orchestra.run.smithi142.stdout: ms inject delay type = client mds osd 2023-01-24T03:29:30.798 INFO:teuthology.orchestra.run.smithi142.stdout: ms inject socket failures = 2500 2023-01-24T03:29:30.798 INFO:teuthology.orchestra.run.smithi142.stdout: osd op complaint time = 180 2023-01-24T03:29:30.799 INFO:teuthology.orchestra.run.smithi142.stdout: rados mon op timeout = 900 2023-01-24T03:29:30.799 INFO:teuthology.orchestra.run.smithi142.stdout: rados osd op timeout = 900 2023-01-24T03:29:30.799 INFO:tasks.ceph:Creating admin key on mon.a... 2023-01-24T03:29:30.800 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-01-24T03:29:30.854 INFO:tasks.ceph:Copying monmap to all nodes... 2023-01-24T03:29:30.854 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-01-24T03:29:30.855 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-01-24T03:29:30.869 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-01-24T03:29:30.870 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-01-24T03:29:30.923 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi112.front.sepia.ceph.com 2023-01-24T03:29:30.924 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-01-24T03:29:30.924 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/ceph/ceph.keyring 2023-01-24T03:29:30.924 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-01-24T03:29:30.996 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-01-24T03:29:30.996 DEBUG:teuthology.orchestra.run.smithi112:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-01-24T03:29:31.051 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi142.front.sepia.ceph.com 2023-01-24T03:29:31.051 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2023-01-24T03:29:31.051 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/ceph/ceph.keyring 2023-01-24T03:29:31.052 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-01-24T03:29:31.086 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2023-01-24T03:29:31.087 DEBUG:teuthology.orchestra.run.smithi142:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-01-24T03:29:31.140 INFO:tasks.ceph:Setting up mon nodes... 2023-01-24T03:29:31.141 INFO:tasks.ceph:Setting up mgr nodes... 2023-01-24T03:29:31.141 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-01-24T03:29:31.186 INFO:teuthology.orchestra.run.smithi112.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-01-24T03:29:31.188 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-01-24T03:29:31.242 INFO:teuthology.orchestra.run.smithi142.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-01-24T03:29:31.244 INFO:tasks.ceph:Setting up mds nodes... 2023-01-24T03:29:31.245 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-01-24T03:29:31.291 INFO:teuthology.orchestra.run.smithi112.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-01-24T03:29:31.294 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-01-24T03:29:31.317 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-01-24T03:29:31.404 INFO:teuthology.orchestra.run.smithi112.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-01-24T03:29:31.406 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-01-24T03:29:31.429 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2023-01-24T03:29:31.515 INFO:teuthology.orchestra.run.smithi112.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2023-01-24T03:29:31.517 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2023-01-24T03:29:31.540 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-01-24T03:29:31.587 INFO:teuthology.orchestra.run.smithi142.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-01-24T03:29:31.589 DEBUG:teuthology.orchestra.run.smithi142:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-01-24T03:29:31.612 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-01-24T03:29:31.698 INFO:teuthology.orchestra.run.smithi142.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-01-24T03:29:31.700 DEBUG:teuthology.orchestra.run.smithi142:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-01-24T03:29:31.723 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2023-01-24T03:29:31.811 INFO:teuthology.orchestra.run.smithi142.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2023-01-24T03:29:31.813 DEBUG:teuthology.orchestra.run.smithi142:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2023-01-24T03:29:31.836 INFO:tasks.ceph_client:Setting up client nodes... 2023-01-24T03:29:31.837 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-01-24T03:29:31.874 INFO:teuthology.orchestra.run.smithi112.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-01-24T03:29:31.885 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-01-24T03:29:31.886 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi112.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi142.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-01-24T03:29:31.886 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-01-24T03:29:31.948 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-01-24T03:29:31.949 INFO:tasks.ceph:role: osd.0 2023-01-24T03:29:31.949 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi112.front.sepia.ceph.com 2023-01-24T03:29:31.950 DEBUG:teuthology.orchestra.run.smithi112:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-01-24T03:29:32.019 INFO:teuthology.orchestra.run.smithi112.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-01-24T03:29:32.019 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-24T03:29:32.020 INFO:teuthology.orchestra.run.smithi112.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-24T03:29:32.020 INFO:teuthology.orchestra.run.smithi112.stdout: = reflink=1 2023-01-24T03:29:32.020 INFO:teuthology.orchestra.run.smithi112.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-24T03:29:32.020 INFO:teuthology.orchestra.run.smithi112.stdout: = sunit=0 swidth=0 blks 2023-01-24T03:29:32.020 INFO:teuthology.orchestra.run.smithi112.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-24T03:29:32.021 INFO:teuthology.orchestra.run.smithi112.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-24T03:29:32.021 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-24T03:29:32.021 INFO:teuthology.orchestra.run.smithi112.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-24T03:29:32.387 INFO:teuthology.orchestra.run.smithi112.stdout:Discarding blocks...Done. 2023-01-24T03:29:32.388 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi112.front.sepia.ceph.com -o noatime 2023-01-24T03:29:32.389 DEBUG:teuthology.orchestra.run.smithi112:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-01-24T03:29:32.528 DEBUG:teuthology.orchestra.run.smithi112:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-01-24T03:29:32.557 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-01-24T03:29:32.620 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-01-24T03:29:32.621 INFO:tasks.ceph:role: osd.1 2023-01-24T03:29:32.621 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi112.front.sepia.ceph.com 2023-01-24T03:29:32.622 DEBUG:teuthology.orchestra.run.smithi112:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-01-24T03:29:32.691 INFO:teuthology.orchestra.run.smithi112.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-01-24T03:29:32.692 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-24T03:29:32.692 INFO:teuthology.orchestra.run.smithi112.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-24T03:29:32.692 INFO:teuthology.orchestra.run.smithi112.stdout: = reflink=1 2023-01-24T03:29:32.692 INFO:teuthology.orchestra.run.smithi112.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-24T03:29:32.693 INFO:teuthology.orchestra.run.smithi112.stdout: = sunit=0 swidth=0 blks 2023-01-24T03:29:32.693 INFO:teuthology.orchestra.run.smithi112.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-24T03:29:32.693 INFO:teuthology.orchestra.run.smithi112.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-24T03:29:32.693 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-24T03:29:32.693 INFO:teuthology.orchestra.run.smithi112.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-24T03:29:33.050 INFO:teuthology.orchestra.run.smithi112.stdout:Discarding blocks...Done. 2023-01-24T03:29:33.053 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi112.front.sepia.ceph.com -o noatime 2023-01-24T03:29:33.053 DEBUG:teuthology.orchestra.run.smithi112:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-01-24T03:29:33.139 DEBUG:teuthology.orchestra.run.smithi112:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-01-24T03:29:33.166 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-01-24T03:29:33.230 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-01-24T03:29:33.231 INFO:tasks.ceph:role: osd.2 2023-01-24T03:29:33.231 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi112.front.sepia.ceph.com 2023-01-24T03:29:33.231 DEBUG:teuthology.orchestra.run.smithi112:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-01-24T03:29:33.301 INFO:teuthology.orchestra.run.smithi112.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-01-24T03:29:33.302 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-24T03:29:33.302 INFO:teuthology.orchestra.run.smithi112.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-24T03:29:33.302 INFO:teuthology.orchestra.run.smithi112.stdout: = reflink=1 2023-01-24T03:29:33.302 INFO:teuthology.orchestra.run.smithi112.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-24T03:29:33.302 INFO:teuthology.orchestra.run.smithi112.stdout: = sunit=0 swidth=0 blks 2023-01-24T03:29:33.303 INFO:teuthology.orchestra.run.smithi112.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-24T03:29:33.303 INFO:teuthology.orchestra.run.smithi112.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-24T03:29:33.303 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-24T03:29:33.303 INFO:teuthology.orchestra.run.smithi112.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-24T03:29:33.665 INFO:teuthology.orchestra.run.smithi112.stdout:Discarding blocks...Done. 2023-01-24T03:29:33.666 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi112.front.sepia.ceph.com -o noatime 2023-01-24T03:29:33.667 DEBUG:teuthology.orchestra.run.smithi112:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-01-24T03:29:33.766 DEBUG:teuthology.orchestra.run.smithi112:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-01-24T03:29:33.792 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-01-24T03:29:33.856 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-01-24T03:29:33.857 INFO:tasks.ceph:role: osd.3 2023-01-24T03:29:33.857 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi112.front.sepia.ceph.com 2023-01-24T03:29:33.857 DEBUG:teuthology.orchestra.run.smithi112:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-01-24T03:29:33.928 INFO:teuthology.orchestra.run.smithi112.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-01-24T03:29:33.928 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-24T03:29:33.928 INFO:teuthology.orchestra.run.smithi112.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-24T03:29:33.929 INFO:teuthology.orchestra.run.smithi112.stdout: = reflink=1 2023-01-24T03:29:33.929 INFO:teuthology.orchestra.run.smithi112.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-24T03:29:33.929 INFO:teuthology.orchestra.run.smithi112.stdout: = sunit=0 swidth=0 blks 2023-01-24T03:29:33.929 INFO:teuthology.orchestra.run.smithi112.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-24T03:29:33.929 INFO:teuthology.orchestra.run.smithi112.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-24T03:29:33.930 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-24T03:29:33.930 INFO:teuthology.orchestra.run.smithi112.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-24T03:29:34.300 INFO:teuthology.orchestra.run.smithi112.stdout:Discarding blocks...Done. 2023-01-24T03:29:34.302 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi112.front.sepia.ceph.com -o noatime 2023-01-24T03:29:34.303 DEBUG:teuthology.orchestra.run.smithi112:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-01-24T03:29:34.401 DEBUG:teuthology.orchestra.run.smithi112:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-01-24T03:29:34.429 DEBUG:teuthology.orchestra.run.smithi112:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-24T03:29:34.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:29:34.506+0000 7f71dabac3c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-01-24T03:29:34.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:29:34.511+0000 7f71dabac3c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-01-24T03:29:34.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:29:34.511+0000 7f71dabac3c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-01-24T03:29:34.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:29:34.511+0000 7f71dabac3c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-01-24T03:29:34.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:29:34.511+0000 7f71dabac3c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-01-24T03:29:34.519 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:29:34.516+0000 7f71dabac3c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-01-24T03:29:37.373 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-01-24T03:29:37.440 DEBUG:teuthology.orchestra.run.smithi112:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-24T03:29:37.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:29:37.519+0000 7f2c51bed3c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-01-24T03:29:37.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:29:37.519+0000 7f2c51bed3c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-01-24T03:29:37.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:29:37.519+0000 7f2c51bed3c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-01-24T03:29:37.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:29:37.519+0000 7f2c51bed3c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-01-24T03:29:37.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:29:37.519+0000 7f2c51bed3c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-01-24T03:29:37.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:29:37.520+0000 7f2c51bed3c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-01-24T03:29:40.382 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-01-24T03:29:40.450 DEBUG:teuthology.orchestra.run.smithi112:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-24T03:29:40.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:29:40.532+0000 7f3613b423c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-01-24T03:29:40.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:29:40.533+0000 7f3613b423c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-01-24T03:29:40.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:29:40.533+0000 7f3613b423c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-01-24T03:29:40.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:29:40.533+0000 7f3613b423c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-01-24T03:29:40.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:29:40.533+0000 7f3613b423c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-01-24T03:29:40.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:29:40.533+0000 7f3613b423c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-01-24T03:29:43.408 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-01-24T03:29:43.436 DEBUG:teuthology.orchestra.run.smithi112:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-24T03:29:43.520 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:29:43.518+0000 7f19613f73c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-01-24T03:29:43.520 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:29:43.518+0000 7f19613f73c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-01-24T03:29:43.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:29:43.518+0000 7f19613f73c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-01-24T03:29:43.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:29:43.518+0000 7f19613f73c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-01-24T03:29:43.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:29:43.518+0000 7f19613f73c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-01-24T03:29:43.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:29:43.518+0000 7f19613f73c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-01-24T03:29:46.380 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-01-24T03:29:46.447 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-01-24T03:29:46.475 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-01-24T03:29:46.476 INFO:tasks.ceph:role: osd.4 2023-01-24T03:29:46.476 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi142.front.sepia.ceph.com 2023-01-24T03:29:46.476 DEBUG:teuthology.orchestra.run.smithi142:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-01-24T03:29:46.550 INFO:teuthology.orchestra.run.smithi142.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-01-24T03:29:46.550 INFO:teuthology.orchestra.run.smithi142.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-24T03:29:46.550 INFO:teuthology.orchestra.run.smithi142.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-24T03:29:46.551 INFO:teuthology.orchestra.run.smithi142.stdout: = reflink=1 2023-01-24T03:29:46.551 INFO:teuthology.orchestra.run.smithi142.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-24T03:29:46.551 INFO:teuthology.orchestra.run.smithi142.stdout: = sunit=0 swidth=0 blks 2023-01-24T03:29:46.551 INFO:teuthology.orchestra.run.smithi142.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-24T03:29:46.551 INFO:teuthology.orchestra.run.smithi142.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-24T03:29:46.552 INFO:teuthology.orchestra.run.smithi142.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-24T03:29:46.552 INFO:teuthology.orchestra.run.smithi142.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-24T03:29:46.914 INFO:teuthology.orchestra.run.smithi142.stdout:Discarding blocks...Done. 2023-01-24T03:29:46.915 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi142.front.sepia.ceph.com -o noatime 2023-01-24T03:29:46.916 DEBUG:teuthology.orchestra.run.smithi142:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-01-24T03:29:47.058 DEBUG:teuthology.orchestra.run.smithi142:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-01-24T03:29:47.089 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-01-24T03:29:47.156 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-01-24T03:29:47.157 INFO:tasks.ceph:role: osd.5 2023-01-24T03:29:47.157 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi142.front.sepia.ceph.com 2023-01-24T03:29:47.157 DEBUG:teuthology.orchestra.run.smithi142:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-01-24T03:29:47.230 INFO:teuthology.orchestra.run.smithi142.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-01-24T03:29:47.230 INFO:teuthology.orchestra.run.smithi142.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-24T03:29:47.230 INFO:teuthology.orchestra.run.smithi142.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-24T03:29:47.231 INFO:teuthology.orchestra.run.smithi142.stdout: = reflink=1 2023-01-24T03:29:47.231 INFO:teuthology.orchestra.run.smithi142.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-24T03:29:47.231 INFO:teuthology.orchestra.run.smithi142.stdout: = sunit=0 swidth=0 blks 2023-01-24T03:29:47.231 INFO:teuthology.orchestra.run.smithi142.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-24T03:29:47.231 INFO:teuthology.orchestra.run.smithi142.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-24T03:29:47.232 INFO:teuthology.orchestra.run.smithi142.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-24T03:29:47.232 INFO:teuthology.orchestra.run.smithi142.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-24T03:29:47.581 INFO:teuthology.orchestra.run.smithi142.stdout:Discarding blocks...Done. 2023-01-24T03:29:47.583 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi142.front.sepia.ceph.com -o noatime 2023-01-24T03:29:47.584 DEBUG:teuthology.orchestra.run.smithi142:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-01-24T03:29:47.674 DEBUG:teuthology.orchestra.run.smithi142:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-01-24T03:29:47.703 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-01-24T03:29:47.769 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-01-24T03:29:47.770 INFO:tasks.ceph:role: osd.6 2023-01-24T03:29:47.770 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi142.front.sepia.ceph.com 2023-01-24T03:29:47.770 DEBUG:teuthology.orchestra.run.smithi142:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-01-24T03:29:47.841 INFO:teuthology.orchestra.run.smithi142.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-01-24T03:29:47.842 INFO:teuthology.orchestra.run.smithi142.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-24T03:29:47.842 INFO:teuthology.orchestra.run.smithi142.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-24T03:29:47.842 INFO:teuthology.orchestra.run.smithi142.stdout: = reflink=1 2023-01-24T03:29:47.842 INFO:teuthology.orchestra.run.smithi142.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-24T03:29:47.843 INFO:teuthology.orchestra.run.smithi142.stdout: = sunit=0 swidth=0 blks 2023-01-24T03:29:47.843 INFO:teuthology.orchestra.run.smithi142.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-24T03:29:47.843 INFO:teuthology.orchestra.run.smithi142.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-24T03:29:47.843 INFO:teuthology.orchestra.run.smithi142.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-24T03:29:47.843 INFO:teuthology.orchestra.run.smithi142.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-24T03:29:48.205 INFO:teuthology.orchestra.run.smithi142.stdout:Discarding blocks...Done. 2023-01-24T03:29:48.207 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi142.front.sepia.ceph.com -o noatime 2023-01-24T03:29:48.207 DEBUG:teuthology.orchestra.run.smithi142:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-01-24T03:29:48.310 DEBUG:teuthology.orchestra.run.smithi142:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-01-24T03:29:48.340 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-01-24T03:29:48.405 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-01-24T03:29:48.406 INFO:tasks.ceph:role: osd.7 2023-01-24T03:29:48.406 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi142.front.sepia.ceph.com 2023-01-24T03:29:48.406 DEBUG:teuthology.orchestra.run.smithi142:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-01-24T03:29:48.480 INFO:teuthology.orchestra.run.smithi142.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-01-24T03:29:48.480 INFO:teuthology.orchestra.run.smithi142.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-24T03:29:48.480 INFO:teuthology.orchestra.run.smithi142.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-24T03:29:48.480 INFO:teuthology.orchestra.run.smithi142.stdout: = reflink=1 2023-01-24T03:29:48.481 INFO:teuthology.orchestra.run.smithi142.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-24T03:29:48.481 INFO:teuthology.orchestra.run.smithi142.stdout: = sunit=0 swidth=0 blks 2023-01-24T03:29:48.481 INFO:teuthology.orchestra.run.smithi142.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-24T03:29:48.481 INFO:teuthology.orchestra.run.smithi142.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-24T03:29:48.481 INFO:teuthology.orchestra.run.smithi142.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-24T03:29:48.482 INFO:teuthology.orchestra.run.smithi142.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-24T03:29:48.848 INFO:teuthology.orchestra.run.smithi142.stdout:Discarding blocks...Done. 2023-01-24T03:29:48.850 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi142.front.sepia.ceph.com -o noatime 2023-01-24T03:29:48.851 DEBUG:teuthology.orchestra.run.smithi142:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-01-24T03:29:48.971 DEBUG:teuthology.orchestra.run.smithi142:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-01-24T03:29:49.001 DEBUG:teuthology.orchestra.run.smithi142:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-24T03:29:49.082 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:29:49.079+0000 7f2ed69d93c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-01-24T03:29:49.085 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:29:49.083+0000 7f2ed69d93c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-01-24T03:29:49.085 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:29:49.083+0000 7f2ed69d93c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-01-24T03:29:49.086 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:29:49.083+0000 7f2ed69d93c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-01-24T03:29:49.086 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:29:49.083+0000 7f2ed69d93c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-01-24T03:29:49.092 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:29:49.090+0000 7f2ed69d93c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-01-24T03:29:51.944 DEBUG:teuthology.orchestra.run.smithi142:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-01-24T03:29:52.011 DEBUG:teuthology.orchestra.run.smithi142:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-24T03:29:52.095 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:29:52.092+0000 7f13216843c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-01-24T03:29:52.095 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:29:52.092+0000 7f13216843c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-01-24T03:29:52.096 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:29:52.093+0000 7f13216843c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-01-24T03:29:52.096 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:29:52.093+0000 7f13216843c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-01-24T03:29:52.096 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:29:52.093+0000 7f13216843c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-01-24T03:29:52.096 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:29:52.093+0000 7f13216843c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-01-24T03:29:54.951 DEBUG:teuthology.orchestra.run.smithi142:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-01-24T03:29:55.019 DEBUG:teuthology.orchestra.run.smithi142:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-24T03:29:55.103 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:29:55.100+0000 7f650256b3c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-01-24T03:29:55.103 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:29:55.100+0000 7f650256b3c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-01-24T03:29:55.103 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:29:55.100+0000 7f650256b3c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-01-24T03:29:55.104 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:29:55.100+0000 7f650256b3c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-01-24T03:29:55.104 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:29:55.100+0000 7f650256b3c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-01-24T03:29:55.104 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:29:55.101+0000 7f650256b3c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-01-24T03:29:57.969 DEBUG:teuthology.orchestra.run.smithi142:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-01-24T03:29:58.038 DEBUG:teuthology.orchestra.run.smithi142:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-24T03:29:58.124 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:29:58.121+0000 7f95837b63c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-01-24T03:29:58.124 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:29:58.122+0000 7f95837b63c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-01-24T03:29:58.125 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:29:58.122+0000 7f95837b63c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-01-24T03:29:58.125 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:29:58.122+0000 7f95837b63c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-01-24T03:29:58.125 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:29:58.122+0000 7f95837b63c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-01-24T03:29:58.126 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:29:58.122+0000 7f95837b63c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-01-24T03:29:58.720 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:29:58.723 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:30:00.981 DEBUG:teuthology.orchestra.run.smithi142:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-01-24T03:30:01.010 INFO:tasks.ceph:Reading keys from all nodes... 2023-01-24T03:30:01.011 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-01-24T03:30:01.011 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-01-24T03:30:01.039 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-01-24T03:30:01.039 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-01-24T03:30:01.106 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-01-24T03:30:01.107 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-01-24T03:30:01.171 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-01-24T03:30:01.171 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2023-01-24T03:30:01.235 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-01-24T03:30:01.235 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-01-24T03:30:01.298 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-01-24T03:30:01.299 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-01-24T03:30:01.362 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-01-24T03:30:01.362 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-01-24T03:30:01.425 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-01-24T03:30:01.425 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-01-24T03:30:01.488 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2023-01-24T03:30:01.489 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-01-24T03:30:01.519 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2023-01-24T03:30:01.520 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-01-24T03:30:01.586 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2023-01-24T03:30:01.586 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-01-24T03:30:01.652 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2023-01-24T03:30:01.653 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2023-01-24T03:30:01.718 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2023-01-24T03:30:01.718 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-01-24T03:30:01.782 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2023-01-24T03:30:01.783 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-01-24T03:30:01.847 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2023-01-24T03:30:01.848 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-01-24T03:30:01.911 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2023-01-24T03:30:01.912 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-01-24T03:30:01.975 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-01-24T03:30:01.975 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-01-24T03:30:01.991 INFO:tasks.ceph:Adding keys to all mons... 2023-01-24T03:30:01.991 DEBUG:teuthology.orchestra.run.smithi112:> sudo tee -a /etc/ceph/ceph.keyring 2023-01-24T03:30:02.033 DEBUG:teuthology.orchestra.run.smithi142:> sudo tee -a /etc/ceph/ceph.keyring 2023-01-24T03:30:02.053 INFO:teuthology.orchestra.run.smithi112.stdout:[mgr.y] 2023-01-24T03:30:02.053 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCbUM9j+y8PCxAATaiAsbV5/9uoDANiMNO7hw== 2023-01-24T03:30:02.054 INFO:teuthology.orchestra.run.smithi112.stdout:[mds.a] 2023-01-24T03:30:02.054 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCbUM9jn5hRERAA8x122RP1JMcgSSlThmO4zA== 2023-01-24T03:30:02.054 INFO:teuthology.orchestra.run.smithi112.stdout:[mds.c] 2023-01-24T03:30:02.054 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCbUM9jYAQFGBAA0EDnb6NBl21Cbu1PZ45zig== 2023-01-24T03:30:02.054 INFO:teuthology.orchestra.run.smithi112.stdout:[mds.e] 2023-01-24T03:30:02.055 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCbUM9jO/mfHhAA7c5aCMOq1yrt9ptucFGE/A== 2023-01-24T03:30:02.055 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.0] 2023-01-24T03:30:02.055 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCeUM9jZpBLHhAA5UAhw+3fPr6tPvsPmJMaqg== 2023-01-24T03:30:02.055 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.1] 2023-01-24T03:30:02.055 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQChUM9jdtcQHxAAEdn8Y605oQmpDDt4GQq1mQ== 2023-01-24T03:30:02.056 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.2] 2023-01-24T03:30:02.056 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCkUM9j8LjMHxAA+hHMiwt+GyYP8EJJ7mckvQ== 2023-01-24T03:30:02.056 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.3] 2023-01-24T03:30:02.056 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCnUM9jlurrHhAALeagY6LGc4nqCI2rbslj9A== 2023-01-24T03:30:02.056 INFO:teuthology.orchestra.run.smithi112.stdout:[mgr.x] 2023-01-24T03:30:02.057 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCbUM9jH3pjDhAAJXzwbYjc25/Bx/DbIpsLJA== 2023-01-24T03:30:02.057 INFO:teuthology.orchestra.run.smithi112.stdout:[mds.b] 2023-01-24T03:30:02.057 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCbUM9jyTvtIhAAe8XsiIjTiCAiBKaLacN7FQ== 2023-01-24T03:30:02.057 INFO:teuthology.orchestra.run.smithi112.stdout:[mds.d] 2023-01-24T03:30:02.057 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCbUM9jUDmPKRAAlIAxOnZdmbfjauxMCLPOmg== 2023-01-24T03:30:02.058 INFO:teuthology.orchestra.run.smithi112.stdout:[mds.f] 2023-01-24T03:30:02.058 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCbUM9jHttGMBAANKfKGjvxx34hlOzy5prnWw== 2023-01-24T03:30:02.058 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.4] 2023-01-24T03:30:02.058 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCtUM9jvg7cBBAAL7w+SmeIVmujZbY3yz7uTA== 2023-01-24T03:30:02.058 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.5] 2023-01-24T03:30:02.059 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCwUM9jxKKWBRAAxqSYoKmo+53s6oHIqQEzcg== 2023-01-24T03:30:02.059 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.6] 2023-01-24T03:30:02.059 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCzUM9jQycPBhAAgjkrgVwISqHLDyytYLs5+g== 2023-01-24T03:30:02.059 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.7] 2023-01-24T03:30:02.059 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQC2UM9jebpQBxAA0aADYz50Hv5VbhSBIcP3QQ== 2023-01-24T03:30:02.060 INFO:teuthology.orchestra.run.smithi112.stdout:[client.0] 2023-01-24T03:30:02.060 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCbUM9jNncGNBAAy+TdDi6Kq/g/lMfp1Ps5OQ== 2023-01-24T03:30:02.061 INFO:teuthology.orchestra.run.smithi142.stdout:[mgr.y] 2023-01-24T03:30:02.061 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQCbUM9j+y8PCxAATaiAsbV5/9uoDANiMNO7hw== 2023-01-24T03:30:02.062 INFO:teuthology.orchestra.run.smithi142.stdout:[mds.a] 2023-01-24T03:30:02.062 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQCbUM9jn5hRERAA8x122RP1JMcgSSlThmO4zA== 2023-01-24T03:30:02.062 INFO:teuthology.orchestra.run.smithi142.stdout:[mds.c] 2023-01-24T03:30:02.062 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQCbUM9jYAQFGBAA0EDnb6NBl21Cbu1PZ45zig== 2023-01-24T03:30:02.062 INFO:teuthology.orchestra.run.smithi142.stdout:[mds.e] 2023-01-24T03:30:02.063 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQCbUM9jO/mfHhAA7c5aCMOq1yrt9ptucFGE/A== 2023-01-24T03:30:02.063 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.0] 2023-01-24T03:30:02.063 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQCeUM9jZpBLHhAA5UAhw+3fPr6tPvsPmJMaqg== 2023-01-24T03:30:02.063 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.1] 2023-01-24T03:30:02.063 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQChUM9jdtcQHxAAEdn8Y605oQmpDDt4GQq1mQ== 2023-01-24T03:30:02.064 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.2] 2023-01-24T03:30:02.064 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQCkUM9j8LjMHxAA+hHMiwt+GyYP8EJJ7mckvQ== 2023-01-24T03:30:02.064 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.3] 2023-01-24T03:30:02.064 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQCnUM9jlurrHhAALeagY6LGc4nqCI2rbslj9A== 2023-01-24T03:30:02.065 INFO:teuthology.orchestra.run.smithi142.stdout:[mgr.x] 2023-01-24T03:30:02.065 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQCbUM9jH3pjDhAAJXzwbYjc25/Bx/DbIpsLJA== 2023-01-24T03:30:02.065 INFO:teuthology.orchestra.run.smithi142.stdout:[mds.b] 2023-01-24T03:30:02.065 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQCbUM9jyTvtIhAAe8XsiIjTiCAiBKaLacN7FQ== 2023-01-24T03:30:02.065 INFO:teuthology.orchestra.run.smithi142.stdout:[mds.d] 2023-01-24T03:30:02.065 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQCbUM9jUDmPKRAAlIAxOnZdmbfjauxMCLPOmg== 2023-01-24T03:30:02.066 INFO:teuthology.orchestra.run.smithi142.stdout:[mds.f] 2023-01-24T03:30:02.066 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQCbUM9jHttGMBAANKfKGjvxx34hlOzy5prnWw== 2023-01-24T03:30:02.066 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.4] 2023-01-24T03:30:02.066 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQCtUM9jvg7cBBAAL7w+SmeIVmujZbY3yz7uTA== 2023-01-24T03:30:02.066 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.5] 2023-01-24T03:30:02.067 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQCwUM9jxKKWBRAAxqSYoKmo+53s6oHIqQEzcg== 2023-01-24T03:30:02.067 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.6] 2023-01-24T03:30:02.067 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQCzUM9jQycPBhAAgjkrgVwISqHLDyytYLs5+g== 2023-01-24T03:30:02.067 INFO:teuthology.orchestra.run.smithi142.stdout:[osd.7] 2023-01-24T03:30:02.067 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQC2UM9jebpQBxAA0aADYz50Hv5VbhSBIcP3QQ== 2023-01-24T03:30:02.068 INFO:teuthology.orchestra.run.smithi142.stdout:[client.0] 2023-01-24T03:30:02.068 INFO:teuthology.orchestra.run.smithi142.stdout: key = AQCbUM9jNncGNBAAy+TdDi6Kq/g/lMfp1Ps5OQ== 2023-01-24T03:30:02.068 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-01-24T03:30:02.106 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-01-24T03:30:02.152 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-01-24T03:30:02.154 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-01-24T03:30:02.203 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-01-24T03:30:02.206 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-01-24T03:30:02.255 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-01-24T03:30:02.257 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-01-24T03:30:02.306 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T03:30:02.309 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T03:30:02.355 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T03:30:02.358 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T03:30:02.404 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T03:30:02.406 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T03:30:02.452 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T03:30:02.455 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T03:30:02.500 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-01-24T03:30:02.503 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-01-24T03:30:02.548 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-01-24T03:30:02.551 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-01-24T03:30:02.600 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-01-24T03:30:02.602 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-01-24T03:30:02.651 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-01-24T03:30:02.654 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-01-24T03:30:02.702 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T03:30:02.704 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T03:30:02.751 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T03:30:02.753 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T03:30:02.801 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T03:30:02.839 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T03:30:02.886 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T03:30:02.888 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T03:30:02.934 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-01-24T03:30:02.936 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-01-24T03:30:02.985 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-01-24T03:30:02.986 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-01-24T03:30:03.009 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-01-24T03:30:03.101 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-01-24T03:30:03.124 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-01-24T03:30:03.147 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-01-24T03:30:03.238 DEBUG:teuthology.orchestra.run.smithi142:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-01-24T03:30:03.261 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-01-24T03:30:03.324 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-01-24T03:30:03.414 DEBUG:teuthology.orchestra.run.smithi142:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-01-24T03:30:03.438 DEBUG:teuthology.orchestra.run.smithi112:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-01-24T03:30:03.441 DEBUG:teuthology.orchestra.run.smithi142:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-01-24T03:30:03.492 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-01-24T03:30:03.492 INFO:tasks.ceph.mon.a:Restarting daemon 2023-01-24T03:30:03.493 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-01-24T03:30:03.496 INFO:tasks.ceph.mon.a:Started 2023-01-24T03:30:03.496 INFO:tasks.ceph.mon.b:Restarting daemon 2023-01-24T03:30:03.496 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-01-24T03:30:03.534 INFO:tasks.ceph.mon.b:Started 2023-01-24T03:30:03.534 INFO:tasks.ceph.mon.c:Restarting daemon 2023-01-24T03:30:03.535 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-01-24T03:30:03.536 INFO:tasks.ceph.mon.c:Started 2023-01-24T03:30:03.537 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-01-24T03:30:03.537 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-01-24T03:30:03.538 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-01-24T03:30:03.540 INFO:tasks.ceph.mgr.y:Started 2023-01-24T03:30:03.540 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-01-24T03:30:03.540 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-01-24T03:30:03.542 INFO:tasks.ceph.mgr.x:Started 2023-01-24T03:30:03.543 INFO:tasks.ceph:Setting crush tunables to default 2023-01-24T03:30:03.543 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster ceph osd crush tunables default 2023-01-24T03:30:03.657 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.654+0000 7f8448f5f700 1 Processor -- start 2023-01-24T03:30:03.657 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.654+0000 7f8448f5f700 1 -- start start 2023-01-24T03:30:03.659 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.655+0000 7f8448f5f700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f844406a9f0 0x7f84440c1540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:03.659 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.655+0000 7f8448f5f700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f84440c4510 0x7f8444067e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:03.660 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.655+0000 7f8448f5f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8444068360 0x7f844406de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:03.660 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.655+0000 7f8448f5f700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84441541c0 con 0x7f844406a9f0 2023-01-24T03:30:03.660 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.655+0000 7f8448f5f700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f844406adc0 con 0x7f84440c4510 2023-01-24T03:30:03.660 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.655+0000 7f8448f5f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8444153360 con 0x7f8444068360 2023-01-24T03:30:03.660 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.655+0000 7f844259c700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f844406a9f0 0x7f84440c1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:03.661 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.655+0000 7f844259c700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f844406a9f0 0x7f84440c1540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:49416/0 (socket says 172.21.15.112:49416) 2023-01-24T03:30:03.661 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.655+0000 7f844259c700 1 -- 172.21.15.112:0/1210404887 learned_addr learned my addr 172.21.15.112:0/1210404887 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:03.661 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.655+0000 7f844259c700 1 -- 172.21.15.112:0/1210404887 >> v2:172.21.15.112:3300/0 conn(0x7f844406a9f0 msgr2=0x7f84440c1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-01-24T03:30:03.661 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.655+0000 7f844259c700 1 -- 172.21.15.112:0/1210404887 >> v2:172.21.15.112:3300/0 conn(0x7f844406a9f0 msgr2=0x7f84440c1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-01-24T03:30:03.662 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.655+0000 7f844259c700 1 --2- 172.21.15.112:0/1210404887 >> v2:172.21.15.112:3300/0 conn(0x7f844406a9f0 0x7f84440c1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-01-24T03:30:03.662 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.655+0000 7f844259c700 1 --2- 172.21.15.112:0/1210404887 >> v2:172.21.15.112:3300/0 conn(0x7f844406a9f0 0x7f84440c1540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-01-24T03:30:03.662 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.655+0000 7f8442d9d700 1 --2- 172.21.15.112:0/1210404887 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8444068360 0x7f844406de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:03.662 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.655+0000 7f8441d9b700 1 --2- 172.21.15.112:0/1210404887 >> v2:172.21.15.142:3300/0 conn(0x7f84440c4510 0x7f8444067e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:03.662 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.656+0000 7f8441d9b700 1 -- 172.21.15.112:0/1210404887 >> v2:172.21.15.142:3300/0 conn(0x7f84440c4510 msgr2=0x7f8444067e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-01-24T03:30:03.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.656+0000 7f8442d9d700 1 -- 172.21.15.112:0/1210404887 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8444068360 msgr2=0x7f844406de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-01-24T03:30:03.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.656+0000 7f8441d9b700 1 -- 172.21.15.112:0/1210404887 >> v2:172.21.15.142:3300/0 conn(0x7f84440c4510 msgr2=0x7f8444067e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-01-24T03:30:03.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.656+0000 7f8442d9d700 1 -- 172.21.15.112:0/1210404887 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8444068360 msgr2=0x7f844406de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-01-24T03:30:03.664 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.656+0000 7f8441d9b700 1 --2- 172.21.15.112:0/1210404887 >> v2:172.21.15.142:3300/0 conn(0x7f84440c4510 0x7f8444067e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-01-24T03:30:03.664 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.656+0000 7f8442d9d700 1 --2- 172.21.15.112:0/1210404887 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8444068360 0x7f844406de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-01-24T03:30:03.664 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.656+0000 7f8441d9b700 1 --2- 172.21.15.112:0/1210404887 >> v2:172.21.15.142:3300/0 conn(0x7f84440c4510 0x7f8444067e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-01-24T03:30:03.664 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.656+0000 7f8442d9d700 1 --2- 172.21.15.112:0/1210404887 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8444068360 0x7f844406de50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-01-24T03:30:03.866 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.856+0000 7f844259c700 1 --2- 172.21.15.112:0/1210404887 >> v2:172.21.15.112:3300/0 conn(0x7f844406a9f0 0x7f84440c1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:03.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.856+0000 7f844259c700 1 -- 172.21.15.112:0/1210404887 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8444068360 msgr2=0x7f844406de50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-01-24T03:30:03.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.856+0000 7f844259c700 1 --2- 172.21.15.112:0/1210404887 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8444068360 0x7f844406de50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:03.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.856+0000 7f844259c700 1 -- 172.21.15.112:0/1210404887 >> v2:172.21.15.142:3300/0 conn(0x7f84440c4510 msgr2=0x7f8444067e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-01-24T03:30:03.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.856+0000 7f844259c700 1 --2- 172.21.15.112:0/1210404887 >> v2:172.21.15.142:3300/0 conn(0x7f84440c4510 0x7f8444067e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:03.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.856+0000 7f844259c700 1 -- 172.21.15.112:0/1210404887 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8444153640 con 0x7f844406a9f0 2023-01-24T03:30:03.869 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.856+0000 7f844259c700 1 --2- 172.21.15.112:0/1210404887 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f844406a9f0 0x7f84440c1540 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f844406d7a0 tx=0x7f8434008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a47e621938bddc25 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:03.869 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.856+0000 7f844159a700 1 -- 172.21.15.112:0/1210404887 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8434011590 con 0x7f844406a9f0 2023-01-24T03:30:03.870 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.856+0000 7f844159a700 1 -- 172.21.15.112:0/1210404887 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84340116f0 con 0x7f844406a9f0 2023-01-24T03:30:03.870 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.856+0000 7f844159a700 1 -- 172.21.15.112:0/1210404887 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8434011ab0 con 0x7f844406a9f0 2023-01-24T03:30:03.870 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.857+0000 7f8448f5f700 1 -- 172.21.15.112:0/1210404887 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f844406a9f0 msgr2=0x7f84440c1540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:03.871 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.857+0000 7f8448f5f700 1 --2- 172.21.15.112:0/1210404887 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f844406a9f0 0x7f84440c1540 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f844406d7a0 tx=0x7f8434008040 comp rx=0 tx=0).stop 2023-01-24T03:30:03.871 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.857+0000 7f8448f5f700 1 -- 172.21.15.112:0/1210404887 shutdown_connections 2023-01-24T03:30:03.871 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.857+0000 7f8448f5f700 1 --2- 172.21.15.112:0/1210404887 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8444068360 0x7f844406de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:03.872 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.857+0000 7f8448f5f700 1 --2- 172.21.15.112:0/1210404887 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f844406a9f0 0x7f84440c1540 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:03.872 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.857+0000 7f8448f5f700 1 --2- 172.21.15.112:0/1210404887 >> v2:172.21.15.142:3300/0 conn(0x7f84440c4510 0x7f8444067e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:03.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.857+0000 7f8448f5f700 1 -- 172.21.15.112:0/1210404887 >> 172.21.15.112:0/1210404887 conn(0x7f844406b550 msgr2=0x7f844406b950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:03.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.857+0000 7f8448f5f700 1 -- 172.21.15.112:0/1210404887 shutdown_connections 2023-01-24T03:30:03.874 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.857+0000 7f8448f5f700 1 -- 172.21.15.112:0/1210404887 wait complete. 2023-01-24T03:30:03.874 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.858+0000 7f8448f5f700 1 Processor -- start 2023-01-24T03:30:03.874 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.858+0000 7f8448f5f700 1 -- start start 2023-01-24T03:30:03.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.859+0000 7f8448f5f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8444068360 0x7f84440c5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:03.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.859+0000 7f8448f5f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f844406a9f0 0x7f84441679f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:03.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.859+0000 7f8448f5f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f84440c4510 0x7f8444166900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:03.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.859+0000 7f8448f5f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84440c6ba0 con 0x7f8444068360 2023-01-24T03:30:03.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.859+0000 7f8448f5f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84441675d0 con 0x7f844406a9f0 2023-01-24T03:30:03.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.859+0000 7f8448f5f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84440c6200 con 0x7f84440c4510 2023-01-24T03:30:03.877 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.859+0000 7f844259c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8444068360 0x7f84440c5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:03.877 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.859+0000 7f844259c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8444068360 0x7f84440c5cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:49442/0 (socket says 172.21.15.112:49442) 2023-01-24T03:30:03.877 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.859+0000 7f844259c700 1 -- 172.21.15.112:0/250970276 learned_addr learned my addr 172.21.15.112:0/250970276 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:03.878 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.859+0000 7f8442d9d700 1 --2- 172.21.15.112:0/250970276 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f84440c4510 0x7f8444166900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:03.878 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.859+0000 7f8441d9b700 1 --2- 172.21.15.112:0/250970276 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f844406a9f0 0x7f84441679f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:03.879 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.859+0000 7f844259c700 1 -- 172.21.15.112:0/250970276 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f84440c4510 msgr2=0x7f8444166900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:03.879 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.859+0000 7f844259c700 1 --2- 172.21.15.112:0/250970276 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f84440c4510 0x7f8444166900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:03.880 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.859+0000 7f844259c700 1 -- 172.21.15.112:0/250970276 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f844406a9f0 msgr2=0x7f84441679f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:03.880 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.859+0000 7f844259c700 1 --2- 172.21.15.112:0/250970276 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f844406a9f0 0x7f84441679f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:03.880 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.859+0000 7f844259c700 1 -- 172.21.15.112:0/250970276 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f843400e040 con 0x7f8444068360 2023-01-24T03:30:03.881 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.859+0000 7f8442d9d700 1 --2- 172.21.15.112:0/250970276 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f84440c4510 0x7f8444166900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:30:03.881 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.859+0000 7f8441d9b700 1 --2- 172.21.15.112:0/250970276 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f844406a9f0 0x7f84441679f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:30:03.881 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.859+0000 7f844259c700 1 --2- 172.21.15.112:0/250970276 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8444068360 0x7f84440c5cc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8434006e70 tx=0x7f8434019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:03.882 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.859+0000 7f84327fc700 1 -- 172.21.15.112:0/250970276 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84340129a0 con 0x7f8444068360 2023-01-24T03:30:03.882 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.860+0000 7f84327fc700 1 -- 172.21.15.112:0/250970276 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8434012b20 con 0x7f8444068360 2023-01-24T03:30:03.882 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.860+0000 7f8448f5f700 1 -- 172.21.15.112:0/250970276 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84440c6400 con 0x7f8444068360 2023-01-24T03:30:03.883 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.860+0000 7f84327fc700 1 -- 172.21.15.112:0/250970276 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843401b420 con 0x7f8444068360 2023-01-24T03:30:03.883 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.860+0000 7f8448f5f700 1 -- 172.21.15.112:0/250970276 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f844414c2b0 con 0x7f8444068360 2023-01-24T03:30:03.884 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.860+0000 7f84327fc700 1 -- 172.21.15.112:0/250970276 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7f8434024030 con 0x7f8444068360 2023-01-24T03:30:03.884 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.860+0000 7f84327fc700 1 -- 172.21.15.112:0/250970276 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f8434027790 con 0x7f8444068360 2023-01-24T03:30:03.884 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.860+0000 7f8448f5f700 1 -- 172.21.15.112:0/250970276 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f844406a670 con 0x7f8444068360 2023-01-24T03:30:03.885 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.862+0000 7f84327fc700 1 -- 172.21.15.112:0/250970276 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70685 (secure 0 0 0) 0x7f8434022070 con 0x7f8444068360 2023-01-24T03:30:03.939 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.936+0000 7f8448f5f700 1 -- 172.21.15.112:0/250970276 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f844405e610 con 0x7f8444068360 2023-01-24T03:30:03.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.944+0000 7f84327fc700 1 -- 172.21.15.112:0/250970276 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f843401bc30 con 0x7f8444068360 2023-01-24T03:30:03.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.945+0000 7f8448f5f700 1 -- 172.21.15.112:0/250970276 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8444068360 msgr2=0x7f84440c5cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:03.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.945+0000 7f8448f5f700 1 --2- 172.21.15.112:0/250970276 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8444068360 0x7f84440c5cc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8434006e70 tx=0x7f8434019000 comp rx=0 tx=0).stop 2023-01-24T03:30:03.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.945+0000 7f8448f5f700 1 -- 172.21.15.112:0/250970276 shutdown_connections 2023-01-24T03:30:03.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.945+0000 7f8448f5f700 1 --2- 172.21.15.112:0/250970276 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8444068360 0x7f84440c5cc0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:03.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.945+0000 7f8448f5f700 1 --2- 172.21.15.112:0/250970276 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f84440c4510 0x7f8444166900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:03.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.945+0000 7f8448f5f700 1 --2- 172.21.15.112:0/250970276 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f844406a9f0 0x7f84441679f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:03.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.945+0000 7f8448f5f700 1 -- 172.21.15.112:0/250970276 >> 172.21.15.112:0/250970276 conn(0x7f844406b550 msgr2=0x7f8444066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:03.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.945+0000 7f8448f5f700 1 -- 172.21.15.112:0/250970276 shutdown_connections 2023-01-24T03:30:03.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:03.945+0000 7f8448f5f700 1 -- 172.21.15.112:0/250970276 wait complete. 2023-01-24T03:30:03.950 INFO:teuthology.orchestra.run.smithi112.stderr:adjusted tunables profile to default 2023-01-24T03:30:03.960 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-01-24T03:30:03.960 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-01-24T03:30:03.961 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-01-24T03:30:03.984 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-01-24T03:30:03.985 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-01-24T03:30:04.048 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-01-24T03:30:04.048 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-01-24T03:30:04.113 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-01-24T03:30:04.113 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-01-24T03:30:04.175 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2023-01-24T03:30:04.176 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-01-24T03:30:04.200 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2023-01-24T03:30:04.200 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-01-24T03:30:04.263 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2023-01-24T03:30:04.264 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-01-24T03:30:04.327 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2023-01-24T03:30:04.328 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-01-24T03:30:04.390 DEBUG:teuthology.orchestra.run.smithi142:> sudo ceph --cluster ceph osd new fe6ad231-a39a-42eb-9a3a-49d008dd35e4 0 2023-01-24T03:30:04.546 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.544+0000 7f5ba2038700 1 Processor -- start 2023-01-24T03:30:04.547 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.544+0000 7f5ba2038700 1 -- start start 2023-01-24T03:30:04.549 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.544+0000 7f5ba2038700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b9c067fe0 0x7f5b9c06a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:04.550 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.544+0000 7f5ba2038700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f5b9c0683b0 0x7f5b9c06ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:04.550 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.544+0000 7f5ba2038700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b9c05dea0 con 0x7f5b9c06f280 2023-01-24T03:30:04.550 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.544+0000 7f5ba2038700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b9c155720 con 0x7f5b9c0683b0 2023-01-24T03:30:04.550 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.544+0000 7f5ba2038700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9c06ad50 con 0x7f5b9c067fe0 2023-01-24T03:30:04.551 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.544+0000 7f5b9b7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b9c067fe0 0x7f5b9c06a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:04.552 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.544+0000 7f5b9affd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f5b9c0683b0 0x7f5b9c06ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:04.552 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.545+0000 7f5b9b7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b9c067fe0 0x7f5b9c06a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:41706/0 (socket says 172.21.15.142:41706) 2023-01-24T03:30:04.552 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.545+0000 7f5b9b7fe700 1 -- 172.21.15.142:0/639151010 learned_addr learned my addr 172.21.15.142:0/639151010 (peer_addr_for_me v2:172.21.15.142:0/0) 2023-01-24T03:30:04.552 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.545+0000 7f5b9affd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f5b9c0683b0 0x7f5b9c06ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48380/0 (socket says 172.21.15.142:48380) 2023-01-24T03:30:04.553 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.545+0000 7f5b9b7fe700 1 -- 172.21.15.142:0/639151010 >> v2:172.21.15.142:3300/0 conn(0x7f5b9c0683b0 msgr2=0x7f5b9c06ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:04.553 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.545+0000 7f5b9b7fe700 1 --2- 172.21.15.142:0/639151010 >> v2:172.21.15.142:3300/0 conn(0x7f5b9c0683b0 0x7f5b9c06ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:04.553 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.545+0000 7f5b9b7fe700 1 -- 172.21.15.142:0/639151010 >> v1:172.21.15.142:6789/0 conn(0x7f5b9c06f280 legacy=0x7f5b9c1521d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:04.553 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.545+0000 7f5b9b7fe700 1 -- 172.21.15.142:0/639151010 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b9c155970 con 0x7f5b9c067fe0 2023-01-24T03:30:04.554 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.545+0000 7f5b9a7fc700 1 -- 172.21.15.142:0/639151010 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3479423875 0 0) 0x7f5b9c05dea0 con 0x7f5b9c06f280 2023-01-24T03:30:04.554 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.545+0000 7f5b9affd700 1 --2- 172.21.15.142:0/639151010 >> v2:172.21.15.142:3300/0 conn(0x7f5b9c0683b0 0x7f5b9c06ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:30:04.554 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.545+0000 7f5b9b7fe700 1 --2- 172.21.15.142:0/639151010 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b9c067fe0 0x7f5b9c06a730 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c006fd0 tx=0x7f5b8c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20b4c783f65a9b5b server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:04.554 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.545+0000 7f5b9a7fc700 1 -- 172.21.15.142:0/639151010 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c011630 con 0x7f5b9c067fe0 2023-01-24T03:30:04.554 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.545+0000 7f5b9a7fc700 1 -- 172.21.15.142:0/639151010 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b8c0117d0 con 0x7f5b9c067fe0 2023-01-24T03:30:04.555 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.545+0000 7f5b9a7fc700 1 -- 172.21.15.142:0/639151010 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c011b90 con 0x7f5b9c067fe0 2023-01-24T03:30:04.555 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.546+0000 7f5ba2038700 1 -- 172.21.15.142:0/639151010 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b9c067fe0 msgr2=0x7f5b9c06a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:04.555 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.546+0000 7f5ba2038700 1 --2- 172.21.15.142:0/639151010 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b9c067fe0 0x7f5b9c06a730 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c006fd0 tx=0x7f5b8c0090f0 comp rx=0 tx=0).stop 2023-01-24T03:30:04.555 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.546+0000 7f5ba2038700 1 -- 172.21.15.142:0/639151010 shutdown_connections 2023-01-24T03:30:04.555 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.546+0000 7f5ba2038700 1 --2- 172.21.15.142:0/639151010 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b9c067fe0 0x7f5b9c06a730 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:04.556 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.546+0000 7f5ba2038700 1 --2- 172.21.15.142:0/639151010 >> v2:172.21.15.142:3300/0 conn(0x7f5b9c0683b0 0x7f5b9c06ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:04.556 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.546+0000 7f5ba2038700 1 -- 172.21.15.142:0/639151010 >> 172.21.15.142:0/639151010 conn(0x7f5b9c06b550 msgr2=0x7f5b9c06b950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:04.556 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.546+0000 7f5ba2038700 1 -- 172.21.15.142:0/639151010 shutdown_connections 2023-01-24T03:30:04.556 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.546+0000 7f5ba2038700 1 -- 172.21.15.142:0/639151010 wait complete. 2023-01-24T03:30:04.556 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.547+0000 7f5ba2038700 1 Processor -- start 2023-01-24T03:30:04.557 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.547+0000 7f5ba2038700 1 -- start start 2023-01-24T03:30:04.559 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.548+0000 7f5ba2038700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b9c067fe0 0x7f5b9c0c35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:04.560 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.548+0000 7f5ba2038700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b9c0683b0 0x7f5b9c0be580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:04.560 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.548+0000 7f5ba2038700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5b9c06f280 0x7f5b9c0beac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:04.560 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.548+0000 7f5ba2038700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9c144fd0 con 0x7f5b9c06f280 2023-01-24T03:30:04.560 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.548+0000 7f5ba2038700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9c169660 con 0x7f5b9c0683b0 2023-01-24T03:30:04.561 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.548+0000 7f5ba2038700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9c168a50 con 0x7f5b9c067fe0 2023-01-24T03:30:04.561 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.548+0000 7f5b9b7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b9c067fe0 0x7f5b9c0c35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:04.561 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.548+0000 7f5b9affd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b9c0683b0 0x7f5b9c0be580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:04.561 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.548+0000 7f5b9b7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b9c067fe0 0x7f5b9c0c35a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:41712/0 (socket says 172.21.15.142:41712) 2023-01-24T03:30:04.561 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.548+0000 7f5b9b7fe700 1 -- 172.21.15.142:0/2189277369 learned_addr learned my addr 172.21.15.142:0/2189277369 (peer_addr_for_me v2:172.21.15.142:0/0) 2023-01-24T03:30:04.562 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.548+0000 7f5b9bfff700 1 --2- 172.21.15.142:0/2189277369 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5b9c06f280 0x7f5b9c0beac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:04.562 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.548+0000 7f5b9affd700 1 -- 172.21.15.142:0/2189277369 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b9c067fe0 msgr2=0x7f5b9c0c35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:04.563 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.548+0000 7f5b9affd700 1 --2- 172.21.15.142:0/2189277369 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b9c067fe0 0x7f5b9c0c35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:04.563 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.548+0000 7f5b9affd700 1 -- 172.21.15.142:0/2189277369 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5b9c06f280 msgr2=0x7f5b9c0beac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:04.563 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.548+0000 7f5b9affd700 1 --2- 172.21.15.142:0/2189277369 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5b9c06f280 0x7f5b9c0beac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:04.563 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.548+0000 7f5b9affd700 1 -- 172.21.15.142:0/2189277369 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b900092a0 con 0x7f5b9c0683b0 2023-01-24T03:30:04.563 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.548+0000 7f5b9b7fe700 1 --2- 172.21.15.142:0/2189277369 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b9c067fe0 0x7f5b9c0c35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:30:04.564 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.548+0000 7f5b9affd700 1 --2- 172.21.15.142:0/2189277369 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b9c0683b0 0x7f5b9c0be580 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5b9000e980 tx=0x7f5b90009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:04.564 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.548+0000 7f5b837fe700 1 -- 172.21.15.142:0/2189277369 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b9000ee70 con 0x7f5b9c0683b0 2023-01-24T03:30:04.564 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.548+0000 7f5b837fe700 1 -- 172.21.15.142:0/2189277369 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b9000c880 con 0x7f5b9c0683b0 2023-01-24T03:30:04.564 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.548+0000 7f5ba2038700 1 -- 172.21.15.142:0/2189277369 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b8c009370 con 0x7f5b9c0683b0 2023-01-24T03:30:04.564 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.548+0000 7f5b837fe700 1 -- 172.21.15.142:0/2189277369 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b9000cc40 con 0x7f5b9c0683b0 2023-01-24T03:30:04.565 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.549+0000 7f5ba2038700 1 -- 172.21.15.142:0/2189277369 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b9c169010 con 0x7f5b9c0683b0 2023-01-24T03:30:04.565 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.549+0000 7f5b837fe700 1 -- 172.21.15.142:0/2189277369 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7f5b90019030 con 0x7f5b9c0683b0 2023-01-24T03:30:04.565 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.549+0000 7f5b837fe700 1 -- 172.21.15.142:0/2189277369 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (secure 0 0 0) 0x7f5b90006540 con 0x7f5b9c0683b0 2023-01-24T03:30:04.565 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.549+0000 7f5ba2038700 1 -- 172.21.15.142:0/2189277369 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b9c1697a0 con 0x7f5b9c0683b0 2023-01-24T03:30:04.566 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.551+0000 7f5b837fe700 1 -- 172.21.15.142:0/2189277369 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70685 (secure 0 0 0) 0x7f5b9c1697a0 con 0x7f5b9c0683b0 2023-01-24T03:30:04.635 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.632+0000 7f5ba2038700 1 -- 172.21.15.142:0/2189277369 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd new", "uuid": "fe6ad231-a39a-42eb-9a3a-49d008dd35e4", "id": 0} v 0) v1 -- 0x7f5b9c05e610 con 0x7f5b9c0683b0 2023-01-24T03:30:04.640 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.638+0000 7f5b837fe700 1 -- 172.21.15.142:0/2189277369 <== mon.1 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fe6ad231-a39a-42eb-9a3a-49d008dd35e4", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5b90021060 con 0x7f5b9c0683b0 2023-01-24T03:30:04.641 INFO:teuthology.orchestra.run.smithi142.stdout:0 2023-01-24T03:30:04.642 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.639+0000 7f5ba2038700 1 -- 172.21.15.142:0/2189277369 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b9c0683b0 msgr2=0x7f5b9c0be580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:04.642 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.639+0000 7f5ba2038700 1 --2- 172.21.15.142:0/2189277369 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b9c0683b0 0x7f5b9c0be580 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5b9000e980 tx=0x7f5b90009060 comp rx=0 tx=0).stop 2023-01-24T03:30:04.643 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.639+0000 7f5ba2038700 1 -- 172.21.15.142:0/2189277369 shutdown_connections 2023-01-24T03:30:04.643 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.639+0000 7f5ba2038700 1 --2- 172.21.15.142:0/2189277369 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5b9c06f280 0x7f5b9c0beac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:04.643 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.639+0000 7f5ba2038700 1 --2- 172.21.15.142:0/2189277369 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b9c067fe0 0x7f5b9c0c35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:04.643 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.639+0000 7f5ba2038700 1 --2- 172.21.15.142:0/2189277369 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b9c0683b0 0x7f5b9c0be580 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:04.643 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.639+0000 7f5ba2038700 1 -- 172.21.15.142:0/2189277369 >> 172.21.15.142:0/2189277369 conn(0x7f5b9c06b550 msgr2=0x7f5b9c0c1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:04.644 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.639+0000 7f5ba2038700 1 -- 172.21.15.142:0/2189277369 shutdown_connections 2023-01-24T03:30:04.644 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.639+0000 7f5ba2038700 1 -- 172.21.15.142:0/2189277369 wait complete. 2023-01-24T03:30:04.653 DEBUG:teuthology.orchestra.run.smithi142:> sudo ceph --cluster ceph osd new 8526a1d2-d540-4021-9766-c4bbb30317ad 1 2023-01-24T03:30:04.767 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.765+0000 7fd265896700 1 Processor -- start 2023-01-24T03:30:04.768 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.765+0000 7fd265896700 1 -- start start 2023-01-24T03:30:04.770 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.766+0000 7fd265896700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2600c8460 0x7fd2601557b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:04.771 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.766+0000 7fd265896700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd2600c3990 0x7fd26015f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:04.771 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.766+0000 7fd265896700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2600c5cc0 con 0x7fd260155cf0 2023-01-24T03:30:04.772 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.766+0000 7fd265896700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2601550a0 con 0x7fd2600c3990 2023-01-24T03:30:04.772 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.766+0000 7fd265896700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd260155320 con 0x7fd2600c8460 2023-01-24T03:30:04.772 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.766+0000 7fd264894700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2600c8460 0x7fd2601557b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:04.772 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.766+0000 7fd265095700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd2600c3990 0x7fd26015f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:04.772 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.766+0000 7fd264894700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2600c8460 0x7fd2601557b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:41718/0 (socket says 172.21.15.142:41718) 2023-01-24T03:30:04.773 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.766+0000 7fd264894700 1 -- 172.21.15.142:0/2188048153 learned_addr learned my addr 172.21.15.142:0/2188048153 (peer_addr_for_me v2:172.21.15.142:0/0) 2023-01-24T03:30:04.773 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.766+0000 7fd265095700 1 -- 172.21.15.142:0/2188048153 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2600c8460 msgr2=0x7fd2601557b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:04.773 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.766+0000 7fd265095700 1 --2- 172.21.15.142:0/2188048153 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2600c8460 0x7fd2601557b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:04.773 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.766+0000 7fd265095700 1 -- 172.21.15.142:0/2188048153 >> v1:172.21.15.142:6789/0 conn(0x7fd260155cf0 legacy=0x7fd2600c3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:04.773 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.766+0000 7fd265095700 1 -- 172.21.15.142:0/2188048153 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2600c2cb0 con 0x7fd2600c3990 2023-01-24T03:30:04.774 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.766+0000 7fd264894700 1 --2- 172.21.15.142:0/2188048153 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2600c8460 0x7fd2601557b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:30:04.774 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.766+0000 7fd265095700 1 --2- 172.21.15.142:0/2188048153 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2600c3990 0x7fd26015f950 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd258009e60 tx=0x7fd258009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20b6b9157587e3ec server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:04.774 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.766+0000 7fd25f7fe700 1 -- 172.21.15.142:0/2188048153 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd25800e070 con 0x7fd2600c3990 2023-01-24T03:30:04.774 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.766+0000 7fd25f7fe700 1 -- 172.21.15.142:0/2188048153 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd25800bbf0 con 0x7fd2600c3990 2023-01-24T03:30:04.774 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.766+0000 7fd25f7fe700 1 -- 172.21.15.142:0/2188048153 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd258012430 con 0x7fd2600c3990 2023-01-24T03:30:04.775 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.767+0000 7fd265896700 1 -- 172.21.15.142:0/2188048153 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2600c3990 msgr2=0x7fd26015f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:04.775 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.767+0000 7fd265896700 1 --2- 172.21.15.142:0/2188048153 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2600c3990 0x7fd26015f950 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd258009e60 tx=0x7fd258009780 comp rx=0 tx=0).stop 2023-01-24T03:30:04.775 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.767+0000 7fd265896700 1 -- 172.21.15.142:0/2188048153 shutdown_connections 2023-01-24T03:30:04.775 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.767+0000 7fd265896700 1 --2- 172.21.15.142:0/2188048153 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2600c3990 0x7fd26015f950 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:04.775 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.767+0000 7fd265896700 1 --2- 172.21.15.142:0/2188048153 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2600c8460 0x7fd2601557b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:04.776 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.767+0000 7fd265896700 1 -- 172.21.15.142:0/2188048153 >> 172.21.15.142:0/2188048153 conn(0x7fd26005d830 msgr2=0x7fd26005dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:04.776 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.767+0000 7fd265896700 1 -- 172.21.15.142:0/2188048153 shutdown_connections 2023-01-24T03:30:04.776 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.767+0000 7fd265896700 1 -- 172.21.15.142:0/2188048153 wait complete. 2023-01-24T03:30:04.776 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.768+0000 7fd265896700 1 Processor -- start 2023-01-24T03:30:04.776 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.768+0000 7fd265896700 1 -- start start 2023-01-24T03:30:04.779 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.769+0000 7fd265896700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2600c3990 0x7fd260165060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:04.780 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.769+0000 7fd265896700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2600c8460 0x7fd260150a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:04.780 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.769+0000 7fd265896700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd260155cf0 0x7fd26014cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:04.780 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.769+0000 7fd265896700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd260152240 con 0x7fd2600c8460 2023-01-24T03:30:04.780 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.769+0000 7fd265896700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd260151350 con 0x7fd2600c3990 2023-01-24T03:30:04.781 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.769+0000 7fd265896700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd260151630 con 0x7fd260155cf0 2023-01-24T03:30:04.781 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.769+0000 7fd264894700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2600c3990 0x7fd260165060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:04.781 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.769+0000 7fd264894700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2600c3990 0x7fd260165060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48394/0 (socket says 172.21.15.142:48394) 2023-01-24T03:30:04.781 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.769+0000 7fd264894700 1 -- 172.21.15.142:0/197955958 learned_addr learned my addr 172.21.15.142:0/197955958 (peer_addr_for_me v2:172.21.15.142:0/0) 2023-01-24T03:30:04.781 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.769+0000 7fd265095700 1 --2- 172.21.15.142:0/197955958 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd260155cf0 0x7fd26014cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:04.782 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.769+0000 7fd25ffff700 1 --2- 172.21.15.142:0/197955958 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2600c8460 0x7fd260150a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:04.782 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.769+0000 7fd264894700 1 -- 172.21.15.142:0/197955958 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd260155cf0 msgr2=0x7fd26014cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:04.782 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.769+0000 7fd264894700 1 --2- 172.21.15.142:0/197955958 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd260155cf0 0x7fd26014cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:04.782 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.769+0000 7fd264894700 1 -- 172.21.15.142:0/197955958 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2600c8460 msgr2=0x7fd260150a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:04.782 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.769+0000 7fd264894700 1 --2- 172.21.15.142:0/197955958 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2600c8460 0x7fd260150a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:04.783 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.769+0000 7fd264894700 1 -- 172.21.15.142:0/197955958 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd254009370 con 0x7fd2600c3990 2023-01-24T03:30:04.783 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.769+0000 7fd265095700 1 --2- 172.21.15.142:0/197955958 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd260155cf0 0x7fd26014cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:30:04.783 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.769+0000 7fd264894700 1 --2- 172.21.15.142:0/197955958 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2600c3990 0x7fd260165060 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd25400ea20 tx=0x7fd2540090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:04.783 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.770+0000 7fd25cff9700 1 -- 172.21.15.142:0/197955958 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd254009650 con 0x7fd2600c3990 2023-01-24T03:30:04.783 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.770+0000 7fd25cff9700 1 -- 172.21.15.142:0/197955958 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd25400ee60 con 0x7fd2600c3990 2023-01-24T03:30:04.784 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.770+0000 7fd265896700 1 -- 172.21.15.142:0/197955958 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd258009a20 con 0x7fd2600c3990 2023-01-24T03:30:04.784 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.770+0000 7fd25cff9700 1 -- 172.21.15.142:0/197955958 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd25400caa0 con 0x7fd2600c3990 2023-01-24T03:30:04.784 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.770+0000 7fd265896700 1 -- 172.21.15.142:0/197955958 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd260151bf0 con 0x7fd2600c3990 2023-01-24T03:30:04.784 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.770+0000 7fd25cff9700 1 -- 172.21.15.142:0/197955958 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7fd254019030 con 0x7fd2600c3990 2023-01-24T03:30:04.785 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.770+0000 7fd25cff9700 1 -- 172.21.15.142:0/197955958 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (secure 0 0 0) 0x7fd2540065b0 con 0x7fd2600c3990 2023-01-24T03:30:04.785 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.771+0000 7fd265896700 1 -- 172.21.15.142:0/197955958 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd26005ea20 con 0x7fd2600c3990 2023-01-24T03:30:04.785 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.773+0000 7fd25cff9700 1 -- 172.21.15.142:0/197955958 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70685 (secure 0 0 0) 0x7fd254021060 con 0x7fd2600c3990 2023-01-24T03:30:04.858 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.855+0000 7fd265896700 1 -- 172.21.15.142:0/197955958 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd new", "uuid": "8526a1d2-d540-4021-9766-c4bbb30317ad", "id": 1} v 0) v1 -- 0x7fd260158a10 con 0x7fd2600c3990 2023-01-24T03:30:04.863 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.860+0000 7fd25cff9700 1 -- 172.21.15.142:0/197955958 <== mon.1 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8526a1d2-d540-4021-9766-c4bbb30317ad", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd254017070 con 0x7fd2600c3990 2023-01-24T03:30:04.863 INFO:teuthology.orchestra.run.smithi142.stdout:1 2023-01-24T03:30:04.864 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.861+0000 7fd265896700 1 -- 172.21.15.142:0/197955958 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2600c3990 msgr2=0x7fd260165060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:04.864 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.861+0000 7fd265896700 1 --2- 172.21.15.142:0/197955958 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2600c3990 0x7fd260165060 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd25400ea20 tx=0x7fd2540090f0 comp rx=0 tx=0).stop 2023-01-24T03:30:04.865 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.861+0000 7fd265896700 1 -- 172.21.15.142:0/197955958 shutdown_connections 2023-01-24T03:30:04.865 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.861+0000 7fd265896700 1 --2- 172.21.15.142:0/197955958 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd260155cf0 0x7fd26014cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:04.865 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.861+0000 7fd265896700 1 --2- 172.21.15.142:0/197955958 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2600c3990 0x7fd260165060 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:04.865 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.861+0000 7fd265896700 1 --2- 172.21.15.142:0/197955958 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2600c8460 0x7fd260150a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:04.865 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.861+0000 7fd265896700 1 -- 172.21.15.142:0/197955958 >> 172.21.15.142:0/197955958 conn(0x7fd26005d830 msgr2=0x7fd2600c6140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:04.866 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.861+0000 7fd265896700 1 -- 172.21.15.142:0/197955958 shutdown_connections 2023-01-24T03:30:04.866 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.861+0000 7fd265896700 1 -- 172.21.15.142:0/197955958 wait complete. 2023-01-24T03:30:04.875 DEBUG:teuthology.orchestra.run.smithi142:> sudo ceph --cluster ceph osd new 041ef8e4-4858-432d-9476-cbdb3a0dc47c 2 2023-01-24T03:30:04.958 INFO:tasks.ceph.mgr.y.smithi112.stderr:2023-01-24T03:30:04.955+0000 7f9fbb5a0000 -1 mgr[py] Module not found: 'prometheus' 2023-01-24T03:30:04.959 INFO:tasks.ceph.mgr.y.smithi112.stderr:2023-01-24T03:30:04.955+0000 7f9fbb5a0000 -1 mgr[py] Traceback (most recent call last): 2023-01-24T03:30:04.959 INFO:tasks.ceph.mgr.y.smithi112.stderr: File "/usr/share/ceph/mgr/prometheus/__init__.py", line 2, in 2023-01-24T03:30:04.959 INFO:tasks.ceph.mgr.y.smithi112.stderr: from .module import Module, StandbyModule 2023-01-24T03:30:04.959 INFO:tasks.ceph.mgr.y.smithi112.stderr: File "/usr/share/ceph/mgr/prometheus/module.py", line 3, in 2023-01-24T03:30:04.960 INFO:tasks.ceph.mgr.y.smithi112.stderr: from packaging.version import Version 2023-01-24T03:30:04.960 INFO:tasks.ceph.mgr.y.smithi112.stderr:ModuleNotFoundError: No module named 'packaging' 2023-01-24T03:30:04.960 INFO:tasks.ceph.mgr.y.smithi112.stderr: 2023-01-24T03:30:04.960 INFO:tasks.ceph.mgr.y.smithi112.stderr:2023-01-24T03:30:04.956+0000 7f9fbb5a0000 -1 mgr[py] Class not found in module 'prometheus' 2023-01-24T03:30:04.961 INFO:tasks.ceph.mgr.y.smithi112.stderr:2023-01-24T03:30:04.956+0000 7f9fbb5a0000 -1 mgr[py] Error loading module 'prometheus': (2) No such file or directory 2023-01-24T03:30:04.964 INFO:tasks.ceph.mgr.x.smithi142.stderr:2023-01-24T03:30:04.962+0000 7f82fde28000 -1 mgr[py] Module not found: 'prometheus' 2023-01-24T03:30:04.966 INFO:tasks.ceph.mgr.x.smithi142.stderr:2023-01-24T03:30:04.962+0000 7f82fde28000 -1 mgr[py] Traceback (most recent call last): 2023-01-24T03:30:04.966 INFO:tasks.ceph.mgr.x.smithi142.stderr: File "/usr/share/ceph/mgr/prometheus/__init__.py", line 2, in 2023-01-24T03:30:04.966 INFO:tasks.ceph.mgr.x.smithi142.stderr: from .module import Module, StandbyModule 2023-01-24T03:30:04.966 INFO:tasks.ceph.mgr.x.smithi142.stderr: File "/usr/share/ceph/mgr/prometheus/module.py", line 3, in 2023-01-24T03:30:04.967 INFO:tasks.ceph.mgr.x.smithi142.stderr: from packaging.version import Version 2023-01-24T03:30:04.967 INFO:tasks.ceph.mgr.x.smithi142.stderr:ModuleNotFoundError: No module named 'packaging' 2023-01-24T03:30:04.967 INFO:tasks.ceph.mgr.x.smithi142.stderr: 2023-01-24T03:30:04.967 INFO:tasks.ceph.mgr.x.smithi142.stderr:2023-01-24T03:30:04.963+0000 7f82fde28000 -1 mgr[py] Class not found in module 'prometheus' 2023-01-24T03:30:04.967 INFO:tasks.ceph.mgr.x.smithi142.stderr:2023-01-24T03:30:04.963+0000 7f82fde28000 -1 mgr[py] Error loading module 'prometheus': (2) No such file or directory 2023-01-24T03:30:04.988 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.986+0000 7fbf302a2700 1 Processor -- start 2023-01-24T03:30:04.989 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.986+0000 7fbf302a2700 1 -- start start 2023-01-24T03:30:04.992 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.987+0000 7fbf302a2700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fbf280694c0 0x7fbf28069890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:04.992 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.987+0000 7fbf302a2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf2805e420 0x7fbf28155f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:04.992 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.987+0000 7fbf302a2700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf28155ab0 con 0x7fbf28068d20 2023-01-24T03:30:04.993 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.987+0000 7fbf302a2700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf28155550 con 0x7fbf280694c0 2023-01-24T03:30:04.993 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.987+0000 7fbf302a2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf28156480 con 0x7fbf2805e420 2023-01-24T03:30:04.993 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.987+0000 7fbf2d83d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf2805e420 0x7fbf28155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:04.993 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.987+0000 7fbf2e83f700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fbf28068d20 0x7fbf280c68d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:41216/0 (socket says 172.21.15.142:41216) 2023-01-24T03:30:04.994 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.987+0000 7fbf2e83f700 1 -- 172.21.15.142:0/1854313479 learned_addr learned my addr 172.21.15.142:0/1854313479 (peer_addr_for_me v1:172.21.15.142:0/0) 2023-01-24T03:30:04.994 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.987+0000 7fbf2e03e700 1 --2- 172.21.15.142:0/1854313479 >> v2:172.21.15.112:3300/0 conn(0x7fbf280694c0 0x7fbf28069890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:04.994 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.987+0000 7fbf2d83d700 1 -- 172.21.15.142:0/1854313479 >> v2:172.21.15.112:3300/0 conn(0x7fbf280694c0 msgr2=0x7fbf28069890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:04.994 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.987+0000 7fbf2d03c700 1 -- 172.21.15.142:0/1854313479 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 588352817 0 0) 0x7fbf28155ab0 con 0x7fbf28068d20 2023-01-24T03:30:04.994 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.987+0000 7fbf2d83d700 1 --2- 172.21.15.142:0/1854313479 >> v2:172.21.15.112:3300/0 conn(0x7fbf280694c0 0x7fbf28069890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:04.995 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.987+0000 7fbf2d83d700 1 -- 172.21.15.142:0/1854313479 >> v1:172.21.15.142:6789/0 conn(0x7fbf28068d20 legacy=0x7fbf280c68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:04.995 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.987+0000 7fbf2d83d700 1 -- 172.21.15.142:0/1854313479 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf280c70e0 con 0x7fbf2805e420 2023-01-24T03:30:04.995 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.988+0000 7fbf2d83d700 1 --2- 172.21.15.142:0/1854313479 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf2805e420 0x7fbf28155f40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbf24009760 tx=0x7fbf24009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6977c6fc47742d45 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:04.995 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.988+0000 7fbf2d03c700 1 -- 172.21.15.142:0/1854313479 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf24011630 con 0x7fbf2805e420 2023-01-24T03:30:04.995 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.988+0000 7fbf2d03c700 1 -- 172.21.15.142:0/1854313479 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf24011790 con 0x7fbf2805e420 2023-01-24T03:30:04.996 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.988+0000 7fbf2d03c700 1 -- 172.21.15.142:0/1854313479 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf24011b50 con 0x7fbf2805e420 2023-01-24T03:30:04.996 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.988+0000 7fbf302a2700 1 -- 172.21.15.142:0/1854313479 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf2805e420 msgr2=0x7fbf28155f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:04.996 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.988+0000 7fbf302a2700 1 --2- 172.21.15.142:0/1854313479 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf2805e420 0x7fbf28155f40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbf24009760 tx=0x7fbf24009060 comp rx=0 tx=0).stop 2023-01-24T03:30:04.996 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.988+0000 7fbf302a2700 1 -- 172.21.15.142:0/1854313479 shutdown_connections 2023-01-24T03:30:04.996 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.988+0000 7fbf302a2700 1 --2- 172.21.15.142:0/1854313479 >> v2:172.21.15.112:3300/0 conn(0x7fbf280694c0 0x7fbf28069890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:04.997 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.988+0000 7fbf302a2700 1 --2- 172.21.15.142:0/1854313479 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf2805e420 0x7fbf28155f40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:04.997 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.988+0000 7fbf302a2700 1 -- 172.21.15.142:0/1854313479 >> 172.21.15.142:0/1854313479 conn(0x7fbf2805d830 msgr2=0x7fbf2805dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:04.997 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.988+0000 7fbf302a2700 1 -- 172.21.15.142:0/1854313479 shutdown_connections 2023-01-24T03:30:04.997 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.988+0000 7fbf302a2700 1 -- 172.21.15.142:0/1854313479 wait complete. 2023-01-24T03:30:04.997 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.989+0000 7fbf302a2700 1 Processor -- start 2023-01-24T03:30:04.998 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.989+0000 7fbf302a2700 1 -- start start 2023-01-24T03:30:05.000 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.990+0000 7fbf302a2700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf2805e420 0x7fbf281522a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:05.000 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.990+0000 7fbf302a2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf28068d20 0x7fbf2814b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:05.000 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.990+0000 7fbf302a2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf280694c0 0x7fbf280c1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:05.001 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.990+0000 7fbf302a2700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf28169280 con 0x7fbf2805e420 2023-01-24T03:30:05.001 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.990+0000 7fbf302a2700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf280cc400 con 0x7fbf280694c0 2023-01-24T03:30:05.001 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.990+0000 7fbf302a2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf280cc6e0 con 0x7fbf28068d20 2023-01-24T03:30:05.001 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.990+0000 7fbf2d83d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf28068d20 0x7fbf2814b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:05.002 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.990+0000 7fbf2e83f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf280694c0 0x7fbf280c1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:05.002 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.990+0000 7fbf2d83d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf28068d20 0x7fbf2814b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:41738/0 (socket says 172.21.15.142:41738) 2023-01-24T03:30:05.002 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.990+0000 7fbf2d83d700 1 -- 172.21.15.142:0/894014948 learned_addr learned my addr 172.21.15.142:0/894014948 (peer_addr_for_me v2:172.21.15.142:0/0) 2023-01-24T03:30:05.002 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.990+0000 7fbf2e03e700 1 --2- 172.21.15.142:0/894014948 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf2805e420 0x7fbf281522a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:05.002 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.990+0000 7fbf2e83f700 1 -- 172.21.15.142:0/894014948 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf28068d20 msgr2=0x7fbf2814b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:05.003 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.990+0000 7fbf2e83f700 1 --2- 172.21.15.142:0/894014948 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf28068d20 0x7fbf2814b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.003 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.990+0000 7fbf2e83f700 1 -- 172.21.15.142:0/894014948 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf2805e420 msgr2=0x7fbf281522a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:05.003 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.990+0000 7fbf2e83f700 1 --2- 172.21.15.142:0/894014948 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf2805e420 0x7fbf281522a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.003 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.990+0000 7fbf2e83f700 1 -- 172.21.15.142:0/894014948 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf240092a0 con 0x7fbf280694c0 2023-01-24T03:30:05.003 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.990+0000 7fbf2d83d700 1 --2- 172.21.15.142:0/894014948 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf28068d20 0x7fbf2814b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:30:05.003 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.991+0000 7fbf2e83f700 1 --2- 172.21.15.142:0/894014948 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf280694c0 0x7fbf280c1d40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fbf28069190 tx=0x7fbf20007eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:05.004 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.991+0000 7fbf167fc700 1 -- 172.21.15.142:0/894014948 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf20005d30 con 0x7fbf280694c0 2023-01-24T03:30:05.004 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.991+0000 7fbf167fc700 1 -- 172.21.15.142:0/894014948 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf20005e90 con 0x7fbf280694c0 2023-01-24T03:30:05.004 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.991+0000 7fbf302a2700 1 -- 172.21.15.142:0/894014948 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf28168950 con 0x7fbf280694c0 2023-01-24T03:30:05.004 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.991+0000 7fbf167fc700 1 -- 172.21.15.142:0/894014948 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf20007320 con 0x7fbf280694c0 2023-01-24T03:30:05.004 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.991+0000 7fbf302a2700 1 -- 172.21.15.142:0/894014948 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf28168f10 con 0x7fbf280694c0 2023-01-24T03:30:05.005 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.991+0000 7fbf167fc700 1 -- 172.21.15.142:0/894014948 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7fbf20019030 con 0x7fbf280694c0 2023-01-24T03:30:05.005 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.991+0000 7fbf167fc700 1 -- 172.21.15.142:0/894014948 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (secure 0 0 0) 0x7fbf20012830 con 0x7fbf280694c0 2023-01-24T03:30:05.005 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.992+0000 7fbf302a2700 1 -- 172.21.15.142:0/894014948 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf281693f0 con 0x7fbf280694c0 2023-01-24T03:30:05.005 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:04.994+0000 7fbf167fc700 1 -- 172.21.15.142:0/894014948 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70685 (secure 0 0 0) 0x7fbf20017070 con 0x7fbf280694c0 2023-01-24T03:30:05.087 INFO:tasks.ceph.mgr.y.smithi112.stderr:2023-01-24T03:30:05.084+0000 7f9fbb5a0000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-01-24T03:30:05.089 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.087+0000 7fbf302a2700 1 -- 172.21.15.142:0/894014948 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd new", "uuid": "041ef8e4-4858-432d-9476-cbdb3a0dc47c", "id": 2} v 0) v1 -- 0x7fbf2806f3a0 con 0x7fbf280694c0 2023-01-24T03:30:05.096 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.094+0000 7fbf167fc700 1 -- 172.21.15.142:0/894014948 <== mon.1 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "041ef8e4-4858-432d-9476-cbdb3a0dc47c", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fbf2001c440 con 0x7fbf280694c0 2023-01-24T03:30:05.097 INFO:teuthology.orchestra.run.smithi142.stdout:2 2023-01-24T03:30:05.098 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.095+0000 7fbf302a2700 1 -- 172.21.15.142:0/894014948 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf280694c0 msgr2=0x7fbf280c1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:05.098 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.095+0000 7fbf302a2700 1 --2- 172.21.15.142:0/894014948 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf280694c0 0x7fbf280c1d40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fbf28069190 tx=0x7fbf20007eb0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.098 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.095+0000 7fbf302a2700 1 -- 172.21.15.142:0/894014948 shutdown_connections 2023-01-24T03:30:05.099 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.096+0000 7fbf302a2700 1 --2- 172.21.15.142:0/894014948 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf2805e420 0x7fbf281522a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.099 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.096+0000 7fbf302a2700 1 --2- 172.21.15.142:0/894014948 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf280694c0 0x7fbf280c1d40 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.099 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.096+0000 7fbf302a2700 1 --2- 172.21.15.142:0/894014948 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf28068d20 0x7fbf2814b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.099 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.096+0000 7fbf302a2700 1 -- 172.21.15.142:0/894014948 >> 172.21.15.142:0/894014948 conn(0x7fbf2805d830 msgr2=0x7fbf280ae9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:05.099 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.096+0000 7fbf302a2700 1 -- 172.21.15.142:0/894014948 shutdown_connections 2023-01-24T03:30:05.100 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.096+0000 7fbf302a2700 1 -- 172.21.15.142:0/894014948 wait complete. 2023-01-24T03:30:05.109 DEBUG:teuthology.orchestra.run.smithi142:> sudo ceph --cluster ceph osd new 987587cf-1357-4a70-ad85-67b189c67729 3 2023-01-24T03:30:05.110 INFO:tasks.ceph.mgr.x.smithi142.stderr:2023-01-24T03:30:05.108+0000 7f82fde28000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-01-24T03:30:05.160 INFO:tasks.ceph.mgr.y.smithi112.stderr:2023-01-24T03:30:05.157+0000 7f9fbb5a0000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-01-24T03:30:05.189 INFO:tasks.ceph.mgr.x.smithi142.stderr:2023-01-24T03:30:05.186+0000 7f82fde28000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-01-24T03:30:05.223 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.221+0000 7f88a035c700 1 Processor -- start 2023-01-24T03:30:05.224 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.221+0000 7f88a035c700 1 -- start start 2023-01-24T03:30:05.226 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.222+0000 7f88a035c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88980694c0 0x7f8898069890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:05.226 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.222+0000 7f88a035c700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8898155ab0 con 0x7f8898068d20 2023-01-24T03:30:05.227 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.222+0000 7f88a035c700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88980c70e0 con 0x7f889805e420 2023-01-24T03:30:05.227 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.222+0000 7f88a035c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8898155550 con 0x7f88980694c0 2023-01-24T03:30:05.227 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.222+0000 7f889d8f7700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f889805e420 0x7f88980c68d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:41224/0 (socket says 172.21.15.142:41224) 2023-01-24T03:30:05.227 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.222+0000 7f889e0f8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88980694c0 0x7f8898069890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:05.227 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.222+0000 7f889d8f7700 1 -- 172.21.15.142:0/2728298913 learned_addr learned my addr 172.21.15.142:0/2728298913 (peer_addr_for_me v1:172.21.15.142:0/0) 2023-01-24T03:30:05.228 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.223+0000 7f889d0f6700 1 -- 172.21.15.142:0/2728298913 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1443852153 0 0) 0x7f88980c70e0 con 0x7f889805e420 2023-01-24T03:30:05.228 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.223+0000 7f889d0f6700 1 -- 172.21.15.142:0/2728298913 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88800033f0 con 0x7f889805e420 2023-01-24T03:30:05.228 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.223+0000 7f889e0f8700 1 -- 172.21.15.142:0/2728298913 >> v1:172.21.15.142:6789/0 conn(0x7f889805e420 legacy=0x7f88980c68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:05.228 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.223+0000 7f889e0f8700 1 -- 172.21.15.142:0/2728298913 >> v1:172.21.15.112:6789/0 conn(0x7f8898068d20 legacy=0x7f88980ae9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:05.228 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.223+0000 7f889e0f8700 1 -- 172.21.15.142:0/2728298913 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88980c73c0 con 0x7f88980694c0 2023-01-24T03:30:05.229 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.223+0000 7f889e0f8700 1 --2- 172.21.15.142:0/2728298913 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88980694c0 0x7f8898069890 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8888009790 tx=0x7f8888009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23d344b3773a5e6f server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:05.229 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.223+0000 7f889d0f6700 1 -- 172.21.15.142:0/2728298913 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8888011630 con 0x7f88980694c0 2023-01-24T03:30:05.229 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.223+0000 7f889d0f6700 1 -- 172.21.15.142:0/2728298913 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8888011790 con 0x7f88980694c0 2023-01-24T03:30:05.229 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.223+0000 7f889d0f6700 1 -- 172.21.15.142:0/2728298913 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8888011b10 con 0x7f88980694c0 2023-01-24T03:30:05.229 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.224+0000 7f88a035c700 1 -- 172.21.15.142:0/2728298913 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88980694c0 msgr2=0x7f8898069890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:05.230 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.224+0000 7f88a035c700 1 --2- 172.21.15.142:0/2728298913 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88980694c0 0x7f8898069890 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8888009790 tx=0x7f8888009060 comp rx=0 tx=0).stop 2023-01-24T03:30:05.230 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.224+0000 7f88a035c700 1 -- 172.21.15.142:0/2728298913 shutdown_connections 2023-01-24T03:30:05.230 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.224+0000 7f88a035c700 1 --2- 172.21.15.142:0/2728298913 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f88980694c0 0x7f8898069890 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.230 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.224+0000 7f88a035c700 1 -- 172.21.15.142:0/2728298913 >> 172.21.15.142:0/2728298913 conn(0x7f889805d830 msgr2=0x7f889805dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:05.230 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.224+0000 7f88a035c700 1 -- 172.21.15.142:0/2728298913 shutdown_connections 2023-01-24T03:30:05.231 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.224+0000 7f88a035c700 1 -- 172.21.15.142:0/2728298913 wait complete. 2023-01-24T03:30:05.234 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.225+0000 7f88a035c700 1 Processor -- start 2023-01-24T03:30:05.234 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.225+0000 7f88a035c700 1 -- start start 2023-01-24T03:30:05.234 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.225+0000 7f88a035c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f889805e420 0x7f88980c4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:05.234 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.225+0000 7f88a035c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8898068d20 0x7f8898150a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:05.234 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.225+0000 7f88a035c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f88980694c0 0x7f8898169230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:05.235 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.225+0000 7f88a035c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889814c8f0 con 0x7f8898068d20 2023-01-24T03:30:05.235 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.225+0000 7f88a035c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88980c56f0 con 0x7f88980694c0 2023-01-24T03:30:05.235 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.225+0000 7f88a035c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f889814cb90 con 0x7f889805e420 2023-01-24T03:30:05.235 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.225+0000 7f889e0f8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f889805e420 0x7f88980c4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:05.235 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.225+0000 7f889e0f8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f889805e420 0x7f88980c4d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:41746/0 (socket says 172.21.15.142:41746) 2023-01-24T03:30:05.236 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.225+0000 7f889e0f8700 1 -- 172.21.15.142:0/1863251510 learned_addr learned my addr 172.21.15.142:0/1863251510 (peer_addr_for_me v2:172.21.15.142:0/0) 2023-01-24T03:30:05.236 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.225+0000 7f889e8f9700 1 --2- 172.21.15.142:0/1863251510 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f88980694c0 0x7f8898169230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:05.236 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.226+0000 7f889d8f7700 1 --2- 172.21.15.142:0/1863251510 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8898068d20 0x7f8898150a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:05.236 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.226+0000 7f889e0f8700 1 -- 172.21.15.142:0/1863251510 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f88980694c0 msgr2=0x7f8898169230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:05.236 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.226+0000 7f889e0f8700 1 --2- 172.21.15.142:0/1863251510 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f88980694c0 0x7f8898169230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.237 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.226+0000 7f889e0f8700 1 -- 172.21.15.142:0/1863251510 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8898068d20 msgr2=0x7f8898150a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:05.237 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.226+0000 7f889e0f8700 1 --2- 172.21.15.142:0/1863251510 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8898068d20 0x7f8898150a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.237 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.226+0000 7f889e0f8700 1 -- 172.21.15.142:0/1863251510 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88880092a0 con 0x7f889805e420 2023-01-24T03:30:05.237 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.226+0000 7f889e0f8700 1 --2- 172.21.15.142:0/1863251510 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f889805e420 0x7f88980c4d70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8888009e30 tx=0x7f8888017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:05.237 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.226+0000 7f88867fc700 1 -- 172.21.15.142:0/1863251510 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8888011900 con 0x7f889805e420 2023-01-24T03:30:05.238 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.226+0000 7f88867fc700 1 -- 172.21.15.142:0/1863251510 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8888011a60 con 0x7f889805e420 2023-01-24T03:30:05.238 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.226+0000 7f88a035c700 1 -- 172.21.15.142:0/1863251510 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f889814ce70 con 0x7f889805e420 2023-01-24T03:30:05.238 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.226+0000 7f88867fc700 1 -- 172.21.15.142:0/1863251510 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f888801a680 con 0x7f889805e420 2023-01-24T03:30:05.238 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.226+0000 7f88a035c700 1 -- 172.21.15.142:0/1863251510 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f889814d430 con 0x7f889805e420 2023-01-24T03:30:05.238 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.226+0000 7f88867fc700 1 -- 172.21.15.142:0/1863251510 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7f888800f050 con 0x7f889805e420 2023-01-24T03:30:05.238 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.226+0000 7f88867fc700 1 -- 172.21.15.142:0/1863251510 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (secure 0 0 0) 0x7f888801acb0 con 0x7f889805e420 2023-01-24T03:30:05.239 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.227+0000 7f88a035c700 1 -- 172.21.15.142:0/1863251510 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f889806f3a0 con 0x7f889805e420 2023-01-24T03:30:05.239 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.229+0000 7f88867fc700 1 -- 172.21.15.142:0/1863251510 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70685 (secure 0 0 0) 0x7f8888034060 con 0x7f889805e420 2023-01-24T03:30:05.313 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.311+0000 7f88a035c700 1 -- 172.21.15.142:0/1863251510 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd new", "uuid": "987587cf-1357-4a70-ad85-67b189c67729", "id": 3} v 0) v1 -- 0x7f88980c2880 con 0x7f889805e420 2023-01-24T03:30:05.319 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.316+0000 7f88867fc700 1 -- 172.21.15.142:0/1863251510 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "987587cf-1357-4a70-ad85-67b189c67729", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8888015020 con 0x7f889805e420 2023-01-24T03:30:05.319 INFO:teuthology.orchestra.run.smithi142.stdout:3 2023-01-24T03:30:05.320 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.317+0000 7f88a035c700 1 -- 172.21.15.142:0/1863251510 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f889805e420 msgr2=0x7f88980c4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:05.320 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.317+0000 7f88a035c700 1 --2- 172.21.15.142:0/1863251510 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f889805e420 0x7f88980c4d70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8888009e30 tx=0x7f8888017000 comp rx=0 tx=0).stop 2023-01-24T03:30:05.321 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.317+0000 7f88a035c700 1 -- 172.21.15.142:0/1863251510 shutdown_connections 2023-01-24T03:30:05.321 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.317+0000 7f88a035c700 1 --2- 172.21.15.142:0/1863251510 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f88980694c0 0x7f8898169230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.321 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.317+0000 7f88a035c700 1 --2- 172.21.15.142:0/1863251510 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f889805e420 0x7f88980c4d70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.321 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.317+0000 7f88a035c700 1 --2- 172.21.15.142:0/1863251510 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8898068d20 0x7f8898150a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.321 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.317+0000 7f88a035c700 1 -- 172.21.15.142:0/1863251510 >> 172.21.15.142:0/1863251510 conn(0x7f889805d830 msgr2=0x7f889814f190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:05.322 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.318+0000 7f88a035c700 1 -- 172.21.15.142:0/1863251510 shutdown_connections 2023-01-24T03:30:05.322 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.318+0000 7f88a035c700 1 -- 172.21.15.142:0/1863251510 wait complete. 2023-01-24T03:30:05.331 DEBUG:teuthology.orchestra.run.smithi142:> sudo ceph --cluster ceph osd new 8503512b-43db-40b8-8b2b-c86cf2e54e17 4 2023-01-24T03:30:05.340 INFO:tasks.ceph.mgr.y.smithi112.stderr:2023-01-24T03:30:05.337+0000 7f9fbb5a0000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-01-24T03:30:05.379 INFO:tasks.ceph.mgr.x.smithi142.stderr:2023-01-24T03:30:05.377+0000 7f82fde28000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-01-24T03:30:05.445 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.443+0000 7f06d9344700 1 Processor -- start 2023-01-24T03:30:05.445 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.443+0000 7f06d9344700 1 -- start start 2023-01-24T03:30:05.448 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.443+0000 7f06d9344700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06d40694c0 0x7f06d4069890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:05.448 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.443+0000 7f06d9344700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f06d405e420 0x7f06d4155f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:05.449 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.443+0000 7f06d9344700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f06d4068d20 0x7f06d40c68d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:05.449 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.443+0000 7f06d9344700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06d4155550 con 0x7f06d405e420 2023-01-24T03:30:05.449 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.443+0000 7f06d9344700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06d4155ab0 con 0x7f06d4068d20 2023-01-24T03:30:05.449 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.443+0000 7f06d9344700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06d4156480 con 0x7f06d40694c0 2023-01-24T03:30:05.449 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.444+0000 7f06d2ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06d40694c0 0x7f06d4069890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:05.450 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.444+0000 7f06d37fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f06d4068d20 0x7f06d40c68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:05.450 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.444+0000 7f06d2ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06d40694c0 0x7f06d4069890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:41752/0 (socket says 172.21.15.142:41752) 2023-01-24T03:30:05.450 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.444+0000 7f06d2ffd700 1 -- 172.21.15.142:0/3175368237 learned_addr learned my addr 172.21.15.142:0/3175368237 (peer_addr_for_me v2:172.21.15.142:0/0) 2023-01-24T03:30:05.450 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.444+0000 7f06d27fc700 1 --2- 172.21.15.142:0/3175368237 >> v2:172.21.15.112:3300/0 conn(0x7f06d405e420 0x7f06d4155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:05.450 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.444+0000 7f06d37fe700 1 -- 172.21.15.142:0/3175368237 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06d40694c0 msgr2=0x7f06d4069890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:05.451 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.444+0000 7f06d37fe700 1 --2- 172.21.15.142:0/3175368237 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06d40694c0 0x7f06d4069890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.451 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.444+0000 7f06d37fe700 1 -- 172.21.15.142:0/3175368237 >> v2:172.21.15.112:3300/0 conn(0x7f06d405e420 msgr2=0x7f06d4155f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:05.451 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.444+0000 7f06d37fe700 1 --2- 172.21.15.142:0/3175368237 >> v2:172.21.15.112:3300/0 conn(0x7f06d405e420 0x7f06d4155f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.451 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.444+0000 7f06d37fe700 1 -- 172.21.15.142:0/3175368237 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06d40c6e10 con 0x7f06d4068d20 2023-01-24T03:30:05.451 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.444+0000 7f06d2ffd700 1 --2- 172.21.15.142:0/3175368237 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06d40694c0 0x7f06d4069890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:30:05.451 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.444+0000 7f06d37fe700 1 --2- 172.21.15.142:0/3175368237 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f06d4068d20 0x7f06d40c68d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f06d40b0a40 tx=0x7f06cc009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4347c0fd2ab8732 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:05.452 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.444+0000 7f06d1ffb700 1 -- 172.21.15.142:0/3175368237 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06cc00e070 con 0x7f06d4068d20 2023-01-24T03:30:05.452 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.444+0000 7f06d1ffb700 1 -- 172.21.15.142:0/3175368237 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06cc00bab0 con 0x7f06d4068d20 2023-01-24T03:30:05.452 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.444+0000 7f06d1ffb700 1 -- 172.21.15.142:0/3175368237 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06cc00be70 con 0x7f06d4068d20 2023-01-24T03:30:05.452 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.445+0000 7f06d9344700 1 -- 172.21.15.142:0/3175368237 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f06d4068d20 msgr2=0x7f06d40c68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:05.452 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.445+0000 7f06d9344700 1 --2- 172.21.15.142:0/3175368237 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f06d4068d20 0x7f06d40c68d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f06d40b0a40 tx=0x7f06cc009780 comp rx=0 tx=0).stop 2023-01-24T03:30:05.453 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.445+0000 7f06d9344700 1 -- 172.21.15.142:0/3175368237 shutdown_connections 2023-01-24T03:30:05.453 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.445+0000 7f06d9344700 1 --2- 172.21.15.142:0/3175368237 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f06d4068d20 0x7f06d40c68d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.453 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.445+0000 7f06d9344700 1 --2- 172.21.15.142:0/3175368237 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06d40694c0 0x7f06d4069890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.453 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.445+0000 7f06d9344700 1 --2- 172.21.15.142:0/3175368237 >> v2:172.21.15.112:3300/0 conn(0x7f06d405e420 0x7f06d4155f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.453 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.445+0000 7f06d9344700 1 -- 172.21.15.142:0/3175368237 >> 172.21.15.142:0/3175368237 conn(0x7f06d405d830 msgr2=0x7f06d405dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:05.454 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.445+0000 7f06d9344700 1 -- 172.21.15.142:0/3175368237 shutdown_connections 2023-01-24T03:30:05.454 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.445+0000 7f06d9344700 1 -- 172.21.15.142:0/3175368237 wait complete. 2023-01-24T03:30:05.454 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.446+0000 7f06d9344700 1 Processor -- start 2023-01-24T03:30:05.455 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.446+0000 7f06d9344700 1 -- start start 2023-01-24T03:30:05.457 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.447+0000 7f06d9344700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f06d405e420 0x7f06d40c3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:05.458 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.447+0000 7f06d9344700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06d4068d20 0x7f06d40c3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:05.458 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.447+0000 7f06d9344700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06d40694c0 0x7f06d4168640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:05.458 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.447+0000 7f06d9344700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d40c5d70 con 0x7f06d4068d20 2023-01-24T03:30:05.459 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.447+0000 7f06d9344700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d40c57d0 con 0x7f06d405e420 2023-01-24T03:30:05.459 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.447+0000 7f06d9344700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06d4162f10 con 0x7f06d40694c0 2023-01-24T03:30:05.459 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.447+0000 7f06d2ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f06d405e420 0x7f06d40c3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:05.459 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.447+0000 7f06d2ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f06d405e420 0x7f06d40c3a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48426/0 (socket says 172.21.15.142:48426) 2023-01-24T03:30:05.459 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.447+0000 7f06d2ffd700 1 -- 172.21.15.142:0/3161482048 learned_addr learned my addr 172.21.15.142:0/3161482048 (peer_addr_for_me v2:172.21.15.142:0/0) 2023-01-24T03:30:05.460 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.447+0000 7f06d37fe700 1 --2- 172.21.15.142:0/3161482048 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06d40694c0 0x7f06d4168640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:05.460 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.447+0000 7f06d27fc700 1 --2- 172.21.15.142:0/3161482048 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06d4068d20 0x7f06d40c3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:05.460 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.447+0000 7f06d2ffd700 1 -- 172.21.15.142:0/3161482048 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06d40694c0 msgr2=0x7f06d4168640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:05.460 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.447+0000 7f06d2ffd700 1 --2- 172.21.15.142:0/3161482048 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06d40694c0 0x7f06d4168640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.460 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.447+0000 7f06d2ffd700 1 -- 172.21.15.142:0/3161482048 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06d4068d20 msgr2=0x7f06d40c3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:05.461 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.447+0000 7f06d2ffd700 1 --2- 172.21.15.142:0/3161482048 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06d4068d20 0x7f06d40c3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.461 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.447+0000 7f06d2ffd700 1 -- 172.21.15.142:0/3161482048 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c4009370 con 0x7f06d405e420 2023-01-24T03:30:05.461 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.447+0000 7f06d37fe700 1 --2- 172.21.15.142:0/3161482048 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06d40694c0 0x7f06d4168640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:30:05.461 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.447+0000 7f06d2ffd700 1 --2- 172.21.15.142:0/3161482048 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f06d405e420 0x7f06d40c3a50 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f06c400ea20 tx=0x7f06c40090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:05.461 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.447+0000 7f06baffd700 1 -- 172.21.15.142:0/3161482048 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c400c850 con 0x7f06d405e420 2023-01-24T03:30:05.461 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.447+0000 7f06baffd700 1 -- 172.21.15.142:0/3161482048 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c400c9b0 con 0x7f06d405e420 2023-01-24T03:30:05.462 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.447+0000 7f06d9344700 1 -- 172.21.15.142:0/3161482048 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06cc009a20 con 0x7f06d405e420 2023-01-24T03:30:05.462 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.447+0000 7f06baffd700 1 -- 172.21.15.142:0/3161482048 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c400cd70 con 0x7f06d405e420 2023-01-24T03:30:05.462 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.447+0000 7f06d9344700 1 -- 172.21.15.142:0/3161482048 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06d40c89c0 con 0x7f06d405e420 2023-01-24T03:30:05.462 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.448+0000 7f06baffd700 1 -- 172.21.15.142:0/3161482048 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7f06c4019030 con 0x7f06d405e420 2023-01-24T03:30:05.462 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.448+0000 7f06baffd700 1 -- 172.21.15.142:0/3161482048 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7f06c4006580 con 0x7f06d405e420 2023-01-24T03:30:05.463 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.448+0000 7f06d9344700 1 -- 172.21.15.142:0/3161482048 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06d406f3a0 con 0x7f06d405e420 2023-01-24T03:30:05.463 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.450+0000 7f06baffd700 1 -- 172.21.15.142:0/3161482048 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70685 (secure 0 0 0) 0x7f06c4021060 con 0x7f06d405e420 2023-01-24T03:30:05.507 INFO:tasks.ceph.mgr.y.smithi112.stderr:2023-01-24T03:30:05.504+0000 7f9fbb5a0000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-01-24T03:30:05.534 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.531+0000 7f06d9344700 1 -- 172.21.15.142:0/3161482048 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd new", "uuid": "8503512b-43db-40b8-8b2b-c86cf2e54e17", "id": 4} v 0) v1 -- 0x7f06d40680e0 con 0x7f06d405e420 2023-01-24T03:30:05.539 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.537+0000 7f06baffd700 1 -- 172.21.15.142:0/3161482048 <== mon.1 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8503512b-43db-40b8-8b2b-c86cf2e54e17", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f06c4017070 con 0x7f06d405e420 2023-01-24T03:30:05.539 INFO:teuthology.orchestra.run.smithi142.stdout:4 2023-01-24T03:30:05.540 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.538+0000 7f06d9344700 1 -- 172.21.15.142:0/3161482048 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f06d405e420 msgr2=0x7f06d40c3a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:05.541 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.538+0000 7f06d9344700 1 --2- 172.21.15.142:0/3161482048 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f06d405e420 0x7f06d40c3a50 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f06c400ea20 tx=0x7f06c40090f0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.541 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.538+0000 7f06d9344700 1 -- 172.21.15.142:0/3161482048 shutdown_connections 2023-01-24T03:30:05.541 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.538+0000 7f06d9344700 1 --2- 172.21.15.142:0/3161482048 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06d40694c0 0x7f06d4168640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.541 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.538+0000 7f06d9344700 1 --2- 172.21.15.142:0/3161482048 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f06d405e420 0x7f06d40c3a50 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.541 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.538+0000 7f06d9344700 1 --2- 172.21.15.142:0/3161482048 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06d4068d20 0x7f06d40c3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.542 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.538+0000 7f06d9344700 1 -- 172.21.15.142:0/3161482048 >> 172.21.15.142:0/3161482048 conn(0x7f06d405d830 msgr2=0x7f06d414f020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:05.542 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.538+0000 7f06d9344700 1 -- 172.21.15.142:0/3161482048 shutdown_connections 2023-01-24T03:30:05.542 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.538+0000 7f06d9344700 1 -- 172.21.15.142:0/3161482048 wait complete. 2023-01-24T03:30:05.552 DEBUG:teuthology.orchestra.run.smithi142:> sudo ceph --cluster ceph osd new d3bc0030-8b08-4645-9ec2-1f8a4e29ef36 5 2023-01-24T03:30:05.557 INFO:tasks.ceph.mgr.x.smithi142.stderr:2023-01-24T03:30:05.555+0000 7f82fde28000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-01-24T03:30:05.581 INFO:tasks.ceph.mgr.y.smithi112.stderr:2023-01-24T03:30:05.579+0000 7f9fbb5a0000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-01-24T03:30:05.637 INFO:tasks.ceph.mgr.x.smithi142.stderr:2023-01-24T03:30:05.635+0000 7f82fde28000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-01-24T03:30:05.667 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.664+0000 7fd06310a700 1 Processor -- start 2023-01-24T03:30:05.667 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.664+0000 7fd06310a700 1 -- start start 2023-01-24T03:30:05.669 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.665+0000 7fd06310a700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fd05c156d50 0x7fd05c0c30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:05.670 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.665+0000 7fd06310a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd05c06c4a0 0x7fd05c15d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:05.670 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.665+0000 7fd06310a700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd05c0c5880 con 0x7fd05c157120 2023-01-24T03:30:05.670 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.665+0000 7fd06310a700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd05c06c0f0 con 0x7fd05c156d50 2023-01-24T03:30:05.670 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.665+0000 7fd06310a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c0c5600 con 0x7fd05c06c4a0 2023-01-24T03:30:05.670 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.665+0000 7fd062909700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd05c06c4a0 0x7fd05c15d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:05.671 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.665+0000 7fd062909700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd05c06c4a0 0x7fd05c15d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:41766/0 (socket says 172.21.15.142:41766) 2023-01-24T03:30:05.671 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.665+0000 7fd062909700 1 -- 172.21.15.142:0/388208236 learned_addr learned my addr 172.21.15.142:0/388208236 (peer_addr_for_me v2:172.21.15.142:0/0) 2023-01-24T03:30:05.671 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.665+0000 7fd062108700 1 --2- 172.21.15.142:0/388208236 >> v2:172.21.15.112:3300/0 conn(0x7fd05c156d50 0x7fd05c0c30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:05.671 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.666+0000 7fd062909700 1 -- 172.21.15.142:0/388208236 >> v2:172.21.15.112:3300/0 conn(0x7fd05c156d50 msgr2=0x7fd05c0c30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:05.671 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.666+0000 7fd062909700 1 --2- 172.21.15.142:0/388208236 >> v2:172.21.15.112:3300/0 conn(0x7fd05c156d50 0x7fd05c0c30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.672 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.666+0000 7fd062909700 1 -- 172.21.15.142:0/388208236 >> v1:172.21.15.112:6789/0 conn(0x7fd05c157120 legacy=0x7fd05c0c6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:05.672 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.666+0000 7fd062909700 1 -- 172.21.15.142:0/388208236 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd05c0bd360 con 0x7fd05c06c4a0 2023-01-24T03:30:05.672 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.666+0000 7fd062909700 1 --2- 172.21.15.142:0/388208236 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd05c06c4a0 0x7fd05c15d840 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd054009f20 tx=0x7fd054009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1828251186d6ba server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:05.672 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.666+0000 7fd061106700 1 -- 172.21.15.142:0/388208236 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05400e070 con 0x7fd05c06c4a0 2023-01-24T03:30:05.672 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.666+0000 7fd061106700 1 -- 172.21.15.142:0/388208236 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd05400b9d0 con 0x7fd05c06c4a0 2023-01-24T03:30:05.673 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.666+0000 7fd061106700 1 -- 172.21.15.142:0/388208236 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0540092e0 con 0x7fd05c06c4a0 2023-01-24T03:30:05.673 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.667+0000 7fd06310a700 1 -- 172.21.15.142:0/388208236 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd05c06c4a0 msgr2=0x7fd05c15d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:05.673 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.667+0000 7fd06310a700 1 --2- 172.21.15.142:0/388208236 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd05c06c4a0 0x7fd05c15d840 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd054009f20 tx=0x7fd054009840 comp rx=0 tx=0).stop 2023-01-24T03:30:05.673 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.667+0000 7fd06310a700 1 -- 172.21.15.142:0/388208236 shutdown_connections 2023-01-24T03:30:05.673 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.667+0000 7fd06310a700 1 --2- 172.21.15.142:0/388208236 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd05c06c4a0 0x7fd05c15d840 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.674 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.667+0000 7fd06310a700 1 --2- 172.21.15.142:0/388208236 >> v2:172.21.15.112:3300/0 conn(0x7fd05c156d50 0x7fd05c0c30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.674 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.667+0000 7fd06310a700 1 -- 172.21.15.142:0/388208236 >> 172.21.15.142:0/388208236 conn(0x7fd05c06ce00 msgr2=0x7fd05c06d200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:05.674 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.667+0000 7fd06310a700 1 -- 172.21.15.142:0/388208236 shutdown_connections 2023-01-24T03:30:05.674 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.667+0000 7fd06310a700 1 -- 172.21.15.142:0/388208236 wait complete. 2023-01-24T03:30:05.677 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.668+0000 7fd06310a700 1 Processor -- start 2023-01-24T03:30:05.678 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.668+0000 7fd06310a700 1 -- start start 2023-01-24T03:30:05.678 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.668+0000 7fd06310a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd05c06c4a0 0x7fd05c152ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:05.678 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.668+0000 7fd06310a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd05c156d50 0x7fd05c153000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:05.678 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.668+0000 7fd06310a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd05c157120 0x7fd05c0a9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:05.678 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.669+0000 7fd06310a700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c0be6d0 con 0x7fd05c157120 2023-01-24T03:30:05.679 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.669+0000 7fd06310a700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c0be970 con 0x7fd05c06c4a0 2023-01-24T03:30:05.679 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.669+0000 7fd06310a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c165840 con 0x7fd05c156d50 2023-01-24T03:30:05.679 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.669+0000 7fd062108700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd05c06c4a0 0x7fd05c152ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:05.680 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.669+0000 7fd061907700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd05c156d50 0x7fd05c153000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:05.680 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.669+0000 7fd062108700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd05c06c4a0 0x7fd05c152ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48438/0 (socket says 172.21.15.142:48438) 2023-01-24T03:30:05.680 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.669+0000 7fd062108700 1 -- 172.21.15.142:0/899100532 learned_addr learned my addr 172.21.15.142:0/899100532 (peer_addr_for_me v2:172.21.15.142:0/0) 2023-01-24T03:30:05.680 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.669+0000 7fd062909700 1 --2- 172.21.15.142:0/899100532 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd05c157120 0x7fd05c0a9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:05.680 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.669+0000 7fd061907700 1 -- 172.21.15.142:0/899100532 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd05c06c4a0 msgr2=0x7fd05c152ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:05.681 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.669+0000 7fd061907700 1 --2- 172.21.15.142:0/899100532 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd05c06c4a0 0x7fd05c152ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.681 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.669+0000 7fd061907700 1 -- 172.21.15.142:0/899100532 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd05c157120 msgr2=0x7fd05c0a9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:05.681 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.669+0000 7fd061907700 1 --2- 172.21.15.142:0/899100532 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd05c157120 0x7fd05c0a9f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.681 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.669+0000 7fd061907700 1 -- 172.21.15.142:0/899100532 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd054009ae0 con 0x7fd05c156d50 2023-01-24T03:30:05.681 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.669+0000 7fd062108700 1 --2- 172.21.15.142:0/899100532 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd05c06c4a0 0x7fd05c152ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:30:05.681 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.669+0000 7fd062909700 1 --2- 172.21.15.142:0/899100532 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd05c157120 0x7fd05c0a9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:30:05.682 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.669+0000 7fd061907700 1 --2- 172.21.15.142:0/899100532 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd05c156d50 0x7fd05c153000 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd05800c410 tx=0x7fd058007d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:05.682 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.669+0000 7fd04a7fc700 1 -- 172.21.15.142:0/899100532 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05800cea0 con 0x7fd05c156d50 2023-01-24T03:30:05.682 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.669+0000 7fd04a7fc700 1 -- 172.21.15.142:0/899100532 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd058012420 con 0x7fd05c156d50 2023-01-24T03:30:05.682 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.669+0000 7fd06310a700 1 -- 172.21.15.142:0/899100532 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd05c165b20 con 0x7fd05c156d50 2023-01-24T03:30:05.682 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.669+0000 7fd04a7fc700 1 -- 172.21.15.142:0/899100532 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0580127c0 con 0x7fd05c156d50 2023-01-24T03:30:05.683 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.669+0000 7fd06310a700 1 -- 172.21.15.142:0/899100532 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd05c0c0580 con 0x7fd05c156d50 2023-01-24T03:30:05.683 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.670+0000 7fd04a7fc700 1 -- 172.21.15.142:0/899100532 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7fd058019030 con 0x7fd05c156d50 2023-01-24T03:30:05.683 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.670+0000 7fd04a7fc700 1 -- 172.21.15.142:0/899100532 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1296+0+0 (secure 0 0 0) 0x7fd05801b930 con 0x7fd05c156d50 2023-01-24T03:30:05.683 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.670+0000 7fd06310a700 1 -- 172.21.15.142:0/899100532 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd05c05ea20 con 0x7fd05c156d50 2023-01-24T03:30:05.683 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.672+0000 7fd04a7fc700 1 -- 172.21.15.142:0/899100532 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70685 (secure 0 0 0) 0x7fd058017070 con 0x7fd05c156d50 2023-01-24T03:30:05.714 INFO:tasks.ceph.mgr.y.smithi112.stderr:2023-01-24T03:30:05.711+0000 7f9fbb5a0000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-01-24T03:30:05.756 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.754+0000 7fd06310a700 1 -- 172.21.15.142:0/899100532 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd new", "uuid": "d3bc0030-8b08-4645-9ec2-1f8a4e29ef36", "id": 5} v 0) v1 -- 0x7fd05c04b340 con 0x7fd05c156d50 2023-01-24T03:30:05.762 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.760+0000 7fd04a7fc700 1 -- 172.21.15.142:0/899100532 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d3bc0030-8b08-4645-9ec2-1f8a4e29ef36", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd05801c440 con 0x7fd05c156d50 2023-01-24T03:30:05.762 INFO:teuthology.orchestra.run.smithi142.stdout:5 2023-01-24T03:30:05.763 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.761+0000 7fd06310a700 1 -- 172.21.15.142:0/899100532 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd05c156d50 msgr2=0x7fd05c153000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:05.764 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.761+0000 7fd06310a700 1 --2- 172.21.15.142:0/899100532 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd05c156d50 0x7fd05c153000 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd05800c410 tx=0x7fd058007d40 comp rx=0 tx=0).stop 2023-01-24T03:30:05.764 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.761+0000 7fd06310a700 1 -- 172.21.15.142:0/899100532 shutdown_connections 2023-01-24T03:30:05.764 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.761+0000 7fd06310a700 1 --2- 172.21.15.142:0/899100532 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd05c157120 0x7fd05c0a9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.764 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.761+0000 7fd06310a700 1 --2- 172.21.15.142:0/899100532 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd05c06c4a0 0x7fd05c152ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.764 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.761+0000 7fd06310a700 1 --2- 172.21.15.142:0/899100532 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd05c156d50 0x7fd05c153000 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.765 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.761+0000 7fd06310a700 1 -- 172.21.15.142:0/899100532 >> 172.21.15.142:0/899100532 conn(0x7fd05c06ce00 msgr2=0x7fd05c0c8010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:05.765 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.761+0000 7fd06310a700 1 -- 172.21.15.142:0/899100532 shutdown_connections 2023-01-24T03:30:05.765 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.761+0000 7fd06310a700 1 -- 172.21.15.142:0/899100532 wait complete. 2023-01-24T03:30:05.775 DEBUG:teuthology.orchestra.run.smithi142:> sudo ceph --cluster ceph osd new c7649e3b-832d-4e6f-8866-b15e5d814476 6 2023-01-24T03:30:05.778 INFO:tasks.ceph.mgr.x.smithi142.stderr:2023-01-24T03:30:05.776+0000 7f82fde28000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-01-24T03:30:05.806 INFO:tasks.ceph.mgr.y.smithi112.stderr:2023-01-24T03:30:05.803+0000 7f9fbb5a0000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-01-24T03:30:05.877 INFO:tasks.ceph.mgr.x.smithi142.stderr:2023-01-24T03:30:05.875+0000 7f82fde28000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-01-24T03:30:05.893 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.891+0000 7f038d17b700 1 Processor -- start 2023-01-24T03:30:05.893 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.891+0000 7f038d17b700 1 -- start start 2023-01-24T03:30:05.897 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.892+0000 7f038d17b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03880b84d0 0x7f03880b88a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:05.897 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.892+0000 7f038d17b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f038806ec80 0x7f038814ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:05.897 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.892+0000 7f038d17b700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f038806c090 con 0x7f03880c8c60 2023-01-24T03:30:05.897 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.892+0000 7f038d17b700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03880c4380 con 0x7f038806ec80 2023-01-24T03:30:05.897 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.892+0000 7f038d17b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03880c4640 con 0x7f03880b84d0 2023-01-24T03:30:05.898 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.892+0000 7f0387fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03880b84d0 0x7f03880b88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:05.898 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.892+0000 7f038c97a700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f038806ec80 0x7f038814ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:05.898 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.892+0000 7f0387fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03880b84d0 0x7f03880b88a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:41774/0 (socket says 172.21.15.142:41774) 2023-01-24T03:30:05.898 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.892+0000 7f0387fff700 1 -- 172.21.15.142:0/862991099 learned_addr learned my addr 172.21.15.142:0/862991099 (peer_addr_for_me v2:172.21.15.142:0/0) 2023-01-24T03:30:05.898 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.892+0000 7f0387fff700 1 -- 172.21.15.142:0/862991099 >> v2:172.21.15.142:3300/0 conn(0x7f038806ec80 msgr2=0x7f038814ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:05.899 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.892+0000 7f0387fff700 1 --2- 172.21.15.142:0/862991099 >> v2:172.21.15.142:3300/0 conn(0x7f038806ec80 0x7f038814ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.899 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.892+0000 7f0387fff700 1 -- 172.21.15.142:0/862991099 >> v1:172.21.15.112:6789/0 conn(0x7f03880c8c60 legacy=0x7f038806c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:05.899 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.892+0000 7f0387fff700 1 -- 172.21.15.142:0/862991099 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03880c4920 con 0x7f03880b84d0 2023-01-24T03:30:05.899 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.892+0000 7f038c97a700 1 --2- 172.21.15.142:0/862991099 >> v2:172.21.15.142:3300/0 conn(0x7f038806ec80 0x7f038814ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:30:05.899 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.892+0000 7f0387fff700 1 --2- 172.21.15.142:0/862991099 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03880b84d0 0x7f03880b88a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f037c006fd0 tx=0x7f037c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14d15c7026d25f61 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:05.900 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.892+0000 7f0386ffd700 1 -- 172.21.15.142:0/862991099 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037c011630 con 0x7f03880b84d0 2023-01-24T03:30:05.900 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.892+0000 7f0386ffd700 1 -- 172.21.15.142:0/862991099 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f037c0117d0 con 0x7f03880b84d0 2023-01-24T03:30:05.900 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.892+0000 7f0386ffd700 1 -- 172.21.15.142:0/862991099 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037c011b90 con 0x7f03880b84d0 2023-01-24T03:30:05.900 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.893+0000 7f038d17b700 1 -- 172.21.15.142:0/862991099 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03880b84d0 msgr2=0x7f03880b88a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:05.900 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.893+0000 7f038d17b700 1 --2- 172.21.15.142:0/862991099 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03880b84d0 0x7f03880b88a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f037c006fd0 tx=0x7f037c0090f0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.901 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.893+0000 7f038d17b700 1 -- 172.21.15.142:0/862991099 shutdown_connections 2023-01-24T03:30:05.901 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.893+0000 7f038d17b700 1 --2- 172.21.15.142:0/862991099 >> v2:172.21.15.142:3300/0 conn(0x7f038806ec80 0x7f038814ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.901 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.893+0000 7f038d17b700 1 --2- 172.21.15.142:0/862991099 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03880b84d0 0x7f03880b88a0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.901 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.893+0000 7f038d17b700 1 -- 172.21.15.142:0/862991099 >> 172.21.15.142:0/862991099 conn(0x7f03880689f0 msgr2=0x7f03880c8550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:05.901 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.893+0000 7f038d17b700 1 -- 172.21.15.142:0/862991099 shutdown_connections 2023-01-24T03:30:05.902 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.893+0000 7f038d17b700 1 -- 172.21.15.142:0/862991099 wait complete. 2023-01-24T03:30:05.902 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.894+0000 7f038d17b700 1 Processor -- start 2023-01-24T03:30:05.902 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.894+0000 7f038d17b700 1 -- start start 2023-01-24T03:30:05.904 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.895+0000 7f038d17b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f038806ec80 0x7f03881678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:05.904 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.895+0000 7f038d17b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f03880b84d0 0x7f0388167e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:05.905 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.895+0000 7f038d17b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03880c8c60 0x7f0388168ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:05.905 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.895+0000 7f038d17b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03880bf4e0 con 0x7f03880b84d0 2023-01-24T03:30:05.906 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.895+0000 7f038d17b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03880bed50 con 0x7f03880c8c60 2023-01-24T03:30:05.906 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.895+0000 7f038d17b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0388167450 con 0x7f038806ec80 2023-01-24T03:30:05.906 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.895+0000 7f0387fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f038806ec80 0x7f03881678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:05.906 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.895+0000 7f038c97a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03880c8c60 0x7f0388168ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:05.906 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.895+0000 7f0387fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f038806ec80 0x7f03881678e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:41780/0 (socket says 172.21.15.142:41780) 2023-01-24T03:30:05.907 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.895+0000 7f0387fff700 1 -- 172.21.15.142:0/1360031904 learned_addr learned my addr 172.21.15.142:0/1360031904 (peer_addr_for_me v2:172.21.15.142:0/0) 2023-01-24T03:30:05.907 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.895+0000 7f03877fe700 1 --2- 172.21.15.142:0/1360031904 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f03880b84d0 0x7f0388167e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:05.907 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.895+0000 7f038c97a700 1 -- 172.21.15.142:0/1360031904 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f038806ec80 msgr2=0x7f03881678e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:05.907 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.895+0000 7f038c97a700 1 --2- 172.21.15.142:0/1360031904 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f038806ec80 0x7f03881678e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.907 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.895+0000 7f038c97a700 1 -- 172.21.15.142:0/1360031904 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f03880b84d0 msgr2=0x7f0388167e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:05.908 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.895+0000 7f038c97a700 1 --2- 172.21.15.142:0/1360031904 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f03880b84d0 0x7f0388167e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.908 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.895+0000 7f038c97a700 1 -- 172.21.15.142:0/1360031904 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0380009a20 con 0x7f03880c8c60 2023-01-24T03:30:05.908 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.895+0000 7f0387fff700 1 --2- 172.21.15.142:0/1360031904 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f038806ec80 0x7f03881678e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:30:05.908 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.896+0000 7f038c97a700 1 --2- 172.21.15.142:0/1360031904 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03880c8c60 0x7f0388168ee0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0380009f10 tx=0x7f038000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:05.908 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.896+0000 7f036ffff700 1 -- 172.21.15.142:0/1360031904 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f038000cb70 con 0x7f03880c8c60 2023-01-24T03:30:05.908 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.896+0000 7f036ffff700 1 -- 172.21.15.142:0/1360031904 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f038000f040 con 0x7f03880c8c60 2023-01-24T03:30:05.909 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.896+0000 7f036ffff700 1 -- 172.21.15.142:0/1360031904 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0380014480 con 0x7f03880c8c60 2023-01-24T03:30:05.909 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.896+0000 7f038d17b700 1 -- 172.21.15.142:0/1360031904 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f037c009370 con 0x7f03880c8c60 2023-01-24T03:30:05.909 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.896+0000 7f038d17b700 1 -- 172.21.15.142:0/1360031904 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03880bf200 con 0x7f03880c8c60 2023-01-24T03:30:05.909 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.897+0000 7f036ffff700 1 -- 172.21.15.142:0/1360031904 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7f0380014a70 con 0x7f03880c8c60 2023-01-24T03:30:05.909 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.897+0000 7f038d17b700 1 -- 172.21.15.142:0/1360031904 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03881675c0 con 0x7f03880c8c60 2023-01-24T03:30:05.910 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.897+0000 7f036ffff700 1 -- 172.21.15.142:0/1360031904 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7f03800139e0 con 0x7f03880c8c60 2023-01-24T03:30:05.910 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.899+0000 7f036ffff700 1 -- 172.21.15.142:0/1360031904 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70685 (secure 0 0 0) 0x7f0380018030 con 0x7f03880c8c60 2023-01-24T03:30:05.982 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.980+0000 7f038d17b700 1 -- 172.21.15.142:0/1360031904 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c7649e3b-832d-4e6f-8866-b15e5d814476", "id": 6} v 0) v1 -- 0x7f038805ea20 con 0x7f03880c8c60 2023-01-24T03:30:05.986 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.984+0000 7f036ffff700 1 -- 172.21.15.142:0/1360031904 <== mon.1 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c7649e3b-832d-4e6f-8866-b15e5d814476", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0380013ba0 con 0x7f03880c8c60 2023-01-24T03:30:05.986 INFO:teuthology.orchestra.run.smithi142.stdout:6 2023-01-24T03:30:05.987 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.985+0000 7f038d17b700 1 -- 172.21.15.142:0/1360031904 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03880c8c60 msgr2=0x7f0388168ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:05.988 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.985+0000 7f038d17b700 1 --2- 172.21.15.142:0/1360031904 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03880c8c60 0x7f0388168ee0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0380009f10 tx=0x7f038000b040 comp rx=0 tx=0).stop 2023-01-24T03:30:05.988 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.985+0000 7f038d17b700 1 -- 172.21.15.142:0/1360031904 shutdown_connections 2023-01-24T03:30:05.988 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.985+0000 7f038d17b700 1 --2- 172.21.15.142:0/1360031904 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03880c8c60 0x7f0388168ee0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.988 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.985+0000 7f038d17b700 1 --2- 172.21.15.142:0/1360031904 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f038806ec80 0x7f03881678e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.988 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.985+0000 7f038d17b700 1 --2- 172.21.15.142:0/1360031904 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f03880b84d0 0x7f0388167e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:05.989 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.985+0000 7f038d17b700 1 -- 172.21.15.142:0/1360031904 >> 172.21.15.142:0/1360031904 conn(0x7f03880689f0 msgr2=0x7f0388157cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:05.989 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.985+0000 7f038d17b700 1 -- 172.21.15.142:0/1360031904 shutdown_connections 2023-01-24T03:30:05.989 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:05.985+0000 7f038d17b700 1 -- 172.21.15.142:0/1360031904 wait complete. 2023-01-24T03:30:05.998 DEBUG:teuthology.orchestra.run.smithi142:> sudo ceph --cluster ceph osd new 2ea5b3bf-884e-4b20-b572-1443efdc7477 7 2023-01-24T03:30:06.111 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.108+0000 7f0cffffe700 1 Processor -- start 2023-01-24T03:30:06.111 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.109+0000 7f0cffffe700 1 -- start start 2023-01-24T03:30:06.117 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.109+0000 7f0cffffe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0cf80694c0 0x7f0cf8069890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:06.117 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.109+0000 7f0cffffe700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf8155ab0 con 0x7f0cf805e420 2023-01-24T03:30:06.117 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.109+0000 7f0cffffe700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf80c70e0 con 0x7f0cf8068d20 2023-01-24T03:30:06.118 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.109+0000 7f0cffffe700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf8155550 con 0x7f0cf80694c0 2023-01-24T03:30:06.118 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.109+0000 7f0cfdd9a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0cf80694c0 0x7f0cf8069890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:06.119 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.109+0000 7f0cfe59b700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f0cf8068d20 0x7f0cf80ae9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.142:41268/0 (socket says 172.21.15.142:41268) 2023-01-24T03:30:06.119 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.109+0000 7f0cfe59b700 1 -- 172.21.15.142:0/3343393422 learned_addr learned my addr 172.21.15.142:0/3343393422 (peer_addr_for_me v1:172.21.15.142:0/0) 2023-01-24T03:30:06.119 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.110+0000 7f0cfcd98700 1 -- 172.21.15.142:0/3343393422 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1995878720 0 0) 0x7f0cf80c70e0 con 0x7f0cf8068d20 2023-01-24T03:30:06.120 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.110+0000 7f0cfdd9a700 1 -- 172.21.15.142:0/3343393422 >> v1:172.21.15.142:6789/0 conn(0x7f0cf8068d20 legacy=0x7f0cf80ae9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:06.120 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.110+0000 7f0cfdd9a700 1 -- 172.21.15.142:0/3343393422 >> v1:172.21.15.112:6789/0 conn(0x7f0cf805e420 legacy=0x7f0cf80c68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:06.120 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.110+0000 7f0cfdd9a700 1 -- 172.21.15.142:0/3343393422 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf80c73c0 con 0x7f0cf80694c0 2023-01-24T03:30:06.121 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.110+0000 7f0cfdd9a700 1 --2- 172.21.15.142:0/3343393422 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0cf80694c0 0x7f0cf8069890 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8009e90 tx=0x7f0ce8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b43900bac3dcad1c server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:06.121 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.110+0000 7f0cfcd98700 1 -- 172.21.15.142:0/3343393422 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce800e070 con 0x7f0cf80694c0 2023-01-24T03:30:06.121 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.110+0000 7f0cfcd98700 1 -- 172.21.15.142:0/3343393422 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce800bc30 con 0x7f0cf80694c0 2023-01-24T03:30:06.121 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.110+0000 7f0cfcd98700 1 -- 172.21.15.142:0/3343393422 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce8012430 con 0x7f0cf80694c0 2023-01-24T03:30:06.121 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.110+0000 7f0cffffe700 1 -- 172.21.15.142:0/3343393422 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0cf80694c0 msgr2=0x7f0cf8069890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:06.122 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.110+0000 7f0cffffe700 1 --2- 172.21.15.142:0/3343393422 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0cf80694c0 0x7f0cf8069890 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8009e90 tx=0x7f0ce8009780 comp rx=0 tx=0).stop 2023-01-24T03:30:06.122 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.110+0000 7f0cffffe700 1 -- 172.21.15.142:0/3343393422 shutdown_connections 2023-01-24T03:30:06.122 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.110+0000 7f0cffffe700 1 --2- 172.21.15.142:0/3343393422 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0cf80694c0 0x7f0cf8069890 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:06.123 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.110+0000 7f0cffffe700 1 -- 172.21.15.142:0/3343393422 >> 172.21.15.142:0/3343393422 conn(0x7f0cf805d830 msgr2=0x7f0cf805dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:06.123 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.111+0000 7f0cffffe700 1 -- 172.21.15.142:0/3343393422 shutdown_connections 2023-01-24T03:30:06.123 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.111+0000 7f0cffffe700 1 -- 172.21.15.142:0/3343393422 wait complete. 2023-01-24T03:30:06.123 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.112+0000 7f0cffffe700 1 Processor -- start 2023-01-24T03:30:06.124 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.112+0000 7f0cffffe700 1 -- start start 2023-01-24T03:30:06.124 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.112+0000 7f0cffffe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0cf805e420 0x7f0cf8169640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:06.124 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.112+0000 7f0cffffe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0cf8068d20 0x7f0cf80aeb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:06.125 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.112+0000 7f0cffffe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0cf80694c0 0x7f0cf814ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:06.125 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.112+0000 7f0cffffe700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf81593c0 con 0x7f0cf8068d20 2023-01-24T03:30:06.125 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.112+0000 7f0cffffe700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf8152440 con 0x7f0cf805e420 2023-01-24T03:30:06.125 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.112+0000 7f0cffffe700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf8152de0 con 0x7f0cf80694c0 2023-01-24T03:30:06.126 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.112+0000 7f0cfe59b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0cf80694c0 0x7f0cf814ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:06.126 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.112+0000 7f0cfdd9a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0cf805e420 0x7f0cf8169640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:06.126 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.112+0000 7f0cfe59b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0cf80694c0 0x7f0cf814ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.142:41794/0 (socket says 172.21.15.142:41794) 2023-01-24T03:30:06.126 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.112+0000 7f0cfe59b700 1 -- 172.21.15.142:0/1691789398 learned_addr learned my addr 172.21.15.142:0/1691789398 (peer_addr_for_me v2:172.21.15.142:0/0) 2023-01-24T03:30:06.126 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.112+0000 7f0cfd599700 1 --2- 172.21.15.142:0/1691789398 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0cf8068d20 0x7f0cf80aeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:06.127 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.113+0000 7f0cfdd9a700 1 -- 172.21.15.142:0/1691789398 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0cf80694c0 msgr2=0x7f0cf814ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:06.127 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.113+0000 7f0cfdd9a700 1 --2- 172.21.15.142:0/1691789398 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0cf80694c0 0x7f0cf814ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:06.127 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.113+0000 7f0cfdd9a700 1 -- 172.21.15.142:0/1691789398 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0cf8068d20 msgr2=0x7f0cf80aeb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:06.128 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.113+0000 7f0cfdd9a700 1 --2- 172.21.15.142:0/1691789398 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0cf8068d20 0x7f0cf80aeb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:06.128 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.113+0000 7f0cfdd9a700 1 -- 172.21.15.142:0/1691789398 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce8009a20 con 0x7f0cf805e420 2023-01-24T03:30:06.128 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.113+0000 7f0cfe59b700 1 --2- 172.21.15.142:0/1691789398 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0cf80694c0 0x7f0cf814ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:30:06.129 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.113+0000 7f0cfdd9a700 1 --2- 172.21.15.142:0/1691789398 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0cf805e420 0x7f0cf8169640 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8016000 tx=0x7f0ce8019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:06.129 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.113+0000 7f0cf5ffb700 1 -- 172.21.15.142:0/1691789398 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce8012db0 con 0x7f0cf805e420 2023-01-24T03:30:06.129 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.113+0000 7f0cf5ffb700 1 -- 172.21.15.142:0/1691789398 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce801b420 con 0x7f0cf805e420 2023-01-24T03:30:06.130 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.113+0000 7f0cf5ffb700 1 -- 172.21.15.142:0/1691789398 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce801b7e0 con 0x7f0cf805e420 2023-01-24T03:30:06.130 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.113+0000 7f0cffffe700 1 -- 172.21.15.142:0/1691789398 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf81691b0 con 0x7f0cf805e420 2023-01-24T03:30:06.130 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.113+0000 7f0cffffe700 1 -- 172.21.15.142:0/1691789398 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf8152820 con 0x7f0cf805e420 2023-01-24T03:30:06.131 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.113+0000 7f0cf5ffb700 1 -- 172.21.15.142:0/1691789398 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7f0ce801bbd0 con 0x7f0cf805e420 2023-01-24T03:30:06.131 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.113+0000 7f0cf5ffb700 1 -- 172.21.15.142:0/1691789398 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1526+0+0 (secure 0 0 0) 0x7f0ce801ce30 con 0x7f0cf805e420 2023-01-24T03:30:06.132 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.114+0000 7f0cffffe700 1 -- 172.21.15.142:0/1691789398 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cf8169490 con 0x7f0cf805e420 2023-01-24T03:30:06.132 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.116+0000 7f0cf5ffb700 1 -- 172.21.15.142:0/1691789398 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70685 (secure 0 0 0) 0x7f0ce802a070 con 0x7f0cf805e420 2023-01-24T03:30:06.198 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.196+0000 7f0cffffe700 1 -- 172.21.15.142:0/1691789398 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2ea5b3bf-884e-4b20-b572-1443efdc7477", "id": 7} v 0) v1 -- 0x7f0cf804b340 con 0x7f0cf805e420 2023-01-24T03:30:06.203 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.200+0000 7f0cf5ffb700 1 -- 172.21.15.142:0/1691789398 <== mon.1 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2ea5b3bf-884e-4b20-b572-1443efdc7477", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0ce801c7c0 con 0x7f0cf805e420 2023-01-24T03:30:06.203 INFO:teuthology.orchestra.run.smithi142.stdout:7 2023-01-24T03:30:06.205 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.202+0000 7f0cffffe700 1 -- 172.21.15.142:0/1691789398 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0cf805e420 msgr2=0x7f0cf8169640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:06.205 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.202+0000 7f0cffffe700 1 --2- 172.21.15.142:0/1691789398 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0cf805e420 0x7f0cf8169640 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8016000 tx=0x7f0ce8019000 comp rx=0 tx=0).stop 2023-01-24T03:30:06.206 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.202+0000 7f0cffffe700 1 -- 172.21.15.142:0/1691789398 shutdown_connections 2023-01-24T03:30:06.206 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.202+0000 7f0cffffe700 1 --2- 172.21.15.142:0/1691789398 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0cf80694c0 0x7f0cf814ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:06.206 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.202+0000 7f0cffffe700 1 --2- 172.21.15.142:0/1691789398 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0cf805e420 0x7f0cf8169640 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:06.207 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.202+0000 7f0cffffe700 1 --2- 172.21.15.142:0/1691789398 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0cf8068d20 0x7f0cf80aeb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:06.207 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.202+0000 7f0cffffe700 1 -- 172.21.15.142:0/1691789398 >> 172.21.15.142:0/1691789398 conn(0x7f0cf805d830 msgr2=0x7f0cf8159ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:06.208 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.202+0000 7f0cffffe700 1 -- 172.21.15.142:0/1691789398 shutdown_connections 2023-01-24T03:30:06.208 INFO:teuthology.orchestra.run.smithi142.stderr:2023-01-24T03:30:06.202+0000 7f0cffffe700 1 -- 172.21.15.142:0/1691789398 wait complete. 2023-01-24T03:30:06.217 INFO:tasks.ceph.osd.0:Restarting daemon 2023-01-24T03:30:06.217 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-01-24T03:30:06.220 INFO:tasks.ceph.osd.0:Started 2023-01-24T03:30:06.220 INFO:tasks.ceph.osd.1:Restarting daemon 2023-01-24T03:30:06.221 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-01-24T03:30:06.222 INFO:tasks.ceph.osd.1:Started 2023-01-24T03:30:06.223 INFO:tasks.ceph.osd.2:Restarting daemon 2023-01-24T03:30:06.223 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-01-24T03:30:06.225 INFO:tasks.ceph.osd.2:Started 2023-01-24T03:30:06.225 INFO:tasks.ceph.osd.3:Restarting daemon 2023-01-24T03:30:06.226 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-01-24T03:30:06.228 INFO:tasks.ceph.osd.3:Started 2023-01-24T03:30:06.228 INFO:tasks.ceph.osd.4:Restarting daemon 2023-01-24T03:30:06.229 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-01-24T03:30:06.232 INFO:tasks.ceph.osd.4:Started 2023-01-24T03:30:06.233 INFO:tasks.ceph.osd.5:Restarting daemon 2023-01-24T03:30:06.233 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-01-24T03:30:06.235 INFO:tasks.ceph.osd.5:Started 2023-01-24T03:30:06.235 INFO:tasks.ceph.osd.6:Restarting daemon 2023-01-24T03:30:06.236 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-01-24T03:30:06.238 INFO:tasks.ceph.osd.6:Started 2023-01-24T03:30:06.238 INFO:tasks.ceph.osd.7:Restarting daemon 2023-01-24T03:30:06.239 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-01-24T03:30:06.241 INFO:tasks.ceph.osd.7:Started 2023-01-24T03:30:06.241 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T03:30:06.331 INFO:tasks.ceph.mgr.y.smithi112.stderr:2023-01-24T03:30:06.328+0000 7f9fbb5a0000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-01-24T03:30:06.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.379+0000 7f1a6ea1b700 1 Processor -- start 2023-01-24T03:30:06.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.379+0000 7f1a6ea1b700 1 -- start start 2023-01-24T03:30:06.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.380+0000 7f1a6ea1b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1a6814c660 0x7f1a680c1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:06.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.380+0000 7f1a6ea1b700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a680c18f0 con 0x7f1a680c22c0 2023-01-24T03:30:06.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.380+0000 7f1a6ea1b700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a680c1030 con 0x7f1a6814c290 2023-01-24T03:30:06.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.380+0000 7f1a6ea1b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a680c2740 con 0x7f1a6814c660 2023-01-24T03:30:06.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.380+0000 7f1a6cfb8700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f1a680c22c0 0x7f1a68155710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:58696/0 (socket says 172.21.15.112:58696) 2023-01-24T03:30:06.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.380+0000 7f1a6cfb8700 1 -- 172.21.15.112:0/1804194290 learned_addr learned my addr 172.21.15.112:0/1804194290 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:06.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.380+0000 7f1a677fe700 1 --2- 172.21.15.112:0/1804194290 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1a6814c660 0x7f1a680c1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:06.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.380+0000 7f1a66ffd700 1 -- 172.21.15.112:0/1804194290 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4042366976 0 0) 0x7f1a680c18f0 con 0x7f1a680c22c0 2023-01-24T03:30:06.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.380+0000 7f1a66ffd700 1 -- 172.21.15.112:0/1804194290 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a540033f0 con 0x7f1a680c22c0 2023-01-24T03:30:06.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.380+0000 7f1a66ffd700 1 -- 172.21.15.112:0/1804194290 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2221980975 0 0) 0x7f1a680c1030 con 0x7f1a6814c290 2023-01-24T03:30:06.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.380+0000 7f1a66ffd700 1 -- 172.21.15.112:0/1804194290 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a680c18f0 con 0x7f1a6814c290 2023-01-24T03:30:06.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.380+0000 7f1a677fe700 1 -- 172.21.15.112:0/1804194290 >> v1:172.21.15.142:6789/0 conn(0x7f1a6814c290 legacy=0x7f1a6806ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:06.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.380+0000 7f1a677fe700 1 -- 172.21.15.112:0/1804194290 >> v1:172.21.15.112:6789/0 conn(0x7f1a680c22c0 legacy=0x7f1a68155710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:06.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.380+0000 7f1a677fe700 1 -- 172.21.15.112:0/1804194290 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a6806bad0 con 0x7f1a6814c660 2023-01-24T03:30:06.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.381+0000 7f1a677fe700 1 --2- 172.21.15.112:0/1804194290 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1a6814c660 0x7f1a680c1d80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1a580097c0 tx=0x7f1a58009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c467683230390e67 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:06.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.381+0000 7f1a66ffd700 1 -- 172.21.15.112:0/1804194290 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a580105b0 con 0x7f1a6814c660 2023-01-24T03:30:06.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.381+0000 7f1a66ffd700 1 -- 172.21.15.112:0/1804194290 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a58010710 con 0x7f1a6814c660 2023-01-24T03:30:06.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.381+0000 7f1a66ffd700 1 -- 172.21.15.112:0/1804194290 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a58010ab0 con 0x7f1a6814c660 2023-01-24T03:30:06.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.381+0000 7f1a6ea1b700 1 -- 172.21.15.112:0/1804194290 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1a6814c660 msgr2=0x7f1a680c1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:06.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.381+0000 7f1a6ea1b700 1 --2- 172.21.15.112:0/1804194290 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1a6814c660 0x7f1a680c1d80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1a580097c0 tx=0x7f1a58009060 comp rx=0 tx=0).stop 2023-01-24T03:30:06.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.381+0000 7f1a6ea1b700 1 -- 172.21.15.112:0/1804194290 shutdown_connections 2023-01-24T03:30:06.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.381+0000 7f1a6ea1b700 1 --2- 172.21.15.112:0/1804194290 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1a6814c660 0x7f1a680c1d80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:06.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.381+0000 7f1a6ea1b700 1 -- 172.21.15.112:0/1804194290 >> 172.21.15.112:0/1804194290 conn(0x7f1a68067f20 msgr2=0x7f1a68068320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:06.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.381+0000 7f1a6ea1b700 1 -- 172.21.15.112:0/1804194290 shutdown_connections 2023-01-24T03:30:06.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.381+0000 7f1a6ea1b700 1 -- 172.21.15.112:0/1804194290 wait complete. 2023-01-24T03:30:06.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.382+0000 7f1a6ea1b700 1 Processor -- start 2023-01-24T03:30:06.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.383+0000 7f1a6ea1b700 1 -- start start 2023-01-24T03:30:06.393 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.383+0000 7f1a6ea1b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1a680c22c0 0x7f1a68163740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:06.393 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.383+0000 7f1a6ea1b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1a6814c290 0x7f1a68151340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:06.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.383+0000 7f1a6ea1b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1a6814c660 0x7f1a68157b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:06.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.383+0000 7f1a6ea1b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a68151880 con 0x7f1a680c22c0 2023-01-24T03:30:06.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.383+0000 7f1a6ea1b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a680c7a30 con 0x7f1a6814c660 2023-01-24T03:30:06.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.383+0000 7f1a6ea1b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a68162ca0 con 0x7f1a6814c290 2023-01-24T03:30:06.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.383+0000 7f1a67fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1a680c22c0 0x7f1a68163740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:06.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.383+0000 7f1a67fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1a680c22c0 0x7f1a68163740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:49478/0 (socket says 172.21.15.112:49478) 2023-01-24T03:30:06.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.383+0000 7f1a67fff700 1 -- 172.21.15.112:0/1140471127 learned_addr learned my addr 172.21.15.112:0/1140471127 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:06.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.383+0000 7f1a677fe700 1 --2- 172.21.15.112:0/1140471127 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1a6814c290 0x7f1a68151340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:06.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.383+0000 7f1a6cfb8700 1 --2- 172.21.15.112:0/1140471127 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1a6814c660 0x7f1a68157b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:06.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.384+0000 7f1a67fff700 1 -- 172.21.15.112:0/1140471127 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1a6814c290 msgr2=0x7f1a68151340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:06.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.384+0000 7f1a67fff700 1 --2- 172.21.15.112:0/1140471127 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1a6814c290 0x7f1a68151340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:06.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.384+0000 7f1a67fff700 1 -- 172.21.15.112:0/1140471127 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1a6814c660 msgr2=0x7f1a68157b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:06.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.384+0000 7f1a67fff700 1 --2- 172.21.15.112:0/1140471127 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1a6814c660 0x7f1a68157b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:06.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.384+0000 7f1a67fff700 1 -- 172.21.15.112:0/1140471127 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a580092a0 con 0x7f1a680c22c0 2023-01-24T03:30:06.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.384+0000 7f1a67fff700 1 --2- 172.21.15.112:0/1140471127 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1a680c22c0 0x7f1a68163740 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c00c490 tx=0x7f1a5c007c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:06.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.384+0000 7f1a4ffff700 1 -- 172.21.15.112:0/1140471127 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c00ce40 con 0x7f1a680c22c0 2023-01-24T03:30:06.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.384+0000 7f1a4ffff700 1 -- 172.21.15.112:0/1140471127 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a5c011590 con 0x7f1a680c22c0 2023-01-24T03:30:06.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.384+0000 7f1a4ffff700 1 -- 172.21.15.112:0/1140471127 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c011950 con 0x7f1a680c22c0 2023-01-24T03:30:06.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.384+0000 7f1a6ea1b700 1 -- 172.21.15.112:0/1140471127 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a68166c20 con 0x7f1a680c22c0 2023-01-24T03:30:06.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.384+0000 7f1a6ea1b700 1 -- 172.21.15.112:0/1140471127 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a68163220 con 0x7f1a680c22c0 2023-01-24T03:30:06.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.384+0000 7f1a4ffff700 1 -- 172.21.15.112:0/1140471127 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7f1a5c010050 con 0x7f1a680c22c0 2023-01-24T03:30:06.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.384+0000 7f1a4ffff700 1 -- 172.21.15.112:0/1140471127 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7f1a5c01ce10 con 0x7f1a680c22c0 2023-01-24T03:30:06.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.385+0000 7f1a6ea1b700 1 -- 172.21.15.112:0/1140471127 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a6804bb80 con 0x7f1a680c22c0 2023-01-24T03:30:06.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.387+0000 7f1a4ffff700 1 -- 172.21.15.112:0/1140471127 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70685 (secure 0 0 0) 0x7f1a5c011d40 con 0x7f1a680c22c0 2023-01-24T03:30:06.411 INFO:tasks.ceph.mgr.y.smithi112.stderr:2023-01-24T03:30:06.408+0000 7f9fbb5a0000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-01-24T03:30:06.428 INFO:tasks.ceph.mgr.x.smithi142.stderr:2023-01-24T03:30:06.426+0000 7f82fde28000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-01-24T03:30:06.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.454+0000 7f1a6ea1b700 1 -- 172.21.15.112:0/1140471127 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a6806c150 con 0x7f1a680c22c0 2023-01-24T03:30:06.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.455+0000 7f1a4ffff700 1 -- 172.21.15.112:0/1140471127 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7f1a5c01c730 con 0x7f1a680c22c0 2023-01-24T03:30:06.458 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:30:06.458 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":10,"fsid":"054b9ac9-b7b0-40e5-b932-6c8a955c5a72","created":"2023-01-24T03:30:03.798533+0000","modified":"2023-01-24T03:30:06.198433+0000","last_up_change":"0.000000","last_in_change":"2023-01-24T03:30:06.198433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"fe6ad231-a39a-42eb-9a3a-49d008dd35e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8526a1d2-d540-4021-9766-c4bbb30317ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"041ef8e4-4858-432d-9476-cbdb3a0dc47c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"987587cf-1357-4a70-ad85-67b189c67729","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8503512b-43db-40b8-8b2b-c86cf2e54e17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d3bc0030-8b08-4645-9ec2-1f8a4e29ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7649e3b-832d-4e6f-8866-b15e5d814476","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2ea5b3bf-884e-4b20-b572-1443efdc7477","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T03:30:06.459 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.456+0000 7f1a6ea1b700 1 -- 172.21.15.112:0/1140471127 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1a680c22c0 msgr2=0x7f1a68163740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:06.460 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.456+0000 7f1a6ea1b700 1 --2- 172.21.15.112:0/1140471127 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1a680c22c0 0x7f1a68163740 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c00c490 tx=0x7f1a5c007c90 comp rx=0 tx=0).stop 2023-01-24T03:30:06.460 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.456+0000 7f1a6ea1b700 1 -- 172.21.15.112:0/1140471127 shutdown_connections 2023-01-24T03:30:06.460 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.456+0000 7f1a6ea1b700 1 --2- 172.21.15.112:0/1140471127 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1a680c22c0 0x7f1a68163740 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:06.460 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.456+0000 7f1a6ea1b700 1 --2- 172.21.15.112:0/1140471127 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1a6814c660 0x7f1a68157b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:06.461 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.456+0000 7f1a6ea1b700 1 --2- 172.21.15.112:0/1140471127 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1a6814c290 0x7f1a68151340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:06.461 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.456+0000 7f1a6ea1b700 1 -- 172.21.15.112:0/1140471127 >> 172.21.15.112:0/1140471127 conn(0x7f1a68067f20 msgr2=0x7f1a68153520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:06.461 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.456+0000 7f1a6ea1b700 1 -- 172.21.15.112:0/1140471127 shutdown_connections 2023-01-24T03:30:06.461 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:06.456+0000 7f1a6ea1b700 1 -- 172.21.15.112:0/1140471127 wait complete. 2023-01-24T03:30:06.470 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-01-24T03:30:06.471 INFO:tasks.ceph:Waiting for OSDs to come up 2023-01-24T03:30:06.489 INFO:tasks.ceph.mgr.y.smithi112.stderr:2023-01-24T03:30:06.486+0000 7f9fbb5a0000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-01-24T03:30:06.504 INFO:tasks.ceph.mgr.x.smithi142.stderr:2023-01-24T03:30:06.502+0000 7f82fde28000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-01-24T03:30:06.580 INFO:tasks.ceph.mgr.x.smithi142.stderr:2023-01-24T03:30:06.578+0000 7f82fde28000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-01-24T03:30:06.827 INFO:tasks.ceph.osd.4.smithi142.stderr:2023-01-24T03:30:06.825+0000 7ff9810133c0 -1 Falling back to public interface 2023-01-24T03:30:06.830 INFO:tasks.ceph.osd.3.smithi112.stderr:2023-01-24T03:30:06.827+0000 7fe93956d3c0 -1 Falling back to public interface 2023-01-24T03:30:06.836 INFO:tasks.ceph.osd.7.smithi142.stderr:2023-01-24T03:30:06.834+0000 7fad094d93c0 -1 Falling back to public interface 2023-01-24T03:30:06.840 INFO:tasks.ceph.osd.1.smithi112.stderr:2023-01-24T03:30:06.837+0000 7f7a1e1803c0 -1 Falling back to public interface 2023-01-24T03:30:06.840 INFO:tasks.ceph.osd.2.smithi112.stderr:2023-01-24T03:30:06.837+0000 7f20c23083c0 -1 Falling back to public interface 2023-01-24T03:30:06.841 INFO:tasks.ceph.osd.0.smithi112.stderr:2023-01-24T03:30:06.837+0000 7f52432cb3c0 -1 Falling back to public interface 2023-01-24T03:30:06.841 INFO:tasks.ceph.osd.2.smithi112.stderr:2023-01-24T03:30:06.837+0000 7f20bc731700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6809/0: (98) Address already in use 2023-01-24T03:30:06.841 INFO:tasks.ceph.osd.2.smithi112.stderr:2023-01-24T03:30:06.838+0000 7f20bc731700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6819/0: (98) Address already in use 2023-01-24T03:30:06.843 INFO:tasks.ceph.osd.5.smithi142.stderr:2023-01-24T03:30:06.841+0000 7f007302b3c0 -1 Falling back to public interface 2023-01-24T03:30:06.855 INFO:tasks.ceph.osd.6.smithi142.stderr:2023-01-24T03:30:06.853+0000 7f5394dc83c0 -1 Falling back to public interface 2023-01-24T03:30:07.055 INFO:tasks.ceph.mgr.y.smithi112.stderr:2023-01-24T03:30:07.052+0000 7f9fbb5a0000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-01-24T03:30:07.154 INFO:tasks.ceph.mgr.x.smithi142.stderr:2023-01-24T03:30:07.151+0000 7f82fde28000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-01-24T03:30:07.190 INFO:tasks.ceph.mgr.y.smithi112.stderr:2023-01-24T03:30:07.187+0000 7f9fbb5a0000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-01-24T03:30:07.278 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-01-24T03:30:07.281 INFO:tasks.ceph.mgr.y.smithi112.stderr:2023-01-24T03:30:07.277+0000 7f9fbb5a0000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-01-24T03:30:07.288 INFO:tasks.ceph.mgr.x.smithi142.stderr:2023-01-24T03:30:07.286+0000 7f82fde28000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-01-24T03:30:07.379 INFO:tasks.ceph.mgr.x.smithi142.stderr:2023-01-24T03:30:07.376+0000 7f82fde28000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-01-24T03:30:07.388 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.385+0000 7f5eb40a1700 1 Processor -- start 2023-01-24T03:30:07.388 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.385+0000 7f5eb40a1700 1 -- start start 2023-01-24T03:30:07.391 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.386+0000 7f5eb40a1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5eac06f000 0x7f5eac1560a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:07.392 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.386+0000 7f5eb40a1700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f5eac06bfd0 0x7f5eac14efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:07.392 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.386+0000 7f5eb40a1700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eac06e460 con 0x7f5eac067720 2023-01-24T03:30:07.392 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.386+0000 7f5eb40a1700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5eac1565e0 con 0x7f5eac06bfd0 2023-01-24T03:30:07.393 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.386+0000 7f5eb40a1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eac0695f0 con 0x7f5eac06f000 2023-01-24T03:30:07.393 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.386+0000 7f5eb1e3d700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f5eac067720 0x7f5eac06e8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:58704/0 (socket says 172.21.15.112:58704) 2023-01-24T03:30:07.393 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.386+0000 7f5eb1e3d700 1 -- 172.21.15.112:0/790074770 learned_addr learned my addr 172.21.15.112:0/790074770 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:07.393 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.386+0000 7f5eb163c700 1 --2- 172.21.15.112:0/790074770 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5eac06f000 0x7f5eac1560a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:07.393 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.386+0000 7f5eb263e700 1 --2- 172.21.15.112:0/790074770 >> v2:172.21.15.142:3300/0 conn(0x7f5eac06bfd0 0x7f5eac14efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:07.394 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.386+0000 7f5eb0e3b700 1 -- 172.21.15.112:0/790074770 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2208383290 0 0) 0x7f5eac06e460 con 0x7f5eac067720 2023-01-24T03:30:07.394 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.386+0000 7f5eb0e3b700 1 -- 172.21.15.112:0/790074770 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e940033f0 con 0x7f5eac067720 2023-01-24T03:30:07.394 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.386+0000 7f5eb263e700 1 -- 172.21.15.112:0/790074770 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5eac06f000 msgr2=0x7f5eac1560a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:07.394 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.386+0000 7f5eb263e700 1 --2- 172.21.15.112:0/790074770 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5eac06f000 0x7f5eac1560a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:07.395 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.386+0000 7f5eb263e700 1 -- 172.21.15.112:0/790074770 >> v1:172.21.15.112:6789/0 conn(0x7f5eac067720 legacy=0x7f5eac06e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:07.395 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.387+0000 7f5eb263e700 1 -- 172.21.15.112:0/790074770 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eac06d890 con 0x7f5eac06bfd0 2023-01-24T03:30:07.395 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.387+0000 7f5eb163c700 1 --2- 172.21.15.112:0/790074770 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5eac06f000 0x7f5eac1560a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:30:07.395 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.387+0000 7f5eb263e700 1 --2- 172.21.15.112:0/790074770 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5eac06bfd0 0x7f5eac14efd0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5ea4009e60 tx=0x7f5ea4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3dd4a3cd2f4a44a0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:07.395 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.387+0000 7f5eb0e3b700 1 -- 172.21.15.112:0/790074770 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea400e070 con 0x7f5eac06bfd0 2023-01-24T03:30:07.396 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.387+0000 7f5eb0e3b700 1 -- 172.21.15.112:0/790074770 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ea400bab0 con 0x7f5eac06bfd0 2023-01-24T03:30:07.396 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.387+0000 7f5eb0e3b700 1 -- 172.21.15.112:0/790074770 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea400be70 con 0x7f5eac06bfd0 2023-01-24T03:30:07.396 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.387+0000 7f5eb40a1700 1 -- 172.21.15.112:0/790074770 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5eac06bfd0 msgr2=0x7f5eac14efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:07.396 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.387+0000 7f5eb40a1700 1 --2- 172.21.15.112:0/790074770 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5eac06bfd0 0x7f5eac14efd0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5ea4009e60 tx=0x7f5ea4009780 comp rx=0 tx=0).stop 2023-01-24T03:30:07.396 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.388+0000 7f5eb40a1700 1 -- 172.21.15.112:0/790074770 shutdown_connections 2023-01-24T03:30:07.397 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.388+0000 7f5eb40a1700 1 --2- 172.21.15.112:0/790074770 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5eac06bfd0 0x7f5eac14efd0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:07.397 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.388+0000 7f5eb40a1700 1 --2- 172.21.15.112:0/790074770 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5eac06f000 0x7f5eac1560a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:07.397 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.388+0000 7f5eb40a1700 1 -- 172.21.15.112:0/790074770 >> 172.21.15.112:0/790074770 conn(0x7f5eac06b4c0 msgr2=0x7f5eac06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:07.397 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.388+0000 7f5eb40a1700 1 -- 172.21.15.112:0/790074770 shutdown_connections 2023-01-24T03:30:07.397 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.388+0000 7f5eb40a1700 1 -- 172.21.15.112:0/790074770 wait complete. 2023-01-24T03:30:07.398 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.389+0000 7f5eb40a1700 1 Processor -- start 2023-01-24T03:30:07.398 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.389+0000 7f5eb40a1700 1 -- start start 2023-01-24T03:30:07.400 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.389+0000 7f5eb40a1700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5eac067720 0x7f5eac151f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:07.400 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.389+0000 7f5eb40a1700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5eac06bfd0 0x7f5eac1524c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:07.401 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.389+0000 7f5eb40a1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5eac06f000 0x7f5eac15c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:07.401 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.389+0000 7f5eb40a1700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eac151850 con 0x7f5eac06bfd0 2023-01-24T03:30:07.401 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.389+0000 7f5eb40a1700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eac159a10 con 0x7f5eac067720 2023-01-24T03:30:07.401 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.389+0000 7f5eb40a1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eac151af0 con 0x7f5eac06f000 2023-01-24T03:30:07.401 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.389+0000 7f5eb163c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5eac06bfd0 0x7f5eac1524c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:07.402 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.390+0000 7f5eb1e3d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5eac067720 0x7f5eac151f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:07.402 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.390+0000 7f5eb163c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5eac06bfd0 0x7f5eac1524c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:49492/0 (socket says 172.21.15.112:49492) 2023-01-24T03:30:07.403 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.390+0000 7f5eb163c700 1 -- 172.21.15.112:0/2837142838 learned_addr learned my addr 172.21.15.112:0/2837142838 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:07.403 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.390+0000 7f5eb263e700 1 --2- 172.21.15.112:0/2837142838 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5eac06f000 0x7f5eac15c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:07.403 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.390+0000 7f5eb1e3d700 1 -- 172.21.15.112:0/2837142838 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5eac06f000 msgr2=0x7f5eac15c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:07.403 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.390+0000 7f5eb1e3d700 1 --2- 172.21.15.112:0/2837142838 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5eac06f000 0x7f5eac15c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:07.403 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.390+0000 7f5eb1e3d700 1 -- 172.21.15.112:0/2837142838 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5eac06bfd0 msgr2=0x7f5eac1524c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:07.404 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.390+0000 7f5eb1e3d700 1 --2- 172.21.15.112:0/2837142838 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5eac06bfd0 0x7f5eac1524c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:07.404 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.390+0000 7f5eb1e3d700 1 -- 172.21.15.112:0/2837142838 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea80092a0 con 0x7f5eac067720 2023-01-24T03:30:07.404 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.390+0000 7f5eb263e700 1 --2- 172.21.15.112:0/2837142838 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5eac06f000 0x7f5eac15c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:30:07.404 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.390+0000 7f5eb1e3d700 1 --2- 172.21.15.112:0/2837142838 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5eac067720 0x7f5eac151f80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c00c410 tx=0x7f5e9c007bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:07.405 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.390+0000 7f5e99ffb700 1 -- 172.21.15.112:0/2837142838 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e9c0070d0 con 0x7f5eac067720 2023-01-24T03:30:07.405 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.390+0000 7f5e99ffb700 1 -- 172.21.15.112:0/2837142838 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e9c007250 con 0x7f5eac067720 2023-01-24T03:30:07.405 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.390+0000 7f5e99ffb700 1 -- 172.21.15.112:0/2837142838 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e9c007610 con 0x7f5eac067720 2023-01-24T03:30:07.405 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.390+0000 7f5eb40a1700 1 -- 172.21.15.112:0/2837142838 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ea4009a20 con 0x7f5eac067720 2023-01-24T03:30:07.405 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.390+0000 7f5e99ffb700 1 -- 172.21.15.112:0/2837142838 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 1) v1 ==== 517+0+0 (secure 0 0 0) 0x7f5e9c019030 con 0x7f5eac067720 2023-01-24T03:30:07.406 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.390+0000 7f5eb40a1700 1 -- 172.21.15.112:0/2837142838 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eac1500c0 con 0x7f5eac067720 2023-01-24T03:30:07.406 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.391+0000 7f5e99ffb700 1 -- 172.21.15.112:0/2837142838 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7f5e9c005e40 con 0x7f5eac067720 2023-01-24T03:30:07.406 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.391+0000 7f5eb40a1700 1 -- 172.21.15.112:0/2837142838 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eac151dd0 con 0x7f5eac067720 2023-01-24T03:30:07.406 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.393+0000 7f5e99ffb700 1 -- 172.21.15.112:0/2837142838 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+70685 (secure 0 0 0) 0x7f5e9c017070 con 0x7f5eac067720 2023-01-24T03:30:07.463 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.460+0000 7f5eb40a1700 1 -- 172.21.15.112:0/2837142838 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5eac04bb80 con 0x7f5eac067720 2023-01-24T03:30:07.464 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.461+0000 7f5e99ffb700 1 -- 172.21.15.112:0/2837142838 <== mon.1 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7f5e9c012680 con 0x7f5eac067720 2023-01-24T03:30:07.465 INFO:teuthology.misc.health.smithi112.stdout: 2023-01-24T03:30:07.465 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":10,"fsid":"054b9ac9-b7b0-40e5-b932-6c8a955c5a72","created":"2023-01-24T03:30:03.798533+0000","modified":"2023-01-24T03:30:06.198433+0000","last_up_change":"0.000000","last_in_change":"2023-01-24T03:30:06.198433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"fe6ad231-a39a-42eb-9a3a-49d008dd35e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8526a1d2-d540-4021-9766-c4bbb30317ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"041ef8e4-4858-432d-9476-cbdb3a0dc47c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"987587cf-1357-4a70-ad85-67b189c67729","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8503512b-43db-40b8-8b2b-c86cf2e54e17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d3bc0030-8b08-4645-9ec2-1f8a4e29ef36","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7649e3b-832d-4e6f-8866-b15e5d814476","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2ea5b3bf-884e-4b20-b572-1443efdc7477","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T03:30:07.466 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.462+0000 7f5eb40a1700 1 -- 172.21.15.112:0/2837142838 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5eac067720 msgr2=0x7f5eac151f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:07.466 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.462+0000 7f5eb40a1700 1 --2- 172.21.15.112:0/2837142838 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5eac067720 0x7f5eac151f80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c00c410 tx=0x7f5e9c007bd0 comp rx=0 tx=0).stop 2023-01-24T03:30:07.466 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.462+0000 7f5eb40a1700 1 -- 172.21.15.112:0/2837142838 shutdown_connections 2023-01-24T03:30:07.467 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.462+0000 7f5eb40a1700 1 --2- 172.21.15.112:0/2837142838 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5eac06f000 0x7f5eac15c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:07.467 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.462+0000 7f5eb40a1700 1 --2- 172.21.15.112:0/2837142838 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5eac067720 0x7f5eac151f80 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:07.467 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.462+0000 7f5eb40a1700 1 --2- 172.21.15.112:0/2837142838 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5eac06bfd0 0x7f5eac1524c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:07.467 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.462+0000 7f5eb40a1700 1 -- 172.21.15.112:0/2837142838 >> 172.21.15.112:0/2837142838 conn(0x7f5eac06b4c0 msgr2=0x7f5eac14eed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:07.467 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.462+0000 7f5eb40a1700 1 -- 172.21.15.112:0/2837142838 shutdown_connections 2023-01-24T03:30:07.468 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:07.462+0000 7f5eb40a1700 1 -- 172.21.15.112:0/2837142838 wait complete. 2023-01-24T03:30:07.475 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-01-24T03:30:07.529 INFO:tasks.ceph.mgr.y.smithi112.stderr:2023-01-24T03:30:07.526+0000 7f9fbb5a0000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-01-24T03:30:07.603 INFO:tasks.ceph.mgr.y.smithi112.stderr:2023-01-24T03:30:07.600+0000 7f9fbb5a0000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-01-24T03:30:07.625 INFO:tasks.ceph.mgr.x.smithi142.stderr:2023-01-24T03:30:07.623+0000 7f82fde28000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-01-24T03:30:07.703 INFO:tasks.ceph.mgr.x.smithi142.stderr:2023-01-24T03:30:07.700+0000 7f82fde28000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-01-24T03:30:08.169 INFO:tasks.ceph.mgr.y.smithi112.stderr:2023-01-24T03:30:08.166+0000 7f9fbb5a0000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-01-24T03:30:08.280 INFO:tasks.ceph.mgr.x.smithi142.stderr:2023-01-24T03:30:08.277+0000 7f82fde28000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-01-24T03:30:08.404 INFO:tasks.ceph.mgr.y.smithi112.stderr:2023-01-24T03:30:08.402+0000 7f9fbb5a0000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-01-24T03:30:08.522 INFO:tasks.ceph.mgr.x.smithi142.stderr:2023-01-24T03:30:08.520+0000 7f82fde28000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-01-24T03:30:08.533 INFO:tasks.ceph.mgr.y.smithi112.stderr:2023-01-24T03:30:08.531+0000 7f9fbb5a0000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-01-24T03:30:08.615 INFO:tasks.ceph.mgr.y.smithi112.stderr:2023-01-24T03:30:08.612+0000 7f9fbb5a0000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-01-24T03:30:08.653 INFO:tasks.ceph.mgr.x.smithi142.stderr:2023-01-24T03:30:08.650+0000 7f82fde28000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-01-24T03:30:08.733 INFO:tasks.ceph.mgr.x.smithi142.stderr:2023-01-24T03:30:08.731+0000 7f82fde28000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-01-24T03:30:08.761 INFO:tasks.ceph.mgr.y.smithi112.stderr:2023-01-24T03:30:08.759+0000 7f9fbb5a0000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-01-24T03:30:08.835 INFO:tasks.ceph.mgr.y.smithi112.stderr:2023-01-24T03:30:08.832+0000 7f9fbb5a0000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-01-24T03:30:08.883 INFO:tasks.ceph.mgr.x.smithi142.stderr:2023-01-24T03:30:08.881+0000 7f82fde28000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-01-24T03:30:08.906 INFO:tasks.ceph.mgr.y.smithi112.stderr:2023-01-24T03:30:08.903+0000 7f9fbb5a0000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-01-24T03:30:08.956 INFO:tasks.ceph.mgr.x.smithi142.stderr:2023-01-24T03:30:08.954+0000 7f82fde28000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-01-24T03:30:08.978 INFO:tasks.ceph.mgr.y.smithi112.stderr:2023-01-24T03:30:08.975+0000 7f9fbb5a0000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-01-24T03:30:08.978 INFO:tasks.ceph.mgr.y.smithi112.stderr:2023-01-24T03:30:08.975+0000 7f9fbb5a0000 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: prometheus 2023-01-24T03:30:09.032 INFO:tasks.ceph.mgr.x.smithi142.stderr:2023-01-24T03:30:09.030+0000 7f82fde28000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-01-24T03:30:09.107 INFO:tasks.ceph.mgr.x.smithi142.stderr:2023-01-24T03:30:09.105+0000 7f82fde28000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-01-24T03:30:09.108 INFO:tasks.ceph.mgr.x.smithi142.stderr:2023-01-24T03:30:09.105+0000 7f82fde28000 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: prometheus 2023-01-24T03:30:10.000 INFO:tasks.ceph.mgr.y.smithi112.stderr:2023-01-24T03:30:09.997+0000 7f9f74a70700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-01-24T03:30:10.000 INFO:tasks.ceph.mgr.y.smithi112.stderr:2023-01-24T03:30:09.997+0000 7f9f74a70700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-01-24T03:30:10.342 INFO:tasks.ceph.osd.0.smithi112.stderr:2023-01-24T03:30:10.339+0000 7f52432cb3c0 -1 osd.0 0 log_to_monitors true 2023-01-24T03:30:10.367 INFO:tasks.ceph.osd.4.smithi142.stderr:2023-01-24T03:30:10.364+0000 7ff9810133c0 -1 osd.4 0 log_to_monitors true 2023-01-24T03:30:10.376 INFO:tasks.ceph.osd.3.smithi112.stderr:2023-01-24T03:30:10.373+0000 7fe93956d3c0 -1 osd.3 0 log_to_monitors true 2023-01-24T03:30:10.377 INFO:tasks.ceph.osd.1.smithi112.stderr:2023-01-24T03:30:10.374+0000 7f7a1e1803c0 -1 osd.1 0 log_to_monitors true 2023-01-24T03:30:10.378 INFO:tasks.ceph.osd.2.smithi112.stderr:2023-01-24T03:30:10.375+0000 7f20c23083c0 -1 osd.2 0 log_to_monitors true 2023-01-24T03:30:10.381 INFO:tasks.ceph.osd.7.smithi142.stderr:2023-01-24T03:30:10.379+0000 7fad094d93c0 -1 osd.7 0 log_to_monitors true 2023-01-24T03:30:10.395 INFO:tasks.ceph.osd.5.smithi142.stderr:2023-01-24T03:30:10.393+0000 7f007302b3c0 -1 osd.5 0 log_to_monitors true 2023-01-24T03:30:10.441 INFO:tasks.ceph.osd.6.smithi142.stderr:2023-01-24T03:30:10.439+0000 7f5394dc83c0 -1 osd.6 0 log_to_monitors true 2023-01-24T03:30:12.932 INFO:tasks.ceph.osd.7.smithi142.stderr:2023-01-24T03:30:12.930+0000 7fad000fb700 -1 osd.7 0 waiting for initial osdmap 2023-01-24T03:30:12.933 INFO:tasks.ceph.osd.1.smithi112.stderr:2023-01-24T03:30:12.930+0000 7f7a14da2700 -1 osd.1 0 waiting for initial osdmap 2023-01-24T03:30:12.938 INFO:tasks.ceph.osd.1.smithi112.stderr:2023-01-24T03:30:12.935+0000 7f7a0db98700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-24T03:30:12.939 INFO:tasks.ceph.osd.7.smithi142.stderr:2023-01-24T03:30:12.936+0000 7facf8ef1700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-24T03:30:12.943 INFO:tasks.ceph.osd.2.smithi112.stderr:2023-01-24T03:30:12.940+0000 7f20b8f2a700 -1 osd.2 0 waiting for initial osdmap 2023-01-24T03:30:12.947 INFO:tasks.ceph.osd.2.smithi112.stderr:2023-01-24T03:30:12.945+0000 7f20b1d20700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-24T03:30:12.951 INFO:tasks.ceph.osd.3.smithi112.stderr:2023-01-24T03:30:12.947+0000 7fe93018f700 -1 osd.3 0 waiting for initial osdmap 2023-01-24T03:30:12.952 INFO:tasks.ceph.osd.6.smithi142.stderr:2023-01-24T03:30:12.950+0000 7f538b9ea700 -1 osd.6 0 waiting for initial osdmap 2023-01-24T03:30:12.953 INFO:tasks.ceph.osd.0.smithi112.stderr:2023-01-24T03:30:12.950+0000 7f5239eed700 -1 osd.0 0 waiting for initial osdmap 2023-01-24T03:30:12.954 INFO:tasks.ceph.osd.3.smithi112.stderr:2023-01-24T03:30:12.951+0000 7fe928f85700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-24T03:30:12.956 INFO:tasks.ceph.osd.6.smithi142.stderr:2023-01-24T03:30:12.954+0000 7f53847e0700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-24T03:30:12.956 INFO:tasks.ceph.osd.2.smithi112.stderr:2023-01-24T03:30:12.953+0000 7f20acebf700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-24T03:30:12.957 INFO:tasks.ceph.osd.0.smithi112.stderr:2023-01-24T03:30:12.954+0000 7f5232ce3700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-24T03:30:12.971 INFO:tasks.ceph.osd.4.smithi142.stderr:2023-01-24T03:30:12.967+0000 7ff977c35700 -1 osd.4 0 waiting for initial osdmap 2023-01-24T03:30:12.971 INFO:tasks.ceph.osd.5.smithi142.stderr:2023-01-24T03:30:12.969+0000 7f0069c4d700 -1 osd.5 0 waiting for initial osdmap 2023-01-24T03:30:12.973 INFO:tasks.ceph.osd.4.smithi142.stderr:2023-01-24T03:30:12.971+0000 7ff970a2b700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-24T03:30:12.975 INFO:tasks.ceph.osd.5.smithi142.stderr:2023-01-24T03:30:12.973+0000 7f0062a43700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-24T03:30:14.282 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-01-24T03:30:14.401 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.398+0000 7f782ec2d700 1 Processor -- start 2023-01-24T03:30:14.401 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.398+0000 7f782ec2d700 1 -- start start 2023-01-24T03:30:14.404 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.399+0000 7f782ec2d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7828067720 0x7f7828155f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:14.404 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.399+0000 7f782ec2d700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78280c3af0 con 0x7f78280bcf90 2023-01-24T03:30:14.404 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.399+0000 7f782ec2d700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f782806bb90 con 0x7f7828068480 2023-01-24T03:30:14.404 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.399+0000 7f782ec2d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7828156440 con 0x7f7828067720 2023-01-24T03:30:14.405 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.399+0000 7f782d1ca700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f78280bcf90 0x7f7828151fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:58950/0 (socket says 172.21.15.112:58950) 2023-01-24T03:30:14.405 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.399+0000 7f782d1ca700 1 -- 172.21.15.112:0/43956626 learned_addr learned my addr 172.21.15.112:0/43956626 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:14.405 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.399+0000 7f782c9c9700 1 --2- 172.21.15.112:0/43956626 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7828067720 0x7f7828155f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:14.405 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.399+0000 7f78277fe700 1 -- 172.21.15.112:0/43956626 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3035384529 0 0) 0x7f78280c3af0 con 0x7f78280bcf90 2023-01-24T03:30:14.405 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.399+0000 7f78277fe700 1 -- 172.21.15.112:0/43956626 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78080033f0 con 0x7f78280bcf90 2023-01-24T03:30:14.406 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.400+0000 7f78277fe700 1 -- 172.21.15.112:0/43956626 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2170858114 0 0) 0x7f7810001490 con 0x7f7828068480 2023-01-24T03:30:14.406 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.400+0000 7f78277fe700 1 -- 172.21.15.112:0/43956626 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78280c3af0 con 0x7f7828068480 2023-01-24T03:30:14.406 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.400+0000 7f782c9c9700 1 -- 172.21.15.112:0/43956626 >> v1:172.21.15.142:6789/0 conn(0x7f7828068480 legacy=0x7f78280bc880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:14.406 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.400+0000 7f782c9c9700 1 -- 172.21.15.112:0/43956626 >> v1:172.21.15.112:6789/0 conn(0x7f78280bcf90 legacy=0x7f7828151fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:14.406 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.400+0000 7f782c9c9700 1 -- 172.21.15.112:0/43956626 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78280c2c00 con 0x7f7828067720 2023-01-24T03:30:14.407 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.400+0000 7f782c9c9700 1 --2- 172.21.15.112:0/43956626 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7828067720 0x7f7828155f00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7818009790 tx=0x7f7818009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8c9d15df07f9e9 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:14.407 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.400+0000 7f78277fe700 1 -- 172.21.15.112:0/43956626 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7818011630 con 0x7f7828067720 2023-01-24T03:30:14.407 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.400+0000 7f78277fe700 1 -- 172.21.15.112:0/43956626 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7818011790 con 0x7f7828067720 2023-01-24T03:30:14.407 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.400+0000 7f78277fe700 1 -- 172.21.15.112:0/43956626 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7818011b10 con 0x7f7828067720 2023-01-24T03:30:14.407 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.401+0000 7f782ec2d700 1 -- 172.21.15.112:0/43956626 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7828067720 msgr2=0x7f7828155f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:14.408 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.401+0000 7f782ec2d700 1 --2- 172.21.15.112:0/43956626 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7828067720 0x7f7828155f00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7818009790 tx=0x7f7818009060 comp rx=0 tx=0).stop 2023-01-24T03:30:14.408 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.401+0000 7f782ec2d700 1 -- 172.21.15.112:0/43956626 shutdown_connections 2023-01-24T03:30:14.408 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.401+0000 7f782ec2d700 1 --2- 172.21.15.112:0/43956626 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7828067720 0x7f7828155f00 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:14.408 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.401+0000 7f782ec2d700 1 -- 172.21.15.112:0/43956626 >> 172.21.15.112:0/43956626 conn(0x7f78280c76f0 msgr2=0x7f78280c7af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:14.408 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.401+0000 7f782ec2d700 1 -- 172.21.15.112:0/43956626 shutdown_connections 2023-01-24T03:30:14.409 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.401+0000 7f782ec2d700 1 -- 172.21.15.112:0/43956626 wait complete. 2023-01-24T03:30:14.411 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.402+0000 7f782ec2d700 1 Processor -- start 2023-01-24T03:30:14.411 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.402+0000 7f782ec2d700 1 -- start start 2023-01-24T03:30:14.412 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.403+0000 7f782ec2d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7828067720 0x7f78281662d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:14.412 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.403+0000 7f782ec2d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7828068480 0x7f78280b9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:14.412 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.403+0000 7f782ec2d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f78280bcf90 0x7f782814efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:14.412 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.403+0000 7f782ec2d700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782806ed90 con 0x7f7828068480 2023-01-24T03:30:14.413 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.403+0000 7f782ec2d700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78280b9200 con 0x7f7828067720 2023-01-24T03:30:14.413 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.403+0000 7f782ec2d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78280b8310 con 0x7f78280bcf90 2023-01-24T03:30:14.413 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.403+0000 7f7827fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7828068480 0x7f78280b9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:14.413 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.403+0000 7f7827fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7828068480 0x7f78280b9690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:49736/0 (socket says 172.21.15.112:49736) 2023-01-24T03:30:14.413 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.403+0000 7f7827fff700 1 -- 172.21.15.112:0/1564371814 learned_addr learned my addr 172.21.15.112:0/1564371814 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:14.414 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.403+0000 7f782c9c9700 1 --2- 172.21.15.112:0/1564371814 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7828067720 0x7f78281662d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:14.414 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.403+0000 7f782d1ca700 1 --2- 172.21.15.112:0/1564371814 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f78280bcf90 0x7f782814efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:14.414 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.403+0000 7f7827fff700 1 -- 172.21.15.112:0/1564371814 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f78280bcf90 msgr2=0x7f782814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:14.414 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.403+0000 7f7827fff700 1 --2- 172.21.15.112:0/1564371814 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f78280bcf90 0x7f782814efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:14.414 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.403+0000 7f7827fff700 1 -- 172.21.15.112:0/1564371814 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7828067720 msgr2=0x7f78281662d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:14.415 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.403+0000 7f7827fff700 1 --2- 172.21.15.112:0/1564371814 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7828067720 0x7f78281662d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:14.415 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.403+0000 7f7827fff700 1 -- 172.21.15.112:0/1564371814 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78180092a0 con 0x7f7828068480 2023-01-24T03:30:14.415 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.403+0000 7f782d1ca700 1 --2- 172.21.15.112:0/1564371814 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f78280bcf90 0x7f782814efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-01-24T03:30:14.415 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.403+0000 7f7827fff700 1 --2- 172.21.15.112:0/1564371814 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7828068480 0x7f78280b9690 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f781000c790 tx=0x7f7810007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:14.415 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.403+0000 7f7824ff9700 1 -- 172.21.15.112:0/1564371814 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7810001ac0 con 0x7f7828068480 2023-01-24T03:30:14.416 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.403+0000 7f7824ff9700 1 -- 172.21.15.112:0/1564371814 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7810001c20 con 0x7f7828068480 2023-01-24T03:30:14.416 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.403+0000 7f782ec2d700 1 -- 172.21.15.112:0/1564371814 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78280b85f0 con 0x7f7828068480 2023-01-24T03:30:14.416 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.403+0000 7f7824ff9700 1 -- 172.21.15.112:0/1564371814 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7810002050 con 0x7f7828068480 2023-01-24T03:30:14.416 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.403+0000 7f782ec2d700 1 -- 172.21.15.112:0/1564371814 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78280b8bb0 con 0x7f7828068480 2023-01-24T03:30:14.416 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.404+0000 7f782ec2d700 1 -- 172.21.15.112:0/1564371814 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f782804bb80 con 0x7f7828068480 2023-01-24T03:30:14.417 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.405+0000 7f7824ff9700 1 -- 172.21.15.112:0/1564371814 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 4) v1 ==== 79852+0+0 (secure 0 0 0) 0x7f7810010070 con 0x7f7828068480 2023-01-24T03:30:14.417 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.405+0000 7f7824ff9700 1 --2- 172.21.15.112:0/1564371814 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f780c061170 0x7f780c063620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:14.417 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.405+0000 7f7824ff9700 1 -- 172.21.15.112:0/1564371814 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4467+0+0 (secure 0 0 0) 0x7f781007cb30 con 0x7f7828068480 2023-01-24T03:30:14.417 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.405+0000 7f782c9c9700 1 --2- 172.21.15.112:0/1564371814 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f780c061170 0x7f780c063620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:14.417 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.405+0000 7f782c9c9700 1 --2- 172.21.15.112:0/1564371814 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f780c061170 0x7f780c063620 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f78180096e0 tx=0x7f7818017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:14.418 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.409+0000 7f7824ff9700 1 -- 172.21.15.112:0/1564371814 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f781004d2e0 con 0x7f7828068480 2023-01-24T03:30:14.598 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.595+0000 7f782ec2d700 1 -- 172.21.15.112:0/1564371814 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f782806bb90 con 0x7f7828068480 2023-01-24T03:30:14.599 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.596+0000 7f7824ff9700 1 -- 172.21.15.112:0/1564371814 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+10444 (secure 0 0 0) 0x7f782806bb90 con 0x7f7828068480 2023-01-24T03:30:14.599 INFO:teuthology.misc.health.smithi112.stdout: 2023-01-24T03:30:14.599 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":13,"fsid":"054b9ac9-b7b0-40e5-b932-6c8a955c5a72","created":"2023-01-24T03:30:03.798533+0000","modified":"2023-01-24T03:30:13.945316+0000","last_up_change":"2023-01-24T03:30:13.945316+0000","last_in_change":"2023-01-24T03:30:06.198433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"fe6ad231-a39a-42eb-9a3a-49d008dd35e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6809","nonce":84512},{"type":"v1","addr":"172.21.15.112:6811","nonce":84512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":84512},{"type":"v1","addr":"172.21.15.112:6817","nonce":84512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6825","nonce":84512},{"type":"v1","addr":"172.21.15.112:6828","nonce":84512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6819","nonce":84512},{"type":"v1","addr":"172.21.15.112:6822","nonce":84512}]},"public_addr":"172.21.15.112:6811/84512","cluster_addr":"172.21.15.112:6817/84512","heartbeat_back_addr":"172.21.15.112:6828/84512","heartbeat_front_addr":"172.21.15.112:6822/84512","state":["exists","up"]},{"osd":1,"uuid":"8526a1d2-d540-4021-9766-c4bbb30317ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":84513},{"type":"v1","addr":"172.21.15.112:6812","nonce":84513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6815","nonce":84513},{"type":"v1","addr":"172.21.15.112:6818","nonce":84513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6827","nonce":84513},{"type":"v1","addr":"172.21.15.112:6830","nonce":84513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6821","nonce":84513},{"type":"v1","addr":"172.21.15.112:6824","nonce":84513}]},"public_addr":"172.21.15.112:6812/84513","cluster_addr":"172.21.15.112:6818/84513","heartbeat_back_addr":"172.21.15.112:6830/84513","heartbeat_front_addr":"172.21.15.112:6824/84513","state":["exists","up"]},{"osd":2,"uuid":"041ef8e4-4858-432d-9476-cbdb3a0dc47c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":84511},{"type":"v1","addr":"172.21.15.112:6813","nonce":84511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":84511},{"type":"v1","addr":"172.21.15.112:6820","nonce":84511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6829","nonce":84511},{"type":"v1","addr":"172.21.15.112:6831","nonce":84511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6823","nonce":84511},{"type":"v1","addr":"172.21.15.112:6826","nonce":84511}]},"public_addr":"172.21.15.112:6813/84511","cluster_addr":"172.21.15.112:6820/84511","heartbeat_back_addr":"172.21.15.112:6831/84511","heartbeat_front_addr":"172.21.15.112:6826/84511","state":["exists","up"]},{"osd":3,"uuid":"987587cf-1357-4a70-ad85-67b189c67729","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":84510},{"type":"v1","addr":"172.21.15.112:6801","nonce":84510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":84510},{"type":"v1","addr":"172.21.15.112:6803","nonce":84510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":84510},{"type":"v1","addr":"172.21.15.112:6807","nonce":84510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":84510},{"type":"v1","addr":"172.21.15.112:6805","nonce":84510}]},"public_addr":"172.21.15.112:6801/84510","cluster_addr":"172.21.15.112:6803/84510","heartbeat_back_addr":"172.21.15.112:6807/84510","heartbeat_front_addr":"172.21.15.112:6805/84510","state":["exists","up"]},{"osd":4,"uuid":"8503512b-43db-40b8-8b2b-c86cf2e54e17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":84614},{"type":"v1","addr":"172.21.15.142:6801","nonce":84614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":84614},{"type":"v1","addr":"172.21.15.142:6803","nonce":84614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":84614},{"type":"v1","addr":"172.21.15.142:6807","nonce":84614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":84614},{"type":"v1","addr":"172.21.15.142:6805","nonce":84614}]},"public_addr":"172.21.15.142:6801/84614","cluster_addr":"172.21.15.142:6803/84614","heartbeat_back_addr":"172.21.15.142:6807/84614","heartbeat_front_addr":"172.21.15.142:6805/84614","state":["exists","up"]},{"osd":5,"uuid":"d3bc0030-8b08-4645-9ec2-1f8a4e29ef36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":84638},{"type":"v1","addr":"172.21.15.142:6817","nonce":84638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":84638},{"type":"v1","addr":"172.21.15.142:6819","nonce":84638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":84638},{"type":"v1","addr":"172.21.15.142:6823","nonce":84638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":84638},{"type":"v1","addr":"172.21.15.142:6821","nonce":84638}]},"public_addr":"172.21.15.142:6817/84638","cluster_addr":"172.21.15.142:6819/84638","heartbeat_back_addr":"172.21.15.142:6823/84638","heartbeat_front_addr":"172.21.15.142:6821/84638","state":["exists","up"]},{"osd":6,"uuid":"c7649e3b-832d-4e6f-8866-b15e5d814476","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":84645},{"type":"v1","addr":"172.21.15.142:6825","nonce":84645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":84645},{"type":"v1","addr":"172.21.15.142:6827","nonce":84645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":84645},{"type":"v1","addr":"172.21.15.142:6831","nonce":84645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":84645},{"type":"v1","addr":"172.21.15.142:6829","nonce":84645}]},"public_addr":"172.21.15.142:6825/84645","cluster_addr":"172.21.15.142:6827/84645","heartbeat_back_addr":"172.21.15.142:6831/84645","heartbeat_front_addr":"172.21.15.142:6829/84645","state":["exists","up"]},{"osd":7,"uuid":"2ea5b3bf-884e-4b20-b572-1443efdc7477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":84615},{"type":"v1","addr":"172.21.15.142:6809","nonce":84615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":84615},{"type":"v1","addr":"172.21.15.142:6811","nonce":84615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":84615},{"type":"v1","addr":"172.21.15.142:6815","nonce":84615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":84615},{"type":"v1","addr":"172.21.15.142:6813","nonce":84615}]},"public_addr":"172.21.15.142:6809/84615","cluster_addr":"172.21.15.142:6811/84615","heartbeat_back_addr":"172.21.15.142:6815/84615","heartbeat_front_addr":"172.21.15.142:6813/84615","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T03:30:14.601 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.598+0000 7f782ec2d700 1 -- 172.21.15.112:0/1564371814 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f780c061170 msgr2=0x7f780c063620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:14.602 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.598+0000 7f782ec2d700 1 --2- 172.21.15.112:0/1564371814 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f780c061170 0x7f780c063620 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f78180096e0 tx=0x7f7818017000 comp rx=0 tx=0).stop 2023-01-24T03:30:14.602 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.598+0000 7f782ec2d700 1 -- 172.21.15.112:0/1564371814 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7828068480 msgr2=0x7f78280b9690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:14.602 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.598+0000 7f782ec2d700 1 --2- 172.21.15.112:0/1564371814 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7828068480 0x7f78280b9690 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f781000c790 tx=0x7f7810007e00 comp rx=0 tx=0).stop 2023-01-24T03:30:14.602 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.598+0000 7f782ec2d700 1 -- 172.21.15.112:0/1564371814 shutdown_connections 2023-01-24T03:30:14.603 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.598+0000 7f782ec2d700 1 --2- 172.21.15.112:0/1564371814 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f780c061170 0x7f780c063620 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:14.603 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.598+0000 7f782ec2d700 1 --2- 172.21.15.112:0/1564371814 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f78280bcf90 0x7f782814efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:14.603 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.598+0000 7f782ec2d700 1 --2- 172.21.15.112:0/1564371814 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7828067720 0x7f78281662d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:14.603 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.598+0000 7f782ec2d700 1 --2- 172.21.15.112:0/1564371814 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7828068480 0x7f78280b9690 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:14.603 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.598+0000 7f782ec2d700 1 -- 172.21.15.112:0/1564371814 >> 172.21.15.112:0/1564371814 conn(0x7f78280c76f0 msgr2=0x7f7828158220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:14.604 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.598+0000 7f782ec2d700 1 -- 172.21.15.112:0/1564371814 shutdown_connections 2023-01-24T03:30:14.604 INFO:teuthology.misc.health.smithi112.stderr:2023-01-24T03:30:14.598+0000 7f782ec2d700 1 -- 172.21.15.112:0/1564371814 wait complete. 2023-01-24T03:30:14.611 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-01-24T03:30:14.612 INFO:tasks.ceph:Creating RBD pool 2023-01-24T03:30:14.612 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster ceph osd pool create rbd 8 2023-01-24T03:30:14.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.723+0000 7fc327fff700 1 Processor -- start 2023-01-24T03:30:14.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.723+0000 7fc327fff700 1 -- start start 2023-01-24T03:30:14.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.724+0000 7fc327fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fc32814f290 0x7fc3280c6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:14.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.724+0000 7fc327fff700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3280c8410 con 0x7fc32814f660 2023-01-24T03:30:14.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.724+0000 7fc327fff700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3280c8cd0 con 0x7fc3280c88a0 2023-01-24T03:30:14.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.724+0000 7fc327fff700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3280c59a0 con 0x7fc32814f290 2023-01-24T03:30:14.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.724+0000 7fc31ffff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fc32814f660 0x7fc3280c2cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:58984/0 (socket says 172.21.15.112:58984) 2023-01-24T03:30:14.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.724+0000 7fc31ffff700 1 -- 172.21.15.112:0/1940717398 learned_addr learned my addr 172.21.15.112:0/1940717398 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:14.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.724+0000 7fc326ffd700 1 --2- 172.21.15.112:0/1940717398 >> v2:172.21.15.142:3300/0 conn(0x7fc32814f290 0x7fc3280c6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:14.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.724+0000 7fc3267fc700 1 -- 172.21.15.112:0/1940717398 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3156563882 0 0) 0x7fc3280c8410 con 0x7fc32814f660 2023-01-24T03:30:14.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.724+0000 7fc3267fc700 1 -- 172.21.15.112:0/1940717398 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3080033f0 con 0x7fc32814f660 2023-01-24T03:30:14.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.724+0000 7fc326ffd700 1 -- 172.21.15.112:0/1940717398 >> v1:172.21.15.142:6789/0 conn(0x7fc3280c88a0 legacy=0x7fc32815fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:14.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.724+0000 7fc3267fc700 1 -- 172.21.15.112:0/1940717398 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3390381054 0 0) 0x7fc3280c8cd0 con 0x7fc3280c88a0 2023-01-24T03:30:14.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.724+0000 7fc326ffd700 1 -- 172.21.15.112:0/1940717398 >> v1:172.21.15.112:6789/0 conn(0x7fc32814f660 legacy=0x7fc3280c2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:14.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.724+0000 7fc326ffd700 1 -- 172.21.15.112:0/1940717398 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3280c53e0 con 0x7fc32814f290 2023-01-24T03:30:14.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.725+0000 7fc326ffd700 1 --2- 172.21.15.112:0/1940717398 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc32814f290 0x7fc3280c6420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc318009790 tx=0x7fc318009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2606f7878eae8df9 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:14.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.725+0000 7fc3267fc700 1 -- 172.21.15.112:0/1940717398 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc318011630 con 0x7fc32814f290 2023-01-24T03:30:14.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.725+0000 7fc3267fc700 1 -- 172.21.15.112:0/1940717398 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc318011790 con 0x7fc32814f290 2023-01-24T03:30:14.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.725+0000 7fc3267fc700 1 -- 172.21.15.112:0/1940717398 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc318011b10 con 0x7fc32814f290 2023-01-24T03:30:14.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.725+0000 7fc327fff700 1 -- 172.21.15.112:0/1940717398 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc32814f290 msgr2=0x7fc3280c6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:14.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.725+0000 7fc327fff700 1 --2- 172.21.15.112:0/1940717398 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc32814f290 0x7fc3280c6420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc318009790 tx=0x7fc318009060 comp rx=0 tx=0).stop 2023-01-24T03:30:14.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.725+0000 7fc327fff700 1 -- 172.21.15.112:0/1940717398 shutdown_connections 2023-01-24T03:30:14.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.725+0000 7fc327fff700 1 --2- 172.21.15.112:0/1940717398 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc32814f290 0x7fc3280c6420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:14.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.725+0000 7fc327fff700 1 -- 172.21.15.112:0/1940717398 >> 172.21.15.112:0/1940717398 conn(0x7fc32805d830 msgr2=0x7fc32805dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:14.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.725+0000 7fc327fff700 1 -- 172.21.15.112:0/1940717398 shutdown_connections 2023-01-24T03:30:14.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.725+0000 7fc327fff700 1 -- 172.21.15.112:0/1940717398 wait complete. 2023-01-24T03:30:14.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.726+0000 7fc327fff700 1 Processor -- start 2023-01-24T03:30:14.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.726+0000 7fc327fff700 1 -- start start 2023-01-24T03:30:14.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.727+0000 7fc327fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc3280c88a0 0x7fc328151430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:14.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.727+0000 7fc327fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc32814f290 0x7fc328067aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:14.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.727+0000 7fc327fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc32814f660 0x7fc328067fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:14.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.727+0000 7fc327fff700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3280c1e80 con 0x7fc32814f660 2023-01-24T03:30:14.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.727+0000 7fc327fff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc328165c40 con 0x7fc3280c88a0 2023-01-24T03:30:14.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.727+0000 7fc327fff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc328165710 con 0x7fc32814f290 2023-01-24T03:30:14.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.727+0000 7fc3277fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc32814f660 0x7fc328067fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:14.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.727+0000 7fc326ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc3280c88a0 0x7fc328151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:14.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.727+0000 7fc3277fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc32814f660 0x7fc328067fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:49772/0 (socket says 172.21.15.112:49772) 2023-01-24T03:30:14.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.727+0000 7fc3277fe700 1 -- 172.21.15.112:0/1011930169 learned_addr learned my addr 172.21.15.112:0/1011930169 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:14.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.727+0000 7fc31ffff700 1 --2- 172.21.15.112:0/1011930169 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc32814f290 0x7fc328067aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:14.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.727+0000 7fc3277fe700 1 -- 172.21.15.112:0/1011930169 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc32814f290 msgr2=0x7fc328067aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:14.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.727+0000 7fc3277fe700 1 --2- 172.21.15.112:0/1011930169 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc32814f290 0x7fc328067aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:14.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.727+0000 7fc3277fe700 1 -- 172.21.15.112:0/1011930169 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc3280c88a0 msgr2=0x7fc328151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:14.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.727+0000 7fc3277fe700 1 --2- 172.21.15.112:0/1011930169 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc3280c88a0 0x7fc328151430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:14.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.727+0000 7fc3277fe700 1 -- 172.21.15.112:0/1011930169 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3180092a0 con 0x7fc32814f660 2023-01-24T03:30:14.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.728+0000 7fc3277fe700 1 --2- 172.21.15.112:0/1011930169 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc32814f660 0x7fc328067fe0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc32000c760 tx=0x7fc320007f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:14.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.728+0000 7fc31effd700 1 -- 172.21.15.112:0/1011930169 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc320011590 con 0x7fc32814f660 2023-01-24T03:30:14.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.728+0000 7fc31effd700 1 -- 172.21.15.112:0/1011930169 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc320011710 con 0x7fc32814f660 2023-01-24T03:30:14.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.728+0000 7fc31effd700 1 -- 172.21.15.112:0/1011930169 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc320011ad0 con 0x7fc32814f660 2023-01-24T03:30:14.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.728+0000 7fc327fff700 1 -- 172.21.15.112:0/1011930169 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3280c33a0 con 0x7fc32814f660 2023-01-24T03:30:14.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.728+0000 7fc327fff700 1 -- 172.21.15.112:0/1011930169 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc328150c30 con 0x7fc32814f660 2023-01-24T03:30:14.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.729+0000 7fc327fff700 1 -- 172.21.15.112:0/1011930169 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc32805ea20 con 0x7fc32814f660 2023-01-24T03:30:14.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.729+0000 7fc31effd700 1 -- 172.21.15.112:0/1011930169 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 4) v1 ==== 79852+0+0 (secure 0 0 0) 0x7fc320010050 con 0x7fc32814f660 2023-01-24T03:30:14.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.729+0000 7fc31effd700 1 --2- 172.21.15.112:0/1011930169 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc30c061170 0x7fc30c063620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:14.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.729+0000 7fc31effd700 1 -- 172.21.15.112:0/1011930169 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4467+0+0 (secure 0 0 0) 0x7fc32007cae0 con 0x7fc32814f660 2023-01-24T03:30:14.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.729+0000 7fc326ffd700 1 --2- 172.21.15.112:0/1011930169 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc30c061170 0x7fc30c063620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:14.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.730+0000 7fc326ffd700 1 --2- 172.21.15.112:0/1011930169 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc30c061170 0x7fc30c063620 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc318011fd0 tx=0x7fc318017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:14.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.733+0000 7fc31effd700 1 -- 172.21.15.112:0/1011930169 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fc32004d290 con 0x7fc32814f660 2023-01-24T03:30:14.916 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:14.912+0000 7fc327fff700 1 -- 172.21.15.112:0/1011930169 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7fc3280c56c0 con 0x7fc32814f660 2023-01-24T03:30:15.041 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.038+0000 7fc31effd700 1 -- 172.21.15.112:0/1011930169 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v14) v1 ==== 109+0+0 (secure 0 0 0) 0x7fc3200508b0 con 0x7fc32814f660 2023-01-24T03:30:15.044 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.041+0000 7fc327fff700 1 -- 172.21.15.112:0/1011930169 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc30c061170 msgr2=0x7fc30c063620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:15.044 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.041+0000 7fc327fff700 1 --2- 172.21.15.112:0/1011930169 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc30c061170 0x7fc30c063620 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc318011fd0 tx=0x7fc318017000 comp rx=0 tx=0).stop 2023-01-24T03:30:15.044 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.041+0000 7fc327fff700 1 -- 172.21.15.112:0/1011930169 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc32814f660 msgr2=0x7fc328067fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:15.044 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.041+0000 7fc327fff700 1 --2- 172.21.15.112:0/1011930169 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc32814f660 0x7fc328067fe0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc32000c760 tx=0x7fc320007f10 comp rx=0 tx=0).stop 2023-01-24T03:30:15.045 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.041+0000 7fc327fff700 1 -- 172.21.15.112:0/1011930169 shutdown_connections 2023-01-24T03:30:15.045 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.041+0000 7fc327fff700 1 --2- 172.21.15.112:0/1011930169 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc30c061170 0x7fc30c063620 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:15.045 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.041+0000 7fc327fff700 1 --2- 172.21.15.112:0/1011930169 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc32814f660 0x7fc328067fe0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:15.045 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.041+0000 7fc327fff700 1 --2- 172.21.15.112:0/1011930169 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc3280c88a0 0x7fc328151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:15.045 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.041+0000 7fc327fff700 1 --2- 172.21.15.112:0/1011930169 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc32814f290 0x7fc328067aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:15.046 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.041+0000 7fc327fff700 1 -- 172.21.15.112:0/1011930169 >> 172.21.15.112:0/1011930169 conn(0x7fc32805d830 msgr2=0x7fc3280cb690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:15.046 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.041+0000 7fc327fff700 1 -- 172.21.15.112:0/1011930169 shutdown_connections 2023-01-24T03:30:15.046 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.041+0000 7fc327fff700 1 -- 172.21.15.112:0/1011930169 wait complete. 2023-01-24T03:30:15.046 INFO:teuthology.orchestra.run.smithi112.stderr:pool 'rbd' created 2023-01-24T03:30:15.055 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-01-24T03:30:15.170 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.167+0000 7f34b931e700 1 Processor -- start 2023-01-24T03:30:15.170 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.167+0000 7f34b931e700 1 -- start start 2023-01-24T03:30:15.178 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.168+0000 7f34b931e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f34b4156eb0 0x7f34b406a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:15.178 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.168+0000 7f34b931e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34b4069cc0 con 0x7f34b40c85a0 2023-01-24T03:30:15.178 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.168+0000 7f34b931e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34b406a690 con 0x7f34b4157280 2023-01-24T03:30:15.178 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.168+0000 7f34b931e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34b4069a20 con 0x7f34b4156eb0 2023-01-24T03:30:15.179 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.168+0000 7f34b27fc700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f34b40c85a0 0x7f34b4067e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:58998/0 (socket says 172.21.15.112:58998) 2023-01-24T03:30:15.179 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.168+0000 7f34b27fc700 1 -- 172.21.15.112:0/4045739560 learned_addr learned my addr 172.21.15.112:0/4045739560 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:15.179 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.168+0000 7f34b1ffb700 1 -- 172.21.15.112:0/4045739560 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1638857792 0 0) 0x7f34b4069cc0 con 0x7f34b40c85a0 2023-01-24T03:30:15.179 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.168+0000 7f34b1ffb700 1 -- 172.21.15.112:0/4045739560 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34a00033f0 con 0x7f34b40c85a0 2023-01-24T03:30:15.180 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.168+0000 7f34b2ffd700 1 --2- 172.21.15.112:0/4045739560 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f34b4156eb0 0x7f34b406a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:15.180 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.169+0000 7f34b1ffb700 1 -- 172.21.15.112:0/4045739560 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4117481023 0 0) 0x7f34a00033f0 con 0x7f34b40c85a0 2023-01-24T03:30:15.180 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.169+0000 7f34b1ffb700 1 -- 172.21.15.112:0/4045739560 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f34b4156eb0 msgr2=0x7f34b406a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:15.180 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.169+0000 7f34b1ffb700 1 --2- 172.21.15.112:0/4045739560 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f34b4156eb0 0x7f34b406a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:15.181 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.169+0000 7f34b1ffb700 1 -- 172.21.15.112:0/4045739560 >> v1:172.21.15.142:6789/0 conn(0x7f34b4157280 legacy=0x7f34b415da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:15.181 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.169+0000 7f34b1ffb700 1 -- 172.21.15.112:0/4045739560 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b40c18e0 con 0x7f34b40c85a0 2023-01-24T03:30:15.181 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.169+0000 7f34b1ffb700 1 -- 172.21.15.112:0/4045739560 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7f34a8001470 con 0x7f34b40c85a0 2023-01-24T03:30:15.181 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.169+0000 7f34b1ffb700 1 -- 172.21.15.112:0/4045739560 >> v1:172.21.15.112:6789/0 conn(0x7f34b40c85a0 legacy=0x7f34b4067e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:15.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.169+0000 7f34b1ffb700 1 --2- 172.21.15.112:0/4045739560 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34a0003910 0x7f34a0003d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:15.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.169+0000 7f34b1ffb700 1 --2- 172.21.15.112:0/4045739560 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34a0004690 0x7f34a0004b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:15.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.169+0000 7f34b1ffb700 1 --2- 172.21.15.112:0/4045739560 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f34b4156eb0 0x7f34a0005060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:15.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.169+0000 7f34b1ffb700 1 -- 172.21.15.112:0/4045739560 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b4069a20 con 0x7f34a0004690 2023-01-24T03:30:15.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.169+0000 7f34b1ffb700 1 -- 172.21.15.112:0/4045739560 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a000b9e0 con 0x7f34a0003910 2023-01-24T03:30:15.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.169+0000 7f34b1ffb700 1 -- 172.21.15.112:0/4045739560 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34a000bba0 con 0x7f34b4156eb0 2023-01-24T03:30:15.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.169+0000 7f34b27fc700 1 --2- 172.21.15.112:0/4045739560 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34a0004690 0x7f34a0004b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:15.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.169+0000 7f34b2ffd700 1 --2- 172.21.15.112:0/4045739560 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34a0003910 0x7f34a0003d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:15.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.169+0000 7f34b37fe700 1 --2- 172.21.15.112:0/4045739560 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f34b4156eb0 0x7f34a0005060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:15.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.169+0000 7f34b27fc700 1 -- 172.21.15.112:0/4045739560 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f34b4156eb0 msgr2=0x7f34a0005060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:15.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.169+0000 7f34b27fc700 1 --2- 172.21.15.112:0/4045739560 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f34b4156eb0 0x7f34a0005060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:15.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.169+0000 7f34b27fc700 1 -- 172.21.15.112:0/4045739560 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34a0003910 msgr2=0x7f34a0003d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:15.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.169+0000 7f34b27fc700 1 --2- 172.21.15.112:0/4045739560 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34a0003910 0x7f34a0003d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:15.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.169+0000 7f34b27fc700 1 -- 172.21.15.112:0/4045739560 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a00033f0 con 0x7f34a0004690 2023-01-24T03:30:15.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.169+0000 7f34b27fc700 1 --2- 172.21.15.112:0/4045739560 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34a0004690 0x7f34a0004b00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f34b406f040 tx=0x7f34a8001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46a967d11d19447e server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:15.185 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.169+0000 7f34b1ffb700 1 -- 172.21.15.112:0/4045739560 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a800ca30 con 0x7f34a0004690 2023-01-24T03:30:15.185 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.169+0000 7f34b1ffb700 1 -- 172.21.15.112:0/4045739560 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34a800cb90 con 0x7f34a0004690 2023-01-24T03:30:15.185 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.169+0000 7f34b1ffb700 1 -- 172.21.15.112:0/4045739560 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a80070d0 con 0x7f34a0004690 2023-01-24T03:30:15.185 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.170+0000 7f34b931e700 1 -- 172.21.15.112:0/4045739560 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34a0004690 msgr2=0x7f34a0004b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:15.185 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.170+0000 7f34b931e700 1 --2- 172.21.15.112:0/4045739560 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34a0004690 0x7f34a0004b00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f34b406f040 tx=0x7f34a8001e70 comp rx=0 tx=0).stop 2023-01-24T03:30:15.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.170+0000 7f34b37fe700 1 -- 172.21.15.112:0/4045739560 reap_dead start 2023-01-24T03:30:15.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.170+0000 7f34b931e700 1 -- 172.21.15.112:0/4045739560 shutdown_connections 2023-01-24T03:30:15.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.170+0000 7f34b931e700 1 -- 172.21.15.112:0/4045739560 >> 172.21.15.112:0/4045739560 conn(0x7f34b406c030 msgr2=0x7f34b40c7e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:15.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.170+0000 7f34b931e700 1 -- 172.21.15.112:0/4045739560 shutdown_connections 2023-01-24T03:30:15.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.170+0000 7f34b931e700 1 -- 172.21.15.112:0/4045739560 wait complete. 2023-01-24T03:30:15.187 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.171+0000 7f34b931e700 1 Processor -- start 2023-01-24T03:30:15.187 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.171+0000 7f34b931e700 1 -- start start 2023-01-24T03:30:15.187 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.173+0000 7f34b931e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34a0004690 0x7f34b40be900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:15.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.173+0000 7f34b931e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f34b40bbe00 0x7f34b4151fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:15.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.173+0000 7f34b931e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34b40bcba0 0x7f34b40bd010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:15.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.173+0000 7f34b931e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b40be470 con 0x7f34a0004690 2023-01-24T03:30:15.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.173+0000 7f34b2ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34a0004690 0x7f34b40be900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:15.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.173+0000 7f34b2ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34a0004690 0x7f34b40be900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:49788/0 (socket says 172.21.15.112:49788) 2023-01-24T03:30:15.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.173+0000 7f34b2ffd700 1 -- 172.21.15.112:0/1468294909 learned_addr learned my addr 172.21.15.112:0/1468294909 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:15.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.173+0000 7f34b931e700 1 -- 172.21.15.112:0/1468294909 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b4152500 con 0x7f34b40bcba0 2023-01-24T03:30:15.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.173+0000 7f34b931e700 1 -- 172.21.15.112:0/1468294909 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34b4163e00 con 0x7f34b40bbe00 2023-01-24T03:30:15.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.173+0000 7f34b27fc700 1 --2- 172.21.15.112:0/1468294909 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f34b40bbe00 0x7f34b4151fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:15.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.173+0000 7f34b37fe700 1 --2- 172.21.15.112:0/1468294909 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34b40bcba0 0x7f34b40bd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:15.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.173+0000 7f34b2ffd700 1 -- 172.21.15.112:0/1468294909 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f34b40bbe00 msgr2=0x7f34b4151fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:15.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.173+0000 7f34b2ffd700 1 --2- 172.21.15.112:0/1468294909 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f34b40bbe00 0x7f34b4151fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:15.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.173+0000 7f34b2ffd700 1 -- 172.21.15.112:0/1468294909 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34b40bcba0 msgr2=0x7f34b40bd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:15.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.173+0000 7f34b2ffd700 1 --2- 172.21.15.112:0/1468294909 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34b40bcba0 0x7f34b40bd010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:15.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.173+0000 7f34b2ffd700 1 -- 172.21.15.112:0/1468294909 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b40bde20 con 0x7f34a0004690 2023-01-24T03:30:15.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.173+0000 7f34b2ffd700 1 --2- 172.21.15.112:0/1468294909 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34a0004690 0x7f34b40be900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f34a4011fb0 tx=0x7f34a400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:15.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.173+0000 7f349affd700 1 -- 172.21.15.112:0/1468294909 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a400e040 con 0x7f34a0004690 2023-01-24T03:30:15.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.173+0000 7f349affd700 1 -- 172.21.15.112:0/1468294909 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34a400f040 con 0x7f34a0004690 2023-01-24T03:30:15.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.173+0000 7f34b931e700 1 -- 172.21.15.112:0/1468294909 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34b40be100 con 0x7f34a0004690 2023-01-24T03:30:15.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.173+0000 7f349affd700 1 -- 172.21.15.112:0/1468294909 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a4015430 con 0x7f34a0004690 2023-01-24T03:30:15.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.173+0000 7f34b931e700 1 -- 172.21.15.112:0/1468294909 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34b4163af0 con 0x7f34a0004690 2023-01-24T03:30:15.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.173+0000 7f34b931e700 1 -- 172.21.15.112:0/1468294909 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34b40bdc80 con 0x7f34a0004690 2023-01-24T03:30:15.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.174+0000 7f349affd700 1 -- 172.21.15.112:0/1468294909 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 4) v1 ==== 79852+0+0 (secure 0 0 0) 0x7f34a4015820 con 0x7f34a0004690 2023-01-24T03:30:15.193 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.174+0000 7f349affd700 1 --2- 172.21.15.112:0/1468294909 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3494061170 0x7f3494063620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:15.193 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.174+0000 7f349affd700 1 -- 172.21.15.112:0/1468294909 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 5184+0+0 (secure 0 0 0) 0x7f34a407e000 con 0x7f34a0004690 2023-01-24T03:30:15.193 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.174+0000 7f34b27fc700 1 --2- 172.21.15.112:0/1468294909 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3494061170 0x7f3494063620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:15.193 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.174+0000 7f34b27fc700 1 --2- 172.21.15.112:0/1468294909 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3494061170 0x7f3494063620 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f34b406f040 tx=0x7f34a8015000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:15.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.178+0000 7f349affd700 1 -- 172.21.15.112:0/1468294909 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f34b40bdc80 con 0x7f34a0004690 2023-01-24T03:30:15.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:15.355+0000 7f34b931e700 1 -- 172.21.15.112:0/1468294909 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f34b40bdc80 con 0x7f34a0004690 2023-01-24T03:30:16.046 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.044+0000 7f349affd700 1 -- 172.21.15.112:0/1468294909 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v15) v1 ==== 173+0+0 (secure 0 0 0) 0x7f34b40bdc80 con 0x7f34a0004690 2023-01-24T03:30:16.052 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.047+0000 7f34b931e700 1 -- 172.21.15.112:0/1468294909 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3494061170 msgr2=0x7f3494063620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:16.052 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.047+0000 7f34b931e700 1 --2- 172.21.15.112:0/1468294909 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3494061170 0x7f3494063620 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f34b406f040 tx=0x7f34a8015000 comp rx=0 tx=0).stop 2023-01-24T03:30:16.052 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.047+0000 7f34b931e700 1 -- 172.21.15.112:0/1468294909 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34a0004690 msgr2=0x7f34b40be900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:16.053 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.047+0000 7f34b931e700 1 --2- 172.21.15.112:0/1468294909 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34a0004690 0x7f34b40be900 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f34a4011fb0 tx=0x7f34a400b040 comp rx=0 tx=0).stop 2023-01-24T03:30:16.053 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.047+0000 7f34b931e700 1 -- 172.21.15.112:0/1468294909 shutdown_connections 2023-01-24T03:30:16.054 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.047+0000 7f34b931e700 1 --2- 172.21.15.112:0/1468294909 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3494061170 0x7f3494063620 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:16.054 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.047+0000 7f34b931e700 1 --2- 172.21.15.112:0/1468294909 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34a0004690 0x7f34b40be900 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:16.054 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.047+0000 7f34b931e700 1 --2- 172.21.15.112:0/1468294909 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f34b40bcba0 0x7f34b40bd010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:16.054 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.047+0000 7f34b931e700 1 --2- 172.21.15.112:0/1468294909 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f34b40bbe00 0x7f34b4151fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:16.054 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.048+0000 7f34b931e700 1 -- 172.21.15.112:0/1468294909 >> 172.21.15.112:0/1468294909 conn(0x7f34b406c030 msgr2=0x7f34b4150390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:16.055 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.048+0000 7f34b931e700 1 -- 172.21.15.112:0/1468294909 shutdown_connections 2023-01-24T03:30:16.055 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.048+0000 7f34b931e700 1 -- 172.21.15.112:0/1468294909 wait complete. 2023-01-24T03:30:16.056 INFO:teuthology.orchestra.run.smithi112.stderr:enabled application 'rbd' on pool 'rbd' 2023-01-24T03:30:16.065 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-01-24T03:30:16.065 INFO:tasks.ceph.mds.a:Restarting daemon 2023-01-24T03:30:16.066 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-01-24T03:30:16.068 INFO:tasks.ceph.mds.a:Started 2023-01-24T03:30:16.068 INFO:tasks.ceph.mds.c:Restarting daemon 2023-01-24T03:30:16.068 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-01-24T03:30:16.070 INFO:tasks.ceph.mds.c:Started 2023-01-24T03:30:16.071 INFO:tasks.ceph.mds.e:Restarting daemon 2023-01-24T03:30:16.071 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-01-24T03:30:16.073 INFO:tasks.ceph.mds.e:Started 2023-01-24T03:30:16.074 INFO:tasks.ceph.mds.b:Restarting daemon 2023-01-24T03:30:16.074 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-01-24T03:30:16.077 INFO:tasks.ceph.mds.b:Started 2023-01-24T03:30:16.077 INFO:tasks.ceph.mds.d:Restarting daemon 2023-01-24T03:30:16.078 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-01-24T03:30:16.080 INFO:tasks.ceph.mds.d:Started 2023-01-24T03:30:16.080 INFO:tasks.ceph.mds.f:Restarting daemon 2023-01-24T03:30:16.080 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-01-24T03:30:16.083 INFO:tasks.ceph.mds.f:Started 2023-01-24T03:30:16.083 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-01-24T03:30:16.083 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T03:30:16.157 INFO:tasks.ceph.mds.c.smithi112.stdout:starting mds.c at 2023-01-24T03:30:16.157 INFO:tasks.ceph.mds.a.smithi112.stdout:starting mds.a at 2023-01-24T03:30:16.162 INFO:tasks.ceph.mds.e.smithi112.stdout:starting mds.e at 2023-01-24T03:30:16.167 INFO:tasks.ceph.mds.d.smithi142.stdout:starting mds.d at 2023-01-24T03:30:16.167 INFO:tasks.ceph.mds.b.smithi142.stdout:starting mds.b at 2023-01-24T03:30:16.169 INFO:tasks.ceph.mds.f.smithi142.stdout:starting mds.f at 2023-01-24T03:30:16.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.218+0000 7fb49562c700 1 Processor -- start 2023-01-24T03:30:16.221 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.218+0000 7fb49562c700 1 -- start start 2023-01-24T03:30:16.223 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.219+0000 7fb49562c700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fb490067fe0 0x7fb49006d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:16.223 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.219+0000 7fb49562c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb49006eec0 0x7fb49006de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:16.224 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.219+0000 7fb49562c700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb49006a000 con 0x7fb49006a500 2023-01-24T03:30:16.224 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.219+0000 7fb49562c700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4900c47b0 con 0x7fb490067fe0 2023-01-24T03:30:16.224 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.219+0000 7fb49562c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4900c3f20 con 0x7fb49006eec0 2023-01-24T03:30:16.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.219+0000 7fb48effd700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fb490067fe0 0x7fb49006d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:16.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.219+0000 7fb48f7fe700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fb49006a500 0x7fb4900bd850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:59092/0 (socket says 172.21.15.112:59092) 2023-01-24T03:30:16.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.219+0000 7fb48f7fe700 1 -- 172.21.15.112:0/655344976 learned_addr learned my addr 172.21.15.112:0/655344976 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:16.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.220+0000 7fb48e7fc700 1 --2- 172.21.15.112:0/655344976 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb49006eec0 0x7fb49006de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:16.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.220+0000 7fb48dffb700 1 -- 172.21.15.112:0/655344976 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 781756198 0 0) 0x7fb49006a000 con 0x7fb49006a500 2023-01-24T03:30:16.226 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.220+0000 7fb48dffb700 1 -- 172.21.15.112:0/655344976 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb47c0033f0 con 0x7fb49006a500 2023-01-24T03:30:16.226 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.220+0000 7fb48effd700 1 -- 172.21.15.112:0/655344976 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb49006eec0 msgr2=0x7fb49006de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:16.226 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.220+0000 7fb48effd700 1 --2- 172.21.15.112:0/655344976 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb49006eec0 0x7fb49006de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:16.226 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.220+0000 7fb48effd700 1 -- 172.21.15.112:0/655344976 >> v1:172.21.15.112:6789/0 conn(0x7fb49006a500 legacy=0x7fb4900bd850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:16.226 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.220+0000 7fb48effd700 1 -- 172.21.15.112:0/655344976 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb49006f420 con 0x7fb490067fe0 2023-01-24T03:30:16.226 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.220+0000 7fb48effd700 1 --2- 172.21.15.112:0/655344976 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb490067fe0 0x7fb49006d930 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb480009760 tx=0x7fb480009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97bf1f8aaa65a661 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:16.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.220+0000 7fb48dffb700 1 -- 172.21.15.112:0/655344976 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4800105b0 con 0x7fb490067fe0 2023-01-24T03:30:16.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.220+0000 7fb48dffb700 1 -- 172.21.15.112:0/655344976 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb480010710 con 0x7fb490067fe0 2023-01-24T03:30:16.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.220+0000 7fb48dffb700 1 -- 172.21.15.112:0/655344976 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb480010a90 con 0x7fb490067fe0 2023-01-24T03:30:16.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.221+0000 7fb49562c700 1 -- 172.21.15.112:0/655344976 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb490067fe0 msgr2=0x7fb49006d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:16.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.221+0000 7fb49562c700 1 --2- 172.21.15.112:0/655344976 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb490067fe0 0x7fb49006d930 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb480009760 tx=0x7fb480009060 comp rx=0 tx=0).stop 2023-01-24T03:30:16.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.221+0000 7fb49562c700 1 -- 172.21.15.112:0/655344976 shutdown_connections 2023-01-24T03:30:16.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.221+0000 7fb49562c700 1 --2- 172.21.15.112:0/655344976 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb490067fe0 0x7fb49006d930 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:16.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.221+0000 7fb49562c700 1 --2- 172.21.15.112:0/655344976 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb49006eec0 0x7fb49006de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:16.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.221+0000 7fb49562c700 1 -- 172.21.15.112:0/655344976 >> 172.21.15.112:0/655344976 conn(0x7fb4900c8940 msgr2=0x7fb4900c4910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:16.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.221+0000 7fb49562c700 1 -- 172.21.15.112:0/655344976 shutdown_connections 2023-01-24T03:30:16.229 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.221+0000 7fb49562c700 1 -- 172.21.15.112:0/655344976 wait complete. 2023-01-24T03:30:16.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.222+0000 7fb49562c700 1 Processor -- start 2023-01-24T03:30:16.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.222+0000 7fb49562c700 1 -- start start 2023-01-24T03:30:16.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.223+0000 7fb49562c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb490067fe0 0x7fb4900a6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:16.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.223+0000 7fb49562c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb49006a500 0x7fb490151660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:16.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.223+0000 7fb49562c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb49006eec0 0x7fb4900a7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:16.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.223+0000 7fb49562c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb490149e20 con 0x7fb490067fe0 2023-01-24T03:30:16.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.223+0000 7fb49562c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4900a94c0 con 0x7fb49006a500 2023-01-24T03:30:16.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.223+0000 7fb49562c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4900a54a0 con 0x7fb49006eec0 2023-01-24T03:30:16.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.223+0000 7fb48effd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb490067fe0 0x7fb4900a6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:16.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.223+0000 7fb48effd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb490067fe0 0x7fb4900a6870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:49874/0 (socket says 172.21.15.112:49874) 2023-01-24T03:30:16.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.223+0000 7fb48effd700 1 -- 172.21.15.112:0/3017851226 learned_addr learned my addr 172.21.15.112:0/3017851226 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:16.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.223+0000 7fb48f7fe700 1 --2- 172.21.15.112:0/3017851226 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb49006eec0 0x7fb4900a7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:16.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.223+0000 7fb48e7fc700 1 --2- 172.21.15.112:0/3017851226 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb49006a500 0x7fb490151660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:16.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.223+0000 7fb48effd700 1 -- 172.21.15.112:0/3017851226 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb49006eec0 msgr2=0x7fb4900a7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:16.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.223+0000 7fb48effd700 1 --2- 172.21.15.112:0/3017851226 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb49006eec0 0x7fb4900a7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:16.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.223+0000 7fb48effd700 1 -- 172.21.15.112:0/3017851226 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb49006a500 msgr2=0x7fb490151660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:16.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.223+0000 7fb48effd700 1 --2- 172.21.15.112:0/3017851226 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb49006a500 0x7fb490151660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:16.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.223+0000 7fb48effd700 1 -- 172.21.15.112:0/3017851226 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4800092a0 con 0x7fb490067fe0 2023-01-24T03:30:16.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.223+0000 7fb48f7fe700 1 --2- 172.21.15.112:0/3017851226 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb49006eec0 0x7fb4900a7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:30:16.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.223+0000 7fb48effd700 1 --2- 172.21.15.112:0/3017851226 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb490067fe0 0x7fb4900a6870 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb480010bb0 tx=0x7fb480016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:16.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.223+0000 7fb476ffd700 1 -- 172.21.15.112:0/3017851226 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4800118b0 con 0x7fb490067fe0 2023-01-24T03:30:16.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.224+0000 7fb476ffd700 1 -- 172.21.15.112:0/3017851226 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb48001f070 con 0x7fb490067fe0 2023-01-24T03:30:16.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.224+0000 7fb476ffd700 1 -- 172.21.15.112:0/3017851226 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb480011c40 con 0x7fb490067fe0 2023-01-24T03:30:16.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.224+0000 7fb49562c700 1 -- 172.21.15.112:0/3017851226 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4900a5d40 con 0x7fb490067fe0 2023-01-24T03:30:16.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.224+0000 7fb49562c700 1 -- 172.21.15.112:0/3017851226 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4900a8220 con 0x7fb490067fe0 2023-01-24T03:30:16.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.225+0000 7fb49562c700 1 -- 172.21.15.112:0/3017851226 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb49004bb80 con 0x7fb490067fe0 2023-01-24T03:30:16.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.225+0000 7fb476ffd700 1 -- 172.21.15.112:0/3017851226 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 4) v1 ==== 79852+0+0 (secure 0 0 0) 0x7fb48000f070 con 0x7fb490067fe0 2023-01-24T03:30:16.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.225+0000 7fb476ffd700 1 --2- 172.21.15.112:0/3017851226 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb470061170 0x7fb470063620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:16.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.226+0000 7fb476ffd700 1 -- 172.21.15.112:0/3017851226 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fb480021070 con 0x7fb490067fe0 2023-01-24T03:30:16.238 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.226+0000 7fb48e7fc700 1 --2- 172.21.15.112:0/3017851226 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb470061170 0x7fb470063620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:16.238 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.226+0000 7fb48e7fc700 1 --2- 172.21.15.112:0/3017851226 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb470061170 0x7fb470063620 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb490152ad0 tx=0x7fb484009060 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:16.238 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.230+0000 7fb476ffd700 1 -- 172.21.15.112:0/3017851226 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fb48004ee00 con 0x7fb490067fe0 2023-01-24T03:30:16.422 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.420+0000 7fb49562c700 1 -- 172.21.15.112:0/3017851226 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4900be0d0 con 0x7fb490067fe0 2023-01-24T03:30:16.423 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.421+0000 7fb476ffd700 1 -- 172.21.15.112:0/3017851226 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+13722 (secure 0 0 0) 0x7fb480052420 con 0x7fb490067fe0 2023-01-24T03:30:16.423 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:30:16.424 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":15,"fsid":"054b9ac9-b7b0-40e5-b932-6c8a955c5a72","created":"2023-01-24T03:30:03.798533+0000","modified":"2023-01-24T03:30:16.037336+0000","last_up_change":"2023-01-24T03:30:13.945316+0000","last_in_change":"2023-01-24T03:30:06.198433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-01-24T03:30:14.931166+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-01-24T03:30:15.023730+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"fe6ad231-a39a-42eb-9a3a-49d008dd35e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6809","nonce":84512},{"type":"v1","addr":"172.21.15.112:6811","nonce":84512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":84512},{"type":"v1","addr":"172.21.15.112:6817","nonce":84512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6825","nonce":84512},{"type":"v1","addr":"172.21.15.112:6828","nonce":84512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6819","nonce":84512},{"type":"v1","addr":"172.21.15.112:6822","nonce":84512}]},"public_addr":"172.21.15.112:6811/84512","cluster_addr":"172.21.15.112:6817/84512","heartbeat_back_addr":"172.21.15.112:6828/84512","heartbeat_front_addr":"172.21.15.112:6822/84512","state":["exists","up"]},{"osd":1,"uuid":"8526a1d2-d540-4021-9766-c4bbb30317ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":84513},{"type":"v1","addr":"172.21.15.112:6812","nonce":84513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6815","nonce":84513},{"type":"v1","addr":"172.21.15.112:6818","nonce":84513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6827","nonce":84513},{"type":"v1","addr":"172.21.15.112:6830","nonce":84513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6821","nonce":84513},{"type":"v1","addr":"172.21.15.112:6824","nonce":84513}]},"public_addr":"172.21.15.112:6812/84513","cluster_addr":"172.21.15.112:6818/84513","heartbeat_back_addr":"172.21.15.112:6830/84513","heartbeat_front_addr":"172.21.15.112:6824/84513","state":["exists","up"]},{"osd":2,"uuid":"041ef8e4-4858-432d-9476-cbdb3a0dc47c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":84511},{"type":"v1","addr":"172.21.15.112:6813","nonce":84511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":84511},{"type":"v1","addr":"172.21.15.112:6820","nonce":84511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6829","nonce":84511},{"type":"v1","addr":"172.21.15.112:6831","nonce":84511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6823","nonce":84511},{"type":"v1","addr":"172.21.15.112:6826","nonce":84511}]},"public_addr":"172.21.15.112:6813/84511","cluster_addr":"172.21.15.112:6820/84511","heartbeat_back_addr":"172.21.15.112:6831/84511","heartbeat_front_addr":"172.21.15.112:6826/84511","state":["exists","up"]},{"osd":3,"uuid":"987587cf-1357-4a70-ad85-67b189c67729","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":84510},{"type":"v1","addr":"172.21.15.112:6801","nonce":84510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":84510},{"type":"v1","addr":"172.21.15.112:6803","nonce":84510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":84510},{"type":"v1","addr":"172.21.15.112:6807","nonce":84510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":84510},{"type":"v1","addr":"172.21.15.112:6805","nonce":84510}]},"public_addr":"172.21.15.112:6801/84510","cluster_addr":"172.21.15.112:6803/84510","heartbeat_back_addr":"172.21.15.112:6807/84510","heartbeat_front_addr":"172.21.15.112:6805/84510","state":["exists","up"]},{"osd":4,"uuid":"8503512b-43db-40b8-8b2b-c86cf2e54e17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":84614},{"type":"v1","addr":"172.21.15.142:6801","nonce":84614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":84614},{"type":"v1","addr":"172.21.15.142:6803","nonce":84614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":84614},{"type":"v1","addr":"172.21.15.142:6807","nonce":84614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":84614},{"type":"v1","addr":"172.21.15.142:6805","nonce":84614}]},"public_addr":"172.21.15.142:6801/84614","cluster_addr":"172.21.15.142:6803/84614","heartbeat_back_addr":"172.21.15.142:6807/84614","heartbeat_front_addr":"172.21.15.142:6805/84614","state":["exists","up"]},{"osd":5,"uuid":"d3bc0030-8b08-4645-9ec2-1f8a4e29ef36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":84638},{"type":"v1","addr":"172.21.15.142:6817","nonce":84638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":84638},{"type":"v1","addr":"172.21.15.142:6819","nonce":84638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":84638},{"type":"v1","addr":"172.21.15.142:6823","nonce":84638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":84638},{"type":"v1","addr":"172.21.15.142:6821","nonce":84638}]},"public_addr":"172.21.15.142:6817/84638","cluster_addr":"172.21.15.142:6819/84638","heartbeat_back_addr":"172.21.15.142:6823/84638","heartbeat_front_addr":"172.21.15.142:6821/84638","state":["exists","up"]},{"osd":6,"uuid":"c7649e3b-832d-4e6f-8866-b15e5d814476","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":84645},{"type":"v1","addr":"172.21.15.142:6825","nonce":84645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":84645},{"type":"v1","addr":"172.21.15.142:6827","nonce":84645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":84645},{"type":"v1","addr":"172.21.15.142:6831","nonce":84645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":84645},{"type":"v1","addr":"172.21.15.142:6829","nonce":84645}]},"public_addr":"172.21.15.142:6825/84645","cluster_addr":"172.21.15.142:6827/84645","heartbeat_back_addr":"172.21.15.142:6831/84645","heartbeat_front_addr":"172.21.15.142:6829/84645","state":["exists","up"]},{"osd":7,"uuid":"2ea5b3bf-884e-4b20-b572-1443efdc7477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":84615},{"type":"v1","addr":"172.21.15.142:6809","nonce":84615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":84615},{"type":"v1","addr":"172.21.15.142:6811","nonce":84615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":84615},{"type":"v1","addr":"172.21.15.142:6815","nonce":84615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":84615},{"type":"v1","addr":"172.21.15.142:6813","nonce":84615}]},"public_addr":"172.21.15.142:6809/84615","cluster_addr":"172.21.15.142:6811/84615","heartbeat_back_addr":"172.21.15.142:6815/84615","heartbeat_front_addr":"172.21.15.142:6813/84615","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.370027+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.335683+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.350667+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.350794+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.391119+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.422933+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.400439+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.354018+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T03:30:16.425 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.423+0000 7fb49562c700 1 -- 172.21.15.112:0/3017851226 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb470061170 msgr2=0x7fb470063620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:16.425 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.423+0000 7fb49562c700 1 --2- 172.21.15.112:0/3017851226 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb470061170 0x7fb470063620 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb490152ad0 tx=0x7fb484009060 comp rx=0 tx=0).stop 2023-01-24T03:30:16.426 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.423+0000 7fb49562c700 1 -- 172.21.15.112:0/3017851226 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb490067fe0 msgr2=0x7fb4900a6870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:16.426 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.423+0000 7fb49562c700 1 --2- 172.21.15.112:0/3017851226 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb490067fe0 0x7fb4900a6870 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb480010bb0 tx=0x7fb480016000 comp rx=0 tx=0).stop 2023-01-24T03:30:16.426 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.423+0000 7fb49562c700 1 -- 172.21.15.112:0/3017851226 shutdown_connections 2023-01-24T03:30:16.426 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.423+0000 7fb49562c700 1 --2- 172.21.15.112:0/3017851226 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb470061170 0x7fb470063620 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:16.426 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.423+0000 7fb49562c700 1 --2- 172.21.15.112:0/3017851226 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb490067fe0 0x7fb4900a6870 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:16.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.423+0000 7fb49562c700 1 --2- 172.21.15.112:0/3017851226 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb49006eec0 0x7fb4900a7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:16.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.423+0000 7fb49562c700 1 --2- 172.21.15.112:0/3017851226 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb49006a500 0x7fb490151660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:16.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.423+0000 7fb49562c700 1 -- 172.21.15.112:0/3017851226 >> 172.21.15.112:0/3017851226 conn(0x7fb4900c8940 msgr2=0x7fb4900be3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:16.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.423+0000 7fb49562c700 1 -- 172.21.15.112:0/3017851226 shutdown_connections 2023-01-24T03:30:16.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.423+0000 7fb49562c700 1 -- 172.21.15.112:0/3017851226 wait complete. 2023-01-24T03:30:16.440 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-01-24T03:30:14.931166+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-01-24T03:30:15.023730+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-01-24T03:30:16.441 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-01-24T03:30:16.559 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.557+0000 7f9aab8a9700 1 Processor -- start 2023-01-24T03:30:16.560 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.558+0000 7f9aab8a9700 1 -- start start 2023-01-24T03:30:16.562 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.558+0000 7f9aab8a9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9aa4155dc0 0x7f9aa406c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:16.563 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.558+0000 7f9aab8a9700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f9aa40c8e30 0x7f9aa4156190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:16.563 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.558+0000 7f9aab8a9700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aa406c190 con 0x7f9aa41566d0 2023-01-24T03:30:16.563 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.558+0000 7f9aab8a9700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9aa406ecf0 con 0x7f9aa40c8e30 2023-01-24T03:30:16.563 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.558+0000 7f9aab8a9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa406f490 con 0x7f9aa4155dc0 2023-01-24T03:30:16.563 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.559+0000 7f9aa8e44700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f9aa40c8e30 0x7f9aa4156190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:16.564 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.559+0000 7f9aa9645700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9aa4155dc0 0x7f9aa406c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:16.564 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.559+0000 7f9aa9e46700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f9aa41566d0 0x7f9aa414bf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.112:44670/0 (socket says 172.21.15.112:44670) 2023-01-24T03:30:16.564 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.559+0000 7f9aa9e46700 1 -- 172.21.15.112:0/439936832 learned_addr learned my addr 172.21.15.112:0/439936832 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:16.564 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.559+0000 7f9aa9645700 1 -- 172.21.15.112:0/439936832 >> v2:172.21.15.142:3300/0 conn(0x7f9aa40c8e30 msgr2=0x7f9aa4156190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:16.564 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.559+0000 7f9aa9645700 1 --2- 172.21.15.112:0/439936832 >> v2:172.21.15.142:3300/0 conn(0x7f9aa40c8e30 0x7f9aa4156190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:16.565 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.559+0000 7f9aa9645700 1 -- 172.21.15.112:0/439936832 >> v1:172.21.15.142:6789/0 conn(0x7f9aa41566d0 legacy=0x7f9aa414bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:16.565 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.559+0000 7f9aa9645700 1 -- 172.21.15.112:0/439936832 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa406eed0 con 0x7f9aa4155dc0 2023-01-24T03:30:16.565 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.559+0000 7f9a93fff700 1 -- 172.21.15.112:0/439936832 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1773998981 0 0) 0x7f9aa406c190 con 0x7f9aa41566d0 2023-01-24T03:30:16.565 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.559+0000 7f9aa9645700 1 --2- 172.21.15.112:0/439936832 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9aa4155dc0 0x7f9aa406c620 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9a94009760 tx=0x7f9a94009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d577e45e0e8788a0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:16.565 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.559+0000 7f9aa8e44700 1 --2- 172.21.15.112:0/439936832 >> v2:172.21.15.142:3300/0 conn(0x7f9aa40c8e30 0x7f9aa4156190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:30:16.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.559+0000 7f9a93fff700 1 -- 172.21.15.112:0/439936832 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a94011630 con 0x7f9aa4155dc0 2023-01-24T03:30:16.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.559+0000 7f9a93fff700 1 -- 172.21.15.112:0/439936832 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a94011790 con 0x7f9aa4155dc0 2023-01-24T03:30:16.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.559+0000 7f9a93fff700 1 -- 172.21.15.112:0/439936832 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a94011b10 con 0x7f9aa4155dc0 2023-01-24T03:30:16.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.560+0000 7f9aab8a9700 1 -- 172.21.15.112:0/439936832 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9aa4155dc0 msgr2=0x7f9aa406c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:16.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.560+0000 7f9aab8a9700 1 --2- 172.21.15.112:0/439936832 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9aa4155dc0 0x7f9aa406c620 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9a94009760 tx=0x7f9a94009060 comp rx=0 tx=0).stop 2023-01-24T03:30:16.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.560+0000 7f9aab8a9700 1 -- 172.21.15.112:0/439936832 shutdown_connections 2023-01-24T03:30:16.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.560+0000 7f9aab8a9700 1 --2- 172.21.15.112:0/439936832 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9aa4155dc0 0x7f9aa406c620 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:16.568 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.560+0000 7f9aab8a9700 1 --2- 172.21.15.112:0/439936832 >> v2:172.21.15.142:3300/0 conn(0x7f9aa40c8e30 0x7f9aa4156190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:16.568 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.560+0000 7f9aab8a9700 1 -- 172.21.15.112:0/439936832 >> 172.21.15.112:0/439936832 conn(0x7f9aa40689f0 msgr2=0x7f9aa40c8720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:16.568 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.560+0000 7f9aab8a9700 1 -- 172.21.15.112:0/439936832 shutdown_connections 2023-01-24T03:30:16.569 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.560+0000 7f9aab8a9700 1 -- 172.21.15.112:0/439936832 wait complete. 2023-01-24T03:30:16.573 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.561+0000 7f9aab8a9700 1 Processor -- start 2023-01-24T03:30:16.573 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.561+0000 7f9aab8a9700 1 -- start start 2023-01-24T03:30:16.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.562+0000 7f9aab8a9700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9aa40c8e30 0x7f9aa4156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:16.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.562+0000 7f9aab8a9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9aa4155dc0 0x7f9aa41667c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:16.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.562+0000 7f9aab8a9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9aa41566d0 0x7f9aa4166d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:16.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.562+0000 7f9aab8a9700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa4155a20 con 0x7f9aa41566d0 2023-01-24T03:30:16.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.562+0000 7f9aab8a9700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa4152d00 con 0x7f9aa40c8e30 2023-01-24T03:30:16.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.562+0000 7f9aab8a9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa4155480 con 0x7f9aa4155dc0 2023-01-24T03:30:16.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.562+0000 7f9aa9645700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9aa40c8e30 0x7f9aa4156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:16.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.562+0000 7f9aa8e44700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9aa4155dc0 0x7f9aa41667c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:16.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.562+0000 7f9aa9e46700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9aa41566d0 0x7f9aa4166d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:16.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.562+0000 7f9aa9645700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9aa40c8e30 0x7f9aa4156dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.112:42388/0 (socket says 172.21.15.112:42388) 2023-01-24T03:30:16.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.562+0000 7f9aa8e44700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9aa4155dc0 0x7f9aa41667c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.112:37198/0 (socket says 172.21.15.112:37198) 2023-01-24T03:30:16.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.562+0000 7f9aa9645700 1 -- 172.21.15.112:0/1859619947 learned_addr learned my addr 172.21.15.112:0/1859619947 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:16.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.562+0000 7f9aa9645700 1 -- 172.21.15.112:0/1859619947 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9aa4155dc0 msgr2=0x7f9aa41667c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:16.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.562+0000 7f9aa9645700 1 --2- 172.21.15.112:0/1859619947 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9aa4155dc0 0x7f9aa41667c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:16.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.562+0000 7f9aa9645700 1 -- 172.21.15.112:0/1859619947 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9aa41566d0 msgr2=0x7f9aa4166d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:16.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.562+0000 7f9aa9645700 1 --2- 172.21.15.112:0/1859619947 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9aa41566d0 0x7f9aa4166d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:16.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.562+0000 7f9aa9645700 1 -- 172.21.15.112:0/1859619947 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa00092a0 con 0x7f9aa40c8e30 2023-01-24T03:30:16.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.563+0000 7f9aa9645700 1 --2- 172.21.15.112:0/1859619947 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9aa40c8e30 0x7f9aa4156dc0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9a94009e00 tx=0x7f9a94017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:16.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.563+0000 7f9a917fa700 1 -- 172.21.15.112:0/1859619947 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a94011900 con 0x7f9aa40c8e30 2023-01-24T03:30:16.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.563+0000 7f9a917fa700 1 -- 172.21.15.112:0/1859619947 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a94011a60 con 0x7f9aa40c8e30 2023-01-24T03:30:16.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.563+0000 7f9aab8a9700 1 -- 172.21.15.112:0/1859619947 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a940092a0 con 0x7f9aa40c8e30 2023-01-24T03:30:16.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.563+0000 7f9a917fa700 1 -- 172.21.15.112:0/1859619947 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a94023640 con 0x7f9aa40c8e30 2023-01-24T03:30:16.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.563+0000 7f9aab8a9700 1 -- 172.21.15.112:0/1859619947 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aa41523d0 con 0x7f9aa40c8e30 2023-01-24T03:30:16.579 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.564+0000 7f9aab8a9700 1 -- 172.21.15.112:0/1859619947 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9aa4166610 con 0x7f9aa40c8e30 2023-01-24T03:30:16.579 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.564+0000 7f9a917fa700 1 -- 172.21.15.112:0/1859619947 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 4) v1 ==== 79852+0+0 (secure 0 0 0) 0x7f9a9402a030 con 0x7f9aa40c8e30 2023-01-24T03:30:16.579 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.564+0000 7f9a917fa700 1 --2- 172.21.15.112:0/1859619947 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9a84061170 0x7f9a84063620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:16.579 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.564+0000 7f9a917fa700 1 -- 172.21.15.112:0/1859619947 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f9a9407e340 con 0x7f9aa40c8e30 2023-01-24T03:30:16.580 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.569+0000 7f9a917fa700 1 -- 172.21.15.112:0/1859619947 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f9a9404e940 con 0x7f9aa40c8e30 2023-01-24T03:30:16.580 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.571+0000 7f9aa8e44700 1 --2- 172.21.15.112:0/1859619947 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9a84061170 0x7f9a84063620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:16.581 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.571+0000 7f9aa8e44700 1 --2- 172.21.15.112:0/1859619947 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9a84061170 0x7f9a84063620 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9aa41675d0 tx=0x7f9aa0009060 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:16.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.755+0000 7f9aab8a9700 1 -- 172.21.15.112:0/1859619947 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f9aa405e820 con 0x7f9aa40c8e30 2023-01-24T03:30:16.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.756+0000 7f9a917fa700 1 -- 172.21.15.112:0/1859619947 <== mon.1 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v15) v1 ==== 92+0+10 (secure 0 0 0) 0x7f9a94051f60 con 0x7f9aa40c8e30 2023-01-24T03:30:16.758 INFO:teuthology.orchestra.run.smithi112.stdout:pg_num: 8 2023-01-24T03:30:16.760 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.758+0000 7f9aab8a9700 1 -- 172.21.15.112:0/1859619947 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9a84061170 msgr2=0x7f9a84063620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:16.761 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.758+0000 7f9aab8a9700 1 --2- 172.21.15.112:0/1859619947 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9a84061170 0x7f9a84063620 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9aa41675d0 tx=0x7f9aa0009060 comp rx=0 tx=0).stop 2023-01-24T03:30:16.761 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.758+0000 7f9aab8a9700 1 -- 172.21.15.112:0/1859619947 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9aa40c8e30 msgr2=0x7f9aa4156dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:16.761 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.758+0000 7f9aab8a9700 1 --2- 172.21.15.112:0/1859619947 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9aa40c8e30 0x7f9aa4156dc0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9a94009e00 tx=0x7f9a94017000 comp rx=0 tx=0).stop 2023-01-24T03:30:16.761 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.758+0000 7f9aab8a9700 1 -- 172.21.15.112:0/1859619947 shutdown_connections 2023-01-24T03:30:16.761 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.758+0000 7f9aab8a9700 1 --2- 172.21.15.112:0/1859619947 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9a84061170 0x7f9a84063620 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:16.762 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.758+0000 7f9aab8a9700 1 --2- 172.21.15.112:0/1859619947 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9aa41566d0 0x7f9aa4166d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:16.762 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.758+0000 7f9aab8a9700 1 --2- 172.21.15.112:0/1859619947 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9aa40c8e30 0x7f9aa4156dc0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:16.762 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.758+0000 7f9aab8a9700 1 --2- 172.21.15.112:0/1859619947 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9aa4155dc0 0x7f9aa41667c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:16.762 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.758+0000 7f9aab8a9700 1 -- 172.21.15.112:0/1859619947 >> 172.21.15.112:0/1859619947 conn(0x7f9aa40689f0 msgr2=0x7f9aa40c2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:16.762 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.758+0000 7f9aab8a9700 1 -- 172.21.15.112:0/1859619947 shutdown_connections 2023-01-24T03:30:16.763 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.758+0000 7f9aab8a9700 1 -- 172.21.15.112:0/1859619947 wait complete. 2023-01-24T03:30:16.771 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-01-24T03:30:16.887 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.885+0000 7f82c759e700 1 Processor -- start 2023-01-24T03:30:16.887 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.885+0000 7f82c759e700 1 -- start start 2023-01-24T03:30:16.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.886+0000 7f82c759e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f82c80c6e40 0x7f82c80c8750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:16.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.886+0000 7f82c759e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82c80674e0 con 0x7f82c80c8c90 2023-01-24T03:30:16.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.886+0000 7f82c759e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82c8068060 con 0x7f82c80c6a70 2023-01-24T03:30:16.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.886+0000 7f82c759e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82c814f800 con 0x7f82c80c6e40 2023-01-24T03:30:16.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.886+0000 7f82c6d9d700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f82c80c8c90 0x7f82c8158550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:59136/0 (socket says 172.21.15.112:59136) 2023-01-24T03:30:16.891 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.886+0000 7f82c6d9d700 1 -- 172.21.15.112:0/3432043724 learned_addr learned my addr 172.21.15.112:0/3432043724 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:16.891 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.886+0000 7f82bffff700 1 --2- 172.21.15.112:0/3432043724 >> v2:172.21.15.142:3300/0 conn(0x7f82c80c6e40 0x7f82c80c8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:16.891 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.886+0000 7f82c5d9b700 1 -- 172.21.15.112:0/3432043724 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1430572659 0 0) 0x7f82c80674e0 con 0x7f82c80c8c90 2023-01-24T03:30:16.891 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.886+0000 7f82c5d9b700 1 -- 172.21.15.112:0/3432043724 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82a80033f0 con 0x7f82c80c8c90 2023-01-24T03:30:16.891 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.886+0000 7f82c5d9b700 1 -- 172.21.15.112:0/3432043724 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1218618861 0 0) 0x7f82c8068060 con 0x7f82c80c6a70 2023-01-24T03:30:16.892 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.886+0000 7f82c5d9b700 1 -- 172.21.15.112:0/3432043724 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82c80674e0 con 0x7f82c80c6a70 2023-01-24T03:30:16.892 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.886+0000 7f82bffff700 1 -- 172.21.15.112:0/3432043724 >> v1:172.21.15.142:6789/0 conn(0x7f82c80c6a70 legacy=0x7f82c80c2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:16.892 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.886+0000 7f82bffff700 1 -- 172.21.15.112:0/3432043724 >> v1:172.21.15.112:6789/0 conn(0x7f82c80c8c90 legacy=0x7f82c8158550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:16.892 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.886+0000 7f82bffff700 1 -- 172.21.15.112:0/3432043724 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c8159270 con 0x7f82c80c6e40 2023-01-24T03:30:16.892 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.886+0000 7f82bffff700 1 --2- 172.21.15.112:0/3432043724 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f82c80c6e40 0x7f82c80c8750 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f82b0009760 tx=0x7f82b0009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d60a534fa7a12f85 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:16.893 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.887+0000 7f82c5d9b700 1 -- 172.21.15.112:0/3432043724 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b00105b0 con 0x7f82c80c6e40 2023-01-24T03:30:16.893 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.887+0000 7f82c5d9b700 1 -- 172.21.15.112:0/3432043724 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82b0010710 con 0x7f82c80c6e40 2023-01-24T03:30:16.893 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.887+0000 7f82c5d9b700 1 -- 172.21.15.112:0/3432043724 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b0010a90 con 0x7f82c80c6e40 2023-01-24T03:30:16.893 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.887+0000 7f82c759e700 1 -- 172.21.15.112:0/3432043724 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f82c80c6e40 msgr2=0x7f82c80c8750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:16.893 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.887+0000 7f82c759e700 1 --2- 172.21.15.112:0/3432043724 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f82c80c6e40 0x7f82c80c8750 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f82b0009760 tx=0x7f82b0009060 comp rx=0 tx=0).stop 2023-01-24T03:30:16.893 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.887+0000 7f82c759e700 1 -- 172.21.15.112:0/3432043724 shutdown_connections 2023-01-24T03:30:16.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.887+0000 7f82c759e700 1 --2- 172.21.15.112:0/3432043724 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f82c80c6e40 0x7f82c80c8750 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:16.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.887+0000 7f82c759e700 1 -- 172.21.15.112:0/3432043724 >> 172.21.15.112:0/3432043724 conn(0x7f82c805d830 msgr2=0x7f82c805dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:16.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.887+0000 7f82c759e700 1 -- 172.21.15.112:0/3432043724 shutdown_connections 2023-01-24T03:30:16.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.887+0000 7f82c759e700 1 -- 172.21.15.112:0/3432043724 wait complete. 2023-01-24T03:30:16.900 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.888+0000 7f82c759e700 1 Processor -- start 2023-01-24T03:30:16.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.888+0000 7f82c759e700 1 -- start start 2023-01-24T03:30:16.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.889+0000 7f82c759e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f82c80c6a70 0x7f82c80ca030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:16.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.889+0000 7f82c759e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82c80c6e40 0x7f82c8168580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:16.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.889+0000 7f82c759e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f82c80c8c90 0x7f82c8168ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:16.902 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.889+0000 7f82c759e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c80cbb70 con 0x7f82c80c6a70 2023-01-24T03:30:16.902 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.889+0000 7f82c759e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c80c2180 con 0x7f82c80c8c90 2023-01-24T03:30:16.902 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.889+0000 7f82c759e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82c80c25d0 con 0x7f82c80c6e40 2023-01-24T03:30:16.902 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.889+0000 7f82c659c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f82c80c6a70 0x7f82c80ca030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:16.902 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.889+0000 7f82c659c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f82c80c6a70 0x7f82c80ca030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:49918/0 (socket says 172.21.15.112:49918) 2023-01-24T03:30:16.903 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.889+0000 7f82c659c700 1 -- 172.21.15.112:0/1401906884 learned_addr learned my addr 172.21.15.112:0/1401906884 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:16.903 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.889+0000 7f82bffff700 1 --2- 172.21.15.112:0/1401906884 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82c80c6e40 0x7f82c8168580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:16.903 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.889+0000 7f82c6d9d700 1 --2- 172.21.15.112:0/1401906884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f82c80c8c90 0x7f82c8168ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:16.903 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.889+0000 7f82c659c700 1 -- 172.21.15.112:0/1401906884 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82c80c6e40 msgr2=0x7f82c8168580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:16.903 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.889+0000 7f82c659c700 1 --2- 172.21.15.112:0/1401906884 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82c80c6e40 0x7f82c8168580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:16.904 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.889+0000 7f82c659c700 1 -- 172.21.15.112:0/1401906884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f82c80c8c90 msgr2=0x7f82c8168ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:16.904 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.889+0000 7f82c659c700 1 --2- 172.21.15.112:0/1401906884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f82c80c8c90 0x7f82c8168ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:16.904 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.889+0000 7f82c659c700 1 -- 172.21.15.112:0/1401906884 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b00092a0 con 0x7f82c80c6a70 2023-01-24T03:30:16.904 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.889+0000 7f82c659c700 1 --2- 172.21.15.112:0/1401906884 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f82c80c6a70 0x7f82c80ca030 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f82b800c490 tx=0x7f82b8007c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:16.904 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.889+0000 7f82be7fc700 1 -- 172.21.15.112:0/1401906884 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b800ce40 con 0x7f82c80c6a70 2023-01-24T03:30:16.905 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.890+0000 7f82be7fc700 1 -- 172.21.15.112:0/1401906884 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82b8011590 con 0x7f82c80c6a70 2023-01-24T03:30:16.905 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.890+0000 7f82c759e700 1 -- 172.21.15.112:0/1401906884 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82c80cc440 con 0x7f82c80c6a70 2023-01-24T03:30:16.905 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.890+0000 7f82be7fc700 1 -- 172.21.15.112:0/1401906884 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b8011950 con 0x7f82c80c6a70 2023-01-24T03:30:16.905 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.890+0000 7f82c759e700 1 -- 172.21.15.112:0/1401906884 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82c80cbd80 con 0x7f82c80c6a70 2023-01-24T03:30:16.905 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.890+0000 7f82c759e700 1 -- 172.21.15.112:0/1401906884 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82c80c2420 con 0x7f82c80c6a70 2023-01-24T03:30:16.906 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.891+0000 7f82be7fc700 1 -- 172.21.15.112:0/1401906884 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 4) v1 ==== 79852+0+0 (secure 0 0 0) 0x7f82b8010050 con 0x7f82c80c6a70 2023-01-24T03:30:16.906 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.891+0000 7f82be7fc700 1 --2- 172.21.15.112:0/1401906884 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f82ac0611a0 0x7f82ac063650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:16.906 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.891+0000 7f82be7fc700 1 -- 172.21.15.112:0/1401906884 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f82b807cb80 con 0x7f82c80c6a70 2023-01-24T03:30:16.906 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.891+0000 7f82bffff700 1 --2- 172.21.15.112:0/1401906884 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f82ac0611a0 0x7f82ac063650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:16.906 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.891+0000 7f82bffff700 1 --2- 172.21.15.112:0/1401906884 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f82ac0611a0 0x7f82ac063650 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f82c8157d10 tx=0x7f82b0016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:16.907 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:16.895+0000 7f82be7fc700 1 -- 172.21.15.112:0/1401906884 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f82b804cff0 con 0x7f82c80c6a70 2023-01-24T03:30:17.084 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.082+0000 7f82c759e700 1 -- 172.21.15.112:0/1401906884 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f82c80894d0 con 0x7f82c80c6a70 2023-01-24T03:30:17.085 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.083+0000 7f82be7fc700 1 -- 172.21.15.112:0/1401906884 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v15) v1 ==== 93+0+10 (secure 0 0 0) 0x7f82b8050610 con 0x7f82c80c6a70 2023-01-24T03:30:17.085 INFO:teuthology.orchestra.run.smithi112.stdout:pg_num: 1 2023-01-24T03:30:17.088 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.085+0000 7f82c759e700 1 -- 172.21.15.112:0/1401906884 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f82ac0611a0 msgr2=0x7f82ac063650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:17.088 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.085+0000 7f82c759e700 1 --2- 172.21.15.112:0/1401906884 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f82ac0611a0 0x7f82ac063650 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f82c8157d10 tx=0x7f82b0016000 comp rx=0 tx=0).stop 2023-01-24T03:30:17.088 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.085+0000 7f82c759e700 1 -- 172.21.15.112:0/1401906884 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f82c80c6a70 msgr2=0x7f82c80ca030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:17.088 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.085+0000 7f82c759e700 1 --2- 172.21.15.112:0/1401906884 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f82c80c6a70 0x7f82c80ca030 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f82b800c490 tx=0x7f82b8007c90 comp rx=0 tx=0).stop 2023-01-24T03:30:17.089 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.085+0000 7f82c759e700 1 -- 172.21.15.112:0/1401906884 shutdown_connections 2023-01-24T03:30:17.089 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.085+0000 7f82c759e700 1 --2- 172.21.15.112:0/1401906884 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f82ac0611a0 0x7f82ac063650 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:17.089 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.085+0000 7f82c759e700 1 --2- 172.21.15.112:0/1401906884 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f82c80c6a70 0x7f82c80ca030 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:17.089 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.085+0000 7f82c759e700 1 --2- 172.21.15.112:0/1401906884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f82c80c8c90 0x7f82c8168ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:17.089 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.085+0000 7f82c759e700 1 --2- 172.21.15.112:0/1401906884 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f82c80c6e40 0x7f82c8168580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:17.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.085+0000 7f82c759e700 1 -- 172.21.15.112:0/1401906884 >> 172.21.15.112:0/1401906884 conn(0x7f82c805d830 msgr2=0x7f82c814efc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:17.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.085+0000 7f82c759e700 1 -- 172.21.15.112:0/1401906884 shutdown_connections 2023-01-24T03:30:17.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.085+0000 7f82c759e700 1 -- 172.21.15.112:0/1401906884 wait complete. 2023-01-24T03:30:17.098 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:30:17.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.213+0000 7fc2a1cc0700 1 Processor -- start 2023-01-24T03:30:17.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.213+0000 7fc2a1cc0700 1 -- start start 2023-01-24T03:30:17.218 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.214+0000 7fc2a1cc0700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fc29c06a550 0x7fc29c06a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:17.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.214+0000 7fc2a1cc0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc29c0681f0 0x7fc29c0c3c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:17.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.214+0000 7fc2a1cc0700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc29c06d930 con 0x7fc29c067e20 2023-01-24T03:30:17.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.214+0000 7fc2a1cc0700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc29c153150 con 0x7fc29c06a550 2023-01-24T03:30:17.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.214+0000 7fc2a1cc0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc29c1528d0 con 0x7fc29c0681f0 2023-01-24T03:30:17.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.214+0000 7fc29b7fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fc29c067e20 0x7fc29c152120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.112:44716/0 (socket says 172.21.15.112:44716) 2023-01-24T03:30:17.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.214+0000 7fc29b7fe700 1 -- 172.21.15.112:0/1120734152 learned_addr learned my addr 172.21.15.112:0/1120734152 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:17.221 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.214+0000 7fc29bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc29c0681f0 0x7fc29c0c3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:17.221 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.214+0000 7fc29affd700 1 --2- 172.21.15.112:0/1120734152 >> v2:172.21.15.142:3300/0 conn(0x7fc29c06a550 0x7fc29c06a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:17.221 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.214+0000 7fc29a7fc700 1 -- 172.21.15.112:0/1120734152 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3724753606 0 0) 0x7fc29c06d930 con 0x7fc29c067e20 2023-01-24T03:30:17.221 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.214+0000 7fc29a7fc700 1 -- 172.21.15.112:0/1120734152 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2880033f0 con 0x7fc29c067e20 2023-01-24T03:30:17.221 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.214+0000 7fc29affd700 1 -- 172.21.15.112:0/1120734152 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc29c0681f0 msgr2=0x7fc29c0c3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:17.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.214+0000 7fc29affd700 1 --2- 172.21.15.112:0/1120734152 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc29c0681f0 0x7fc29c0c3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:17.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.214+0000 7fc29affd700 1 -- 172.21.15.112:0/1120734152 >> v1:172.21.15.142:6789/0 conn(0x7fc29c067e20 legacy=0x7fc29c152120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:17.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.215+0000 7fc29affd700 1 -- 172.21.15.112:0/1120734152 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc29c152b90 con 0x7fc29c06a550 2023-01-24T03:30:17.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.215+0000 7fc29bfff700 1 --2- 172.21.15.112:0/1120734152 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc29c0681f0 0x7fc29c0c3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:30:17.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.215+0000 7fc29affd700 1 --2- 172.21.15.112:0/1120734152 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc29c06a550 0x7fc29c06a9f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc290009760 tx=0x7fc290009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98e3957da74b18cb server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:17.223 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.215+0000 7fc29a7fc700 1 -- 172.21.15.112:0/1120734152 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2900105b0 con 0x7fc29c06a550 2023-01-24T03:30:17.223 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.215+0000 7fc29a7fc700 1 -- 172.21.15.112:0/1120734152 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc290010710 con 0x7fc29c06a550 2023-01-24T03:30:17.223 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.215+0000 7fc29a7fc700 1 -- 172.21.15.112:0/1120734152 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc290010ab0 con 0x7fc29c06a550 2023-01-24T03:30:17.223 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.215+0000 7fc2a1cc0700 1 -- 172.21.15.112:0/1120734152 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc29c06a550 msgr2=0x7fc29c06a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:17.223 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.215+0000 7fc2a1cc0700 1 --2- 172.21.15.112:0/1120734152 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc29c06a550 0x7fc29c06a9f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc290009760 tx=0x7fc290009060 comp rx=0 tx=0).stop 2023-01-24T03:30:17.224 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.215+0000 7fc2a1cc0700 1 -- 172.21.15.112:0/1120734152 shutdown_connections 2023-01-24T03:30:17.224 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.215+0000 7fc2a1cc0700 1 --2- 172.21.15.112:0/1120734152 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc29c0681f0 0x7fc29c0c3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:17.224 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.215+0000 7fc2a1cc0700 1 --2- 172.21.15.112:0/1120734152 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc29c06a550 0x7fc29c06a9f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:17.224 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.215+0000 7fc2a1cc0700 1 -- 172.21.15.112:0/1120734152 >> 172.21.15.112:0/1120734152 conn(0x7fc29c0c8940 msgr2=0x7fc29c0c4910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:17.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.215+0000 7fc2a1cc0700 1 -- 172.21.15.112:0/1120734152 shutdown_connections 2023-01-24T03:30:17.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.215+0000 7fc2a1cc0700 1 -- 172.21.15.112:0/1120734152 wait complete. 2023-01-24T03:30:17.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.217+0000 7fc2a1cc0700 1 Processor -- start 2023-01-24T03:30:17.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.217+0000 7fc2a1cc0700 1 -- start start 2023-01-24T03:30:17.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.217+0000 7fc2a1cc0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc29c067e20 0x7fc29c165bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:17.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.217+0000 7fc2a1cc0700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc29c0681f0 0x7fc29c14ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:17.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.217+0000 7fc2a1cc0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc29c06a550 0x7fc29c0c01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:17.229 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.217+0000 7fc2a1cc0700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29c150b80 con 0x7fc29c067e20 2023-01-24T03:30:17.229 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.217+0000 7fc2a1cc0700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29c150e60 con 0x7fc29c0681f0 2023-01-24T03:30:17.229 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.217+0000 7fc2a1cc0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc29c151d40 con 0x7fc29c06a550 2023-01-24T03:30:17.229 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.217+0000 7fc29b7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc29c067e20 0x7fc29c165bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:17.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.217+0000 7fc29b7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc29c067e20 0x7fc29c165bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:49942/0 (socket says 172.21.15.112:49942) 2023-01-24T03:30:17.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.217+0000 7fc29b7fe700 1 -- 172.21.15.112:0/3758357050 learned_addr learned my addr 172.21.15.112:0/3758357050 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:17.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.217+0000 7fc29affd700 1 --2- 172.21.15.112:0/3758357050 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc29c0681f0 0x7fc29c14ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:17.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.217+0000 7fc29bfff700 1 --2- 172.21.15.112:0/3758357050 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc29c06a550 0x7fc29c0c01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:17.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.218+0000 7fc29b7fe700 1 -- 172.21.15.112:0/3758357050 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc29c06a550 msgr2=0x7fc29c0c01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:17.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.218+0000 7fc29b7fe700 1 --2- 172.21.15.112:0/3758357050 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc29c06a550 0x7fc29c0c01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:17.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.218+0000 7fc29b7fe700 1 -- 172.21.15.112:0/3758357050 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc29c0681f0 msgr2=0x7fc29c14ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:17.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.218+0000 7fc29b7fe700 1 --2- 172.21.15.112:0/3758357050 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc29c0681f0 0x7fc29c14ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:17.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.218+0000 7fc29b7fe700 1 -- 172.21.15.112:0/3758357050 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc294009a20 con 0x7fc29c067e20 2023-01-24T03:30:17.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.218+0000 7fc29b7fe700 1 --2- 172.21.15.112:0/3758357050 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc29c067e20 0x7fc29c165bd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc28c00c590 tx=0x7fc28c007c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:17.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.218+0000 7fc2837fe700 1 -- 172.21.15.112:0/3758357050 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc28c0072e0 con 0x7fc29c067e20 2023-01-24T03:30:17.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.218+0000 7fc2837fe700 1 -- 172.21.15.112:0/3758357050 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc28c007480 con 0x7fc29c067e20 2023-01-24T03:30:17.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.218+0000 7fc2a1cc0700 1 -- 172.21.15.112:0/3758357050 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2900092a0 con 0x7fc29c067e20 2023-01-24T03:30:17.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.218+0000 7fc2837fe700 1 -- 172.21.15.112:0/3758357050 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc28c0059d0 con 0x7fc29c067e20 2023-01-24T03:30:17.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.218+0000 7fc2a1cc0700 1 -- 172.21.15.112:0/3758357050 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc29c1524b0 con 0x7fc29c067e20 2023-01-24T03:30:17.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.219+0000 7fc2a1cc0700 1 -- 172.21.15.112:0/3758357050 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc29c151ba0 con 0x7fc29c067e20 2023-01-24T03:30:17.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.219+0000 7fc2837fe700 1 -- 172.21.15.112:0/3758357050 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 4) v1 ==== 79852+0+0 (secure 0 0 0) 0x7fc28c019030 con 0x7fc29c067e20 2023-01-24T03:30:17.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.219+0000 7fc2837fe700 1 --2- 172.21.15.112:0/3758357050 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc27c0611a0 0x7fc27c063650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:17.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.220+0000 7fc29affd700 1 --2- 172.21.15.112:0/3758357050 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc27c0611a0 0x7fc27c063650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:17.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.220+0000 7fc2837fe700 1 -- 172.21.15.112:0/3758357050 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fc28c07c1f0 con 0x7fc29c067e20 2023-01-24T03:30:17.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.220+0000 7fc29affd700 1 --2- 172.21.15.112:0/3758357050 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc27c0611a0 0x7fc27c063650 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc290009760 tx=0x7fc290016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:17.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.224+0000 7fc2837fe700 1 -- 172.21.15.112:0/3758357050 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fc28c04a1d0 con 0x7fc29c067e20 2023-01-24T03:30:17.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.445+0000 7fc2a1cc0700 1 -- 172.21.15.112:0/3758357050 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc29c168740 con 0x7fc29c067e20 2023-01-24T03:30:17.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.445+0000 7fc2837fe700 1 -- 172.21.15.112:0/3758357050 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+5042 (secure 0 0 0) 0x7fc29c151ba0 con 0x7fc29c067e20 2023-01-24T03:30:17.447 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:30:17.448 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/869267413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":869267413},{"type":"v1","addr":"172.21.15.142:6834","nonce":869267413}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4293,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/2762621058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2762621058},{"type":"v1","addr":"172.21.15.142:6835","nonce":2762621058}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-01-24T03:30:17.451 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.447+0000 7fc2a1cc0700 1 -- 172.21.15.112:0/3758357050 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc27c0611a0 msgr2=0x7fc27c063650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:17.451 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.447+0000 7fc2a1cc0700 1 --2- 172.21.15.112:0/3758357050 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc27c0611a0 0x7fc27c063650 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc290009760 tx=0x7fc290016000 comp rx=0 tx=0).stop 2023-01-24T03:30:17.452 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.447+0000 7fc2a1cc0700 1 -- 172.21.15.112:0/3758357050 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc29c067e20 msgr2=0x7fc29c165bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:17.452 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.447+0000 7fc2a1cc0700 1 --2- 172.21.15.112:0/3758357050 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc29c067e20 0x7fc29c165bd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc28c00c590 tx=0x7fc28c007c80 comp rx=0 tx=0).stop 2023-01-24T03:30:17.452 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.447+0000 7fc2a1cc0700 1 -- 172.21.15.112:0/3758357050 shutdown_connections 2023-01-24T03:30:17.452 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.447+0000 7fc2a1cc0700 1 --2- 172.21.15.112:0/3758357050 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc27c0611a0 0x7fc27c063650 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:17.453 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.447+0000 7fc2a1cc0700 1 --2- 172.21.15.112:0/3758357050 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc29c067e20 0x7fc29c165bd0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:17.453 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.447+0000 7fc2a1cc0700 1 --2- 172.21.15.112:0/3758357050 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc29c06a550 0x7fc29c0c01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:17.453 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.447+0000 7fc2a1cc0700 1 --2- 172.21.15.112:0/3758357050 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc29c0681f0 0x7fc29c14ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:17.453 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.447+0000 7fc2a1cc0700 1 -- 172.21.15.112:0/3758357050 >> 172.21.15.112:0/3758357050 conn(0x7fc29c0c8940 msgr2=0x7fc29c0c71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:17.453 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.447+0000 7fc2a1cc0700 1 -- 172.21.15.112:0/3758357050 shutdown_connections 2023-01-24T03:30:17.454 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.447+0000 7fc2a1cc0700 1 -- 172.21.15.112:0/3758357050 wait complete. 2023-01-24T03:30:17.454 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 3 2023-01-24T03:30:17.461 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T03:30:17.573 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.571+0000 7f95a98f7700 1 Processor -- start 2023-01-24T03:30:17.573 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.571+0000 7f95a98f7700 1 -- start start 2023-01-24T03:30:17.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.571+0000 7f95a98f7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95a4068b10 0x7f95a40cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:17.579 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.571+0000 7f95a98f7700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f95a40aef60 0x7f95a40b14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:17.579 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.571+0000 7f95a98f7700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a41560e0 con 0x7f95a40aeb90 2023-01-24T03:30:17.579 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.571+0000 7f95a98f7700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95a4155530 con 0x7f95a40aef60 2023-01-24T03:30:17.580 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.571+0000 7f95a98f7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95a4069ac0 con 0x7f95a4068b10 2023-01-24T03:30:17.580 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.572+0000 7f95a37fe700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f95a40aef60 0x7f95a40b14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:17.580 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.572+0000 7f95a37fe700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f95a40aef60 0x7f95a40b14c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:49952/0 (socket says 172.21.15.112:49952) 2023-01-24T03:30:17.580 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.572+0000 7f95a37fe700 1 -- 172.21.15.112:0/3586763492 learned_addr learned my addr 172.21.15.112:0/3586763492 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:17.580 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.572+0000 7f95a27fc700 1 --2- 172.21.15.112:0/3586763492 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95a4068b10 0x7f95a40cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:17.581 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.572+0000 7f95a37fe700 1 -- 172.21.15.112:0/3586763492 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95a4068b10 msgr2=0x7f95a40cc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:17.581 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.572+0000 7f95a37fe700 1 --2- 172.21.15.112:0/3586763492 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95a4068b10 0x7f95a40cc3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:17.581 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.572+0000 7f95a37fe700 1 -- 172.21.15.112:0/3586763492 >> v1:172.21.15.142:6789/0 conn(0x7f95a40aeb90 legacy=0x7f95a4068400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:17.581 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.572+0000 7f95a37fe700 1 -- 172.21.15.112:0/3586763492 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a4069d80 con 0x7f95a40aef60 2023-01-24T03:30:17.582 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.572+0000 7f95a37fe700 1 --2- 172.21.15.112:0/3586763492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f95a40aef60 0x7f95a40b14c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f959c009f20 tx=0x7f959c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e523b967043709f3 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:17.582 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.572+0000 7f95a1ffb700 1 -- 172.21.15.112:0/3586763492 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959c00e070 con 0x7f95a40aef60 2023-01-24T03:30:17.582 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.572+0000 7f95a1ffb700 1 -- 172.21.15.112:0/3586763492 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f959c00ba80 con 0x7f95a40aef60 2023-01-24T03:30:17.582 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.572+0000 7f95a1ffb700 1 -- 172.21.15.112:0/3586763492 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959c0092e0 con 0x7f95a40aef60 2023-01-24T03:30:17.583 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.573+0000 7f95a98f7700 1 -- 172.21.15.112:0/3586763492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f95a40aef60 msgr2=0x7f95a40b14c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:17.583 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.573+0000 7f95a98f7700 1 --2- 172.21.15.112:0/3586763492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f95a40aef60 0x7f95a40b14c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f959c009f20 tx=0x7f959c009840 comp rx=0 tx=0).stop 2023-01-24T03:30:17.583 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.573+0000 7f95a98f7700 1 -- 172.21.15.112:0/3586763492 shutdown_connections 2023-01-24T03:30:17.583 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.573+0000 7f95a98f7700 1 --2- 172.21.15.112:0/3586763492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f95a40aef60 0x7f95a40b14c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:17.583 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.573+0000 7f95a98f7700 1 --2- 172.21.15.112:0/3586763492 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95a4068b10 0x7f95a40cc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:17.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.573+0000 7f95a98f7700 1 -- 172.21.15.112:0/3586763492 >> 172.21.15.112:0/3586763492 conn(0x7f95a405e300 msgr2=0x7f95a40cba20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:17.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.573+0000 7f95a98f7700 1 -- 172.21.15.112:0/3586763492 shutdown_connections 2023-01-24T03:30:17.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.573+0000 7f95a98f7700 1 -- 172.21.15.112:0/3586763492 wait complete. 2023-01-24T03:30:17.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.574+0000 7f95a98f7700 1 Processor -- start 2023-01-24T03:30:17.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.574+0000 7f95a98f7700 1 -- start start 2023-01-24T03:30:17.586 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.575+0000 7f95a98f7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95a4068b10 0x7f95a40ca480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:17.586 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.575+0000 7f95a98f7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95a40aeb90 0x7f95a40c8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:17.587 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.575+0000 7f95a98f7700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f95a40aef60 0x7f95a414cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:17.587 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.575+0000 7f95a98f7700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a40c7180 con 0x7f95a40aef60 2023-01-24T03:30:17.587 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.575+0000 7f95a98f7700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a40c7c60 con 0x7f95a40aeb90 2023-01-24T03:30:17.588 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.575+0000 7f95a98f7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95a4169700 con 0x7f95a4068b10 2023-01-24T03:30:17.588 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.575+0000 7f95a37fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f95a40aef60 0x7f95a414cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:17.589 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.575+0000 7f95a37fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f95a40aef60 0x7f95a414cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:49960/0 (socket says 172.21.15.112:49960) 2023-01-24T03:30:17.589 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.575+0000 7f95a37fe700 1 -- 172.21.15.112:0/331344829 learned_addr learned my addr 172.21.15.112:0/331344829 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:17.589 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.575+0000 7f95a2ffd700 1 --2- 172.21.15.112:0/331344829 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95a4068b10 0x7f95a40ca480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:17.590 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.575+0000 7f95a27fc700 1 --2- 172.21.15.112:0/331344829 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95a40aeb90 0x7f95a40c8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:17.590 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.575+0000 7f95a37fe700 1 -- 172.21.15.112:0/331344829 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95a4068b10 msgr2=0x7f95a40ca480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:17.590 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.575+0000 7f95a37fe700 1 --2- 172.21.15.112:0/331344829 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95a4068b10 0x7f95a40ca480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:17.591 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.575+0000 7f95a37fe700 1 -- 172.21.15.112:0/331344829 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95a40aeb90 msgr2=0x7f95a40c8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:17.591 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.575+0000 7f95a37fe700 1 --2- 172.21.15.112:0/331344829 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95a40aeb90 0x7f95a40c8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:17.591 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.575+0000 7f95a37fe700 1 -- 172.21.15.112:0/331344829 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f959c009ae0 con 0x7f95a40aef60 2023-01-24T03:30:17.592 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.575+0000 7f95a37fe700 1 --2- 172.21.15.112:0/331344829 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f95a40aef60 0x7f95a414cfd0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f959c00bf20 tx=0x7f959c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:17.592 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.575+0000 7f958affd700 1 -- 172.21.15.112:0/331344829 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959c007190 con 0x7f95a40aef60 2023-01-24T03:30:17.593 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.575+0000 7f958affd700 1 -- 172.21.15.112:0/331344829 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f959c007920 con 0x7f95a40aef60 2023-01-24T03:30:17.593 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.575+0000 7f95a98f7700 1 -- 172.21.15.112:0/331344829 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95a4168af0 con 0x7f95a40aef60 2023-01-24T03:30:17.593 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.575+0000 7f958affd700 1 -- 172.21.15.112:0/331344829 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959c007ce0 con 0x7f95a40aef60 2023-01-24T03:30:17.594 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.575+0000 7f95a98f7700 1 -- 172.21.15.112:0/331344829 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95a41690b0 con 0x7f95a40aef60 2023-01-24T03:30:17.594 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.576+0000 7f95a98f7700 1 -- 172.21.15.112:0/331344829 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95a4169840 con 0x7f95a40aef60 2023-01-24T03:30:17.594 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.576+0000 7f958affd700 1 -- 172.21.15.112:0/331344829 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 4) v1 ==== 79852+0+0 (secure 0 0 0) 0x7f959c01d420 con 0x7f95a40aef60 2023-01-24T03:30:17.595 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.577+0000 7f958affd700 1 --2- 172.21.15.112:0/331344829 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9584061170 0x7f9584063620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:17.595 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.577+0000 7f958affd700 1 -- 172.21.15.112:0/331344829 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f959c07f090 con 0x7f95a40aef60 2023-01-24T03:30:17.595 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.577+0000 7f95a2ffd700 1 --2- 172.21.15.112:0/331344829 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9584061170 0x7f9584063620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:17.596 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.577+0000 7f95a2ffd700 1 --2- 172.21.15.112:0/331344829 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9584061170 0x7f9584063620 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9594003900 tx=0x7f9594007d40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:17.596 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.581+0000 7f958affd700 1 -- 172.21.15.112:0/331344829 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f95a4169840 con 0x7f95a40aef60 2023-01-24T03:30:17.762 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.760+0000 7f958affd700 1 -- 172.21.15.112:0/331344829 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f959c04f980 con 0x7f95a40aef60 2023-01-24T03:30:17.768 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.766+0000 7f95a98f7700 1 -- 172.21.15.112:0/331344829 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95a4169840 con 0x7f95a40aef60 2023-01-24T03:30:17.768 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.766+0000 7f958affd700 1 -- 172.21.15.112:0/331344829 <== mon.0 v2:172.21.15.112:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+13696 (secure 0 0 0) 0x7f95a4169840 con 0x7f95a40aef60 2023-01-24T03:30:17.769 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:30:17.769 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":16,"fsid":"054b9ac9-b7b0-40e5-b932-6c8a955c5a72","created":"2023-01-24T03:30:03.798533+0000","modified":"2023-01-24T03:30:17.042376+0000","last_up_change":"2023-01-24T03:30:13.945316+0000","last_in_change":"2023-01-24T03:30:06.198433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-01-24T03:30:14.931166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-01-24T03:30:15.023730+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"fe6ad231-a39a-42eb-9a3a-49d008dd35e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6809","nonce":84512},{"type":"v1","addr":"172.21.15.112:6811","nonce":84512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":84512},{"type":"v1","addr":"172.21.15.112:6817","nonce":84512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6825","nonce":84512},{"type":"v1","addr":"172.21.15.112:6828","nonce":84512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6819","nonce":84512},{"type":"v1","addr":"172.21.15.112:6822","nonce":84512}]},"public_addr":"172.21.15.112:6811/84512","cluster_addr":"172.21.15.112:6817/84512","heartbeat_back_addr":"172.21.15.112:6828/84512","heartbeat_front_addr":"172.21.15.112:6822/84512","state":["exists","up"]},{"osd":1,"uuid":"8526a1d2-d540-4021-9766-c4bbb30317ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":84513},{"type":"v1","addr":"172.21.15.112:6812","nonce":84513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6815","nonce":84513},{"type":"v1","addr":"172.21.15.112:6818","nonce":84513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6827","nonce":84513},{"type":"v1","addr":"172.21.15.112:6830","nonce":84513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6821","nonce":84513},{"type":"v1","addr":"172.21.15.112:6824","nonce":84513}]},"public_addr":"172.21.15.112:6812/84513","cluster_addr":"172.21.15.112:6818/84513","heartbeat_back_addr":"172.21.15.112:6830/84513","heartbeat_front_addr":"172.21.15.112:6824/84513","state":["exists","up"]},{"osd":2,"uuid":"041ef8e4-4858-432d-9476-cbdb3a0dc47c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":84511},{"type":"v1","addr":"172.21.15.112:6813","nonce":84511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":84511},{"type":"v1","addr":"172.21.15.112:6820","nonce":84511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6829","nonce":84511},{"type":"v1","addr":"172.21.15.112:6831","nonce":84511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6823","nonce":84511},{"type":"v1","addr":"172.21.15.112:6826","nonce":84511}]},"public_addr":"172.21.15.112:6813/84511","cluster_addr":"172.21.15.112:6820/84511","heartbeat_back_addr":"172.21.15.112:6831/84511","heartbeat_front_addr":"172.21.15.112:6826/84511","state":["exists","up"]},{"osd":3,"uuid":"987587cf-1357-4a70-ad85-67b189c67729","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":84510},{"type":"v1","addr":"172.21.15.112:6801","nonce":84510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":84510},{"type":"v1","addr":"172.21.15.112:6803","nonce":84510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":84510},{"type":"v1","addr":"172.21.15.112:6807","nonce":84510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":84510},{"type":"v1","addr":"172.21.15.112:6805","nonce":84510}]},"public_addr":"172.21.15.112:6801/84510","cluster_addr":"172.21.15.112:6803/84510","heartbeat_back_addr":"172.21.15.112:6807/84510","heartbeat_front_addr":"172.21.15.112:6805/84510","state":["exists","up"]},{"osd":4,"uuid":"8503512b-43db-40b8-8b2b-c86cf2e54e17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":84614},{"type":"v1","addr":"172.21.15.142:6801","nonce":84614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":84614},{"type":"v1","addr":"172.21.15.142:6803","nonce":84614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":84614},{"type":"v1","addr":"172.21.15.142:6807","nonce":84614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":84614},{"type":"v1","addr":"172.21.15.142:6805","nonce":84614}]},"public_addr":"172.21.15.142:6801/84614","cluster_addr":"172.21.15.142:6803/84614","heartbeat_back_addr":"172.21.15.142:6807/84614","heartbeat_front_addr":"172.21.15.142:6805/84614","state":["exists","up"]},{"osd":5,"uuid":"d3bc0030-8b08-4645-9ec2-1f8a4e29ef36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":84638},{"type":"v1","addr":"172.21.15.142:6817","nonce":84638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":84638},{"type":"v1","addr":"172.21.15.142:6819","nonce":84638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":84638},{"type":"v1","addr":"172.21.15.142:6823","nonce":84638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":84638},{"type":"v1","addr":"172.21.15.142:6821","nonce":84638}]},"public_addr":"172.21.15.142:6817/84638","cluster_addr":"172.21.15.142:6819/84638","heartbeat_back_addr":"172.21.15.142:6823/84638","heartbeat_front_addr":"172.21.15.142:6821/84638","state":["exists","up"]},{"osd":6,"uuid":"c7649e3b-832d-4e6f-8866-b15e5d814476","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":84645},{"type":"v1","addr":"172.21.15.142:6825","nonce":84645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":84645},{"type":"v1","addr":"172.21.15.142:6827","nonce":84645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":84645},{"type":"v1","addr":"172.21.15.142:6831","nonce":84645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":84645},{"type":"v1","addr":"172.21.15.142:6829","nonce":84645}]},"public_addr":"172.21.15.142:6825/84645","cluster_addr":"172.21.15.142:6827/84645","heartbeat_back_addr":"172.21.15.142:6831/84645","heartbeat_front_addr":"172.21.15.142:6829/84645","state":["exists","up"]},{"osd":7,"uuid":"2ea5b3bf-884e-4b20-b572-1443efdc7477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":84615},{"type":"v1","addr":"172.21.15.142:6809","nonce":84615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":84615},{"type":"v1","addr":"172.21.15.142:6811","nonce":84615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":84615},{"type":"v1","addr":"172.21.15.142:6815","nonce":84615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":84615},{"type":"v1","addr":"172.21.15.142:6813","nonce":84615}]},"public_addr":"172.21.15.142:6809/84615","cluster_addr":"172.21.15.142:6811/84615","heartbeat_back_addr":"172.21.15.142:6815/84615","heartbeat_front_addr":"172.21.15.142:6813/84615","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.370027+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.335683+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.350667+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.350794+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.391119+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.422933+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.400439+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.354018+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T03:30:17.771 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.769+0000 7f95a98f7700 1 -- 172.21.15.112:0/331344829 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9584061170 msgr2=0x7f9584063620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:17.771 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.769+0000 7f95a98f7700 1 --2- 172.21.15.112:0/331344829 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9584061170 0x7f9584063620 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9594003900 tx=0x7f9594007d40 comp rx=0 tx=0).stop 2023-01-24T03:30:17.772 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.769+0000 7f95a98f7700 1 -- 172.21.15.112:0/331344829 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f95a40aef60 msgr2=0x7f95a414cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:17.772 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.769+0000 7f95a98f7700 1 --2- 172.21.15.112:0/331344829 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f95a40aef60 0x7f95a414cfd0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f959c00bf20 tx=0x7f959c018000 comp rx=0 tx=0).stop 2023-01-24T03:30:17.772 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.769+0000 7f95a98f7700 1 -- 172.21.15.112:0/331344829 shutdown_connections 2023-01-24T03:30:17.772 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.769+0000 7f95a98f7700 1 --2- 172.21.15.112:0/331344829 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9584061170 0x7f9584063620 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:17.772 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.769+0000 7f95a98f7700 1 --2- 172.21.15.112:0/331344829 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f95a40aef60 0x7f95a414cfd0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:17.773 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.769+0000 7f95a98f7700 1 --2- 172.21.15.112:0/331344829 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f95a4068b10 0x7f95a40ca480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:17.773 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.769+0000 7f95a98f7700 1 --2- 172.21.15.112:0/331344829 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f95a40aeb90 0x7f95a40c8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:17.773 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.769+0000 7f95a98f7700 1 -- 172.21.15.112:0/331344829 >> 172.21.15.112:0/331344829 conn(0x7f95a405e300 msgr2=0x7f95a41535d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:17.773 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.769+0000 7f95a98f7700 1 -- 172.21.15.112:0/331344829 shutdown_connections 2023-01-24T03:30:17.773 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.769+0000 7f95a98f7700 1 -- 172.21.15.112:0/331344829 wait complete. 2023-01-24T03:30:17.783 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-01-24T03:30:14.931166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-01-24T03:30:15.023730+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-01-24T03:30:17.784 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-01-24T03:30:17.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.899+0000 7fc6e1da1700 1 Processor -- start 2023-01-24T03:30:17.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.899+0000 7fc6e1da1700 1 -- start start 2023-01-24T03:30:17.909 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.899+0000 7fc6e1da1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc6dc14f3f0 0x7fc6dc0c4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:17.909 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.899+0000 7fc6e1da1700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fc6dc155ba0 0x7fc6dc0c8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:17.910 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.899+0000 7fc6e1da1700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6dc0c82e0 con 0x7fc6dc0c6420 2023-01-24T03:30:17.910 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.899+0000 7fc6e1da1700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6dc0c8cb0 con 0x7fc6dc155ba0 2023-01-24T03:30:17.910 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.899+0000 7fc6e1da1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc0c2980 con 0x7fc6dc14f3f0 2023-01-24T03:30:17.911 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.900+0000 7fc6dbfff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fc6dc0c6420 0x7fc6dc0c2cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:59192/0 (socket says 172.21.15.112:59192) 2023-01-24T03:30:17.911 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.900+0000 7fc6dbfff700 1 -- 172.21.15.112:0/610321441 learned_addr learned my addr 172.21.15.112:0/610321441 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:17.912 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.900+0000 7fc6e0d9f700 1 --2- 172.21.15.112:0/610321441 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc6dc14f3f0 0x7fc6dc0c4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:17.912 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.900+0000 7fc6e15a0700 1 --2- 172.21.15.112:0/610321441 >> v2:172.21.15.142:3300/0 conn(0x7fc6dc155ba0 0x7fc6dc0c8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:17.912 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.900+0000 7fc6db7fe700 1 -- 172.21.15.112:0/610321441 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1022198419 0 0) 0x7fc6dc0c82e0 con 0x7fc6dc0c6420 2023-01-24T03:30:17.913 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.900+0000 7fc6db7fe700 1 -- 172.21.15.112:0/610321441 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6bc0033f0 con 0x7fc6dc0c6420 2023-01-24T03:30:17.913 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.900+0000 7fc6db7fe700 1 -- 172.21.15.112:0/610321441 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1755256482 0 0) 0x7fc6bc0033f0 con 0x7fc6dc0c6420 2023-01-24T03:30:17.913 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.900+0000 7fc6db7fe700 1 -- 172.21.15.112:0/610321441 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc6dc14f3f0 msgr2=0x7fc6dc0c4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:17.914 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.900+0000 7fc6db7fe700 1 --2- 172.21.15.112:0/610321441 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc6dc14f3f0 0x7fc6dc0c4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:17.914 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.900+0000 7fc6db7fe700 1 -- 172.21.15.112:0/610321441 >> v2:172.21.15.142:3300/0 conn(0x7fc6dc155ba0 msgr2=0x7fc6dc0c8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:17.914 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.900+0000 7fc6db7fe700 1 --2- 172.21.15.112:0/610321441 >> v2:172.21.15.142:3300/0 conn(0x7fc6dc155ba0 0x7fc6dc0c8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:17.915 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.900+0000 7fc6db7fe700 1 -- 172.21.15.112:0/610321441 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6dc156040 con 0x7fc6dc0c6420 2023-01-24T03:30:17.915 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.900+0000 7fc6db7fe700 1 -- 172.21.15.112:0/610321441 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7fc6c40017f0 con 0x7fc6dc0c6420 2023-01-24T03:30:17.915 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.900+0000 7fc6db7fe700 1 -- 172.21.15.112:0/610321441 >> v1:172.21.15.112:6789/0 conn(0x7fc6dc0c6420 legacy=0x7fc6dc0c2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:17.915 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.900+0000 7fc6db7fe700 1 --2- 172.21.15.112:0/610321441 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc6bc0038e0 0x7fc6bc003cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:17.916 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.900+0000 7fc6db7fe700 1 --2- 172.21.15.112:0/610321441 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc6bc004690 0x7fc6bc004b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:17.916 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.900+0000 7fc6db7fe700 1 --2- 172.21.15.112:0/610321441 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc6dc14f3f0 0x7fc6bc005040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:17.917 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.900+0000 7fc6db7fe700 1 -- 172.21.15.112:0/610321441 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc0c8cb0 con 0x7fc6bc0038e0 2023-01-24T03:30:17.917 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.900+0000 7fc6db7fe700 1 -- 172.21.15.112:0/610321441 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc0c2980 con 0x7fc6bc004690 2023-01-24T03:30:17.917 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.900+0000 7fc6db7fe700 1 -- 172.21.15.112:0/610321441 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6bc00b8e0 con 0x7fc6dc14f3f0 2023-01-24T03:30:17.918 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.900+0000 7fc6e0d9f700 1 --2- 172.21.15.112:0/610321441 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc6bc0038e0 0x7fc6bc003cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:17.918 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.900+0000 7fc6e15a0700 1 --2- 172.21.15.112:0/610321441 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc6dc14f3f0 0x7fc6bc005040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:17.918 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.900+0000 7fc6dbfff700 1 --2- 172.21.15.112:0/610321441 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc6bc004690 0x7fc6bc004b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:17.918 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.900+0000 7fc6e0d9f700 1 -- 172.21.15.112:0/610321441 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc6dc14f3f0 msgr2=0x7fc6bc005040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:17.918 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.900+0000 7fc6e0d9f700 1 --2- 172.21.15.112:0/610321441 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc6dc14f3f0 0x7fc6bc005040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:17.919 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.900+0000 7fc6e0d9f700 1 -- 172.21.15.112:0/610321441 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc6bc004690 msgr2=0x7fc6bc004b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:17.919 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.900+0000 7fc6e0d9f700 1 --2- 172.21.15.112:0/610321441 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc6bc004690 0x7fc6bc004b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:17.919 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.900+0000 7fc6e0d9f700 1 -- 172.21.15.112:0/610321441 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6bc0033f0 con 0x7fc6bc0038e0 2023-01-24T03:30:17.919 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.900+0000 7fc6e15a0700 1 --2- 172.21.15.112:0/610321441 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc6dc14f3f0 0x7fc6bc005040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:30:17.919 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.900+0000 7fc6dbfff700 1 --2- 172.21.15.112:0/610321441 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc6bc004690 0x7fc6bc004b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:30:17.920 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.901+0000 7fc6e0d9f700 1 --2- 172.21.15.112:0/610321441 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc6bc0038e0 0x7fc6bc003cf0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc009ab0 tx=0x7fc6cc009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b17a7459f6f27341 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:17.920 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.901+0000 7fc6db7fe700 1 -- 172.21.15.112:0/610321441 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc009430 con 0x7fc6bc0038e0 2023-01-24T03:30:17.920 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.901+0000 7fc6db7fe700 1 -- 172.21.15.112:0/610321441 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6cc00e640 con 0x7fc6bc0038e0 2023-01-24T03:30:17.920 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.901+0000 7fc6db7fe700 1 -- 172.21.15.112:0/610321441 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc00ea00 con 0x7fc6bc0038e0 2023-01-24T03:30:17.920 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.901+0000 7fc6e1da1700 1 -- 172.21.15.112:0/610321441 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc6bc0038e0 msgr2=0x7fc6bc003cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:17.921 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.901+0000 7fc6e1da1700 1 --2- 172.21.15.112:0/610321441 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc6bc0038e0 0x7fc6bc003cf0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc009ab0 tx=0x7fc6cc009060 comp rx=0 tx=0).stop 2023-01-24T03:30:17.921 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.901+0000 7fc6e15a0700 1 -- 172.21.15.112:0/610321441 reap_dead start 2023-01-24T03:30:17.921 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.901+0000 7fc6e1da1700 1 -- 172.21.15.112:0/610321441 shutdown_connections 2023-01-24T03:30:17.921 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.901+0000 7fc6e1da1700 1 -- 172.21.15.112:0/610321441 >> 172.21.15.112:0/610321441 conn(0x7fc6dc05d830 msgr2=0x7fc6dc05dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:17.921 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.901+0000 7fc6e1da1700 1 -- 172.21.15.112:0/610321441 shutdown_connections 2023-01-24T03:30:17.922 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.901+0000 7fc6e1da1700 1 -- 172.21.15.112:0/610321441 wait complete. 2023-01-24T03:30:17.922 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.902+0000 7fc6e1da1700 1 Processor -- start 2023-01-24T03:30:17.922 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.902+0000 7fc6e1da1700 1 -- start start 2023-01-24T03:30:17.922 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.903+0000 7fc6e1da1700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc6bc0038e0 0x7fc6dc0cad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:17.922 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.903+0000 7fc6e1da1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc6dc0c1980 0x7fc6dc0bf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:17.922 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.903+0000 7fc6e1da1700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc6dc0bfaf0 0x7fc6dc1687d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:17.923 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.903+0000 7fc6e1da1700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc0c6c60 con 0x7fc6dc0bfaf0 2023-01-24T03:30:17.923 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.903+0000 7fc6e1da1700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc148930 con 0x7fc6bc0038e0 2023-01-24T03:30:17.923 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.903+0000 7fc6e1da1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc149340 con 0x7fc6dc0c1980 2023-01-24T03:30:17.923 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.903+0000 7fc6e15a0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc6dc0bfaf0 0x7fc6dc1687d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:17.923 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.903+0000 7fc6e15a0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc6dc0bfaf0 0x7fc6dc1687d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:49984/0 (socket says 172.21.15.112:49984) 2023-01-24T03:30:17.924 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.903+0000 7fc6e15a0700 1 -- 172.21.15.112:0/1102279178 learned_addr learned my addr 172.21.15.112:0/1102279178 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:17.924 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.903+0000 7fc6dbfff700 1 --2- 172.21.15.112:0/1102279178 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc6dc0c1980 0x7fc6dc0bf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:17.924 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.903+0000 7fc6e0d9f700 1 --2- 172.21.15.112:0/1102279178 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc6bc0038e0 0x7fc6dc0cad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:17.924 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.903+0000 7fc6e15a0700 1 -- 172.21.15.112:0/1102279178 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc6dc0c1980 msgr2=0x7fc6dc0bf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:17.924 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.903+0000 7fc6e15a0700 1 --2- 172.21.15.112:0/1102279178 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc6dc0c1980 0x7fc6dc0bf5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:17.925 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.903+0000 7fc6e15a0700 1 -- 172.21.15.112:0/1102279178 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc6bc0038e0 msgr2=0x7fc6dc0cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:17.925 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.903+0000 7fc6e15a0700 1 --2- 172.21.15.112:0/1102279178 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc6bc0038e0 0x7fc6dc0cad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:17.925 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.903+0000 7fc6e15a0700 1 -- 172.21.15.112:0/1102279178 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6dc0c4f70 con 0x7fc6dc0bfaf0 2023-01-24T03:30:17.925 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.904+0000 7fc6e15a0700 1 --2- 172.21.15.112:0/1102279178 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc6dc0bfaf0 0x7fc6dc1687d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc05e5e0 tx=0x7fc6d4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:17.925 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.904+0000 7fc6d8ff9700 1 -- 172.21.15.112:0/1102279178 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d4011eb0 con 0x7fc6dc0bfaf0 2023-01-24T03:30:17.926 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.904+0000 7fc6d8ff9700 1 -- 172.21.15.112:0/1102279178 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6d4015460 con 0x7fc6dc0bfaf0 2023-01-24T03:30:17.926 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.904+0000 7fc6e1da1700 1 -- 172.21.15.112:0/1102279178 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6dc1481d0 con 0x7fc6dc0bfaf0 2023-01-24T03:30:17.926 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.904+0000 7fc6d8ff9700 1 -- 172.21.15.112:0/1102279178 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d4015840 con 0x7fc6dc0bfaf0 2023-01-24T03:30:17.926 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.904+0000 7fc6e1da1700 1 -- 172.21.15.112:0/1102279178 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6dc148b40 con 0x7fc6dc0bfaf0 2023-01-24T03:30:17.926 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.905+0000 7fc6e1da1700 1 -- 172.21.15.112:0/1102279178 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6dc1486d0 con 0x7fc6dc0bfaf0 2023-01-24T03:30:17.927 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.905+0000 7fc6d8ff9700 1 -- 172.21.15.112:0/1102279178 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fc6d4024070 con 0x7fc6dc0bfaf0 2023-01-24T03:30:17.927 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.905+0000 7fc6d8ff9700 1 --2- 172.21.15.112:0/1102279178 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc6c00611c0 0x7fc6c0063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:17.927 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.905+0000 7fc6d8ff9700 1 -- 172.21.15.112:0/1102279178 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fc6d407cf80 con 0x7fc6dc0bfaf0 2023-01-24T03:30:17.927 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.905+0000 7fc6e0d9f700 1 --2- 172.21.15.112:0/1102279178 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc6c00611c0 0x7fc6c0063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:17.927 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.906+0000 7fc6e0d9f700 1 --2- 172.21.15.112:0/1102279178 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc6c00611c0 0x7fc6c0063670 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc009ab0 tx=0x7fc6cc01a000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:17.928 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:17.910+0000 7fc6d8ff9700 1 -- 172.21.15.112:0/1102279178 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fc6d404d400 con 0x7fc6dc0bfaf0 2023-01-24T03:30:18.089 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.086+0000 7fc6e1da1700 1 -- 172.21.15.112:0/1102279178 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fc6dc05ea20 con 0x7fc6dc0bfaf0 2023-01-24T03:30:18.089 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.087+0000 7fc6d8ff9700 1 -- 172.21.15.112:0/1102279178 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v16) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc6d4022020 con 0x7fc6dc0bfaf0 2023-01-24T03:30:18.090 INFO:teuthology.orchestra.run.smithi112.stdout:pg_num: 8 2023-01-24T03:30:18.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.089+0000 7fc6e1da1700 1 -- 172.21.15.112:0/1102279178 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc6c00611c0 msgr2=0x7fc6c0063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:18.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.089+0000 7fc6e1da1700 1 --2- 172.21.15.112:0/1102279178 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc6c00611c0 0x7fc6c0063670 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc009ab0 tx=0x7fc6cc01a000 comp rx=0 tx=0).stop 2023-01-24T03:30:18.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.089+0000 7fc6e1da1700 1 -- 172.21.15.112:0/1102279178 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc6dc0bfaf0 msgr2=0x7fc6dc1687d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:18.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.089+0000 7fc6e1da1700 1 --2- 172.21.15.112:0/1102279178 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc6dc0bfaf0 0x7fc6dc1687d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc05e5e0 tx=0x7fc6d4008040 comp rx=0 tx=0).stop 2023-01-24T03:30:18.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.089+0000 7fc6e1da1700 1 -- 172.21.15.112:0/1102279178 shutdown_connections 2023-01-24T03:30:18.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.089+0000 7fc6e1da1700 1 --2- 172.21.15.112:0/1102279178 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc6c00611c0 0x7fc6c0063670 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:18.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.089+0000 7fc6e1da1700 1 --2- 172.21.15.112:0/1102279178 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc6dc0bfaf0 0x7fc6dc1687d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:18.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.089+0000 7fc6e1da1700 1 --2- 172.21.15.112:0/1102279178 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc6bc0038e0 0x7fc6dc0cad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:18.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.089+0000 7fc6e1da1700 1 --2- 172.21.15.112:0/1102279178 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc6dc0c1980 0x7fc6dc0bf5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:18.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.089+0000 7fc6e1da1700 1 -- 172.21.15.112:0/1102279178 >> 172.21.15.112:0/1102279178 conn(0x7fc6dc05d830 msgr2=0x7fc6dc0c2ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:18.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.089+0000 7fc6e1da1700 1 -- 172.21.15.112:0/1102279178 shutdown_connections 2023-01-24T03:30:18.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.089+0000 7fc6e1da1700 1 -- 172.21.15.112:0/1102279178 wait complete. 2023-01-24T03:30:18.103 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-01-24T03:30:18.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.217+0000 7fd0ed070700 1 Processor -- start 2023-01-24T03:30:18.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.217+0000 7fd0ed070700 1 -- start start 2023-01-24T03:30:18.226 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.218+0000 7fd0ed070700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd0e814d4c0 0x7fd0e80c4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:18.226 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.218+0000 7fd0ed070700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0e806d4f0 con 0x7fd0e805dff0 2023-01-24T03:30:18.226 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.218+0000 7fd0ed070700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0e806cee0 con 0x7fd0e806d980 2023-01-24T03:30:18.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.218+0000 7fd0ed070700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e806de00 con 0x7fd0e814d4c0 2023-01-24T03:30:18.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.218+0000 7fd0e659c700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fd0e805dff0 0x7fd0e8153240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:59212/0 (socket says 172.21.15.112:59212) 2023-01-24T03:30:18.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.218+0000 7fd0e659c700 1 -- 172.21.15.112:0/4015074154 learned_addr learned my addr 172.21.15.112:0/4015074154 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:18.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.218+0000 7fd0e6d9d700 1 --2- 172.21.15.112:0/4015074154 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd0e814d4c0 0x7fd0e80c4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:18.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.218+0000 7fd0e5d9b700 1 -- 172.21.15.112:0/4015074154 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4172847216 0 0) 0x7fd0e806d4f0 con 0x7fd0e805dff0 2023-01-24T03:30:18.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.219+0000 7fd0e5d9b700 1 -- 172.21.15.112:0/4015074154 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0c80033f0 con 0x7fd0e805dff0 2023-01-24T03:30:18.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.219+0000 7fd0e5d9b700 1 -- 172.21.15.112:0/4015074154 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3856899687 0 0) 0x7fd0c80033f0 con 0x7fd0e805dff0 2023-01-24T03:30:18.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.219+0000 7fd0e5d9b700 1 -- 172.21.15.112:0/4015074154 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd0e814d4c0 msgr2=0x7fd0e80c4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:18.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.219+0000 7fd0e5d9b700 1 --2- 172.21.15.112:0/4015074154 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd0e814d4c0 0x7fd0e80c4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:18.229 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.219+0000 7fd0e5d9b700 1 -- 172.21.15.112:0/4015074154 >> v1:172.21.15.142:6789/0 conn(0x7fd0e806d980 legacy=0x7fd0e80c32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:18.229 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.219+0000 7fd0e5d9b700 1 -- 172.21.15.112:0/4015074154 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e8153af0 con 0x7fd0e805dff0 2023-01-24T03:30:18.229 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.219+0000 7fd0e5d9b700 1 -- 172.21.15.112:0/4015074154 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7fd0d0001470 con 0x7fd0e805dff0 2023-01-24T03:30:18.229 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.219+0000 7fd0e5d9b700 1 -- 172.21.15.112:0/4015074154 >> v1:172.21.15.112:6789/0 conn(0x7fd0e805dff0 legacy=0x7fd0e8153240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:18.229 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.219+0000 7fd0e5d9b700 1 --2- 172.21.15.112:0/4015074154 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd0e814d4c0 0x7fd0c8003910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:18.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.219+0000 7fd0e5d9b700 1 --2- 172.21.15.112:0/4015074154 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd0c8003e50 0x7fd0c8004690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:18.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.219+0000 7fd0e5d9b700 1 --2- 172.21.15.112:0/4015074154 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0c8004bd0 0x7fd0c8005040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:18.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.219+0000 7fd0e5d9b700 1 -- 172.21.15.112:0/4015074154 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e806de00 con 0x7fd0c8004bd0 2023-01-24T03:30:18.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.219+0000 7fd0e5d9b700 1 -- 172.21.15.112:0/4015074154 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c800b9c0 con 0x7fd0c8003e50 2023-01-24T03:30:18.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.219+0000 7fd0e5d9b700 1 -- 172.21.15.112:0/4015074154 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c800bb80 con 0x7fd0e814d4c0 2023-01-24T03:30:18.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.219+0000 7fd0e759e700 1 --2- 172.21.15.112:0/4015074154 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0c8004bd0 0x7fd0c8005040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:18.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.219+0000 7fd0e6d9d700 1 --2- 172.21.15.112:0/4015074154 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd0e814d4c0 0x7fd0c8003910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:18.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.219+0000 7fd0e659c700 1 --2- 172.21.15.112:0/4015074154 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd0c8003e50 0x7fd0c8004690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:18.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.219+0000 7fd0e759e700 1 -- 172.21.15.112:0/4015074154 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd0e814d4c0 msgr2=0x7fd0c8003910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:18.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.219+0000 7fd0e759e700 1 --2- 172.21.15.112:0/4015074154 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd0e814d4c0 0x7fd0c8003910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:18.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.219+0000 7fd0e759e700 1 -- 172.21.15.112:0/4015074154 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd0c8003e50 msgr2=0x7fd0c8004690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:18.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.219+0000 7fd0e759e700 1 --2- 172.21.15.112:0/4015074154 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd0c8003e50 0x7fd0c8004690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:18.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.219+0000 7fd0e759e700 1 -- 172.21.15.112:0/4015074154 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c80033f0 con 0x7fd0c8004bd0 2023-01-24T03:30:18.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.219+0000 7fd0e759e700 1 --2- 172.21.15.112:0/4015074154 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0c8004bd0 0x7fd0c8005040 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0001820 tx=0x7fd0e0007b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42a3fd932e2ad072 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:18.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.219+0000 7fd0e5d9b700 1 -- 172.21.15.112:0/4015074154 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e0007ea0 con 0x7fd0c8004bd0 2023-01-24T03:30:18.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.220+0000 7fd0e5d9b700 1 -- 172.21.15.112:0/4015074154 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0e0012420 con 0x7fd0c8004bd0 2023-01-24T03:30:18.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.220+0000 7fd0e5d9b700 1 -- 172.21.15.112:0/4015074154 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e00127c0 con 0x7fd0c8004bd0 2023-01-24T03:30:18.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.220+0000 7fd0ed070700 1 -- 172.21.15.112:0/4015074154 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0c8004bd0 msgr2=0x7fd0c8005040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:18.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.220+0000 7fd0ed070700 1 --2- 172.21.15.112:0/4015074154 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0c8004bd0 0x7fd0c8005040 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0001820 tx=0x7fd0e0007b90 comp rx=0 tx=0).stop 2023-01-24T03:30:18.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.220+0000 7fd0e759e700 1 -- 172.21.15.112:0/4015074154 reap_dead start 2023-01-24T03:30:18.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.220+0000 7fd0ed070700 1 -- 172.21.15.112:0/4015074154 shutdown_connections 2023-01-24T03:30:18.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.220+0000 7fd0ed070700 1 -- 172.21.15.112:0/4015074154 >> 172.21.15.112:0/4015074154 conn(0x7fd0e806c4f0 msgr2=0x7fd0e805d830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:18.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.220+0000 7fd0ed070700 1 -- 172.21.15.112:0/4015074154 shutdown_connections 2023-01-24T03:30:18.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.220+0000 7fd0ed070700 1 -- 172.21.15.112:0/4015074154 wait complete. 2023-01-24T03:30:18.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.221+0000 7fd0ed070700 1 Processor -- start 2023-01-24T03:30:18.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.221+0000 7fd0ed070700 1 -- start start 2023-01-24T03:30:18.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.222+0000 7fd0ed070700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd0c8004bd0 0x7fd0e8151f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:18.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.222+0000 7fd0ed070700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0e81510e0 0x7fd0e8154720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:18.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.222+0000 7fd0ed070700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd0e8157030 0x7fd0e8154c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:18.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.222+0000 7fd0ed070700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e8151690 con 0x7fd0e81510e0 2023-01-24T03:30:18.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.222+0000 7fd0ed070700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e8152fd0 con 0x7fd0c8004bd0 2023-01-24T03:30:18.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.222+0000 7fd0ed070700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e8163b90 con 0x7fd0e8157030 2023-01-24T03:30:18.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.222+0000 7fd0e659c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0e81510e0 0x7fd0e8154720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:18.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.222+0000 7fd0e659c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0e81510e0 0x7fd0e8154720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50000/0 (socket says 172.21.15.112:50000) 2023-01-24T03:30:18.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.222+0000 7fd0e659c700 1 -- 172.21.15.112:0/1080838170 learned_addr learned my addr 172.21.15.112:0/1080838170 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:18.238 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.222+0000 7fd0e759e700 1 --2- 172.21.15.112:0/1080838170 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd0e8157030 0x7fd0e8154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:18.238 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.222+0000 7fd0e6d9d700 1 --2- 172.21.15.112:0/1080838170 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd0c8004bd0 0x7fd0e8151f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:18.238 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.222+0000 7fd0e659c700 1 -- 172.21.15.112:0/1080838170 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd0e8157030 msgr2=0x7fd0e8154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:18.238 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.222+0000 7fd0e659c700 1 --2- 172.21.15.112:0/1080838170 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd0e8157030 0x7fd0e8154c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:18.238 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.222+0000 7fd0e659c700 1 -- 172.21.15.112:0/1080838170 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd0c8004bd0 msgr2=0x7fd0e8151f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:18.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.222+0000 7fd0e659c700 1 --2- 172.21.15.112:0/1080838170 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd0c8004bd0 0x7fd0e8151f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:18.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.222+0000 7fd0e659c700 1 -- 172.21.15.112:0/1080838170 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e81518a0 con 0x7fd0e81510e0 2023-01-24T03:30:18.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.222+0000 7fd0e659c700 1 --2- 172.21.15.112:0/1080838170 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0e81510e0 0x7fd0e8154720 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd0d0001d00 tx=0x7fd0d0001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:18.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.222+0000 7fd0deffd700 1 -- 172.21.15.112:0/1080838170 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d000f610 con 0x7fd0e81510e0 2023-01-24T03:30:18.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.222+0000 7fd0deffd700 1 -- 172.21.15.112:0/1080838170 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d000f7d0 con 0x7fd0e81510e0 2023-01-24T03:30:18.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.223+0000 7fd0ed070700 1 -- 172.21.15.112:0/1080838170 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0e8163860 con 0x7fd0e81510e0 2023-01-24T03:30:18.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.223+0000 7fd0deffd700 1 -- 172.21.15.112:0/1080838170 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d000fb90 con 0x7fd0e81510e0 2023-01-24T03:30:18.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.223+0000 7fd0ed070700 1 -- 172.21.15.112:0/1080838170 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0e8149530 con 0x7fd0e81510e0 2023-01-24T03:30:18.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.223+0000 7fd0ed070700 1 -- 172.21.15.112:0/1080838170 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0e8153dd0 con 0x7fd0e81510e0 2023-01-24T03:30:18.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.224+0000 7fd0deffd700 1 -- 172.21.15.112:0/1080838170 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fd0d001d460 con 0x7fd0e81510e0 2023-01-24T03:30:18.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.224+0000 7fd0deffd700 1 --2- 172.21.15.112:0/1080838170 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd0cc0611c0 0x7fd0cc063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:18.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.224+0000 7fd0deffd700 1 -- 172.21.15.112:0/1080838170 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fd0d007d420 con 0x7fd0e81510e0 2023-01-24T03:30:18.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.224+0000 7fd0e6d9d700 1 --2- 172.21.15.112:0/1080838170 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd0cc0611c0 0x7fd0cc063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:18.242 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.224+0000 7fd0e6d9d700 1 --2- 172.21.15.112:0/1080838170 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd0cc0611c0 0x7fd0cc063670 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd0d80094d0 tx=0x7fd0d800b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:18.242 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.228+0000 7fd0deffd700 1 -- 172.21.15.112:0/1080838170 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fd0d004d840 con 0x7fd0e81510e0 2023-01-24T03:30:18.407 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.404+0000 7fd0ed070700 1 -- 172.21.15.112:0/1080838170 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd0e804b340 con 0x7fd0e81510e0 2023-01-24T03:30:18.407 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.405+0000 7fd0deffd700 1 -- 172.21.15.112:0/1080838170 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd0d0050e60 con 0x7fd0e81510e0 2023-01-24T03:30:18.408 INFO:teuthology.orchestra.run.smithi112.stdout:pg_num: 1 2023-01-24T03:30:18.410 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.407+0000 7fd0ed070700 1 -- 172.21.15.112:0/1080838170 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd0cc0611c0 msgr2=0x7fd0cc063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:18.410 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.407+0000 7fd0ed070700 1 --2- 172.21.15.112:0/1080838170 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd0cc0611c0 0x7fd0cc063670 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd0d80094d0 tx=0x7fd0d800b040 comp rx=0 tx=0).stop 2023-01-24T03:30:18.410 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.407+0000 7fd0ed070700 1 -- 172.21.15.112:0/1080838170 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0e81510e0 msgr2=0x7fd0e8154720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:18.411 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.407+0000 7fd0ed070700 1 --2- 172.21.15.112:0/1080838170 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0e81510e0 0x7fd0e8154720 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd0d0001d00 tx=0x7fd0d0001e70 comp rx=0 tx=0).stop 2023-01-24T03:30:18.411 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.407+0000 7fd0ed070700 1 -- 172.21.15.112:0/1080838170 shutdown_connections 2023-01-24T03:30:18.411 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.407+0000 7fd0ed070700 1 --2- 172.21.15.112:0/1080838170 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd0cc0611c0 0x7fd0cc063670 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:18.412 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.407+0000 7fd0ed070700 1 --2- 172.21.15.112:0/1080838170 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd0e8157030 0x7fd0e8154c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:18.412 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.407+0000 7fd0ed070700 1 --2- 172.21.15.112:0/1080838170 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd0c8004bd0 0x7fd0e8151f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:18.413 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.407+0000 7fd0ed070700 1 --2- 172.21.15.112:0/1080838170 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0e81510e0 0x7fd0e8154720 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:18.413 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.407+0000 7fd0ed070700 1 -- 172.21.15.112:0/1080838170 >> 172.21.15.112:0/1080838170 conn(0x7fd0e806c4f0 msgr2=0x7fd0e80c7b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:18.413 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.407+0000 7fd0ed070700 1 -- 172.21.15.112:0/1080838170 shutdown_connections 2023-01-24T03:30:18.414 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.407+0000 7fd0ed070700 1 -- 172.21.15.112:0/1080838170 wait complete. 2023-01-24T03:30:18.421 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-01-24T03:30:18.542 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.532+0000 7fd2bd68f700 1 Processor -- start 2023-01-24T03:30:18.542 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.532+0000 7fd2bd68f700 1 -- start start 2023-01-24T03:30:18.543 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.532+0000 7fd2bd68f700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fd2b806cf50 0x7fd2b80c1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:18.543 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.532+0000 7fd2bd68f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2b806d320 0x7fd2b80c2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:18.543 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.532+0000 7fd2bd68f700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd2b80c3390 0x7fd2b80c4790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:18.544 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.532+0000 7fd2bd68f700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2b8153e80 con 0x7fd2b806cf50 2023-01-24T03:30:18.544 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.532+0000 7fd2bd68f700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2b80c2870 con 0x7fd2b80c3390 2023-01-24T03:30:18.544 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.532+0000 7fd2bd68f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b8153270 con 0x7fd2b806d320 2023-01-24T03:30:18.545 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.533+0000 7fd2b6ffd700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fd2b806cf50 0x7fd2b80c1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:18.545 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.533+0000 7fd2b6ffd700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fd2b806cf50 0x7fd2b80c1560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50008/0 (socket says 172.21.15.112:50008) 2023-01-24T03:30:18.545 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.533+0000 7fd2b6ffd700 1 -- 172.21.15.112:0/3515822422 learned_addr learned my addr 172.21.15.112:0/3515822422 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:18.546 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.533+0000 7fd2b67fc700 1 --2- 172.21.15.112:0/3515822422 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2b806d320 0x7fd2b80c2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:18.546 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.533+0000 7fd2b77fe700 1 --2- 172.21.15.112:0/3515822422 >> v2:172.21.15.142:3300/0 conn(0x7fd2b80c3390 0x7fd2b80c4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:18.546 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.533+0000 7fd2b6ffd700 1 -- 172.21.15.112:0/3515822422 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2b806d320 msgr2=0x7fd2b80c2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:18.547 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.533+0000 7fd2b6ffd700 1 --2- 172.21.15.112:0/3515822422 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2b806d320 0x7fd2b80c2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:18.547 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.533+0000 7fd2b6ffd700 1 -- 172.21.15.112:0/3515822422 >> v2:172.21.15.142:3300/0 conn(0x7fd2b80c3390 msgr2=0x7fd2b80c4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:18.547 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.533+0000 7fd2b6ffd700 1 --2- 172.21.15.112:0/3515822422 >> v2:172.21.15.142:3300/0 conn(0x7fd2b80c3390 0x7fd2b80c4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:18.548 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.533+0000 7fd2b6ffd700 1 -- 172.21.15.112:0/3515822422 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b80c37f0 con 0x7fd2b806cf50 2023-01-24T03:30:18.548 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.533+0000 7fd2b67fc700 1 --2- 172.21.15.112:0/3515822422 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2b806d320 0x7fd2b80c2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:30:18.548 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.533+0000 7fd2b6ffd700 1 --2- 172.21.15.112:0/3515822422 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2b806cf50 0x7fd2b80c1560 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd2b80c4490 tx=0x7fd2b0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6b9868021771935 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:18.549 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.533+0000 7fd2b5ffb700 1 -- 172.21.15.112:0/3515822422 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b000e070 con 0x7fd2b806cf50 2023-01-24T03:30:18.549 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.533+0000 7fd2b5ffb700 1 -- 172.21.15.112:0/3515822422 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b000ba50 con 0x7fd2b806cf50 2023-01-24T03:30:18.549 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.533+0000 7fd2b5ffb700 1 -- 172.21.15.112:0/3515822422 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b00092e0 con 0x7fd2b806cf50 2023-01-24T03:30:18.549 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.534+0000 7fd2bd68f700 1 -- 172.21.15.112:0/3515822422 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2b806cf50 msgr2=0x7fd2b80c1560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:18.550 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.534+0000 7fd2bd68f700 1 --2- 172.21.15.112:0/3515822422 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2b806cf50 0x7fd2b80c1560 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd2b80c4490 tx=0x7fd2b0009840 comp rx=0 tx=0).stop 2023-01-24T03:30:18.550 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.534+0000 7fd2bd68f700 1 -- 172.21.15.112:0/3515822422 shutdown_connections 2023-01-24T03:30:18.550 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.534+0000 7fd2bd68f700 1 --2- 172.21.15.112:0/3515822422 >> v2:172.21.15.142:3300/0 conn(0x7fd2b80c3390 0x7fd2b80c4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:18.550 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.534+0000 7fd2bd68f700 1 --2- 172.21.15.112:0/3515822422 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2b806cf50 0x7fd2b80c1560 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:18.550 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.534+0000 7fd2bd68f700 1 --2- 172.21.15.112:0/3515822422 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2b806d320 0x7fd2b80c2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:18.551 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.534+0000 7fd2bd68f700 1 -- 172.21.15.112:0/3515822422 >> 172.21.15.112:0/3515822422 conn(0x7fd2b80c9820 msgr2=0x7fd2b80c9c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:18.551 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.534+0000 7fd2bd68f700 1 -- 172.21.15.112:0/3515822422 shutdown_connections 2023-01-24T03:30:18.551 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.534+0000 7fd2bd68f700 1 -- 172.21.15.112:0/3515822422 wait complete. 2023-01-24T03:30:18.551 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.535+0000 7fd2bd68f700 1 Processor -- start 2023-01-24T03:30:18.552 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.535+0000 7fd2bd68f700 1 -- start start 2023-01-24T03:30:18.552 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.536+0000 7fd2bd68f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2b806cf50 0x7fd2b81636c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:18.552 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.536+0000 7fd2bd68f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2b806d320 0x7fd2b81511c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:18.552 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.536+0000 7fd2bd68f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2b80c3390 0x7fd2b8166830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:18.553 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.536+0000 7fd2bd68f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b8163230 con 0x7fd2b806cf50 2023-01-24T03:30:18.553 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.536+0000 7fd2bd68f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b80c4950 con 0x7fd2b80c3390 2023-01-24T03:30:18.553 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.536+0000 7fd2bd68f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b80c0420 con 0x7fd2b806d320 2023-01-24T03:30:18.553 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.536+0000 7fd2b6ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2b806cf50 0x7fd2b81636c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:18.553 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.536+0000 7fd2b6ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2b806cf50 0x7fd2b81636c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50014/0 (socket says 172.21.15.112:50014) 2023-01-24T03:30:18.554 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.536+0000 7fd2b6ffd700 1 -- 172.21.15.112:0/4151968059 learned_addr learned my addr 172.21.15.112:0/4151968059 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:18.554 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.536+0000 7fd2b67fc700 1 --2- 172.21.15.112:0/4151968059 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2b806d320 0x7fd2b81511c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:18.554 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.536+0000 7fd2b77fe700 1 --2- 172.21.15.112:0/4151968059 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2b80c3390 0x7fd2b8166830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:18.554 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.536+0000 7fd2b6ffd700 1 -- 172.21.15.112:0/4151968059 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2b806d320 msgr2=0x7fd2b81511c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:18.555 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.536+0000 7fd2b6ffd700 1 --2- 172.21.15.112:0/4151968059 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2b806d320 0x7fd2b81511c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:18.555 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.536+0000 7fd2b6ffd700 1 -- 172.21.15.112:0/4151968059 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2b80c3390 msgr2=0x7fd2b8166830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:18.555 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.536+0000 7fd2b6ffd700 1 --2- 172.21.15.112:0/4151968059 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2b80c3390 0x7fd2b8166830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:18.555 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.536+0000 7fd2b6ffd700 1 -- 172.21.15.112:0/4151968059 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b0009ae0 con 0x7fd2b806cf50 2023-01-24T03:30:18.555 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.536+0000 7fd2b77fe700 1 --2- 172.21.15.112:0/4151968059 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2b80c3390 0x7fd2b8166830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:30:18.556 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.536+0000 7fd2b6ffd700 1 --2- 172.21.15.112:0/4151968059 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2b806cf50 0x7fd2b81636c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd2b0009fd0 tx=0x7fd2b0018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:18.556 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.536+0000 7fd29effd700 1 -- 172.21.15.112:0/4151968059 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b00094e0 con 0x7fd2b806cf50 2023-01-24T03:30:18.556 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.536+0000 7fd29effd700 1 -- 172.21.15.112:0/4151968059 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b0007820 con 0x7fd2b806cf50 2023-01-24T03:30:18.556 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.536+0000 7fd29effd700 1 -- 172.21.15.112:0/4151968059 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b0007be0 con 0x7fd2b806cf50 2023-01-24T03:30:18.556 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.536+0000 7fd2bd68f700 1 -- 172.21.15.112:0/4151968059 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2b80beaf0 con 0x7fd2b806cf50 2023-01-24T03:30:18.557 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.536+0000 7fd2bd68f700 1 -- 172.21.15.112:0/4151968059 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2b8162ec0 con 0x7fd2b806cf50 2023-01-24T03:30:18.557 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.537+0000 7fd2bd68f700 1 -- 172.21.15.112:0/4151968059 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2b804bb80 con 0x7fd2b806cf50 2023-01-24T03:30:18.557 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.537+0000 7fd29effd700 1 -- 172.21.15.112:0/4151968059 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fd2b001d420 con 0x7fd2b806cf50 2023-01-24T03:30:18.557 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.538+0000 7fd29effd700 1 --2- 172.21.15.112:0/4151968059 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd2980611c0 0x7fd298063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:18.557 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.538+0000 7fd29effd700 1 -- 172.21.15.112:0/4151968059 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fd2b007f0d0 con 0x7fd2b806cf50 2023-01-24T03:30:18.558 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.538+0000 7fd2b67fc700 1 --2- 172.21.15.112:0/4151968059 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd2980611c0 0x7fd298063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:18.558 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.538+0000 7fd2b67fc700 1 --2- 172.21.15.112:0/4151968059 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd2980611c0 0x7fd298063670 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd2b80c58c0 tx=0x7fd2ac008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:18.558 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.542+0000 7fd29effd700 1 -- 172.21.15.112:0/4151968059 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fd2b004f4f0 con 0x7fd2b806cf50 2023-01-24T03:30:18.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.722+0000 7fd2bd68f700 1 -- 172.21.15.112:0/4151968059 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fd2b81534e0 con 0x7fd2b806cf50 2023-01-24T03:30:18.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.722+0000 7fd29effd700 1 -- 172.21.15.112:0/4151968059 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+126 (secure 0 0 0) 0x7fd2b002a090 con 0x7fd2b806cf50 2023-01-24T03:30:18.725 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:30:18.726 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-01-24T03:30:18.726 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-01-24T03:30:18.727 INFO:teuthology.orchestra.run.smithi112.stdout: "poolnum": 1, 2023-01-24T03:30:18.727 INFO:teuthology.orchestra.run.smithi112.stdout: "poolname": "rbd" 2023-01-24T03:30:18.727 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-01-24T03:30:18.728 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-01-24T03:30:18.728 INFO:teuthology.orchestra.run.smithi112.stdout: "poolnum": 2, 2023-01-24T03:30:18.728 INFO:teuthology.orchestra.run.smithi112.stdout: "poolname": ".mgr" 2023-01-24T03:30:18.729 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-01-24T03:30:18.729 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-01-24T03:30:18.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.724+0000 7fd2bd68f700 1 -- 172.21.15.112:0/4151968059 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd2980611c0 msgr2=0x7fd298063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:18.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.724+0000 7fd2bd68f700 1 --2- 172.21.15.112:0/4151968059 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd2980611c0 0x7fd298063670 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd2b80c58c0 tx=0x7fd2ac008040 comp rx=0 tx=0).stop 2023-01-24T03:30:18.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.724+0000 7fd2bd68f700 1 -- 172.21.15.112:0/4151968059 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2b806cf50 msgr2=0x7fd2b81636c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:18.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.724+0000 7fd2bd68f700 1 --2- 172.21.15.112:0/4151968059 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2b806cf50 0x7fd2b81636c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd2b0009fd0 tx=0x7fd2b0018000 comp rx=0 tx=0).stop 2023-01-24T03:30:18.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.724+0000 7fd2bd68f700 1 -- 172.21.15.112:0/4151968059 shutdown_connections 2023-01-24T03:30:18.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.724+0000 7fd2bd68f700 1 --2- 172.21.15.112:0/4151968059 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd2980611c0 0x7fd298063670 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:18.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.724+0000 7fd2bd68f700 1 --2- 172.21.15.112:0/4151968059 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2b806cf50 0x7fd2b81636c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:18.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.724+0000 7fd2bd68f700 1 --2- 172.21.15.112:0/4151968059 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2b80c3390 0x7fd2b8166830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:18.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.724+0000 7fd2bd68f700 1 --2- 172.21.15.112:0/4151968059 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd2b806d320 0x7fd2b81511c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:18.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.724+0000 7fd2bd68f700 1 -- 172.21.15.112:0/4151968059 >> 172.21.15.112:0/4151968059 conn(0x7fd2b80c9820 msgr2=0x7fd2b80c4f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:18.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.724+0000 7fd2bd68f700 1 -- 172.21.15.112:0/4151968059 shutdown_connections 2023-01-24T03:30:18.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.724+0000 7fd2bd68f700 1 -- 172.21.15.112:0/4151968059 wait complete. 2023-01-24T03:30:18.738 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-01-24T03:30:18.739 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-01-24T03:30:18.855 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.853+0000 7f3e6e423700 1 Processor -- start 2023-01-24T03:30:18.855 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.853+0000 7f3e6e423700 1 -- start start 2023-01-24T03:30:18.863 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.853+0000 7f3e6e423700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f3e681560e0 0x7f3e6815e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:18.863 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.853+0000 7f3e6e423700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e680c4a30 con 0x7f3e680c4520 2023-01-24T03:30:18.863 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.853+0000 7f3e6e423700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e680c5e80 con 0x7f3e6806ce70 2023-01-24T03:30:18.864 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.853+0000 7f3e6e423700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e680c16b0 con 0x7f3e681560e0 2023-01-24T03:30:18.864 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.854+0000 7f3e6cc20700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f3e680c4520 0x7f3e681559d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:59242/0 (socket says 172.21.15.112:59242) 2023-01-24T03:30:18.864 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.854+0000 7f3e6cc20700 1 -- 172.21.15.112:0/54397210 learned_addr learned my addr 172.21.15.112:0/54397210 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:18.864 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.854+0000 7f3e6dc22700 1 --2- 172.21.15.112:0/54397210 >> v2:172.21.15.112:3300/0 conn(0x7f3e681560e0 0x7f3e6815e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:18.864 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.854+0000 7f3e6dc22700 1 -- 172.21.15.112:0/54397210 >> v1:172.21.15.142:6789/0 conn(0x7f3e6806ce70 legacy=0x7f3e680c3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:18.865 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.854+0000 7f3e6dc22700 1 -- 172.21.15.112:0/54397210 >> v1:172.21.15.112:6789/0 conn(0x7f3e680c4520 legacy=0x7f3e681559d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:18.865 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.854+0000 7f3e6dc22700 1 -- 172.21.15.112:0/54397210 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e680c3b80 con 0x7f3e681560e0 2023-01-24T03:30:18.865 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.854+0000 7f3e6dc22700 1 --2- 172.21.15.112:0/54397210 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e681560e0 0x7f3e6815e940 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3e60009760 tx=0x7f3e60009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb192f8cf8a0b975 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:18.865 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.854+0000 7f3e67fff700 1 -- 172.21.15.112:0/54397210 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e60011630 con 0x7f3e681560e0 2023-01-24T03:30:18.865 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.854+0000 7f3e67fff700 1 -- 172.21.15.112:0/54397210 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e60011790 con 0x7f3e681560e0 2023-01-24T03:30:18.866 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.854+0000 7f3e67fff700 1 -- 172.21.15.112:0/54397210 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e60011b10 con 0x7f3e681560e0 2023-01-24T03:30:18.866 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.855+0000 7f3e6e423700 1 -- 172.21.15.112:0/54397210 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e681560e0 msgr2=0x7f3e6815e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:18.866 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.855+0000 7f3e6e423700 1 --2- 172.21.15.112:0/54397210 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e681560e0 0x7f3e6815e940 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3e60009760 tx=0x7f3e60009060 comp rx=0 tx=0).stop 2023-01-24T03:30:18.866 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.855+0000 7f3e6e423700 1 -- 172.21.15.112:0/54397210 shutdown_connections 2023-01-24T03:30:18.866 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.855+0000 7f3e6e423700 1 --2- 172.21.15.112:0/54397210 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e681560e0 0x7f3e6815e940 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:18.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.855+0000 7f3e6e423700 1 -- 172.21.15.112:0/54397210 >> 172.21.15.112:0/54397210 conn(0x7f3e6805e2b0 msgr2=0x7f3e680c96e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:18.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.855+0000 7f3e6e423700 1 -- 172.21.15.112:0/54397210 shutdown_connections 2023-01-24T03:30:18.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.855+0000 7f3e6e423700 1 -- 172.21.15.112:0/54397210 wait complete. 2023-01-24T03:30:18.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.856+0000 7f3e6e423700 1 Processor -- start 2023-01-24T03:30:18.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.856+0000 7f3e6e423700 1 -- start start 2023-01-24T03:30:18.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.856+0000 7f3e6e423700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e6806ce70 0x7f3e680a7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:18.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.856+0000 7f3e6e423700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3e680c4520 0x7f3e680ca140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:18.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.857+0000 7f3e6e423700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e681560e0 0x7f3e680aa010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:18.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.857+0000 7f3e6e423700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e68163cd0 con 0x7f3e681560e0 2023-01-24T03:30:18.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.857+0000 7f3e6e423700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e680a8510 con 0x7f3e6806ce70 2023-01-24T03:30:18.869 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.857+0000 7f3e6e423700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e681630c0 con 0x7f3e680c4520 2023-01-24T03:30:18.869 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.857+0000 7f3e6dc22700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e681560e0 0x7f3e680aa010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:18.869 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.857+0000 7f3e6dc22700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e681560e0 0x7f3e680aa010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50032/0 (socket says 172.21.15.112:50032) 2023-01-24T03:30:18.869 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.857+0000 7f3e6dc22700 1 -- 172.21.15.112:0/1787796592 learned_addr learned my addr 172.21.15.112:0/1787796592 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:18.869 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.857+0000 7f3e6cc20700 1 --2- 172.21.15.112:0/1787796592 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3e680c4520 0x7f3e680ca140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:18.870 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.857+0000 7f3e6d421700 1 --2- 172.21.15.112:0/1787796592 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e6806ce70 0x7f3e680a7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:18.870 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.857+0000 7f3e6dc22700 1 -- 172.21.15.112:0/1787796592 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3e680c4520 msgr2=0x7f3e680ca140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:18.870 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.857+0000 7f3e6dc22700 1 --2- 172.21.15.112:0/1787796592 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3e680c4520 0x7f3e680ca140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:18.870 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.857+0000 7f3e6dc22700 1 -- 172.21.15.112:0/1787796592 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e6806ce70 msgr2=0x7f3e680a7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:18.870 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.857+0000 7f3e6dc22700 1 --2- 172.21.15.112:0/1787796592 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e6806ce70 0x7f3e680a7e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:18.870 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.857+0000 7f3e6dc22700 1 -- 172.21.15.112:0/1787796592 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e600092a0 con 0x7f3e681560e0 2023-01-24T03:30:18.871 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.857+0000 7f3e6dc22700 1 --2- 172.21.15.112:0/1787796592 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e681560e0 0x7f3e680aa010 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3e60009eb0 tx=0x7f3e60017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:18.871 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.857+0000 7f3e657fa700 1 -- 172.21.15.112:0/1787796592 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e60011b10 con 0x7f3e681560e0 2023-01-24T03:30:18.871 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.857+0000 7f3e657fa700 1 -- 172.21.15.112:0/1787796592 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e600253f0 con 0x7f3e681560e0 2023-01-24T03:30:18.871 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.857+0000 7f3e6e423700 1 -- 172.21.15.112:0/1787796592 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e681633a0 con 0x7f3e681560e0 2023-01-24T03:30:18.872 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.857+0000 7f3e657fa700 1 -- 172.21.15.112:0/1787796592 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e600257b0 con 0x7f3e681560e0 2023-01-24T03:30:18.872 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.857+0000 7f3e6e423700 1 -- 172.21.15.112:0/1787796592 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e68163960 con 0x7f3e681560e0 2023-01-24T03:30:18.872 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.858+0000 7f3e6e423700 1 -- 172.21.15.112:0/1787796592 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e6806c640 con 0x7f3e681560e0 2023-01-24T03:30:18.872 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.859+0000 7f3e657fa700 1 -- 172.21.15.112:0/1787796592 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f3e6000f050 con 0x7f3e681560e0 2023-01-24T03:30:18.872 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.859+0000 7f3e657fa700 1 --2- 172.21.15.112:0/1787796592 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3e480611f0 0x7f3e480636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:18.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.859+0000 7f3e657fa700 1 -- 172.21.15.112:0/1787796592 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f3e6007efc0 con 0x7f3e681560e0 2023-01-24T03:30:18.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.859+0000 7f3e6d421700 1 --2- 172.21.15.112:0/1787796592 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3e480611f0 0x7f3e480636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:18.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.860+0000 7f3e6d421700 1 --2- 172.21.15.112:0/1787796592 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3e480611f0 0x7f3e480636a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3e58004290 tx=0x7f3e58001560 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:18.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:18.863+0000 7f3e657fa700 1 -- 172.21.15.112:0/1787796592 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f3e6004f3d0 con 0x7f3e681560e0 2023-01-24T03:30:19.044 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.042+0000 7f3e6e423700 1 -- 172.21.15.112:0/1787796592 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f3e6804b340 con 0x7f3e681560e0 2023-01-24T03:30:19.592 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.590+0000 7f3e657fa700 1 -- 172.21.15.112:0/1787796592 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3e600529f0 con 0x7f3e681560e0 2023-01-24T03:30:19.596 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.594+0000 7f3e6e423700 1 -- 172.21.15.112:0/1787796592 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3e480611f0 msgr2=0x7f3e480636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:19.597 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.594+0000 7f3e6e423700 1 --2- 172.21.15.112:0/1787796592 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3e480611f0 0x7f3e480636a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3e58004290 tx=0x7f3e58001560 comp rx=0 tx=0).stop 2023-01-24T03:30:19.597 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.594+0000 7f3e6e423700 1 -- 172.21.15.112:0/1787796592 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e681560e0 msgr2=0x7f3e680aa010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:19.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.594+0000 7f3e6e423700 1 --2- 172.21.15.112:0/1787796592 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e681560e0 0x7f3e680aa010 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3e60009eb0 tx=0x7f3e60017000 comp rx=0 tx=0).stop 2023-01-24T03:30:19.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.594+0000 7f3e6e423700 1 -- 172.21.15.112:0/1787796592 shutdown_connections 2023-01-24T03:30:19.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.594+0000 7f3e6e423700 1 --2- 172.21.15.112:0/1787796592 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3e480611f0 0x7f3e480636a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:19.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.594+0000 7f3e6e423700 1 --2- 172.21.15.112:0/1787796592 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e681560e0 0x7f3e680aa010 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:19.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.594+0000 7f3e6e423700 1 --2- 172.21.15.112:0/1787796592 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3e6806ce70 0x7f3e680a7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:19.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.594+0000 7f3e6e423700 1 --2- 172.21.15.112:0/1787796592 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3e680c4520 0x7f3e680ca140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:19.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.594+0000 7f3e6e423700 1 -- 172.21.15.112:0/1787796592 >> 172.21.15.112:0/1787796592 conn(0x7f3e6805e2b0 msgr2=0x7f3e68153ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:19.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.594+0000 7f3e6e423700 1 -- 172.21.15.112:0/1787796592 shutdown_connections 2023-01-24T03:30:19.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.594+0000 7f3e6e423700 1 -- 172.21.15.112:0/1787796592 wait complete. 2023-01-24T03:30:19.600 INFO:teuthology.orchestra.run.smithi112.stderr:pool 'cephfs_metadata' created 2023-01-24T03:30:19.613 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2023-01-24T03:30:19.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.735+0000 7f437ffff700 1 Processor -- start 2023-01-24T03:30:19.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.735+0000 7f437ffff700 1 -- start start 2023-01-24T03:30:19.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.736+0000 7f437ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f43800c1c60 0x7f4380152b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:19.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.736+0000 7f437ffff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f438006deb0 0x7f4380066e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:19.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.736+0000 7f437ffff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f438006f170 0x7f43800c2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:19.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.736+0000 7f437ffff700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43801530a0 con 0x7f438006deb0 2023-01-24T03:30:19.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.736+0000 7f437ffff700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43800c40b0 con 0x7f438006f170 2023-01-24T03:30:19.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.736+0000 7f437ffff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43800c37c0 con 0x7f43800c1c60 2023-01-24T03:30:19.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.736+0000 7f4377fff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f438006deb0 0x7f4380066e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:19.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.736+0000 7f4377fff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f438006deb0 0x7f4380066e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50040/0 (socket says 172.21.15.112:50040) 2023-01-24T03:30:19.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.736+0000 7f4377fff700 1 -- 172.21.15.112:0/3524861980 learned_addr learned my addr 172.21.15.112:0/3524861980 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:19.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.736+0000 7f437effd700 1 --2- 172.21.15.112:0/3524861980 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f43800c1c60 0x7f4380152b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:19.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.736+0000 7f437f7fe700 1 --2- 172.21.15.112:0/3524861980 >> v2:172.21.15.142:3300/0 conn(0x7f438006f170 0x7f43800c2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:19.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.736+0000 7f4377fff700 1 -- 172.21.15.112:0/3524861980 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f43800c1c60 msgr2=0x7f4380152b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:19.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.736+0000 7f4377fff700 1 --2- 172.21.15.112:0/3524861980 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f43800c1c60 0x7f4380152b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:19.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.736+0000 7f4377fff700 1 -- 172.21.15.112:0/3524861980 >> v2:172.21.15.142:3300/0 conn(0x7f438006f170 msgr2=0x7f43800c2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:19.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.736+0000 7f4377fff700 1 --2- 172.21.15.112:0/3524861980 >> v2:172.21.15.142:3300/0 conn(0x7f438006f170 0x7f43800c2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:19.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.736+0000 7f4377fff700 1 -- 172.21.15.112:0/3524861980 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f438006e360 con 0x7f438006deb0 2023-01-24T03:30:19.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.736+0000 7f4377fff700 1 --2- 172.21.15.112:0/3524861980 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f438006deb0 0x7f4380066e10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4380071f40 tx=0x7f43680090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82ebe3248d6da7e3 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:19.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.736+0000 7f437e7fc700 1 -- 172.21.15.112:0/3524861980 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f436800b690 con 0x7f438006deb0 2023-01-24T03:30:19.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.736+0000 7f437e7fc700 1 -- 172.21.15.112:0/3524861980 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f436800b830 con 0x7f438006deb0 2023-01-24T03:30:19.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.736+0000 7f437e7fc700 1 -- 172.21.15.112:0/3524861980 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f436800bbf0 con 0x7f438006deb0 2023-01-24T03:30:19.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.737+0000 7f437ffff700 1 -- 172.21.15.112:0/3524861980 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f438006deb0 msgr2=0x7f4380066e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:19.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.737+0000 7f437ffff700 1 --2- 172.21.15.112:0/3524861980 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f438006deb0 0x7f4380066e10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4380071f40 tx=0x7f43680090f0 comp rx=0 tx=0).stop 2023-01-24T03:30:19.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.737+0000 7f437ffff700 1 -- 172.21.15.112:0/3524861980 shutdown_connections 2023-01-24T03:30:19.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.737+0000 7f437ffff700 1 --2- 172.21.15.112:0/3524861980 >> v2:172.21.15.142:3300/0 conn(0x7f438006f170 0x7f43800c2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:19.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.737+0000 7f437ffff700 1 --2- 172.21.15.112:0/3524861980 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f43800c1c60 0x7f4380152b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:19.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.737+0000 7f437ffff700 1 --2- 172.21.15.112:0/3524861980 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f438006deb0 0x7f4380066e10 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:19.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.737+0000 7f437ffff700 1 -- 172.21.15.112:0/3524861980 >> 172.21.15.112:0/3524861980 conn(0x7f43800c4920 msgr2=0x7f43800c4d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:19.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.737+0000 7f437ffff700 1 -- 172.21.15.112:0/3524861980 shutdown_connections 2023-01-24T03:30:19.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.737+0000 7f437ffff700 1 -- 172.21.15.112:0/3524861980 wait complete. 2023-01-24T03:30:19.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.738+0000 7f437ffff700 1 Processor -- start 2023-01-24T03:30:19.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.738+0000 7f437ffff700 1 -- start start 2023-01-24T03:30:19.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.739+0000 7f437ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f438006deb0 0x7f438014f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:19.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.739+0000 7f437ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f438006f170 0x7f43800bcc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:19.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.739+0000 7f437ffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43800c1c60 0x7f43800bd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:19.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.739+0000 7f437ffff700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4380151f00 con 0x7f438006deb0 2023-01-24T03:30:19.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.739+0000 7f437ffff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438014dd30 con 0x7f43800c1c60 2023-01-24T03:30:19.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.739+0000 7f437ffff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43801516f0 con 0x7f438006f170 2023-01-24T03:30:19.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.739+0000 7f437effd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f438006deb0 0x7f438014f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:19.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.739+0000 7f437effd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f438006deb0 0x7f438014f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50042/0 (socket says 172.21.15.112:50042) 2023-01-24T03:30:19.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.739+0000 7f437effd700 1 -- 172.21.15.112:0/2198290761 learned_addr learned my addr 172.21.15.112:0/2198290761 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:19.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.739+0000 7f4377fff700 1 --2- 172.21.15.112:0/2198290761 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f438006f170 0x7f43800bcc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:19.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.739+0000 7f437f7fe700 1 --2- 172.21.15.112:0/2198290761 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43800c1c60 0x7f43800bd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:19.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.739+0000 7f437effd700 1 -- 172.21.15.112:0/2198290761 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f438006f170 msgr2=0x7f43800bcc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:19.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.739+0000 7f437effd700 1 --2- 172.21.15.112:0/2198290761 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f438006f170 0x7f43800bcc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:19.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.739+0000 7f437effd700 1 -- 172.21.15.112:0/2198290761 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43800c1c60 msgr2=0x7f43800bd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:19.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.739+0000 7f437effd700 1 --2- 172.21.15.112:0/2198290761 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43800c1c60 0x7f43800bd1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:19.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.739+0000 7f437effd700 1 -- 172.21.15.112:0/2198290761 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4368009370 con 0x7f438006deb0 2023-01-24T03:30:19.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.739+0000 7f437f7fe700 1 --2- 172.21.15.112:0/2198290761 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43800c1c60 0x7f43800bd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:30:19.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.739+0000 7f4377fff700 1 --2- 172.21.15.112:0/2198290761 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f438006f170 0x7f43800bcc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:30:19.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.739+0000 7f437effd700 1 --2- 172.21.15.112:0/2198290761 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f438006deb0 0x7f438014f070 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f437000d890 tx=0x7f4370008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:19.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.739+0000 7f4376ffd700 1 -- 172.21.15.112:0/2198290761 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4370014620 con 0x7f438006deb0 2023-01-24T03:30:19.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.739+0000 7f4376ffd700 1 -- 172.21.15.112:0/2198290761 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4370014780 con 0x7f438006deb0 2023-01-24T03:30:19.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.739+0000 7f437ffff700 1 -- 172.21.15.112:0/2198290761 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f438014d6d0 con 0x7f438006deb0 2023-01-24T03:30:19.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.739+0000 7f4376ffd700 1 -- 172.21.15.112:0/2198290761 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4370014b20 con 0x7f438006deb0 2023-01-24T03:30:19.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.739+0000 7f437ffff700 1 -- 172.21.15.112:0/2198290761 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4380151c20 con 0x7f438006deb0 2023-01-24T03:30:19.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.740+0000 7f437ffff700 1 -- 172.21.15.112:0/2198290761 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f438005ea20 con 0x7f438006deb0 2023-01-24T03:30:19.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.740+0000 7f4376ffd700 1 -- 172.21.15.112:0/2198290761 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f437001d400 con 0x7f438006deb0 2023-01-24T03:30:19.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.741+0000 7f4376ffd700 1 --2- 172.21.15.112:0/2198290761 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f43640611c0 0x7f4364063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:19.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.741+0000 7f4376ffd700 1 -- 172.21.15.112:0/2198290761 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5576+0+0 (secure 0 0 0) 0x7f437007d9e0 con 0x7f438006deb0 2023-01-24T03:30:19.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.741+0000 7f4377fff700 1 --2- 172.21.15.112:0/2198290761 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f43640611c0 0x7f4364063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:19.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.741+0000 7f4377fff700 1 --2- 172.21.15.112:0/2198290761 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f43640611c0 0x7f4364063670 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4368009340 tx=0x7f4368016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:19.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.745+0000 7f4376ffd700 1 -- 172.21.15.112:0/2198290761 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f437004dba0 con 0x7f438006deb0 2023-01-24T03:30:19.927 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:19.926+0000 7f437ffff700 1 -- 172.21.15.112:0/2198290761 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f438006e8c0 con 0x7f438006deb0 2023-01-24T03:30:20.593 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.592+0000 7f4376ffd700 1 -- 172.21.15.112:0/2198290761 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v18) v1 ==== 171+0+0 (secure 0 0 0) 0x7f43700511c0 con 0x7f438006deb0 2023-01-24T03:30:20.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.596+0000 7f437ffff700 1 -- 172.21.15.112:0/2198290761 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f43640611c0 msgr2=0x7f4364063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:20.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.596+0000 7f437ffff700 1 --2- 172.21.15.112:0/2198290761 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f43640611c0 0x7f4364063670 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4368009340 tx=0x7f4368016000 comp rx=0 tx=0).stop 2023-01-24T03:30:20.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.596+0000 7f437ffff700 1 -- 172.21.15.112:0/2198290761 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f438006deb0 msgr2=0x7f438014f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:20.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.596+0000 7f437ffff700 1 --2- 172.21.15.112:0/2198290761 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f438006deb0 0x7f438014f070 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f437000d890 tx=0x7f4370008040 comp rx=0 tx=0).stop 2023-01-24T03:30:20.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.597+0000 7f437ffff700 1 -- 172.21.15.112:0/2198290761 shutdown_connections 2023-01-24T03:30:20.600 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.597+0000 7f437ffff700 1 --2- 172.21.15.112:0/2198290761 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f43640611c0 0x7f4364063670 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:20.600 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.597+0000 7f437ffff700 1 --2- 172.21.15.112:0/2198290761 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f438006deb0 0x7f438014f070 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:20.600 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.597+0000 7f437ffff700 1 --2- 172.21.15.112:0/2198290761 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43800c1c60 0x7f43800bd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:20.600 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.597+0000 7f437ffff700 1 --2- 172.21.15.112:0/2198290761 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f438006f170 0x7f43800bcc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:20.600 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.597+0000 7f437ffff700 1 -- 172.21.15.112:0/2198290761 >> 172.21.15.112:0/2198290761 conn(0x7f43800c4920 msgr2=0x7f4380067710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:20.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.599+0000 7f437ffff700 1 -- 172.21.15.112:0/2198290761 shutdown_connections 2023-01-24T03:30:20.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.600+0000 7f437ffff700 1 -- 172.21.15.112:0/2198290761 wait complete. 2023-01-24T03:30:20.612 INFO:teuthology.orchestra.run.smithi112.stderr:pool 'cephfs_data' created 2023-01-24T03:30:20.621 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-01-24T03:30:20.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.735+0000 7faecf037700 1 Processor -- start 2023-01-24T03:30:20.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.735+0000 7faecf037700 1 -- start start 2023-01-24T03:30:20.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.735+0000 7faecf037700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faec80c8e20 0x7faec8069960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:20.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.735+0000 7faecf037700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7faec81569e0 0x7faec80ae9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:20.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.735+0000 7faecf037700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faec81591f0 con 0x7faec805e340 2023-01-24T03:30:20.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.735+0000 7faecf037700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faec8159a10 con 0x7faec81569e0 2023-01-24T03:30:20.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.735+0000 7faecf037700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faec8156e80 con 0x7faec80c8e20 2023-01-24T03:30:20.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.735+0000 7faecd5d4700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7faec81569e0 0x7faec80ae9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:20.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.736+0000 7faecd5d4700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7faec81569e0 0x7faec80ae9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50052/0 (socket says 172.21.15.112:50052) 2023-01-24T03:30:20.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.736+0000 7faecd5d4700 1 -- 172.21.15.112:0/435695433 learned_addr learned my addr 172.21.15.112:0/435695433 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:20.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.736+0000 7faeccdd3700 1 --2- 172.21.15.112:0/435695433 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faec80c8e20 0x7faec8069960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:20.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.736+0000 7faecd5d4700 1 -- 172.21.15.112:0/435695433 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faec80c8e20 msgr2=0x7faec8069960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:20.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.736+0000 7faecd5d4700 1 --2- 172.21.15.112:0/435695433 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faec80c8e20 0x7faec8069960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:20.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.736+0000 7faecd5d4700 1 -- 172.21.15.112:0/435695433 >> v1:172.21.15.142:6789/0 conn(0x7faec805e340 legacy=0x7faec8158ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:20.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.736+0000 7faecd5d4700 1 -- 172.21.15.112:0/435695433 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faec8159450 con 0x7faec81569e0 2023-01-24T03:30:20.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.736+0000 7faeccdd3700 1 --2- 172.21.15.112:0/435695433 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faec80c8e20 0x7faec8069960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:30:20.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.736+0000 7faecd5d4700 1 --2- 172.21.15.112:0/435695433 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faec81569e0 0x7faec80ae9f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7faec0009f20 tx=0x7faec0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efa67b177721ed3 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:20.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.736+0000 7faec77fe700 1 -- 172.21.15.112:0/435695433 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec000e070 con 0x7faec81569e0 2023-01-24T03:30:20.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.736+0000 7faec77fe700 1 -- 172.21.15.112:0/435695433 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faec000ba80 con 0x7faec81569e0 2023-01-24T03:30:20.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.736+0000 7faec77fe700 1 -- 172.21.15.112:0/435695433 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec00092e0 con 0x7faec81569e0 2023-01-24T03:30:20.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.737+0000 7faecf037700 1 -- 172.21.15.112:0/435695433 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faec81569e0 msgr2=0x7faec80ae9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:20.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.737+0000 7faecf037700 1 --2- 172.21.15.112:0/435695433 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faec81569e0 0x7faec80ae9f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7faec0009f20 tx=0x7faec0009840 comp rx=0 tx=0).stop 2023-01-24T03:30:20.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.737+0000 7faecf037700 1 -- 172.21.15.112:0/435695433 shutdown_connections 2023-01-24T03:30:20.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.737+0000 7faecf037700 1 --2- 172.21.15.112:0/435695433 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faec81569e0 0x7faec80ae9f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:20.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.737+0000 7faecf037700 1 --2- 172.21.15.112:0/435695433 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faec80c8e20 0x7faec8069960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:20.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.737+0000 7faecf037700 1 -- 172.21.15.112:0/435695433 >> 172.21.15.112:0/435695433 conn(0x7faec805d830 msgr2=0x7faec805dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:20.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.737+0000 7faecf037700 1 -- 172.21.15.112:0/435695433 shutdown_connections 2023-01-24T03:30:20.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.737+0000 7faecf037700 1 -- 172.21.15.112:0/435695433 wait complete. 2023-01-24T03:30:20.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.738+0000 7faecf037700 1 Processor -- start 2023-01-24T03:30:20.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.738+0000 7faecf037700 1 -- start start 2023-01-24T03:30:20.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.739+0000 7faecf037700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faec805e340 0x7faec8168fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:20.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.739+0000 7faecf037700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faec80c8e20 0x7faec81577b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:20.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.739+0000 7faecf037700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faec81569e0 0x7faec8165890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:20.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.739+0000 7faecf037700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faec8169510 con 0x7faec81569e0 2023-01-24T03:30:20.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.739+0000 7faecf037700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faec8165400 con 0x7faec805e340 2023-01-24T03:30:20.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.739+0000 7faecf037700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faec80c7c40 con 0x7faec80c8e20 2023-01-24T03:30:20.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.739+0000 7faecd5d4700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faec81569e0 0x7faec8165890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:20.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.739+0000 7faecd5d4700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faec81569e0 0x7faec8165890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50060/0 (socket says 172.21.15.112:50060) 2023-01-24T03:30:20.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.739+0000 7faecd5d4700 1 -- 172.21.15.112:0/3118994522 learned_addr learned my addr 172.21.15.112:0/3118994522 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:20.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.739+0000 7faec7fff700 1 --2- 172.21.15.112:0/3118994522 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faec80c8e20 0x7faec81577b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:20.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.739+0000 7faeccdd3700 1 --2- 172.21.15.112:0/3118994522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faec805e340 0x7faec8168fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:20.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.739+0000 7faecd5d4700 1 -- 172.21.15.112:0/3118994522 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faec80c8e20 msgr2=0x7faec81577b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:20.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.739+0000 7faecd5d4700 1 --2- 172.21.15.112:0/3118994522 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faec80c8e20 0x7faec81577b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:20.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.739+0000 7faecd5d4700 1 -- 172.21.15.112:0/3118994522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faec805e340 msgr2=0x7faec8168fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:20.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.739+0000 7faecd5d4700 1 --2- 172.21.15.112:0/3118994522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faec805e340 0x7faec8168fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:20.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.739+0000 7faecd5d4700 1 -- 172.21.15.112:0/3118994522 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faec0009ae0 con 0x7faec81569e0 2023-01-24T03:30:20.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.739+0000 7faecd5d4700 1 --2- 172.21.15.112:0/3118994522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faec81569e0 0x7faec8165890 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7faec000b6c0 tx=0x7faec0018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:20.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.739+0000 7faec4ff9700 1 -- 172.21.15.112:0/3118994522 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec00032f0 con 0x7faec81569e0 2023-01-24T03:30:20.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.739+0000 7faec4ff9700 1 -- 172.21.15.112:0/3118994522 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faec00092e0 con 0x7faec81569e0 2023-01-24T03:30:20.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.739+0000 7faec4ff9700 1 -- 172.21.15.112:0/3118994522 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec001d420 con 0x7faec81569e0 2023-01-24T03:30:20.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.739+0000 7faecf037700 1 -- 172.21.15.112:0/3118994522 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faec81650b0 con 0x7faec81569e0 2023-01-24T03:30:20.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.739+0000 7faecf037700 1 -- 172.21.15.112:0/3118994522 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faec80c81c0 con 0x7faec81569e0 2023-01-24T03:30:20.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.740+0000 7faecf037700 1 -- 172.21.15.112:0/3118994522 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faec806f3a0 con 0x7faec81569e0 2023-01-24T03:30:20.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.741+0000 7faec4ff9700 1 -- 172.21.15.112:0/3118994522 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7faec001d810 con 0x7faec81569e0 2023-01-24T03:30:20.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.741+0000 7faec4ff9700 1 --2- 172.21.15.112:0/3118994522 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7faeac0611c0 0x7faeac063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:20.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.741+0000 7faeccdd3700 1 --2- 172.21.15.112:0/3118994522 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7faeac0611c0 0x7faeac063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:20.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.741+0000 7faec4ff9700 1 -- 172.21.15.112:0/3118994522 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5958+0+0 (secure 0 0 0) 0x7faec007f2e0 con 0x7faec81569e0 2023-01-24T03:30:20.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.742+0000 7faeccdd3700 1 --2- 172.21.15.112:0/3118994522 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7faeac0611c0 0x7faeac063670 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7faeb8006d80 tx=0x7faeb8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:20.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.746+0000 7faec4ff9700 1 -- 172.21.15.112:0/3118994522 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7faec004f470 con 0x7faec81569e0 2023-01-24T03:30:20.972 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.970+0000 7faecf037700 1 -- 172.21.15.112:0/3118994522 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7faec8152f40 con 0x7faec81569e0 2023-01-24T03:30:20.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.978+0000 7faec4ff9700 1 -- 172.21.15.112:0/3118994522 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 3 and data pool 4 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7faec0052a90 con 0x7faec81569e0 2023-01-24T03:30:20.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.982+0000 7faecf037700 1 -- 172.21.15.112:0/3118994522 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7faeac0611c0 msgr2=0x7faeac063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:20.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.982+0000 7faecf037700 1 --2- 172.21.15.112:0/3118994522 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7faeac0611c0 0x7faeac063670 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7faeb8006d80 tx=0x7faeb8008040 comp rx=0 tx=0).stop 2023-01-24T03:30:20.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.982+0000 7faecf037700 1 -- 172.21.15.112:0/3118994522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faec81569e0 msgr2=0x7faec8165890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:20.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.982+0000 7faecf037700 1 --2- 172.21.15.112:0/3118994522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faec81569e0 0x7faec8165890 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7faec000b6c0 tx=0x7faec0018000 comp rx=0 tx=0).stop 2023-01-24T03:30:20.986 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.983+0000 7faecf037700 1 -- 172.21.15.112:0/3118994522 shutdown_connections 2023-01-24T03:30:20.986 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.983+0000 7faecf037700 1 --2- 172.21.15.112:0/3118994522 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7faeac0611c0 0x7faeac063670 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:20.986 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.983+0000 7faecf037700 1 --2- 172.21.15.112:0/3118994522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faec81569e0 0x7faec8165890 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:20.986 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.983+0000 7faecf037700 1 --2- 172.21.15.112:0/3118994522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faec805e340 0x7faec8168fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:20.987 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.983+0000 7faecf037700 1 --2- 172.21.15.112:0/3118994522 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faec80c8e20 0x7faec81577b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:20.987 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.983+0000 7faecf037700 1 -- 172.21.15.112:0/3118994522 >> 172.21.15.112:0/3118994522 conn(0x7faec805d830 msgr2=0x7faec80ae9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:20.987 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.985+0000 7faecf037700 1 -- 172.21.15.112:0/3118994522 shutdown_connections 2023-01-24T03:30:20.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:20.986+0000 7faecf037700 1 -- 172.21.15.112:0/3118994522 wait complete. 2023-01-24T03:30:20.991 INFO:teuthology.orchestra.run.smithi112.stderr:new fs with metadata pool 3 and data pool 4 2023-01-24T03:30:21.007 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2023-01-24T03:30:21.008 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2023-01-24T03:30:21.127 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.122+0000 7fe4782ff700 1 Processor -- start 2023-01-24T03:30:21.127 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.122+0000 7fe4782ff700 1 -- start start 2023-01-24T03:30:21.127 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.123+0000 7fe4782ff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe47005df40 0x7fe4701534e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:21.127 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.123+0000 7fe4782ff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fe470153a20 0x7fe47006b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:21.128 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.123+0000 7fe4782ff700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4701575a0 con 0x7fe4700c0dd0 2023-01-24T03:30:21.128 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.123+0000 7fe4782ff700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe470156780 con 0x7fe470153a20 2023-01-24T03:30:21.128 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.123+0000 7fe4782ff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4700c22c0 con 0x7fe47005df40 2023-01-24T03:30:21.128 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.123+0000 7fe47689c700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fe470153a20 0x7fe47006b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:21.128 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.123+0000 7fe47689c700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fe470153a20 0x7fe47006b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50070/0 (socket says 172.21.15.112:50070) 2023-01-24T03:30:21.129 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.123+0000 7fe47689c700 1 -- 172.21.15.112:0/2029143154 learned_addr learned my addr 172.21.15.112:0/2029143154 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:21.129 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.123+0000 7fe47589a700 1 --2- 172.21.15.112:0/2029143154 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe47005df40 0x7fe4701534e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:21.129 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.124+0000 7fe47689c700 1 -- 172.21.15.112:0/2029143154 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe47005df40 msgr2=0x7fe4701534e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:21.129 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.124+0000 7fe47689c700 1 --2- 172.21.15.112:0/2029143154 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe47005df40 0x7fe4701534e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:21.129 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.124+0000 7fe47689c700 1 -- 172.21.15.112:0/2029143154 >> v1:172.21.15.142:6789/0 conn(0x7fe4700c0dd0 legacy=0x7fe4700c11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:21.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.124+0000 7fe47689c700 1 -- 172.21.15.112:0/2029143154 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4700c18b0 con 0x7fe470153a20 2023-01-24T03:30:21.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.124+0000 7fe47689c700 1 --2- 172.21.15.112:0/2029143154 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe470153a20 0x7fe47006b3c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe468006fd0 tx=0x7fe4680090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1d05fbc673b9383 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:21.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.124+0000 7fe475099700 1 -- 172.21.15.112:0/2029143154 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe468011630 con 0x7fe470153a20 2023-01-24T03:30:21.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.124+0000 7fe475099700 1 -- 172.21.15.112:0/2029143154 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe468011790 con 0x7fe470153a20 2023-01-24T03:30:21.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.124+0000 7fe475099700 1 -- 172.21.15.112:0/2029143154 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe468011b50 con 0x7fe470153a20 2023-01-24T03:30:21.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.124+0000 7fe4782ff700 1 -- 172.21.15.112:0/2029143154 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe470153a20 msgr2=0x7fe47006b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:21.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.124+0000 7fe4782ff700 1 --2- 172.21.15.112:0/2029143154 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe470153a20 0x7fe47006b3c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe468006fd0 tx=0x7fe4680090f0 comp rx=0 tx=0).stop 2023-01-24T03:30:21.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.124+0000 7fe4782ff700 1 -- 172.21.15.112:0/2029143154 shutdown_connections 2023-01-24T03:30:21.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.124+0000 7fe4782ff700 1 --2- 172.21.15.112:0/2029143154 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe470153a20 0x7fe47006b3c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:21.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.124+0000 7fe4782ff700 1 --2- 172.21.15.112:0/2029143154 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe47005df40 0x7fe4701534e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:21.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.124+0000 7fe4782ff700 1 -- 172.21.15.112:0/2029143154 >> 172.21.15.112:0/2029143154 conn(0x7fe47006db00 msgr2=0x7fe47005d830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:21.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.124+0000 7fe4782ff700 1 -- 172.21.15.112:0/2029143154 shutdown_connections 2023-01-24T03:30:21.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.124+0000 7fe4782ff700 1 -- 172.21.15.112:0/2029143154 wait complete. 2023-01-24T03:30:21.135 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.125+0000 7fe4782ff700 1 Processor -- start 2023-01-24T03:30:21.135 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.125+0000 7fe4782ff700 1 -- start start 2023-01-24T03:30:21.135 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.126+0000 7fe4782ff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe47005df40 0x7fe470148a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:21.136 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.126+0000 7fe4782ff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4700c0dd0 0x7fe4701543a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:21.136 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.126+0000 7fe4782ff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe470153a20 0x7fe470143060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:21.136 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.126+0000 7fe4782ff700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe470156630 con 0x7fe470153a20 2023-01-24T03:30:21.136 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.126+0000 7fe4782ff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe470151de0 con 0x7fe4700c0dd0 2023-01-24T03:30:21.136 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.126+0000 7fe4782ff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe470155900 con 0x7fe47005df40 2023-01-24T03:30:21.137 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.126+0000 7fe47689c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe470153a20 0x7fe470143060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:21.137 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.126+0000 7fe47689c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe470153a20 0x7fe470143060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50078/0 (socket says 172.21.15.112:50078) 2023-01-24T03:30:21.137 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.126+0000 7fe47689c700 1 -- 172.21.15.112:0/2366446896 learned_addr learned my addr 172.21.15.112:0/2366446896 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:21.137 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.126+0000 7fe47589a700 1 --2- 172.21.15.112:0/2366446896 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4700c0dd0 0x7fe4701543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:21.137 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.126+0000 7fe47609b700 1 --2- 172.21.15.112:0/2366446896 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe47005df40 0x7fe470148a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:21.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.126+0000 7fe47689c700 1 -- 172.21.15.112:0/2366446896 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe47005df40 msgr2=0x7fe470148a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:21.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.126+0000 7fe47689c700 1 --2- 172.21.15.112:0/2366446896 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe47005df40 0x7fe470148a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:21.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.126+0000 7fe47689c700 1 -- 172.21.15.112:0/2366446896 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4700c0dd0 msgr2=0x7fe4701543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:21.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.126+0000 7fe47689c700 1 --2- 172.21.15.112:0/2366446896 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4700c0dd0 0x7fe4701543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:21.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.126+0000 7fe47689c700 1 -- 172.21.15.112:0/2366446896 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe468009370 con 0x7fe470153a20 2023-01-24T03:30:21.139 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.127+0000 7fe47689c700 1 --2- 172.21.15.112:0/2366446896 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe470153a20 0x7fe470143060 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe468011fd0 tx=0x7fe468016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:21.139 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.127+0000 7fe46e7fc700 1 -- 172.21.15.112:0/2366446896 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46801ce70 con 0x7fe470153a20 2023-01-24T03:30:21.139 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.127+0000 7fe46e7fc700 1 -- 172.21.15.112:0/2366446896 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe46801a450 con 0x7fe470153a20 2023-01-24T03:30:21.139 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.127+0000 7fe46e7fc700 1 -- 172.21.15.112:0/2366446896 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46801a7f0 con 0x7fe470153a20 2023-01-24T03:30:21.139 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.127+0000 7fe4782ff700 1 -- 172.21.15.112:0/2366446896 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4700c1660 con 0x7fe470153a20 2023-01-24T03:30:21.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.127+0000 7fe4782ff700 1 -- 172.21.15.112:0/2366446896 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe470069960 con 0x7fe470153a20 2023-01-24T03:30:21.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.128+0000 7fe4782ff700 1 -- 172.21.15.112:0/2366446896 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4700c64b0 con 0x7fe470153a20 2023-01-24T03:30:21.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.128+0000 7fe46e7fc700 1 -- 172.21.15.112:0/2366446896 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fe46802b020 con 0x7fe470153a20 2023-01-24T03:30:21.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.128+0000 7fe46e7fc700 1 --2- 172.21.15.112:0/2366446896 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fe4500611c0 0x7fe450063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:21.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.128+0000 7fe46e7fc700 1 -- 172.21.15.112:0/2366446896 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fe46807f460 con 0x7fe470153a20 2023-01-24T03:30:21.141 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.128+0000 7fe47609b700 1 --2- 172.21.15.112:0/2366446896 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fe4500611c0 0x7fe450063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:21.141 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.129+0000 7fe47609b700 1 --2- 172.21.15.112:0/2366446896 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fe4500611c0 0x7fe450063670 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe460004290 tx=0x7fe460001560 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:21.141 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.133+0000 7fe46e7fc700 1 -- 172.21.15.112:0/2366446896 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fe4700c64b0 con 0x7fe470153a20 2023-01-24T03:30:21.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.320+0000 7fe4782ff700 1 -- 172.21.15.112:0/2366446896 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7fe4700c64b0 con 0x7fe470153a20 2023-01-24T03:30:21.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.979+0000 7fe46e7fc700 1 -- 172.21.15.112:0/2366446896 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v20) v1 ==== 157+0+0 (secure 0 0 0) 0x7fe4700c64b0 con 0x7fe470153a20 2023-01-24T03:30:21.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.981+0000 7fe4782ff700 1 -- 172.21.15.112:0/2366446896 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fe4500611c0 msgr2=0x7fe450063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:21.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.981+0000 7fe4782ff700 1 --2- 172.21.15.112:0/2366446896 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fe4500611c0 0x7fe450063670 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe460004290 tx=0x7fe460001560 comp rx=0 tx=0).stop 2023-01-24T03:30:21.985 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.981+0000 7fe4782ff700 1 -- 172.21.15.112:0/2366446896 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe470153a20 msgr2=0x7fe470143060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:21.985 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.981+0000 7fe4782ff700 1 --2- 172.21.15.112:0/2366446896 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe470153a20 0x7fe470143060 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe468011fd0 tx=0x7fe468016000 comp rx=0 tx=0).stop 2023-01-24T03:30:21.985 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.981+0000 7fe4782ff700 1 -- 172.21.15.112:0/2366446896 shutdown_connections 2023-01-24T03:30:21.985 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.981+0000 7fe4782ff700 1 --2- 172.21.15.112:0/2366446896 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fe4500611c0 0x7fe450063670 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:21.985 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.981+0000 7fe4782ff700 1 --2- 172.21.15.112:0/2366446896 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe470153a20 0x7fe470143060 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:21.986 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.981+0000 7fe4782ff700 1 --2- 172.21.15.112:0/2366446896 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe47005df40 0x7fe470148a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:21.986 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.981+0000 7fe4782ff700 1 --2- 172.21.15.112:0/2366446896 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4700c0dd0 0x7fe4701543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:21.986 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.981+0000 7fe4782ff700 1 -- 172.21.15.112:0/2366446896 >> 172.21.15.112:0/2366446896 conn(0x7fe47006db00 msgr2=0x7fe47005d830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:21.986 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.981+0000 7fe4782ff700 1 -- 172.21.15.112:0/2366446896 shutdown_connections 2023-01-24T03:30:21.986 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:21.981+0000 7fe4782ff700 1 -- 172.21.15.112:0/2366446896 wait complete. 2023-01-24T03:30:21.997 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2023-01-24T03:30:22.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.120+0000 7f272d920700 1 Processor -- start 2023-01-24T03:30:22.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.120+0000 7f272d920700 1 -- start start 2023-01-24T03:30:22.125 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.121+0000 7f272d920700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f272814d4c0 0x7f27280c4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:22.125 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.121+0000 7f272d920700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f272806d980 0x7f27280c32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:22.126 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.121+0000 7f272d920700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f272806d4f0 con 0x7f272805dff0 2023-01-24T03:30:22.126 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.121+0000 7f272d920700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f272806dd80 con 0x7f272806d980 2023-01-24T03:30:22.126 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.121+0000 7f272d920700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f272806cee0 con 0x7f272814d4c0 2023-01-24T03:30:22.126 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.121+0000 7f27277fe700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f272806d980 0x7f27280c32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:22.127 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.121+0000 7f27267fc700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f272805dff0 0x7f2728153240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:59302/0 (socket says 172.21.15.112:59302) 2023-01-24T03:30:22.127 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.121+0000 7f27277fe700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f272806d980 0x7f27280c32a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50086/0 (socket says 172.21.15.112:50086) 2023-01-24T03:30:22.127 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.121+0000 7f27267fc700 1 -- 172.21.15.112:0/2566124558 learned_addr learned my addr 172.21.15.112:0/2566124558 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:22.127 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.121+0000 7f27277fe700 1 -- 172.21.15.112:0/2566124558 >> v2:172.21.15.142:3300/0 conn(0x7f272814d4c0 msgr2=0x7f27280c4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:22.127 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.121+0000 7f2726ffd700 1 --2- 172.21.15.112:0/2566124558 >> v2:172.21.15.142:3300/0 conn(0x7f272814d4c0 0x7f27280c4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:22.128 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.121+0000 7f27277fe700 1 --2- 172.21.15.112:0/2566124558 >> v2:172.21.15.142:3300/0 conn(0x7f272814d4c0 0x7f27280c4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:22.128 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.121+0000 7f27277fe700 1 -- 172.21.15.112:0/2566124558 >> v1:172.21.15.112:6789/0 conn(0x7f272805dff0 legacy=0x7f2728153240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:22.128 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.121+0000 7f2725ffb700 1 -- 172.21.15.112:0/2566124558 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2378861293 0 0) 0x7f272806d4f0 con 0x7f272805dff0 2023-01-24T03:30:22.128 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.121+0000 7f27277fe700 1 -- 172.21.15.112:0/2566124558 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2728153af0 con 0x7f272806d980 2023-01-24T03:30:22.128 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.121+0000 7f27277fe700 1 --2- 172.21.15.112:0/2566124558 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f272806d980 0x7f27280c32a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f27280c7140 tx=0x7f2720009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64244113ffc9babf server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:22.129 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.121+0000 7f2725ffb700 1 -- 172.21.15.112:0/2566124558 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2720011630 con 0x7f272806d980 2023-01-24T03:30:22.129 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.122+0000 7f2725ffb700 1 -- 172.21.15.112:0/2566124558 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2720011790 con 0x7f272806d980 2023-01-24T03:30:22.129 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.122+0000 7f2725ffb700 1 -- 172.21.15.112:0/2566124558 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2720011b50 con 0x7f272806d980 2023-01-24T03:30:22.129 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.122+0000 7f272d920700 1 -- 172.21.15.112:0/2566124558 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f272806d980 msgr2=0x7f27280c32a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:22.129 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.122+0000 7f272d920700 1 --2- 172.21.15.112:0/2566124558 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f272806d980 0x7f27280c32a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f27280c7140 tx=0x7f2720009060 comp rx=0 tx=0).stop 2023-01-24T03:30:22.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.122+0000 7f272d920700 1 -- 172.21.15.112:0/2566124558 shutdown_connections 2023-01-24T03:30:22.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.122+0000 7f272d920700 1 --2- 172.21.15.112:0/2566124558 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f272806d980 0x7f27280c32a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:22.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.122+0000 7f272d920700 1 --2- 172.21.15.112:0/2566124558 >> v2:172.21.15.142:3300/0 conn(0x7f272814d4c0 0x7f27280c4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:22.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.122+0000 7f272d920700 1 -- 172.21.15.112:0/2566124558 >> 172.21.15.112:0/2566124558 conn(0x7f272806c4f0 msgr2=0x7f272805d830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:22.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.122+0000 7f272d920700 1 -- 172.21.15.112:0/2566124558 shutdown_connections 2023-01-24T03:30:22.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.122+0000 7f272d920700 1 -- 172.21.15.112:0/2566124558 wait complete. 2023-01-24T03:30:22.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.123+0000 7f272d920700 1 Processor -- start 2023-01-24T03:30:22.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.123+0000 7f272d920700 1 -- start start 2023-01-24T03:30:22.137 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.124+0000 7f272d920700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f272805dff0 0x7f27280be9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:22.137 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.124+0000 7f272d920700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f272806d980 0x7f2728163c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:22.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.124+0000 7f272d920700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f272814d4c0 0x7f27281670e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:22.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.124+0000 7f272d920700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27280c0420 con 0x7f272814d4c0 2023-01-24T03:30:22.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.124+0000 7f272d920700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27281477c0 con 0x7f272805dff0 2023-01-24T03:30:22.139 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.124+0000 7f272d920700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27281631c0 con 0x7f272806d980 2023-01-24T03:30:22.139 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.124+0000 7f27277fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f272814d4c0 0x7f27281670e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:22.139 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.124+0000 7f27277fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f272814d4c0 0x7f27281670e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50092/0 (socket says 172.21.15.112:50092) 2023-01-24T03:30:22.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.124+0000 7f27277fe700 1 -- 172.21.15.112:0/4142424206 learned_addr learned my addr 172.21.15.112:0/4142424206 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:22.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.124+0000 7f2726ffd700 1 --2- 172.21.15.112:0/4142424206 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f272805dff0 0x7f27280be9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:22.141 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.124+0000 7f27267fc700 1 --2- 172.21.15.112:0/4142424206 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f272806d980 0x7f2728163c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:22.141 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.124+0000 7f27277fe700 1 -- 172.21.15.112:0/4142424206 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f272806d980 msgr2=0x7f2728163c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:22.141 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.124+0000 7f27277fe700 1 --2- 172.21.15.112:0/4142424206 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f272806d980 0x7f2728163c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:22.142 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.124+0000 7f27277fe700 1 -- 172.21.15.112:0/4142424206 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f272805dff0 msgr2=0x7f27280be9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:22.142 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.124+0000 7f27277fe700 1 --2- 172.21.15.112:0/4142424206 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f272805dff0 0x7f27280be9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:22.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.124+0000 7f27277fe700 1 -- 172.21.15.112:0/4142424206 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27200092a0 con 0x7f272814d4c0 2023-01-24T03:30:22.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.125+0000 7f27277fe700 1 --2- 172.21.15.112:0/4142424206 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f272814d4c0 0x7f27281670e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2728067880 tx=0x7f2720017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:22.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.125+0000 7f271effd700 1 -- 172.21.15.112:0/4142424206 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27200119d0 con 0x7f272814d4c0 2023-01-24T03:30:22.144 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.125+0000 7f271effd700 1 -- 172.21.15.112:0/4142424206 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2720011b30 con 0x7f272814d4c0 2023-01-24T03:30:22.144 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.125+0000 7f272d920700 1 -- 172.21.15.112:0/4142424206 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27280c1070 con 0x7f272814d4c0 2023-01-24T03:30:22.144 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.125+0000 7f271effd700 1 -- 172.21.15.112:0/4142424206 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2720025670 con 0x7f272814d4c0 2023-01-24T03:30:22.145 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.125+0000 7f272d920700 1 -- 172.21.15.112:0/4142424206 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27280c1630 con 0x7f272814d4c0 2023-01-24T03:30:22.145 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.126+0000 7f272d920700 1 -- 172.21.15.112:0/4142424206 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2728153dd0 con 0x7f272814d4c0 2023-01-24T03:30:22.145 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.126+0000 7f271effd700 1 -- 172.21.15.112:0/4142424206 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f272002c030 con 0x7f272814d4c0 2023-01-24T03:30:22.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.126+0000 7f271effd700 1 --2- 172.21.15.112:0/4142424206 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f270c0611c0 0x7f270c063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:22.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.126+0000 7f271effd700 1 -- 172.21.15.112:0/4142424206 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6284+0+0 (secure 0 0 0) 0x7f272007edf0 con 0x7f272814d4c0 2023-01-24T03:30:22.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.126+0000 7f2726ffd700 1 --2- 172.21.15.112:0/4142424206 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f270c0611c0 0x7f270c063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:22.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.127+0000 7f2726ffd700 1 --2- 172.21.15.112:0/4142424206 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f270c0611c0 0x7f270c063670 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f271800e420 tx=0x7f27180075e0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:22.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.130+0000 7f271effd700 1 -- 172.21.15.112:0/4142424206 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f272004ede0 con 0x7f272814d4c0 2023-01-24T03:30:22.313 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:22.311+0000 7f272d920700 1 -- 172.21.15.112:0/4142424206 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f272804b340 con 0x7f272814d4c0 2023-01-24T03:30:23.987 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:23.985+0000 7f271effd700 1 -- 172.21.15.112:0/4142424206 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v22) v1 ==== 228+0+0 (secure 0 0 0) 0x7f2720015100 con 0x7f272814d4c0 2023-01-24T03:30:23.990 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:23.989+0000 7f272d920700 1 -- 172.21.15.112:0/4142424206 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f270c0611c0 msgr2=0x7f270c063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:23.991 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:23.989+0000 7f272d920700 1 --2- 172.21.15.112:0/4142424206 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f270c0611c0 0x7f270c063670 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f271800e420 tx=0x7f27180075e0 comp rx=0 tx=0).stop 2023-01-24T03:30:23.991 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:23.989+0000 7f272d920700 1 -- 172.21.15.112:0/4142424206 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f272814d4c0 msgr2=0x7f27281670e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:23.991 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:23.989+0000 7f272d920700 1 --2- 172.21.15.112:0/4142424206 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f272814d4c0 0x7f27281670e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2728067880 tx=0x7f2720017000 comp rx=0 tx=0).stop 2023-01-24T03:30:23.992 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:23.989+0000 7f272d920700 1 -- 172.21.15.112:0/4142424206 shutdown_connections 2023-01-24T03:30:23.992 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:23.989+0000 7f272d920700 1 --2- 172.21.15.112:0/4142424206 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f270c0611c0 0x7f270c063670 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:23.992 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:23.989+0000 7f272d920700 1 --2- 172.21.15.112:0/4142424206 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f272814d4c0 0x7f27281670e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:23.992 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:23.989+0000 7f272d920700 1 --2- 172.21.15.112:0/4142424206 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f272805dff0 0x7f27280be9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:23.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:23.989+0000 7f272d920700 1 --2- 172.21.15.112:0/4142424206 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f272806d980 0x7f2728163c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:23.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:23.989+0000 7f272d920700 1 -- 172.21.15.112:0/4142424206 >> 172.21.15.112:0/4142424206 conn(0x7f272806c4f0 msgr2=0x7f2728149230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:23.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:23.989+0000 7f272d920700 1 -- 172.21.15.112:0/4142424206 shutdown_connections 2023-01-24T03:30:23.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:23.989+0000 7f272d920700 1 -- 172.21.15.112:0/4142424206 wait complete. 2023-01-24T03:30:23.997 INFO:teuthology.orchestra.run.smithi112.stderr:pool 'cephfs_data_ec' created 2023-01-24T03:30:24.012 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2023-01-24T03:30:24.134 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.132+0000 7f6629b02700 1 Processor -- start 2023-01-24T03:30:24.134 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.132+0000 7f6629b02700 1 -- start start 2023-01-24T03:30:24.136 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.132+0000 7f6629b02700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f6624157fc0 0x7f6624089800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:24.137 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.132+0000 7f6629b02700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f6624158390 0x7f6624155ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:24.137 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.132+0000 7f6629b02700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f662405e620 con 0x7f66240695c0 2023-01-24T03:30:24.137 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.132+0000 7f6629b02700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6624069130 con 0x7f6624158390 2023-01-24T03:30:24.137 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.132+0000 7f6629b02700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6624156420 con 0x7f6624157fc0 2023-01-24T03:30:24.137 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.133+0000 7f6622ffd700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f6624158390 0x7f6624155ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:24.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.133+0000 7f6622ffd700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f6624158390 0x7f6624155ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50100/0 (socket says 172.21.15.112:50100) 2023-01-24T03:30:24.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.133+0000 7f6622ffd700 1 -- 172.21.15.112:0/4163483247 learned_addr learned my addr 172.21.15.112:0/4163483247 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:24.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.133+0000 7f66237fe700 1 --2- 172.21.15.112:0/4163483247 >> v2:172.21.15.142:3300/0 conn(0x7f6624157fc0 0x7f6624089800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:24.139 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.133+0000 7f6622ffd700 1 -- 172.21.15.112:0/4163483247 >> v2:172.21.15.142:3300/0 conn(0x7f6624157fc0 msgr2=0x7f6624089800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:24.139 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.133+0000 7f6622ffd700 1 --2- 172.21.15.112:0/4163483247 >> v2:172.21.15.142:3300/0 conn(0x7f6624157fc0 0x7f6624089800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:24.139 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.133+0000 7f6622ffd700 1 -- 172.21.15.112:0/4163483247 >> v1:172.21.15.142:6789/0 conn(0x7f66240695c0 legacy=0x7f66240b0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:24.139 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.133+0000 7f6622ffd700 1 -- 172.21.15.112:0/4163483247 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6624156980 con 0x7f6624158390 2023-01-24T03:30:24.139 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.133+0000 7f6622ffd700 1 --2- 172.21.15.112:0/4163483247 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6624158390 0x7f6624155ee0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6624067280 tx=0x7f660c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c18b2bcf29a8edda server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:24.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.133+0000 7f66227fc700 1 -- 172.21.15.112:0/4163483247 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660c011630 con 0x7f6624158390 2023-01-24T03:30:24.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.133+0000 7f66227fc700 1 -- 172.21.15.112:0/4163483247 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f660c011790 con 0x7f6624158390 2023-01-24T03:30:24.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.133+0000 7f66227fc700 1 -- 172.21.15.112:0/4163483247 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660c011b50 con 0x7f6624158390 2023-01-24T03:30:24.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.134+0000 7f6629b02700 1 -- 172.21.15.112:0/4163483247 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6624158390 msgr2=0x7f6624155ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:24.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.134+0000 7f6629b02700 1 --2- 172.21.15.112:0/4163483247 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6624158390 0x7f6624155ee0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6624067280 tx=0x7f660c0090f0 comp rx=0 tx=0).stop 2023-01-24T03:30:24.141 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.134+0000 7f6629b02700 1 -- 172.21.15.112:0/4163483247 shutdown_connections 2023-01-24T03:30:24.141 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.134+0000 7f6629b02700 1 --2- 172.21.15.112:0/4163483247 >> v2:172.21.15.142:3300/0 conn(0x7f6624157fc0 0x7f6624089800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:24.141 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.134+0000 7f6629b02700 1 --2- 172.21.15.112:0/4163483247 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6624158390 0x7f6624155ee0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:24.141 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.134+0000 7f6629b02700 1 -- 172.21.15.112:0/4163483247 >> 172.21.15.112:0/4163483247 conn(0x7f662405d830 msgr2=0x7f662405dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:24.141 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.134+0000 7f6629b02700 1 -- 172.21.15.112:0/4163483247 shutdown_connections 2023-01-24T03:30:24.142 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.134+0000 7f6629b02700 1 -- 172.21.15.112:0/4163483247 wait complete. 2023-01-24T03:30:24.142 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.135+0000 7f6629b02700 1 Processor -- start 2023-01-24T03:30:24.142 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.135+0000 7f6629b02700 1 -- start start 2023-01-24T03:30:24.145 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.136+0000 7f6629b02700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66240695c0 0x7f66240c3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:24.145 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.136+0000 7f6629b02700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6624157fc0 0x7f66240c43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:24.145 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.136+0000 7f6629b02700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6624158390 0x7f66240c01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:24.145 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.136+0000 7f6629b02700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66240c2350 con 0x7f66240695c0 2023-01-24T03:30:24.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.136+0000 7f6629b02700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66240c1460 con 0x7f6624158390 2023-01-24T03:30:24.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.136+0000 7f6629b02700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66240c1740 con 0x7f6624157fc0 2023-01-24T03:30:24.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.136+0000 7f66237fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66240695c0 0x7f66240c3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:24.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.136+0000 7f66237fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66240695c0 0x7f66240c3e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50104/0 (socket says 172.21.15.112:50104) 2023-01-24T03:30:24.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.136+0000 7f66237fe700 1 -- 172.21.15.112:0/602662993 learned_addr learned my addr 172.21.15.112:0/602662993 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:24.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.136+0000 7f6622ffd700 1 --2- 172.21.15.112:0/602662993 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6624157fc0 0x7f66240c43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:24.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.136+0000 7f6623fff700 1 --2- 172.21.15.112:0/602662993 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6624158390 0x7f66240c01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:24.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.136+0000 7f66237fe700 1 -- 172.21.15.112:0/602662993 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6624157fc0 msgr2=0x7f66240c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:24.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.136+0000 7f66237fe700 1 --2- 172.21.15.112:0/602662993 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6624157fc0 0x7f66240c43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:24.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.136+0000 7f66237fe700 1 -- 172.21.15.112:0/602662993 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6624158390 msgr2=0x7f66240c01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:24.148 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.136+0000 7f66237fe700 1 --2- 172.21.15.112:0/602662993 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6624158390 0x7f66240c01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:24.148 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.136+0000 7f66237fe700 1 -- 172.21.15.112:0/602662993 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f660c009370 con 0x7f66240695c0 2023-01-24T03:30:24.148 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.136+0000 7f66237fe700 1 --2- 172.21.15.112:0/602662993 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66240695c0 0x7f66240c3e90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6614009ef0 tx=0x7f661400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:24.148 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.136+0000 7f661b7fe700 1 -- 172.21.15.112:0/602662993 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f661400cce0 con 0x7f66240695c0 2023-01-24T03:30:24.149 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.136+0000 7f661b7fe700 1 -- 172.21.15.112:0/602662993 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f661400f040 con 0x7f66240695c0 2023-01-24T03:30:24.149 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.136+0000 7f6629b02700 1 -- 172.21.15.112:0/602662993 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66240c1a20 con 0x7f66240695c0 2023-01-24T03:30:24.149 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.136+0000 7f661b7fe700 1 -- 172.21.15.112:0/602662993 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66140146b0 con 0x7f66240695c0 2023-01-24T03:30:24.149 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.136+0000 7f6629b02700 1 -- 172.21.15.112:0/602662993 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66240c1fe0 con 0x7f66240695c0 2023-01-24T03:30:24.149 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.137+0000 7f6629b02700 1 -- 172.21.15.112:0/602662993 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f662404bb80 con 0x7f66240695c0 2023-01-24T03:30:24.149 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.138+0000 7f661b7fe700 1 -- 172.21.15.112:0/602662993 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f6614014a20 con 0x7f66240695c0 2023-01-24T03:30:24.150 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.138+0000 7f661b7fe700 1 --2- 172.21.15.112:0/602662993 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f66080611c0 0x7f6608063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:24.150 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.138+0000 7f661b7fe700 1 -- 172.21.15.112:0/602662993 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6777+0+0 (secure 0 0 0) 0x7f661407e680 con 0x7f66240695c0 2023-01-24T03:30:24.150 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.138+0000 7f6622ffd700 1 --2- 172.21.15.112:0/602662993 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f66080611c0 0x7f6608063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:24.150 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.138+0000 7f6622ffd700 1 --2- 172.21.15.112:0/602662993 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f66080611c0 0x7f6608063670 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6624067280 tx=0x7f660c016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:24.150 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.142+0000 7f661b7fe700 1 -- 172.21.15.112:0/602662993 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f661404e4e0 con 0x7f66240695c0 2023-01-24T03:30:24.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.321+0000 7f6629b02700 1 -- 172.21.15.112:0/602662993 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f66240680e0 con 0x7f66240695c0 2023-01-24T03:30:24.990 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.988+0000 7f661b7fe700 1 -- 172.21.15.112:0/602662993 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 5 allow_ec_overwrites to true v23) v1 ==== 169+0+0 (secure 0 0 0) 0x7f6614014cb0 con 0x7f66240695c0 2023-01-24T03:30:24.998 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.993+0000 7f6629b02700 1 -- 172.21.15.112:0/602662993 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f66080611c0 msgr2=0x7f6608063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:24.999 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.993+0000 7f6629b02700 1 --2- 172.21.15.112:0/602662993 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f66080611c0 0x7f6608063670 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6624067280 tx=0x7f660c016000 comp rx=0 tx=0).stop 2023-01-24T03:30:24.999 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.993+0000 7f6629b02700 1 -- 172.21.15.112:0/602662993 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66240695c0 msgr2=0x7f66240c3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:24.999 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.993+0000 7f6629b02700 1 --2- 172.21.15.112:0/602662993 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66240695c0 0x7f66240c3e90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6614009ef0 tx=0x7f661400b040 comp rx=0 tx=0).stop 2023-01-24T03:30:24.999 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.994+0000 7f6629b02700 1 -- 172.21.15.112:0/602662993 shutdown_connections 2023-01-24T03:30:25.000 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.994+0000 7f6629b02700 1 --2- 172.21.15.112:0/602662993 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f66080611c0 0x7f6608063670 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:25.000 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.994+0000 7f6629b02700 1 --2- 172.21.15.112:0/602662993 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66240695c0 0x7f66240c3e90 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:25.000 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.994+0000 7f6629b02700 1 --2- 172.21.15.112:0/602662993 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6624158390 0x7f66240c01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:25.000 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.994+0000 7f6629b02700 1 --2- 172.21.15.112:0/602662993 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6624157fc0 0x7f66240c43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:25.000 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.994+0000 7f6629b02700 1 -- 172.21.15.112:0/602662993 >> 172.21.15.112:0/602662993 conn(0x7f662405d830 msgr2=0x7f662414f090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:25.001 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.996+0000 7f6629b02700 1 -- 172.21.15.112:0/602662993 shutdown_connections 2023-01-24T03:30:25.001 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:24.997+0000 7f6629b02700 1 -- 172.21.15.112:0/602662993 wait complete. 2023-01-24T03:30:25.003 INFO:teuthology.orchestra.run.smithi112.stderr:set pool 5 allow_ec_overwrites to true 2023-01-24T03:30:25.025 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2023-01-24T03:30:25.148 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.138+0000 7f982b561700 1 Processor -- start 2023-01-24T03:30:25.149 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.138+0000 7f982b561700 1 -- start start 2023-01-24T03:30:25.149 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.138+0000 7f982b561700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f982405e420 0x7f9824069890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:25.149 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.139+0000 7f982b561700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9824155ab0 con 0x7f9824068d20 2023-01-24T03:30:25.150 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.139+0000 7f982b561700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98240c70e0 con 0x7f98240694c0 2023-01-24T03:30:25.150 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.139+0000 7f982b561700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9824155550 con 0x7f982405e420 2023-01-24T03:30:25.151 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.139+0000 7f9829afe700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f9824068d20 0x7f98240ae9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:59342/0 (socket says 172.21.15.112:59342) 2023-01-24T03:30:25.151 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.139+0000 7f9829afe700 1 -- 172.21.15.112:0/71786016 learned_addr learned my addr 172.21.15.112:0/71786016 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:25.151 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.139+0000 7f9828afc700 1 --2- 172.21.15.112:0/71786016 >> v2:172.21.15.142:3300/0 conn(0x7f982405e420 0x7f9824069890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:25.151 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.139+0000 7f9813fff700 1 -- 172.21.15.112:0/71786016 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2637816583 0 0) 0x7f9824155ab0 con 0x7f9824068d20 2023-01-24T03:30:25.152 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.139+0000 7f9813fff700 1 -- 172.21.15.112:0/71786016 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f980c0033f0 con 0x7f9824068d20 2023-01-24T03:30:25.152 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.139+0000 7f9813fff700 1 -- 172.21.15.112:0/71786016 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 259976650 0 0) 0x7f980c0033f0 con 0x7f9824068d20 2023-01-24T03:30:25.152 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.139+0000 7f9813fff700 1 -- 172.21.15.112:0/71786016 >> v2:172.21.15.142:3300/0 conn(0x7f982405e420 msgr2=0x7f9824069890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:25.153 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.139+0000 7f9813fff700 1 --2- 172.21.15.112:0/71786016 >> v2:172.21.15.142:3300/0 conn(0x7f982405e420 0x7f9824069890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:25.153 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.139+0000 7f9813fff700 1 -- 172.21.15.112:0/71786016 >> v1:172.21.15.142:6789/0 conn(0x7f98240694c0 legacy=0x7f98240c68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:25.153 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.139+0000 7f9813fff700 1 -- 172.21.15.112:0/71786016 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98240c73c0 con 0x7f9824068d20 2023-01-24T03:30:25.154 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.139+0000 7f9813fff700 1 -- 172.21.15.112:0/71786016 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7f9820001470 con 0x7f9824068d20 2023-01-24T03:30:25.154 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.139+0000 7f9813fff700 1 -- 172.21.15.112:0/71786016 >> v1:172.21.15.112:6789/0 conn(0x7f9824068d20 legacy=0x7f98240ae9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:25.154 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.139+0000 7f9813fff700 1 --2- 172.21.15.112:0/71786016 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f980c0038e0 0x7f980c003cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:25.154 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.139+0000 7f9813fff700 1 --2- 172.21.15.112:0/71786016 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f980c004690 0x7f980c004b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:25.155 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.139+0000 7f9813fff700 1 --2- 172.21.15.112:0/71786016 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f980c005040 0x7f980c00b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:25.155 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.139+0000 7f9813fff700 1 -- 172.21.15.112:0/71786016 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980c00bf20 con 0x7f980c005040 2023-01-24T03:30:25.155 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.139+0000 7f9813fff700 1 -- 172.21.15.112:0/71786016 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980c00c090 con 0x7f980c004690 2023-01-24T03:30:25.156 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.139+0000 7f9813fff700 1 -- 172.21.15.112:0/71786016 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f980c00c250 con 0x7f980c0038e0 2023-01-24T03:30:25.156 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.139+0000 7f9829afe700 1 --2- 172.21.15.112:0/71786016 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f980c005040 0x7f980c00b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:25.157 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.139+0000 7f9828afc700 1 --2- 172.21.15.112:0/71786016 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f980c004690 0x7f980c004b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:25.157 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.139+0000 7f98292fd700 1 --2- 172.21.15.112:0/71786016 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f980c0038e0 0x7f980c003cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:25.157 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.140+0000 7f9829afe700 1 -- 172.21.15.112:0/71786016 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f980c0038e0 msgr2=0x7f980c003cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:25.158 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.140+0000 7f9829afe700 1 --2- 172.21.15.112:0/71786016 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f980c0038e0 0x7f980c003cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:25.158 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.140+0000 7f9829afe700 1 -- 172.21.15.112:0/71786016 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f980c004690 msgr2=0x7f980c004b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:25.158 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.140+0000 7f9829afe700 1 --2- 172.21.15.112:0/71786016 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f980c004690 0x7f980c004b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:25.159 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.140+0000 7f9829afe700 1 -- 172.21.15.112:0/71786016 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980c0033f0 con 0x7f980c005040 2023-01-24T03:30:25.159 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.140+0000 7f9829afe700 1 -- 172.21.15.112:0/71786016 reap_dead start 2023-01-24T03:30:25.159 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.140+0000 7f9829afe700 1 --2- 172.21.15.112:0/71786016 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f980c005040 0x7f980c00b8e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f982000c190 tx=0x7f9820001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48c98c17acf3d68d server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:25.159 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.140+0000 7f9813fff700 1 -- 172.21.15.112:0/71786016 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982000cac0 con 0x7f980c005040 2023-01-24T03:30:25.160 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.140+0000 7f9813fff700 1 -- 172.21.15.112:0/71786016 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f982000cc20 con 0x7f980c005040 2023-01-24T03:30:25.160 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.140+0000 7f9813fff700 1 -- 172.21.15.112:0/71786016 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98200070d0 con 0x7f980c005040 2023-01-24T03:30:25.160 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.140+0000 7f982b561700 1 -- 172.21.15.112:0/71786016 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f980c005040 msgr2=0x7f980c00b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:25.161 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.140+0000 7f982b561700 1 --2- 172.21.15.112:0/71786016 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f980c005040 0x7f980c00b8e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f982000c190 tx=0x7f9820001e50 comp rx=0 tx=0).stop 2023-01-24T03:30:25.161 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.140+0000 7f982b561700 1 -- 172.21.15.112:0/71786016 shutdown_connections 2023-01-24T03:30:25.161 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.140+0000 7f982b561700 1 --2- 172.21.15.112:0/71786016 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f980c005040 0x7f980c00b8e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:25.162 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.140+0000 7f982b561700 1 -- 172.21.15.112:0/71786016 >> 172.21.15.112:0/71786016 conn(0x7f982405d830 msgr2=0x7f982405dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:25.162 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.140+0000 7f982b561700 1 -- 172.21.15.112:0/71786016 shutdown_connections 2023-01-24T03:30:25.162 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.140+0000 7f982b561700 1 -- 172.21.15.112:0/71786016 wait complete. 2023-01-24T03:30:25.163 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.141+0000 7f982b561700 1 Processor -- start 2023-01-24T03:30:25.163 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.141+0000 7f982b561700 1 -- start start 2023-01-24T03:30:25.163 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.142+0000 7f982b561700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f980c005040 0x7f98240c14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:25.164 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.142+0000 7f982b561700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f98240c2220 0x7f98240c2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:25.164 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.142+0000 7f982b561700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f98240c3400 0x7f982414eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:25.164 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.142+0000 7f982b561700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9824169dd0 con 0x7f980c005040 2023-01-24T03:30:25.165 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.142+0000 7f982b561700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98240c3870 con 0x7f98240c2220 2023-01-24T03:30:25.165 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.142+0000 7f982b561700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9824168fb0 con 0x7f98240c3400 2023-01-24T03:30:25.165 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.142+0000 7f98292fd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f980c005040 0x7f98240c14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:25.165 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.142+0000 7f98292fd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f980c005040 0x7f98240c14d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50130/0 (socket says 172.21.15.112:50130) 2023-01-24T03:30:25.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.142+0000 7f98292fd700 1 -- 172.21.15.112:0/2083927707 learned_addr learned my addr 172.21.15.112:0/2083927707 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:25.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.143+0000 7f98292fd700 1 -- 172.21.15.112:0/2083927707 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f98240c3400 msgr2=0x7f982414eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:25.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.143+0000 7f98292fd700 1 --2- 172.21.15.112:0/2083927707 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f98240c3400 0x7f982414eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:25.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.143+0000 7f98292fd700 1 -- 172.21.15.112:0/2083927707 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f98240c2220 msgr2=0x7f98240c2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:25.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.143+0000 7f98292fd700 1 --2- 172.21.15.112:0/2083927707 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f98240c2220 0x7f98240c2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:25.167 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.143+0000 7f98292fd700 1 -- 172.21.15.112:0/2083927707 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9824169250 con 0x7f980c005040 2023-01-24T03:30:25.167 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.143+0000 7f98292fd700 1 --2- 172.21.15.112:0/2083927707 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f980c005040 0x7f98240c14d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f981c007aa0 tx=0x7f981c007f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:25.167 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.143+0000 7f98117fa700 1 -- 172.21.15.112:0/2083927707 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f981c001a10 con 0x7f980c005040 2023-01-24T03:30:25.167 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.143+0000 7f98117fa700 1 -- 172.21.15.112:0/2083927707 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f981c001b70 con 0x7f980c005040 2023-01-24T03:30:25.167 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.143+0000 7f982b561700 1 -- 172.21.15.112:0/2083927707 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9824169530 con 0x7f980c005040 2023-01-24T03:30:25.168 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.143+0000 7f98117fa700 1 -- 172.21.15.112:0/2083927707 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f981c002070 con 0x7f980c005040 2023-01-24T03:30:25.168 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.143+0000 7f982b561700 1 -- 172.21.15.112:0/2083927707 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9824169af0 con 0x7f980c005040 2023-01-24T03:30:25.168 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.144+0000 7f982b561700 1 -- 172.21.15.112:0/2083927707 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98240c2d10 con 0x7f980c005040 2023-01-24T03:30:25.168 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.144+0000 7f98117fa700 1 -- 172.21.15.112:0/2083927707 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f981c0023e0 con 0x7f980c005040 2023-01-24T03:30:25.168 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.145+0000 7f98117fa700 1 --2- 172.21.15.112:0/2083927707 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f98040611c0 0x7f9804063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:25.169 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.145+0000 7f9828afc700 1 --2- 172.21.15.112:0/2083927707 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f98040611c0 0x7f9804063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:25.169 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.145+0000 7f98117fa700 1 -- 172.21.15.112:0/2083927707 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6777+0+0 (secure 0 0 0) 0x7f981c07d910 con 0x7f980c005040 2023-01-24T03:30:25.169 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.145+0000 7f9828afc700 1 --2- 172.21.15.112:0/2083927707 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f98040611c0 0x7f9804063670 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9814006f70 tx=0x7f9814009060 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:25.170 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.149+0000 7f98117fa700 1 -- 172.21.15.112:0/2083927707 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f981c04d7f0 con 0x7f980c005040 2023-01-24T03:30:25.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.372+0000 7f982b561700 1 -- 172.21.15.112:0/2083927707 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f982406f3a0 con 0x7f980c005040 2023-01-24T03:30:25.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.376+0000 7f98117fa700 1 -- 172.21.15.112:0/2083927707 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 5 to fsmap v5) v1 ==== 137+0+0 (secure 0 0 0) 0x7f982406f3a0 con 0x7f980c005040 2023-01-24T03:30:25.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.378+0000 7f982b561700 1 -- 172.21.15.112:0/2083927707 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f98040611c0 msgr2=0x7f9804063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:25.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.378+0000 7f982b561700 1 --2- 172.21.15.112:0/2083927707 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f98040611c0 0x7f9804063670 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9814006f70 tx=0x7f9814009060 comp rx=0 tx=0).stop 2023-01-24T03:30:25.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.378+0000 7f982b561700 1 -- 172.21.15.112:0/2083927707 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f980c005040 msgr2=0x7f98240c14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:25.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.378+0000 7f982b561700 1 --2- 172.21.15.112:0/2083927707 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f980c005040 0x7f98240c14d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f981c007aa0 tx=0x7f981c007f30 comp rx=0 tx=0).stop 2023-01-24T03:30:25.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.378+0000 7f982b561700 1 -- 172.21.15.112:0/2083927707 shutdown_connections 2023-01-24T03:30:25.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.378+0000 7f982b561700 1 --2- 172.21.15.112:0/2083927707 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f98040611c0 0x7f9804063670 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:25.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.378+0000 7f982b561700 1 --2- 172.21.15.112:0/2083927707 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f980c005040 0x7f98240c14d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:25.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.378+0000 7f982b561700 1 --2- 172.21.15.112:0/2083927707 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f98240c3400 0x7f982414eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:25.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.378+0000 7f982b561700 1 --2- 172.21.15.112:0/2083927707 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f98240c2220 0x7f98240c2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:25.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.378+0000 7f982b561700 1 -- 172.21.15.112:0/2083927707 >> 172.21.15.112:0/2083927707 conn(0x7f982405d830 msgr2=0x7f98240ae9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:25.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.378+0000 7f982b561700 1 -- 172.21.15.112:0/2083927707 shutdown_connections 2023-01-24T03:30:25.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.378+0000 7f982b561700 1 -- 172.21.15.112:0/2083927707 wait complete. 2023-01-24T03:30:25.385 INFO:teuthology.orchestra.run.smithi112.stderr:added data pool 5 to fsmap 2023-01-24T03:30:25.391 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:30:25.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.504+0000 7f8ff0b64700 1 Processor -- start 2023-01-24T03:30:25.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.504+0000 7f8ff0b64700 1 -- start start 2023-01-24T03:30:25.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.505+0000 7f8ff0b64700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f8fec0c5680 0x7f8fec156730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:25.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.505+0000 7f8ff0b64700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8fec155e20 0x7f8fec156c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:25.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.505+0000 7f8ff0b64700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f8fec0c5a50 0x7f8fec0cb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:25.510 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.505+0000 7f8ff0b64700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fec0c4eb0 con 0x7f8fec0c5a50 2023-01-24T03:30:25.510 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.505+0000 7f8ff0b64700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fec0c3d10 con 0x7f8fec0c5680 2023-01-24T03:30:25.510 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.505+0000 7f8ff0b64700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fec0c2e20 con 0x7f8fec155e20 2023-01-24T03:30:25.510 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.505+0000 7f8febfff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f8fec0c5a50 0x7f8fec0cb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:25.510 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.505+0000 7f8febfff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f8fec0c5a50 0x7f8fec0cb870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50140/0 (socket says 172.21.15.112:50140) 2023-01-24T03:30:25.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.505+0000 7f8febfff700 1 -- 172.21.15.112:0/4139713491 learned_addr learned my addr 172.21.15.112:0/4139713491 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:25.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.505+0000 7f8feaffd700 1 --2- 172.21.15.112:0/4139713491 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8fec155e20 0x7f8fec156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:25.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.505+0000 7f8feb7fe700 1 --2- 172.21.15.112:0/4139713491 >> v2:172.21.15.142:3300/0 conn(0x7f8fec0c5680 0x7f8fec156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:25.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.505+0000 7f8febfff700 1 -- 172.21.15.112:0/4139713491 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8fec155e20 msgr2=0x7f8fec156c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:25.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.505+0000 7f8febfff700 1 --2- 172.21.15.112:0/4139713491 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8fec155e20 0x7f8fec156c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:25.512 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.505+0000 7f8febfff700 1 -- 172.21.15.112:0/4139713491 >> v2:172.21.15.142:3300/0 conn(0x7f8fec0c5680 msgr2=0x7f8fec156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:25.512 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.505+0000 7f8febfff700 1 --2- 172.21.15.112:0/4139713491 >> v2:172.21.15.142:3300/0 conn(0x7f8fec0c5680 0x7f8fec156730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:25.512 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.505+0000 7f8febfff700 1 -- 172.21.15.112:0/4139713491 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fec0c3100 con 0x7f8fec0c5a50 2023-01-24T03:30:25.512 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.505+0000 7f8febfff700 1 --2- 172.21.15.112:0/4139713491 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8fec0c5a50 0x7f8fec0cb870 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4009f20 tx=0x7f8fe4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=671391ea4f549d9d server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:25.512 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.505+0000 7f8fea7fc700 1 -- 172.21.15.112:0/4139713491 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe400e070 con 0x7f8fec0c5a50 2023-01-24T03:30:25.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.505+0000 7f8fea7fc700 1 -- 172.21.15.112:0/4139713491 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe400ba40 con 0x7f8fec0c5a50 2023-01-24T03:30:25.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.505+0000 7f8fea7fc700 1 -- 172.21.15.112:0/4139713491 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe40092e0 con 0x7f8fec0c5a50 2023-01-24T03:30:25.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.506+0000 7f8ff0b64700 1 -- 172.21.15.112:0/4139713491 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8fec0c5a50 msgr2=0x7f8fec0cb870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:25.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.506+0000 7f8ff0b64700 1 --2- 172.21.15.112:0/4139713491 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8fec0c5a50 0x7f8fec0cb870 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4009f20 tx=0x7f8fe4009840 comp rx=0 tx=0).stop 2023-01-24T03:30:25.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.506+0000 7f8ff0b64700 1 -- 172.21.15.112:0/4139713491 shutdown_connections 2023-01-24T03:30:25.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.506+0000 7f8ff0b64700 1 --2- 172.21.15.112:0/4139713491 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8fec0c5a50 0x7f8fec0cb870 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:25.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.506+0000 7f8ff0b64700 1 --2- 172.21.15.112:0/4139713491 >> v2:172.21.15.142:3300/0 conn(0x7f8fec0c5680 0x7f8fec156730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:25.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.506+0000 7f8ff0b64700 1 --2- 172.21.15.112:0/4139713491 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8fec155e20 0x7f8fec156c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:25.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.506+0000 7f8ff0b64700 1 -- 172.21.15.112:0/4139713491 >> 172.21.15.112:0/4139713491 conn(0x7f8fec0685b0 msgr2=0x7f8fec0689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:25.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.506+0000 7f8ff0b64700 1 -- 172.21.15.112:0/4139713491 shutdown_connections 2023-01-24T03:30:25.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.506+0000 7f8ff0b64700 1 -- 172.21.15.112:0/4139713491 wait complete. 2023-01-24T03:30:25.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.507+0000 7f8ff0b64700 1 Processor -- start 2023-01-24T03:30:25.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.507+0000 7f8ff0b64700 1 -- start start 2023-01-24T03:30:25.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.508+0000 7f8ff0b64700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8fec0c5680 0x7f8fec14aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:25.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.508+0000 7f8ff0b64700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8fec0c5a50 0x7f8fec154d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:25.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.508+0000 7f8ff0b64700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8fec155e20 0x7f8fec164b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:25.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.508+0000 7f8ff0b64700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fec0c2390 con 0x7f8fec155e20 2023-01-24T03:30:25.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.508+0000 7f8ff0b64700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fec0cad80 con 0x7f8fec0c5a50 2023-01-24T03:30:25.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.508+0000 7f8ff0b64700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fec0cb000 con 0x7f8fec0c5680 2023-01-24T03:30:25.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.508+0000 7f8febfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8fec155e20 0x7f8fec164b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:25.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.508+0000 7f8feb7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8fec0c5680 0x7f8fec14aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:25.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.508+0000 7f8febfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8fec155e20 0x7f8fec164b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50148/0 (socket says 172.21.15.112:50148) 2023-01-24T03:30:25.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.508+0000 7f8febfff700 1 -- 172.21.15.112:0/2303166831 learned_addr learned my addr 172.21.15.112:0/2303166831 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:25.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.508+0000 7f8feaffd700 1 --2- 172.21.15.112:0/2303166831 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8fec0c5a50 0x7f8fec154d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:25.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.508+0000 7f8febfff700 1 -- 172.21.15.112:0/2303166831 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8fec0c5680 msgr2=0x7f8fec14aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:25.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.508+0000 7f8febfff700 1 --2- 172.21.15.112:0/2303166831 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8fec0c5680 0x7f8fec14aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:25.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.508+0000 7f8febfff700 1 -- 172.21.15.112:0/2303166831 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8fec0c5a50 msgr2=0x7f8fec154d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:25.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.508+0000 7f8febfff700 1 --2- 172.21.15.112:0/2303166831 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8fec0c5a50 0x7f8fec154d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:25.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.508+0000 7f8febfff700 1 -- 172.21.15.112:0/2303166831 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe4009ae0 con 0x7f8fec155e20 2023-01-24T03:30:25.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.508+0000 7f8febfff700 1 --2- 172.21.15.112:0/2303166831 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8fec155e20 0x7f8fec164b50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8fe400b6c0 tx=0x7f8fe4018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:25.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.508+0000 7f8fd37fe700 1 -- 172.21.15.112:0/2303166831 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe4009940 con 0x7f8fec155e20 2023-01-24T03:30:25.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.509+0000 7f8fd37fe700 1 -- 172.21.15.112:0/2303166831 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe4008d10 con 0x7f8fec155e20 2023-01-24T03:30:25.525 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.509+0000 7f8ff0b64700 1 -- 172.21.15.112:0/2303166831 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fec0cb2e0 con 0x7f8fec155e20 2023-01-24T03:30:25.525 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.509+0000 7f8fd37fe700 1 -- 172.21.15.112:0/2303166831 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe401d670 con 0x7f8fec155e20 2023-01-24T03:30:25.525 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.509+0000 7f8ff0b64700 1 -- 172.21.15.112:0/2303166831 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fec0cb8a0 con 0x7f8fec155e20 2023-01-24T03:30:25.525 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.509+0000 7f8ff0b64700 1 -- 172.21.15.112:0/2303166831 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fec05ea20 con 0x7f8fec155e20 2023-01-24T03:30:25.525 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.510+0000 7f8fd37fe700 1 -- 172.21.15.112:0/2303166831 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f8fe402e070 con 0x7f8fec155e20 2023-01-24T03:30:25.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.510+0000 7f8fd37fe700 1 --2- 172.21.15.112:0/2303166831 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f8fcc0611f0 0x7f8fcc0636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:25.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.510+0000 7f8fd37fe700 1 -- 172.21.15.112:0/2303166831 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f8fe407f470 con 0x7f8fec155e20 2023-01-24T03:30:25.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.510+0000 7f8feb7fe700 1 --2- 172.21.15.112:0/2303166831 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f8fcc0611f0 0x7f8fcc0636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:25.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.510+0000 7f8feb7fe700 1 --2- 172.21.15.112:0/2303166831 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f8fcc0611f0 0x7f8fcc0636a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc00bb10 tx=0x7f8fdc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:25.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.514+0000 7f8fd37fe700 1 -- 172.21.15.112:0/2303166831 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f8fe404f2b0 con 0x7f8fec155e20 2023-01-24T03:30:25.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.732+0000 7f8ff0b64700 1 -- 172.21.15.112:0/2303166831 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8fec04b340 con 0x7f8fec155e20 2023-01-24T03:30:25.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.733+0000 7f8fd37fe700 1 -- 172.21.15.112:0/2303166831 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+6113 (secure 0 0 0) 0x7f8fe40528d0 con 0x7f8fec155e20 2023-01-24T03:30:25.735 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:30:25.735 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":5,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/869267413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":869267413},{"type":"v1","addr":"172.21.15.142:6834","nonce":869267413}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":5,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:30:25.373571+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:creating","state_seq":1,"addr":"172.21.15.142:6835/2762621058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2762621058},{"type":"v1","addr":"172.21.15.142:6835","nonce":2762621058}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:30:25.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.735+0000 7f8ff0b64700 1 -- 172.21.15.112:0/2303166831 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f8fcc0611f0 msgr2=0x7f8fcc0636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:25.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.735+0000 7f8ff0b64700 1 --2- 172.21.15.112:0/2303166831 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f8fcc0611f0 0x7f8fcc0636a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc00bb10 tx=0x7f8fdc008040 comp rx=0 tx=0).stop 2023-01-24T03:30:25.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.735+0000 7f8ff0b64700 1 -- 172.21.15.112:0/2303166831 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8fec155e20 msgr2=0x7f8fec164b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:25.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.735+0000 7f8ff0b64700 1 --2- 172.21.15.112:0/2303166831 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8fec155e20 0x7f8fec164b50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8fe400b6c0 tx=0x7f8fe4018000 comp rx=0 tx=0).stop 2023-01-24T03:30:25.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.735+0000 7f8ff0b64700 1 -- 172.21.15.112:0/2303166831 shutdown_connections 2023-01-24T03:30:25.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.735+0000 7f8ff0b64700 1 --2- 172.21.15.112:0/2303166831 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f8fcc0611f0 0x7f8fcc0636a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:25.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.735+0000 7f8ff0b64700 1 --2- 172.21.15.112:0/2303166831 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8fec155e20 0x7f8fec164b50 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:25.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.735+0000 7f8ff0b64700 1 --2- 172.21.15.112:0/2303166831 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8fec0c5680 0x7f8fec14aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:25.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.735+0000 7f8ff0b64700 1 --2- 172.21.15.112:0/2303166831 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8fec0c5a50 0x7f8fec154d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:25.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.735+0000 7f8ff0b64700 1 -- 172.21.15.112:0/2303166831 >> 172.21.15.112:0/2303166831 conn(0x7f8fec0685b0 msgr2=0x7f8fec14bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:25.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.735+0000 7f8ff0b64700 1 -- 172.21.15.112:0/2303166831 shutdown_connections 2023-01-24T03:30:25.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.735+0000 7f8ff0b64700 1 -- 172.21.15.112:0/2303166831 wait complete. 2023-01-24T03:30:25.740 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 5 2023-01-24T03:30:25.749 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T03:30:25.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.858+0000 7fa918cac700 1 Processor -- start 2023-01-24T03:30:25.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.858+0000 7fa918cac700 1 -- start start 2023-01-24T03:30:25.863 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.859+0000 7fa918cac700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa9140c7e40 0x7fa9140b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:25.863 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.859+0000 7fa918cac700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa914156210 con 0x7fa9140ca3e0 2023-01-24T03:30:25.863 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.859+0000 7fa918cac700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9140b1170 con 0x7fa914155da0 2023-01-24T03:30:25.863 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.859+0000 7fa918cac700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9140b1450 con 0x7fa9140c7e40 2023-01-24T03:30:25.864 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.859+0000 7fa912d9d700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fa9140ca3e0 0x7fa9140ca820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:59376/0 (socket says 172.21.15.112:59376) 2023-01-24T03:30:25.864 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.859+0000 7fa912d9d700 1 -- 172.21.15.112:0/1141084240 learned_addr learned my addr 172.21.15.112:0/1141084240 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:25.864 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.859+0000 7fa91259c700 1 --2- 172.21.15.112:0/1141084240 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa9140c7e40 0x7fa9140b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:25.864 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.859+0000 7fa91159a700 1 -- 172.21.15.112:0/1141084240 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3224078305 0 0) 0x7fa914156210 con 0x7fa9140ca3e0 2023-01-24T03:30:25.865 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.859+0000 7fa91159a700 1 -- 172.21.15.112:0/1141084240 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8f40033f0 con 0x7fa9140ca3e0 2023-01-24T03:30:25.865 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.860+0000 7fa91159a700 1 -- 172.21.15.112:0/1141084240 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1243959099 0 0) 0x7fa9140b1170 con 0x7fa914155da0 2023-01-24T03:30:25.865 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.860+0000 7fa91159a700 1 -- 172.21.15.112:0/1141084240 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa914156210 con 0x7fa914155da0 2023-01-24T03:30:25.865 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.860+0000 7fa91159a700 1 -- 172.21.15.112:0/1141084240 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1014510291 0 0) 0x7fa8f40033f0 con 0x7fa9140ca3e0 2023-01-24T03:30:25.865 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.860+0000 7fa91259c700 1 -- 172.21.15.112:0/1141084240 >> v1:172.21.15.142:6789/0 conn(0x7fa914155da0 legacy=0x7fa9140ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:25.865 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.860+0000 7fa91259c700 1 -- 172.21.15.112:0/1141084240 >> v1:172.21.15.112:6789/0 conn(0x7fa9140ca3e0 legacy=0x7fa9140ca820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:25.866 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.860+0000 7fa91259c700 1 -- 172.21.15.112:0/1141084240 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9140b1730 con 0x7fa9140c7e40 2023-01-24T03:30:25.866 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.860+0000 7fa91259c700 1 --2- 172.21.15.112:0/1141084240 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa9140c7e40 0x7fa9140b0aa0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa904009e90 tx=0x7fa904009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a41186fb65262b09 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:25.866 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.860+0000 7fa91159a700 1 -- 172.21.15.112:0/1141084240 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90400e070 con 0x7fa9140c7e40 2023-01-24T03:30:25.866 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.860+0000 7fa91159a700 1 -- 172.21.15.112:0/1141084240 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa90400b870 con 0x7fa9140c7e40 2023-01-24T03:30:25.866 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.860+0000 7fa91159a700 1 -- 172.21.15.112:0/1141084240 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90c001470 con 0x7fa9140c7e40 2023-01-24T03:30:25.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.860+0000 7fa918cac700 1 -- 172.21.15.112:0/1141084240 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa9140c7e40 msgr2=0x7fa9140b0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:25.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.860+0000 7fa918cac700 1 --2- 172.21.15.112:0/1141084240 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa9140c7e40 0x7fa9140b0aa0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa904009e90 tx=0x7fa904009780 comp rx=0 tx=0).stop 2023-01-24T03:30:25.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.861+0000 7fa918cac700 1 -- 172.21.15.112:0/1141084240 shutdown_connections 2023-01-24T03:30:25.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.861+0000 7fa918cac700 1 --2- 172.21.15.112:0/1141084240 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa9140c7e40 0x7fa9140b0aa0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:25.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.861+0000 7fa918cac700 1 -- 172.21.15.112:0/1141084240 >> 172.21.15.112:0/1141084240 conn(0x7fa91405e2c0 msgr2=0x7fa9140cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:25.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.861+0000 7fa918cac700 1 -- 172.21.15.112:0/1141084240 shutdown_connections 2023-01-24T03:30:25.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.861+0000 7fa918cac700 1 -- 172.21.15.112:0/1141084240 wait complete. 2023-01-24T03:30:25.871 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.862+0000 7fa918cac700 1 Processor -- start 2023-01-24T03:30:25.871 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.862+0000 7fa918cac700 1 -- start start 2023-01-24T03:30:25.871 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.862+0000 7fa918cac700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa9140c7e40 0x7fa914166190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:25.871 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.862+0000 7fa918cac700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9140ca3e0 0x7fa914150660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:25.871 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.862+0000 7fa918cac700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa914155da0 0x7fa9140c4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:25.872 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.862+0000 7fa918cac700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa914068550 con 0x7fa914155da0 2023-01-24T03:30:25.872 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.862+0000 7fa918cac700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9141659b0 con 0x7fa9140ca3e0 2023-01-24T03:30:25.872 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.862+0000 7fa918cac700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9140c1990 con 0x7fa9140c7e40 2023-01-24T03:30:25.872 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.862+0000 7fa912d9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa914155da0 0x7fa9140c4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:25.872 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.862+0000 7fa912d9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa914155da0 0x7fa9140c4380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50162/0 (socket says 172.21.15.112:50162) 2023-01-24T03:30:25.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.862+0000 7fa912d9d700 1 -- 172.21.15.112:0/2707593340 learned_addr learned my addr 172.21.15.112:0/2707593340 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:25.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.863+0000 7fa91259c700 1 --2- 172.21.15.112:0/2707593340 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa9140c7e40 0x7fa914166190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:25.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.863+0000 7fa911d9b700 1 --2- 172.21.15.112:0/2707593340 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9140ca3e0 0x7fa914150660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:25.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.863+0000 7fa912d9d700 1 -- 172.21.15.112:0/2707593340 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa9140c7e40 msgr2=0x7fa914166190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:25.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.863+0000 7fa912d9d700 1 --2- 172.21.15.112:0/2707593340 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa9140c7e40 0x7fa914166190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:25.874 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.863+0000 7fa912d9d700 1 -- 172.21.15.112:0/2707593340 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9140ca3e0 msgr2=0x7fa914150660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:25.874 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.863+0000 7fa912d9d700 1 --2- 172.21.15.112:0/2707593340 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9140ca3e0 0x7fa914150660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:25.874 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.863+0000 7fa912d9d700 1 -- 172.21.15.112:0/2707593340 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa904009a20 con 0x7fa914155da0 2023-01-24T03:30:25.874 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.863+0000 7fa912d9d700 1 --2- 172.21.15.112:0/2707593340 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa914155da0 0x7fa9140c4380 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa90c00c800 tx=0x7fa90c001a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:25.874 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.863+0000 7fa90a7fc700 1 -- 172.21.15.112:0/2707593340 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90c007360 con 0x7fa914155da0 2023-01-24T03:30:25.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.863+0000 7fa90a7fc700 1 -- 172.21.15.112:0/2707593340 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa90c0074c0 con 0x7fa914155da0 2023-01-24T03:30:25.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.863+0000 7fa918cac700 1 -- 172.21.15.112:0/2707593340 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9140c1c70 con 0x7fa914155da0 2023-01-24T03:30:25.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.863+0000 7fa90a7fc700 1 -- 172.21.15.112:0/2707593340 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90c004900 con 0x7fa914155da0 2023-01-24T03:30:25.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.863+0000 7fa918cac700 1 -- 172.21.15.112:0/2707593340 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9140c2230 con 0x7fa914155da0 2023-01-24T03:30:25.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.864+0000 7fa918cac700 1 -- 172.21.15.112:0/2707593340 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa91404bb80 con 0x7fa914155da0 2023-01-24T03:30:25.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.864+0000 7fa90a7fc700 1 -- 172.21.15.112:0/2707593340 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fa90c01a030 con 0x7fa914155da0 2023-01-24T03:30:25.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.865+0000 7fa90a7fc700 1 --2- 172.21.15.112:0/2707593340 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa8f80611c0 0x7fa8f8063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:25.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.865+0000 7fa91259c700 1 --2- 172.21.15.112:0/2707593340 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa8f80611c0 0x7fa8f8063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:25.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.865+0000 7fa90a7fc700 1 -- 172.21.15.112:0/2707593340 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fa90c07cc50 con 0x7fa914155da0 2023-01-24T03:30:25.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.865+0000 7fa91259c700 1 --2- 172.21.15.112:0/2707593340 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa8f80611c0 0x7fa8f8063670 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa9040097c0 tx=0x7fa904017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:25.877 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:25.869+0000 7fa90a7fc700 1 -- 172.21.15.112:0/2707593340 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fa90c04cb10 con 0x7fa914155da0 2023-01-24T03:30:26.055 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.053+0000 7fa918cac700 1 -- 172.21.15.112:0/2707593340 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa91406efe0 con 0x7fa914155da0 2023-01-24T03:30:26.055 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.054+0000 7fa90a7fc700 1 -- 172.21.15.112:0/2707593340 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18755 (secure 0 0 0) 0x7fa90c018020 con 0x7fa914155da0 2023-01-24T03:30:26.056 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:30:26.056 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":24,"fsid":"054b9ac9-b7b0-40e5-b932-6c8a955c5a72","created":"2023-01-24T03:30:03.798533+0000","modified":"2023-01-24T03:30:25.373340+0000","last_up_change":"2023-01-24T03:30:13.945316+0000","last_in_change":"2023-01-24T03:30:06.198433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-01-24T03:30:14.931166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-01-24T03:30:15.023730+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-01-24T03:30:19.060597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-01-24T03:30:19.943686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-01-24T03:30:22.999036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"fe6ad231-a39a-42eb-9a3a-49d008dd35e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6809","nonce":84512},{"type":"v1","addr":"172.21.15.112:6811","nonce":84512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":84512},{"type":"v1","addr":"172.21.15.112:6817","nonce":84512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6825","nonce":84512},{"type":"v1","addr":"172.21.15.112:6828","nonce":84512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6819","nonce":84512},{"type":"v1","addr":"172.21.15.112:6822","nonce":84512}]},"public_addr":"172.21.15.112:6811/84512","cluster_addr":"172.21.15.112:6817/84512","heartbeat_back_addr":"172.21.15.112:6828/84512","heartbeat_front_addr":"172.21.15.112:6822/84512","state":["exists","up"]},{"osd":1,"uuid":"8526a1d2-d540-4021-9766-c4bbb30317ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":84513},{"type":"v1","addr":"172.21.15.112:6812","nonce":84513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6815","nonce":84513},{"type":"v1","addr":"172.21.15.112:6818","nonce":84513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6827","nonce":84513},{"type":"v1","addr":"172.21.15.112:6830","nonce":84513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6821","nonce":84513},{"type":"v1","addr":"172.21.15.112:6824","nonce":84513}]},"public_addr":"172.21.15.112:6812/84513","cluster_addr":"172.21.15.112:6818/84513","heartbeat_back_addr":"172.21.15.112:6830/84513","heartbeat_front_addr":"172.21.15.112:6824/84513","state":["exists","up"]},{"osd":2,"uuid":"041ef8e4-4858-432d-9476-cbdb3a0dc47c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":84511},{"type":"v1","addr":"172.21.15.112:6813","nonce":84511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":84511},{"type":"v1","addr":"172.21.15.112:6820","nonce":84511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6829","nonce":84511},{"type":"v1","addr":"172.21.15.112:6831","nonce":84511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6823","nonce":84511},{"type":"v1","addr":"172.21.15.112:6826","nonce":84511}]},"public_addr":"172.21.15.112:6813/84511","cluster_addr":"172.21.15.112:6820/84511","heartbeat_back_addr":"172.21.15.112:6831/84511","heartbeat_front_addr":"172.21.15.112:6826/84511","state":["exists","up"]},{"osd":3,"uuid":"987587cf-1357-4a70-ad85-67b189c67729","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":84510},{"type":"v1","addr":"172.21.15.112:6801","nonce":84510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":84510},{"type":"v1","addr":"172.21.15.112:6803","nonce":84510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":84510},{"type":"v1","addr":"172.21.15.112:6807","nonce":84510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":84510},{"type":"v1","addr":"172.21.15.112:6805","nonce":84510}]},"public_addr":"172.21.15.112:6801/84510","cluster_addr":"172.21.15.112:6803/84510","heartbeat_back_addr":"172.21.15.112:6807/84510","heartbeat_front_addr":"172.21.15.112:6805/84510","state":["exists","up"]},{"osd":4,"uuid":"8503512b-43db-40b8-8b2b-c86cf2e54e17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":84614},{"type":"v1","addr":"172.21.15.142:6801","nonce":84614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":84614},{"type":"v1","addr":"172.21.15.142:6803","nonce":84614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":84614},{"type":"v1","addr":"172.21.15.142:6807","nonce":84614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":84614},{"type":"v1","addr":"172.21.15.142:6805","nonce":84614}]},"public_addr":"172.21.15.142:6801/84614","cluster_addr":"172.21.15.142:6803/84614","heartbeat_back_addr":"172.21.15.142:6807/84614","heartbeat_front_addr":"172.21.15.142:6805/84614","state":["exists","up"]},{"osd":5,"uuid":"d3bc0030-8b08-4645-9ec2-1f8a4e29ef36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":84638},{"type":"v1","addr":"172.21.15.142:6817","nonce":84638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":84638},{"type":"v1","addr":"172.21.15.142:6819","nonce":84638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":84638},{"type":"v1","addr":"172.21.15.142:6823","nonce":84638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":84638},{"type":"v1","addr":"172.21.15.142:6821","nonce":84638}]},"public_addr":"172.21.15.142:6817/84638","cluster_addr":"172.21.15.142:6819/84638","heartbeat_back_addr":"172.21.15.142:6823/84638","heartbeat_front_addr":"172.21.15.142:6821/84638","state":["exists","up"]},{"osd":6,"uuid":"c7649e3b-832d-4e6f-8866-b15e5d814476","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":84645},{"type":"v1","addr":"172.21.15.142:6825","nonce":84645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":84645},{"type":"v1","addr":"172.21.15.142:6827","nonce":84645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":84645},{"type":"v1","addr":"172.21.15.142:6831","nonce":84645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":84645},{"type":"v1","addr":"172.21.15.142:6829","nonce":84645}]},"public_addr":"172.21.15.142:6825/84645","cluster_addr":"172.21.15.142:6827/84645","heartbeat_back_addr":"172.21.15.142:6831/84645","heartbeat_front_addr":"172.21.15.142:6829/84645","state":["exists","up"]},{"osd":7,"uuid":"2ea5b3bf-884e-4b20-b572-1443efdc7477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":84615},{"type":"v1","addr":"172.21.15.142:6809","nonce":84615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":84615},{"type":"v1","addr":"172.21.15.142:6811","nonce":84615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":84615},{"type":"v1","addr":"172.21.15.142:6815","nonce":84615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":84615},{"type":"v1","addr":"172.21.15.142:6813","nonce":84615}]},"public_addr":"172.21.15.142:6809/84615","cluster_addr":"172.21.15.142:6811/84615","heartbeat_back_addr":"172.21.15.142:6815/84615","heartbeat_front_addr":"172.21.15.142:6813/84615","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.370027+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.335683+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.350667+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.350794+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.391119+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.422933+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.400439+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.354018+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T03:30:26.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.056+0000 7fa918cac700 1 -- 172.21.15.112:0/2707593340 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa8f80611c0 msgr2=0x7fa8f8063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:26.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.056+0000 7fa918cac700 1 --2- 172.21.15.112:0/2707593340 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa8f80611c0 0x7fa8f8063670 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa9040097c0 tx=0x7fa904017000 comp rx=0 tx=0).stop 2023-01-24T03:30:26.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.056+0000 7fa918cac700 1 -- 172.21.15.112:0/2707593340 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa914155da0 msgr2=0x7fa9140c4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:26.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.056+0000 7fa918cac700 1 --2- 172.21.15.112:0/2707593340 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa914155da0 0x7fa9140c4380 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa90c00c800 tx=0x7fa90c001a20 comp rx=0 tx=0).stop 2023-01-24T03:30:26.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.056+0000 7fa918cac700 1 -- 172.21.15.112:0/2707593340 shutdown_connections 2023-01-24T03:30:26.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.056+0000 7fa918cac700 1 --2- 172.21.15.112:0/2707593340 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa8f80611c0 0x7fa8f8063670 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:26.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.056+0000 7fa918cac700 1 --2- 172.21.15.112:0/2707593340 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa914155da0 0x7fa9140c4380 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:26.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.056+0000 7fa918cac700 1 --2- 172.21.15.112:0/2707593340 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa9140c7e40 0x7fa914166190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:26.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.056+0000 7fa918cac700 1 --2- 172.21.15.112:0/2707593340 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9140ca3e0 0x7fa914150660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:26.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.056+0000 7fa918cac700 1 -- 172.21.15.112:0/2707593340 >> 172.21.15.112:0/2707593340 conn(0x7fa91405e2c0 msgr2=0x7fa9140ae9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:26.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.056+0000 7fa918cac700 1 -- 172.21.15.112:0/2707593340 shutdown_connections 2023-01-24T03:30:26.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.056+0000 7fa918cac700 1 -- 172.21.15.112:0/2707593340 wait complete. 2023-01-24T03:30:26.070 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T03:30:26.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.180+0000 7fd73da24700 1 Processor -- start 2023-01-24T03:30:26.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.180+0000 7fd73da24700 1 -- start start 2023-01-24T03:30:26.187 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.180+0000 7fd73da24700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fd7380c3080 0x7fd73805df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:26.187 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.180+0000 7fd73da24700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd73814d5d0 0x7fd738153400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:26.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.180+0000 7fd73da24700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7381566c0 con 0x7fd7380c15b0 2023-01-24T03:30:26.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.180+0000 7fd73da24700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7381569a0 con 0x7fd7380c3080 2023-01-24T03:30:26.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.180+0000 7fd73da24700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd738156c80 con 0x7fd73814d5d0 2023-01-24T03:30:26.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.181+0000 7fd736ffd700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fd7380c3080 0x7fd73805df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:26.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.181+0000 7fd7367fc700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fd7380c15b0 0x7fd73814cec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:59386/0 (socket says 172.21.15.112:59386) 2023-01-24T03:30:26.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.181+0000 7fd7367fc700 1 -- 172.21.15.112:0/246727666 learned_addr learned my addr 172.21.15.112:0/246727666 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:26.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.181+0000 7fd735ffb700 1 -- 172.21.15.112:0/246727666 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3388322821 0 0) 0x7fd7381566c0 con 0x7fd7380c15b0 2023-01-24T03:30:26.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.181+0000 7fd735ffb700 1 -- 172.21.15.112:0/246727666 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7180033f0 con 0x7fd7380c15b0 2023-01-24T03:30:26.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.181+0000 7fd7377fe700 1 --2- 172.21.15.112:0/246727666 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd73814d5d0 0x7fd738153400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:26.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.181+0000 7fd736ffd700 1 -- 172.21.15.112:0/246727666 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd73814d5d0 msgr2=0x7fd738153400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:26.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.181+0000 7fd736ffd700 1 --2- 172.21.15.112:0/246727666 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd73814d5d0 0x7fd738153400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:26.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.181+0000 7fd736ffd700 1 -- 172.21.15.112:0/246727666 >> v1:172.21.15.112:6789/0 conn(0x7fd7380c15b0 legacy=0x7fd73814cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:26.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.181+0000 7fd736ffd700 1 -- 172.21.15.112:0/246727666 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd738156f60 con 0x7fd7380c3080 2023-01-24T03:30:26.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.181+0000 7fd736ffd700 1 --2- 172.21.15.112:0/246727666 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd7380c3080 0x7fd73805df40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd728009e60 tx=0x7fd728009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd82de123f7b7f06 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:26.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.181+0000 7fd735ffb700 1 -- 172.21.15.112:0/246727666 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd72800e070 con 0x7fd7380c3080 2023-01-24T03:30:26.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.181+0000 7fd735ffb700 1 -- 172.21.15.112:0/246727666 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd72800bb60 con 0x7fd7380c3080 2023-01-24T03:30:26.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.181+0000 7fd735ffb700 1 -- 172.21.15.112:0/246727666 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd728012430 con 0x7fd7380c3080 2023-01-24T03:30:26.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.182+0000 7fd73da24700 1 -- 172.21.15.112:0/246727666 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd7380c3080 msgr2=0x7fd73805df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:26.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.182+0000 7fd73da24700 1 --2- 172.21.15.112:0/246727666 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd7380c3080 0x7fd73805df40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd728009e60 tx=0x7fd728009780 comp rx=0 tx=0).stop 2023-01-24T03:30:26.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.182+0000 7fd73da24700 1 -- 172.21.15.112:0/246727666 shutdown_connections 2023-01-24T03:30:26.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.182+0000 7fd73da24700 1 --2- 172.21.15.112:0/246727666 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd73814d5d0 0x7fd738153400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:26.193 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.182+0000 7fd73da24700 1 --2- 172.21.15.112:0/246727666 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd7380c3080 0x7fd73805df40 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:26.193 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.182+0000 7fd73da24700 1 -- 172.21.15.112:0/246727666 >> 172.21.15.112:0/246727666 conn(0x7fd73806daf0 msgr2=0x7fd73805d830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:26.193 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.182+0000 7fd73da24700 1 -- 172.21.15.112:0/246727666 shutdown_connections 2023-01-24T03:30:26.193 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.182+0000 7fd73da24700 1 -- 172.21.15.112:0/246727666 wait complete. 2023-01-24T03:30:26.193 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.183+0000 7fd73da24700 1 Processor -- start 2023-01-24T03:30:26.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.183+0000 7fd73da24700 1 -- start start 2023-01-24T03:30:26.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.184+0000 7fd73da24700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd7380c15b0 0x7fd738067f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:26.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.184+0000 7fd73da24700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd7380c3080 0x7fd7380c9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:26.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.184+0000 7fd73da24700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd73814d5d0 0x7fd7380c7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:26.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.184+0000 7fd73da24700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7380c8540 con 0x7fd7380c3080 2023-01-24T03:30:26.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.184+0000 7fd73da24700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7380c87e0 con 0x7fd7380c15b0 2023-01-24T03:30:26.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.184+0000 7fd73da24700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd738165b20 con 0x7fd73814d5d0 2023-01-24T03:30:26.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.184+0000 7fd7367fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd7380c3080 0x7fd7380c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:26.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.184+0000 7fd7367fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd7380c3080 0x7fd7380c9830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50172/0 (socket says 172.21.15.112:50172) 2023-01-24T03:30:26.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.184+0000 7fd7367fc700 1 -- 172.21.15.112:0/1636190661 learned_addr learned my addr 172.21.15.112:0/1636190661 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:26.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.184+0000 7fd7377fe700 1 --2- 172.21.15.112:0/1636190661 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd73814d5d0 0x7fd7380c7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:26.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.184+0000 7fd736ffd700 1 --2- 172.21.15.112:0/1636190661 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd7380c15b0 0x7fd738067f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:26.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.184+0000 7fd7367fc700 1 -- 172.21.15.112:0/1636190661 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd73814d5d0 msgr2=0x7fd7380c7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:26.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.184+0000 7fd7367fc700 1 --2- 172.21.15.112:0/1636190661 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd73814d5d0 0x7fd7380c7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:26.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.184+0000 7fd7367fc700 1 -- 172.21.15.112:0/1636190661 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd7380c15b0 msgr2=0x7fd738067f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:26.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.184+0000 7fd7367fc700 1 --2- 172.21.15.112:0/1636190661 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd7380c15b0 0x7fd738067f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:26.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.184+0000 7fd7367fc700 1 -- 172.21.15.112:0/1636190661 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd728009a20 con 0x7fd7380c3080 2023-01-24T03:30:26.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.184+0000 7fd736ffd700 1 --2- 172.21.15.112:0/1636190661 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd7380c15b0 0x7fd738067f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:30:26.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.184+0000 7fd7367fc700 1 --2- 172.21.15.112:0/1636190661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd7380c3080 0x7fd7380c9830 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd72000c420 tx=0x7fd720007c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:26.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.184+0000 7fd72effd700 1 -- 172.21.15.112:0/1636190661 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7200072e0 con 0x7fd7380c3080 2023-01-24T03:30:26.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.184+0000 7fd72effd700 1 -- 172.21.15.112:0/1636190661 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd720007440 con 0x7fd7380c3080 2023-01-24T03:30:26.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.184+0000 7fd73da24700 1 -- 172.21.15.112:0/1636190661 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd738165e00 con 0x7fd7380c3080 2023-01-24T03:30:26.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.184+0000 7fd72effd700 1 -- 172.21.15.112:0/1636190661 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd720005780 con 0x7fd7380c3080 2023-01-24T03:30:26.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.184+0000 7fd73da24700 1 -- 172.21.15.112:0/1636190661 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7380c9030 con 0x7fd7380c3080 2023-01-24T03:30:26.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.185+0000 7fd73da24700 1 -- 172.21.15.112:0/1636190661 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd73804bb80 con 0x7fd7380c3080 2023-01-24T03:30:26.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.185+0000 7fd72effd700 1 -- 172.21.15.112:0/1636190661 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fd720019030 con 0x7fd7380c3080 2023-01-24T03:30:26.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.186+0000 7fd72effd700 1 --2- 172.21.15.112:0/1636190661 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd71c0611f0 0x7fd71c0636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:26.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.186+0000 7fd736ffd700 1 --2- 172.21.15.112:0/1636190661 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd71c0611f0 0x7fd71c0636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:26.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.186+0000 7fd72effd700 1 -- 172.21.15.112:0/1636190661 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fd72007ca90 con 0x7fd7380c3080 2023-01-24T03:30:26.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.186+0000 7fd736ffd700 1 --2- 172.21.15.112:0/1636190661 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd71c0611f0 0x7fd71c0636a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd7280097c0 tx=0x7fd728019000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:26.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.190+0000 7fd72effd700 1 -- 172.21.15.112:0/1636190661 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fd72004c8d0 con 0x7fd7380c3080 2023-01-24T03:30:26.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.369+0000 7fd73da24700 1 -- 172.21.15.112:0/1636190661 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd738156630 con 0x7fd7380c3080 2023-01-24T03:30:26.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.370+0000 7fd72effd700 1 -- 172.21.15.112:0/1636190661 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18755 (secure 0 0 0) 0x7fd720007d90 con 0x7fd7380c3080 2023-01-24T03:30:26.372 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:30:26.372 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":24,"fsid":"054b9ac9-b7b0-40e5-b932-6c8a955c5a72","created":"2023-01-24T03:30:03.798533+0000","modified":"2023-01-24T03:30:25.373340+0000","last_up_change":"2023-01-24T03:30:13.945316+0000","last_in_change":"2023-01-24T03:30:06.198433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-01-24T03:30:14.931166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-01-24T03:30:15.023730+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-01-24T03:30:19.060597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-01-24T03:30:19.943686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-01-24T03:30:22.999036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"fe6ad231-a39a-42eb-9a3a-49d008dd35e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6809","nonce":84512},{"type":"v1","addr":"172.21.15.112:6811","nonce":84512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":84512},{"type":"v1","addr":"172.21.15.112:6817","nonce":84512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6825","nonce":84512},{"type":"v1","addr":"172.21.15.112:6828","nonce":84512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6819","nonce":84512},{"type":"v1","addr":"172.21.15.112:6822","nonce":84512}]},"public_addr":"172.21.15.112:6811/84512","cluster_addr":"172.21.15.112:6817/84512","heartbeat_back_addr":"172.21.15.112:6828/84512","heartbeat_front_addr":"172.21.15.112:6822/84512","state":["exists","up"]},{"osd":1,"uuid":"8526a1d2-d540-4021-9766-c4bbb30317ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":84513},{"type":"v1","addr":"172.21.15.112:6812","nonce":84513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6815","nonce":84513},{"type":"v1","addr":"172.21.15.112:6818","nonce":84513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6827","nonce":84513},{"type":"v1","addr":"172.21.15.112:6830","nonce":84513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6821","nonce":84513},{"type":"v1","addr":"172.21.15.112:6824","nonce":84513}]},"public_addr":"172.21.15.112:6812/84513","cluster_addr":"172.21.15.112:6818/84513","heartbeat_back_addr":"172.21.15.112:6830/84513","heartbeat_front_addr":"172.21.15.112:6824/84513","state":["exists","up"]},{"osd":2,"uuid":"041ef8e4-4858-432d-9476-cbdb3a0dc47c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":84511},{"type":"v1","addr":"172.21.15.112:6813","nonce":84511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":84511},{"type":"v1","addr":"172.21.15.112:6820","nonce":84511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6829","nonce":84511},{"type":"v1","addr":"172.21.15.112:6831","nonce":84511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6823","nonce":84511},{"type":"v1","addr":"172.21.15.112:6826","nonce":84511}]},"public_addr":"172.21.15.112:6813/84511","cluster_addr":"172.21.15.112:6820/84511","heartbeat_back_addr":"172.21.15.112:6831/84511","heartbeat_front_addr":"172.21.15.112:6826/84511","state":["exists","up"]},{"osd":3,"uuid":"987587cf-1357-4a70-ad85-67b189c67729","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":84510},{"type":"v1","addr":"172.21.15.112:6801","nonce":84510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":84510},{"type":"v1","addr":"172.21.15.112:6803","nonce":84510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":84510},{"type":"v1","addr":"172.21.15.112:6807","nonce":84510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":84510},{"type":"v1","addr":"172.21.15.112:6805","nonce":84510}]},"public_addr":"172.21.15.112:6801/84510","cluster_addr":"172.21.15.112:6803/84510","heartbeat_back_addr":"172.21.15.112:6807/84510","heartbeat_front_addr":"172.21.15.112:6805/84510","state":["exists","up"]},{"osd":4,"uuid":"8503512b-43db-40b8-8b2b-c86cf2e54e17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":84614},{"type":"v1","addr":"172.21.15.142:6801","nonce":84614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":84614},{"type":"v1","addr":"172.21.15.142:6803","nonce":84614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":84614},{"type":"v1","addr":"172.21.15.142:6807","nonce":84614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":84614},{"type":"v1","addr":"172.21.15.142:6805","nonce":84614}]},"public_addr":"172.21.15.142:6801/84614","cluster_addr":"172.21.15.142:6803/84614","heartbeat_back_addr":"172.21.15.142:6807/84614","heartbeat_front_addr":"172.21.15.142:6805/84614","state":["exists","up"]},{"osd":5,"uuid":"d3bc0030-8b08-4645-9ec2-1f8a4e29ef36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":84638},{"type":"v1","addr":"172.21.15.142:6817","nonce":84638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":84638},{"type":"v1","addr":"172.21.15.142:6819","nonce":84638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":84638},{"type":"v1","addr":"172.21.15.142:6823","nonce":84638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":84638},{"type":"v1","addr":"172.21.15.142:6821","nonce":84638}]},"public_addr":"172.21.15.142:6817/84638","cluster_addr":"172.21.15.142:6819/84638","heartbeat_back_addr":"172.21.15.142:6823/84638","heartbeat_front_addr":"172.21.15.142:6821/84638","state":["exists","up"]},{"osd":6,"uuid":"c7649e3b-832d-4e6f-8866-b15e5d814476","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":84645},{"type":"v1","addr":"172.21.15.142:6825","nonce":84645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":84645},{"type":"v1","addr":"172.21.15.142:6827","nonce":84645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":84645},{"type":"v1","addr":"172.21.15.142:6831","nonce":84645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":84645},{"type":"v1","addr":"172.21.15.142:6829","nonce":84645}]},"public_addr":"172.21.15.142:6825/84645","cluster_addr":"172.21.15.142:6827/84645","heartbeat_back_addr":"172.21.15.142:6831/84645","heartbeat_front_addr":"172.21.15.142:6829/84645","state":["exists","up"]},{"osd":7,"uuid":"2ea5b3bf-884e-4b20-b572-1443efdc7477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":84615},{"type":"v1","addr":"172.21.15.142:6809","nonce":84615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":84615},{"type":"v1","addr":"172.21.15.142:6811","nonce":84615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":84615},{"type":"v1","addr":"172.21.15.142:6815","nonce":84615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":84615},{"type":"v1","addr":"172.21.15.142:6813","nonce":84615}]},"public_addr":"172.21.15.142:6809/84615","cluster_addr":"172.21.15.142:6811/84615","heartbeat_back_addr":"172.21.15.142:6815/84615","heartbeat_front_addr":"172.21.15.142:6813/84615","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.370027+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.335683+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.350667+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.350794+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.391119+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.422933+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.400439+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.354018+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T03:30:26.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.372+0000 7fd73da24700 1 -- 172.21.15.112:0/1636190661 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd71c0611f0 msgr2=0x7fd71c0636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:26.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.372+0000 7fd73da24700 1 --2- 172.21.15.112:0/1636190661 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd71c0611f0 0x7fd71c0636a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd7280097c0 tx=0x7fd728019000 comp rx=0 tx=0).stop 2023-01-24T03:30:26.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.372+0000 7fd73da24700 1 -- 172.21.15.112:0/1636190661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd7380c3080 msgr2=0x7fd7380c9830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:26.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.372+0000 7fd73da24700 1 --2- 172.21.15.112:0/1636190661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd7380c3080 0x7fd7380c9830 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd72000c420 tx=0x7fd720007c50 comp rx=0 tx=0).stop 2023-01-24T03:30:26.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.372+0000 7fd73da24700 1 -- 172.21.15.112:0/1636190661 shutdown_connections 2023-01-24T03:30:26.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.372+0000 7fd73da24700 1 --2- 172.21.15.112:0/1636190661 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd71c0611f0 0x7fd71c0636a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:26.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.372+0000 7fd73da24700 1 --2- 172.21.15.112:0/1636190661 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd73814d5d0 0x7fd7380c7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:26.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.372+0000 7fd73da24700 1 --2- 172.21.15.112:0/1636190661 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd7380c15b0 0x7fd738067f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:26.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.372+0000 7fd73da24700 1 --2- 172.21.15.112:0/1636190661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd7380c3080 0x7fd7380c9830 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:26.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.372+0000 7fd73da24700 1 -- 172.21.15.112:0/1636190661 >> 172.21.15.112:0/1636190661 conn(0x7fd73806daf0 msgr2=0x7fd738155d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:26.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.372+0000 7fd73da24700 1 -- 172.21.15.112:0/1636190661 shutdown_connections 2023-01-24T03:30:26.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.372+0000 7fd73da24700 1 -- 172.21.15.112:0/1636190661 wait complete. 2023-01-24T03:30:26.398 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:30:26.510 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.508+0000 7fd8fe485700 1 Processor -- start 2023-01-24T03:30:26.510 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.508+0000 7fd8fe485700 1 -- start start 2023-01-24T03:30:26.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.509+0000 7fd8fe485700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fd8f80c7e40 0x7fd8f80b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:26.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.509+0000 7fd8fe485700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd8f80ca3e0 0x7fd8f80ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:26.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.509+0000 7fd8fe485700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8f8156240 con 0x7fd8f8155da0 2023-01-24T03:30:26.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.509+0000 7fd8fe485700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8f80b1170 con 0x7fd8f80c7e40 2023-01-24T03:30:26.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.509+0000 7fd8fe485700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f80b1450 con 0x7fd8f80ca3e0 2023-01-24T03:30:26.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.509+0000 7fd8f7fff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fd8f80c7e40 0x7fd8f80b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:26.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.509+0000 7fd8f77fe700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fd8f8155da0 0x7fd8f80ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:59400/0 (socket says 172.21.15.112:59400) 2023-01-24T03:30:26.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.509+0000 7fd8f77fe700 1 -- 172.21.15.112:0/4138604270 learned_addr learned my addr 172.21.15.112:0/4138604270 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:26.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.509+0000 7fd8f7fff700 1 -- 172.21.15.112:0/4138604270 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd8f80ca3e0 msgr2=0x7fd8f80ca7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:26.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.509+0000 7fd8f7fff700 1 --2- 172.21.15.112:0/4138604270 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd8f80ca3e0 0x7fd8f80ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:26.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.509+0000 7fd8f7fff700 1 -- 172.21.15.112:0/4138604270 >> v1:172.21.15.112:6789/0 conn(0x7fd8f8155da0 legacy=0x7fd8f80ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:26.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.509+0000 7fd8f7fff700 1 -- 172.21.15.112:0/4138604270 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f80b1730 con 0x7fd8f80c7e40 2023-01-24T03:30:26.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.509+0000 7fd8f6ffd700 1 -- 172.21.15.112:0/4138604270 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2797043152 0 0) 0x7fd8f8156240 con 0x7fd8f8155da0 2023-01-24T03:30:26.519 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.509+0000 7fd8f7fff700 1 --2- 172.21.15.112:0/4138604270 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8f80c7e40 0x7fd8f80b0aa0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec009760 tx=0x7fd8ec009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acc6e1bbb702f11d server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:26.519 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.509+0000 7fd8f6ffd700 1 -- 172.21.15.112:0/4138604270 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec011630 con 0x7fd8f80c7e40 2023-01-24T03:30:26.519 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.509+0000 7fd8f6ffd700 1 -- 172.21.15.112:0/4138604270 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8ec011790 con 0x7fd8f80c7e40 2023-01-24T03:30:26.519 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.509+0000 7fd8f6ffd700 1 -- 172.21.15.112:0/4138604270 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec011b30 con 0x7fd8f80c7e40 2023-01-24T03:30:26.520 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.510+0000 7fd8fe485700 1 -- 172.21.15.112:0/4138604270 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8f80c7e40 msgr2=0x7fd8f80b0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:26.520 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.510+0000 7fd8fe485700 1 --2- 172.21.15.112:0/4138604270 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8f80c7e40 0x7fd8f80b0aa0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec009760 tx=0x7fd8ec009060 comp rx=0 tx=0).stop 2023-01-24T03:30:26.520 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.510+0000 7fd8fe485700 1 -- 172.21.15.112:0/4138604270 shutdown_connections 2023-01-24T03:30:26.520 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.510+0000 7fd8fe485700 1 --2- 172.21.15.112:0/4138604270 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd8f80ca3e0 0x7fd8f80ca7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:26.520 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.510+0000 7fd8fe485700 1 --2- 172.21.15.112:0/4138604270 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8f80c7e40 0x7fd8f80b0aa0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:26.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.510+0000 7fd8fe485700 1 -- 172.21.15.112:0/4138604270 >> 172.21.15.112:0/4138604270 conn(0x7fd8f805e2c0 msgr2=0x7fd8f80cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:26.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.510+0000 7fd8fe485700 1 -- 172.21.15.112:0/4138604270 shutdown_connections 2023-01-24T03:30:26.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.510+0000 7fd8fe485700 1 -- 172.21.15.112:0/4138604270 wait complete. 2023-01-24T03:30:26.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.511+0000 7fd8fe485700 1 Processor -- start 2023-01-24T03:30:26.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.511+0000 7fd8fe485700 1 -- start start 2023-01-24T03:30:26.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.512+0000 7fd8fe485700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8f80c7e40 0x7fd8f8068550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:26.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.512+0000 7fd8fe485700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8f80ca3e0 0x7fd8f8148f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:26.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.512+0000 7fd8fe485700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd8f8155da0 0x7fd8f814b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:26.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.512+0000 7fd8fe485700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f80c6160 con 0x7fd8f80c7e40 2023-01-24T03:30:26.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.512+0000 7fd8fe485700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f8150370 con 0x7fd8f80ca3e0 2023-01-24T03:30:26.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.512+0000 7fd8fe485700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f8152680 con 0x7fd8f8155da0 2023-01-24T03:30:26.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.512+0000 7fd8f7fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8f80c7e40 0x7fd8f8068550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:26.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.512+0000 7fd8f7fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8f80c7e40 0x7fd8f8068550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50184/0 (socket says 172.21.15.112:50184) 2023-01-24T03:30:26.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.512+0000 7fd8f7fff700 1 -- 172.21.15.112:0/1822502802 learned_addr learned my addr 172.21.15.112:0/1822502802 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:26.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.512+0000 7fd8f77fe700 1 --2- 172.21.15.112:0/1822502802 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8f80ca3e0 0x7fd8f8148f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:26.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.512+0000 7fd8fca22700 1 --2- 172.21.15.112:0/1822502802 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd8f8155da0 0x7fd8f814b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:26.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.512+0000 7fd8f7fff700 1 -- 172.21.15.112:0/1822502802 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd8f8155da0 msgr2=0x7fd8f814b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:26.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.512+0000 7fd8f7fff700 1 --2- 172.21.15.112:0/1822502802 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd8f8155da0 0x7fd8f814b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:26.525 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.512+0000 7fd8f7fff700 1 -- 172.21.15.112:0/1822502802 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8f80ca3e0 msgr2=0x7fd8f8148f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:26.525 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.512+0000 7fd8f7fff700 1 --2- 172.21.15.112:0/1822502802 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8f80ca3e0 0x7fd8f8148f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:26.525 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.512+0000 7fd8f7fff700 1 -- 172.21.15.112:0/1822502802 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8ec0092a0 con 0x7fd8f80c7e40 2023-01-24T03:30:26.525 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.512+0000 7fd8f7fff700 1 --2- 172.21.15.112:0/1822502802 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8f80c7e40 0x7fd8f8068550 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8089340 tx=0x7fd8ec017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:26.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.512+0000 7fd8dffff700 1 -- 172.21.15.112:0/1822502802 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec011b10 con 0x7fd8f80c7e40 2023-01-24T03:30:26.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.512+0000 7fd8dffff700 1 -- 172.21.15.112:0/1822502802 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8ec01f070 con 0x7fd8f80c7e40 2023-01-24T03:30:26.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.512+0000 7fd8fe485700 1 -- 172.21.15.112:0/1822502802 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8f81528f0 con 0x7fd8f80c7e40 2023-01-24T03:30:26.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.512+0000 7fd8dffff700 1 -- 172.21.15.112:0/1822502802 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec01b660 con 0x7fd8f80c7e40 2023-01-24T03:30:26.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.513+0000 7fd8fe485700 1 -- 172.21.15.112:0/1822502802 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8f8150000 con 0x7fd8f80c7e40 2023-01-24T03:30:26.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.513+0000 7fd8fe485700 1 -- 172.21.15.112:0/1822502802 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8f804bb80 con 0x7fd8f80c7e40 2023-01-24T03:30:26.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.514+0000 7fd8dffff700 1 -- 172.21.15.112:0/1822502802 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fd8ec00f050 con 0x7fd8f80c7e40 2023-01-24T03:30:26.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.514+0000 7fd8dffff700 1 --2- 172.21.15.112:0/1822502802 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd8d80611f0 0x7fd8d80636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:26.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.514+0000 7fd8f77fe700 1 --2- 172.21.15.112:0/1822502802 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd8d80611f0 0x7fd8d80636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:26.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.514+0000 7fd8dffff700 1 -- 172.21.15.112:0/1822502802 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fd8ec07ec10 con 0x7fd8f80c7e40 2023-01-24T03:30:26.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.514+0000 7fd8f77fe700 1 --2- 172.21.15.112:0/1822502802 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd8d80611f0 0x7fd8d80636a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd8f80b0e60 tx=0x7fd8e8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:26.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.518+0000 7fd8dffff700 1 -- 172.21.15.112:0/1822502802 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fd8ec04ea50 con 0x7fd8f80c7e40 2023-01-24T03:30:26.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.736+0000 7fd8fe485700 1 -- 172.21.15.112:0/1822502802 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8f814b1f0 con 0x7fd8f80c7e40 2023-01-24T03:30:26.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.737+0000 7fd8dffff700 1 -- 172.21.15.112:0/1822502802 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+6113 (secure 0 0 0) 0x7fd8ec021090 con 0x7fd8f80c7e40 2023-01-24T03:30:26.739 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:30:26.739 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":5,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/869267413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":869267413},{"type":"v1","addr":"172.21.15.142:6834","nonce":869267413}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":5,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:30:25.373571+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:creating","state_seq":1,"addr":"172.21.15.142:6835/2762621058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2762621058},{"type":"v1","addr":"172.21.15.142:6835","nonce":2762621058}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:30:26.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.739+0000 7fd8fe485700 1 -- 172.21.15.112:0/1822502802 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd8d80611f0 msgr2=0x7fd8d80636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:26.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.739+0000 7fd8fe485700 1 --2- 172.21.15.112:0/1822502802 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd8d80611f0 0x7fd8d80636a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd8f80b0e60 tx=0x7fd8e8009040 comp rx=0 tx=0).stop 2023-01-24T03:30:26.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.739+0000 7fd8fe485700 1 -- 172.21.15.112:0/1822502802 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8f80c7e40 msgr2=0x7fd8f8068550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:26.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.739+0000 7fd8fe485700 1 --2- 172.21.15.112:0/1822502802 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8f80c7e40 0x7fd8f8068550 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8089340 tx=0x7fd8ec017000 comp rx=0 tx=0).stop 2023-01-24T03:30:26.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.739+0000 7fd8fe485700 1 -- 172.21.15.112:0/1822502802 shutdown_connections 2023-01-24T03:30:26.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.739+0000 7fd8fe485700 1 --2- 172.21.15.112:0/1822502802 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd8d80611f0 0x7fd8d80636a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:26.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.739+0000 7fd8fe485700 1 --2- 172.21.15.112:0/1822502802 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd8f80c7e40 0x7fd8f8068550 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:26.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.739+0000 7fd8fe485700 1 --2- 172.21.15.112:0/1822502802 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd8f8155da0 0x7fd8f814b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:26.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.739+0000 7fd8fe485700 1 --2- 172.21.15.112:0/1822502802 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd8f80ca3e0 0x7fd8f8148f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:26.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.739+0000 7fd8fe485700 1 -- 172.21.15.112:0/1822502802 >> 172.21.15.112:0/1822502802 conn(0x7fd8f805e2c0 msgr2=0x7fd8f80ae9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:26.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.739+0000 7fd8fe485700 1 -- 172.21.15.112:0/1822502802 shutdown_connections 2023-01-24T03:30:26.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:26.739+0000 7fd8fe485700 1 -- 172.21.15.112:0/1822502802 wait complete. 2023-01-24T03:30:26.744 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 5 2023-01-24T03:30:27.356 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 5, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-01-24T03:30:20.971710+0000', 'modified': '2023-01-24T03:30:25.373571+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4293}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4293': {'gid': 4293, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.142:6835/2762621058', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6833', 'nonce': 2762621058}, {'type': 'v1', 'addr': '172.21.15.142:6835', 'nonce': 2762621058}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [4, 5], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-01-24T03:30:27.357 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4293:up:creating 2023-01-24T03:30:28.358 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:30:28.483 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.481+0000 7f7c1897e700 1 Processor -- start 2023-01-24T03:30:28.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.481+0000 7f7c1897e700 1 -- start start 2023-01-24T03:30:28.486 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.482+0000 7f7c1897e700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f7c1414d390 0x7f7c14153220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:28.486 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.482+0000 7f7c1897e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7c1414d760 0x7f7c140c1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:28.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.482+0000 7f7c1897e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c14066ec0 con 0x7f7c14153760 2023-01-24T03:30:28.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.482+0000 7f7c1897e700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c140c3a00 con 0x7f7c1414d390 2023-01-24T03:30:28.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.482+0000 7f7c1897e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c141571c0 con 0x7f7c1414d760 2023-01-24T03:30:28.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.482+0000 7f7c1259c700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f7c1414d390 0x7f7c14153220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:28.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.482+0000 7f7c1259c700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f7c1414d390 0x7f7c14153220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50192/0 (socket says 172.21.15.112:50192) 2023-01-24T03:30:28.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.482+0000 7f7c1259c700 1 -- 172.21.15.112:0/1510181911 learned_addr learned my addr 172.21.15.112:0/1510181911 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:28.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.483+0000 7f7c11d9b700 1 --2- 172.21.15.112:0/1510181911 >> v2:172.21.15.142:3300/0 conn(0x7f7c1414d760 0x7f7c140c1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:28.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.483+0000 7f7c1259c700 1 -- 172.21.15.112:0/1510181911 >> v2:172.21.15.142:3300/0 conn(0x7f7c1414d760 msgr2=0x7f7c140c1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:28.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.483+0000 7f7c1259c700 1 --2- 172.21.15.112:0/1510181911 >> v2:172.21.15.142:3300/0 conn(0x7f7c1414d760 0x7f7c140c1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:28.489 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.483+0000 7f7c1259c700 1 -- 172.21.15.112:0/1510181911 >> v1:172.21.15.142:6789/0 conn(0x7f7c14153760 legacy=0x7f7c140c50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:28.489 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.483+0000 7f7c1259c700 1 -- 172.21.15.112:0/1510181911 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c14157650 con 0x7f7c1414d390 2023-01-24T03:30:28.489 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.483+0000 7f7c11d9b700 1 --2- 172.21.15.112:0/1510181911 >> v2:172.21.15.142:3300/0 conn(0x7f7c1414d760 0x7f7c140c1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-01-24T03:30:28.489 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.483+0000 7f7c1259c700 1 --2- 172.21.15.112:0/1510181911 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c1414d390 0x7f7c14153220 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7c1406e0b0 tx=0x7f7c04009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa6308690df2c15 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:28.489 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.483+0000 7f7c1159a700 1 -- 172.21.15.112:0/1510181911 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0400e070 con 0x7f7c1414d390 2023-01-24T03:30:28.490 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.483+0000 7f7c1159a700 1 -- 172.21.15.112:0/1510181911 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c0400ba10 con 0x7f7c1414d390 2023-01-24T03:30:28.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.483+0000 7f7c1159a700 1 -- 172.21.15.112:0/1510181911 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c040092e0 con 0x7f7c1414d390 2023-01-24T03:30:28.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.484+0000 7f7c1897e700 1 -- 172.21.15.112:0/1510181911 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c1414d390 msgr2=0x7f7c14153220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:28.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.484+0000 7f7c1897e700 1 --2- 172.21.15.112:0/1510181911 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c1414d390 0x7f7c14153220 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7c1406e0b0 tx=0x7f7c04009840 comp rx=0 tx=0).stop 2023-01-24T03:30:28.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.484+0000 7f7c1897e700 1 -- 172.21.15.112:0/1510181911 shutdown_connections 2023-01-24T03:30:28.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.484+0000 7f7c1897e700 1 --2- 172.21.15.112:0/1510181911 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c1414d390 0x7f7c14153220 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:28.492 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.484+0000 7f7c1897e700 1 --2- 172.21.15.112:0/1510181911 >> v2:172.21.15.142:3300/0 conn(0x7f7c1414d760 0x7f7c140c1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:28.492 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.484+0000 7f7c1897e700 1 -- 172.21.15.112:0/1510181911 >> 172.21.15.112:0/1510181911 conn(0x7f7c14067f20 msgr2=0x7f7c14068320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:28.492 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.484+0000 7f7c1897e700 1 -- 172.21.15.112:0/1510181911 shutdown_connections 2023-01-24T03:30:28.492 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.484+0000 7f7c1897e700 1 -- 172.21.15.112:0/1510181911 wait complete. 2023-01-24T03:30:28.492 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.485+0000 7f7c1897e700 1 Processor -- start 2023-01-24T03:30:28.493 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.485+0000 7f7c1897e700 1 -- start start 2023-01-24T03:30:28.496 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.485+0000 7f7c1897e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c1414d390 0x7f7c14155570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:28.496 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.485+0000 7f7c1897e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c1414d760 0x7f7c1414e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:28.496 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.485+0000 7f7c1897e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c14153760 0x7f7c14148b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:28.497 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.485+0000 7f7c1897e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1414e2d0 con 0x7f7c14153760 2023-01-24T03:30:28.497 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.485+0000 7f7c1897e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1414eca0 con 0x7f7c1414d760 2023-01-24T03:30:28.497 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.485+0000 7f7c1897e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c14162db0 con 0x7f7c1414d390 2023-01-24T03:30:28.497 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.485+0000 7f7c12d9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c14153760 0x7f7c14148b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:28.497 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.486+0000 7f7c12d9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c14153760 0x7f7c14148b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50202/0 (socket says 172.21.15.112:50202) 2023-01-24T03:30:28.498 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.486+0000 7f7c12d9d700 1 -- 172.21.15.112:0/2112973043 learned_addr learned my addr 172.21.15.112:0/2112973043 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:28.498 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.486+0000 7f7c11d9b700 1 --2- 172.21.15.112:0/2112973043 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c1414d760 0x7f7c1414e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:28.498 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.486+0000 7f7c1259c700 1 --2- 172.21.15.112:0/2112973043 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c1414d390 0x7f7c14155570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:28.498 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.486+0000 7f7c12d9d700 1 -- 172.21.15.112:0/2112973043 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c1414d390 msgr2=0x7f7c14155570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:28.498 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.486+0000 7f7c12d9d700 1 --2- 172.21.15.112:0/2112973043 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c1414d390 0x7f7c14155570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:28.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.486+0000 7f7c12d9d700 1 -- 172.21.15.112:0/2112973043 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c1414d760 msgr2=0x7f7c1414e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:28.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.486+0000 7f7c12d9d700 1 --2- 172.21.15.112:0/2112973043 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c1414d760 0x7f7c1414e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:28.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.486+0000 7f7c12d9d700 1 -- 172.21.15.112:0/2112973043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c04009ae0 con 0x7f7c14153760 2023-01-24T03:30:28.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.486+0000 7f7c12d9d700 1 --2- 172.21.15.112:0/2112973043 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c14153760 0x7f7c14148b70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c0094a0 tx=0x7f7c0c006ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:28.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.486+0000 7f7c0a7fc700 1 -- 172.21.15.112:0/2112973043 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c00e540 con 0x7f7c14153760 2023-01-24T03:30:28.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.486+0000 7f7c0a7fc700 1 -- 172.21.15.112:0/2112973043 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c0c00e6c0 con 0x7f7c14153760 2023-01-24T03:30:28.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.486+0000 7f7c1897e700 1 -- 172.21.15.112:0/2112973043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c14153070 con 0x7f7c14153760 2023-01-24T03:30:28.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.486+0000 7f7c0a7fc700 1 -- 172.21.15.112:0/2112973043 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c00ea80 con 0x7f7c14153760 2023-01-24T03:30:28.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.486+0000 7f7c1897e700 1 -- 172.21.15.112:0/2112973043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c1414dc80 con 0x7f7c14153760 2023-01-24T03:30:28.502 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.487+0000 7f7c1897e700 1 -- 172.21.15.112:0/2112973043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c1414e480 con 0x7f7c14153760 2023-01-24T03:30:28.502 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.487+0000 7f7c0a7fc700 1 -- 172.21.15.112:0/2112973043 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f7c0c019030 con 0x7f7c14153760 2023-01-24T03:30:28.502 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.488+0000 7f7c0a7fc700 1 --2- 172.21.15.112:0/2112973043 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7bf80611c0 0x7f7bf8063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:28.502 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.488+0000 7f7c0a7fc700 1 -- 172.21.15.112:0/2112973043 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f7c0c07cc30 con 0x7f7c14153760 2023-01-24T03:30:28.502 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.488+0000 7f7c1259c700 1 --2- 172.21.15.112:0/2112973043 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7bf80611c0 0x7f7bf8063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:28.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.488+0000 7f7c1259c700 1 --2- 172.21.15.112:0/2112973043 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7bf80611c0 0x7f7bf8063670 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7c04009880 tx=0x7f7c04017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:28.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.492+0000 7f7c0a7fc700 1 -- 172.21.15.112:0/2112973043 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f7c1414e480 con 0x7f7c14153760 2023-01-24T03:30:28.716 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.715+0000 7f7c1897e700 1 -- 172.21.15.112:0/2112973043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c1414e410 con 0x7f7c14153760 2023-01-24T03:30:28.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.715+0000 7f7c0a7fc700 1 -- 172.21.15.112:0/2112973043 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+6111 (secure 0 0 0) 0x7f7c0c017020 con 0x7f7c14153760 2023-01-24T03:30:28.718 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:30:28.718 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/869267413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":869267413},{"type":"v1","addr":"172.21.15.142:6834","nonce":869267413}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:30:26.989184+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.142:6835/2762621058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2762621058},{"type":"v1","addr":"172.21.15.142:6835","nonce":2762621058}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2023-01-24T03:30:28.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.717+0000 7f7c1897e700 1 -- 172.21.15.112:0/2112973043 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7bf80611c0 msgr2=0x7f7bf8063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:28.720 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.717+0000 7f7c1897e700 1 --2- 172.21.15.112:0/2112973043 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7bf80611c0 0x7f7bf8063670 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7c04009880 tx=0x7f7c04017000 comp rx=0 tx=0).stop 2023-01-24T03:30:28.720 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.717+0000 7f7c1897e700 1 -- 172.21.15.112:0/2112973043 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c14153760 msgr2=0x7f7c14148b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:28.720 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.717+0000 7f7c1897e700 1 --2- 172.21.15.112:0/2112973043 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c14153760 0x7f7c14148b70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c0094a0 tx=0x7f7c0c006ea0 comp rx=0 tx=0).stop 2023-01-24T03:30:28.720 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.717+0000 7f7c1897e700 1 -- 172.21.15.112:0/2112973043 shutdown_connections 2023-01-24T03:30:28.720 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.717+0000 7f7c1897e700 1 --2- 172.21.15.112:0/2112973043 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7bf80611c0 0x7f7bf8063670 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:28.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.717+0000 7f7c1897e700 1 --2- 172.21.15.112:0/2112973043 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c14153760 0x7f7c14148b70 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:28.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.717+0000 7f7c1897e700 1 --2- 172.21.15.112:0/2112973043 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7c1414d390 0x7f7c14155570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:28.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.717+0000 7f7c1897e700 1 --2- 172.21.15.112:0/2112973043 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c1414d760 0x7f7c1414e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:28.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.717+0000 7f7c1897e700 1 -- 172.21.15.112:0/2112973043 >> 172.21.15.112:0/2112973043 conn(0x7f7c14067f20 msgr2=0x7f7c1406ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:28.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.717+0000 7f7c1897e700 1 -- 172.21.15.112:0/2112973043 shutdown_connections 2023-01-24T03:30:28.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:28.717+0000 7f7c1897e700 1 -- 172.21.15.112:0/2112973043 wait complete. 2023-01-24T03:30:28.722 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 7 2023-01-24T03:30:28.808 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:30:28.811 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:30:29.335 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-01-24T03:30:20.971710+0000', 'modified': '2023-01-24T03:30:26.989184+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4293}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4293': {'gid': 4293, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.142:6835/2762621058', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6833', 'nonce': 2762621058}, {'type': 'v1', 'addr': '172.21.15.142:6835', 'nonce': 2762621058}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [4, 5], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 1} 2023-01-24T03:30:29.335 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-01-24T03:30:29.336 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-01-24T03:30:29.451 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.449+0000 7fb24bea6700 1 Processor -- start 2023-01-24T03:30:29.452 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.450+0000 7fb24bea6700 1 -- start start 2023-01-24T03:30:29.454 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.450+0000 7fb24bea6700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb24405e420 0x7fb244069890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:29.455 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.450+0000 7fb24bea6700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fb244068d20 0x7fb244155f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:29.455 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.450+0000 7fb24bea6700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb244155ab0 con 0x7fb2440694c0 2023-01-24T03:30:29.455 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.450+0000 7fb24bea6700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb244155550 con 0x7fb244068d20 2023-01-24T03:30:29.455 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.450+0000 7fb24bea6700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb244156480 con 0x7fb24405e420 2023-01-24T03:30:29.455 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.450+0000 7fb24a443700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fb244068d20 0x7fb244155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:29.456 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.450+0000 7fb24a443700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fb244068d20 0x7fb244155f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50210/0 (socket says 172.21.15.112:50210) 2023-01-24T03:30:29.456 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.450+0000 7fb24a443700 1 -- 172.21.15.112:0/3909679745 learned_addr learned my addr 172.21.15.112:0/3909679745 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:29.456 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.451+0000 7fb249441700 1 --2- 172.21.15.112:0/3909679745 >> v2:172.21.15.142:3300/0 conn(0x7fb24405e420 0x7fb244069890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:29.456 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.451+0000 7fb24a443700 1 -- 172.21.15.112:0/3909679745 >> v2:172.21.15.142:3300/0 conn(0x7fb24405e420 msgr2=0x7fb244069890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:29.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.451+0000 7fb24a443700 1 --2- 172.21.15.112:0/3909679745 >> v2:172.21.15.142:3300/0 conn(0x7fb24405e420 0x7fb244069890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:29.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.451+0000 7fb24a443700 1 -- 172.21.15.112:0/3909679745 >> v1:172.21.15.142:6789/0 conn(0x7fb2440694c0 legacy=0x7fb2440c68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:29.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.451+0000 7fb24a443700 1 -- 172.21.15.112:0/3909679745 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2440c70e0 con 0x7fb244068d20 2023-01-24T03:30:29.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.451+0000 7fb24a443700 1 --2- 172.21.15.112:0/3909679745 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb244068d20 0x7fb244155f40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb2440ae770 tx=0x7fb23c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=215f77d1ffe9be3c server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:29.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.451+0000 7fb248c40700 1 -- 172.21.15.112:0/3909679745 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23c011630 con 0x7fb244068d20 2023-01-24T03:30:29.458 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.451+0000 7fb248c40700 1 -- 172.21.15.112:0/3909679745 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb23c011790 con 0x7fb244068d20 2023-01-24T03:30:29.458 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.451+0000 7fb248c40700 1 -- 172.21.15.112:0/3909679745 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23c011b50 con 0x7fb244068d20 2023-01-24T03:30:29.458 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.451+0000 7fb24bea6700 1 -- 172.21.15.112:0/3909679745 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb244068d20 msgr2=0x7fb244155f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:29.458 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.451+0000 7fb24bea6700 1 --2- 172.21.15.112:0/3909679745 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb244068d20 0x7fb244155f40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb2440ae770 tx=0x7fb23c0090f0 comp rx=0 tx=0).stop 2023-01-24T03:30:29.458 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.452+0000 7fb24bea6700 1 -- 172.21.15.112:0/3909679745 shutdown_connections 2023-01-24T03:30:29.459 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.452+0000 7fb24bea6700 1 --2- 172.21.15.112:0/3909679745 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb244068d20 0x7fb244155f40 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:29.459 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.452+0000 7fb24bea6700 1 --2- 172.21.15.112:0/3909679745 >> v2:172.21.15.142:3300/0 conn(0x7fb24405e420 0x7fb244069890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:29.459 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.452+0000 7fb24bea6700 1 -- 172.21.15.112:0/3909679745 >> 172.21.15.112:0/3909679745 conn(0x7fb24405d830 msgr2=0x7fb24405dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:29.459 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.452+0000 7fb24bea6700 1 -- 172.21.15.112:0/3909679745 shutdown_connections 2023-01-24T03:30:29.460 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.452+0000 7fb24bea6700 1 -- 172.21.15.112:0/3909679745 wait complete. 2023-01-24T03:30:29.460 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.453+0000 7fb24bea6700 1 Processor -- start 2023-01-24T03:30:29.460 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.453+0000 7fb24bea6700 1 -- start start 2023-01-24T03:30:29.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.453+0000 7fb24bea6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb24405e420 0x7fb244169500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:29.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.453+0000 7fb24bea6700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb244068d20 0x7fb2440caa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:29.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.453+0000 7fb24bea6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2440694c0 0x7fb2440caf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:29.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.453+0000 7fb24bea6700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb244169a40 con 0x7fb2440694c0 2023-01-24T03:30:29.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.453+0000 7fb24bea6700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb24415b9d0 con 0x7fb244068d20 2023-01-24T03:30:29.471 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.453+0000 7fb24bea6700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb244166110 con 0x7fb24405e420 2023-01-24T03:30:29.471 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.453+0000 7fb24a443700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2440694c0 0x7fb2440caf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:29.471 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.453+0000 7fb24a443700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2440694c0 0x7fb2440caf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50214/0 (socket says 172.21.15.112:50214) 2023-01-24T03:30:29.471 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.453+0000 7fb24a443700 1 -- 172.21.15.112:0/2814138161 learned_addr learned my addr 172.21.15.112:0/2814138161 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:29.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.454+0000 7fb24a443700 1 -- 172.21.15.112:0/2814138161 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb24405e420 msgr2=0x7fb244169500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:29.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.454+0000 7fb249441700 1 --2- 172.21.15.112:0/2814138161 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb244068d20 0x7fb2440caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:29.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.454+0000 7fb24a443700 1 --2- 172.21.15.112:0/2814138161 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb24405e420 0x7fb244169500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:29.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.454+0000 7fb24a443700 1 -- 172.21.15.112:0/2814138161 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb244068d20 msgr2=0x7fb2440caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:29.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.454+0000 7fb24a443700 1 --2- 172.21.15.112:0/2814138161 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb244068d20 0x7fb2440caa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:29.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.454+0000 7fb24a443700 1 -- 172.21.15.112:0/2814138161 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb23c009370 con 0x7fb2440694c0 2023-01-24T03:30:29.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.454+0000 7fb24a443700 1 --2- 172.21.15.112:0/2814138161 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2440694c0 0x7fb2440caf60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb23c009130 tx=0x7fb23c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:29.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.454+0000 7fb241ffb700 1 -- 172.21.15.112:0/2814138161 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23c011630 con 0x7fb2440694c0 2023-01-24T03:30:29.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.454+0000 7fb241ffb700 1 -- 172.21.15.112:0/2814138161 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb23c020070 con 0x7fb2440694c0 2023-01-24T03:30:29.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.454+0000 7fb24bea6700 1 -- 172.21.15.112:0/2814138161 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb244165520 con 0x7fb2440694c0 2023-01-24T03:30:29.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.454+0000 7fb241ffb700 1 -- 172.21.15.112:0/2814138161 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23c0276b0 con 0x7fb2440694c0 2023-01-24T03:30:29.474 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.454+0000 7fb24bea6700 1 -- 172.21.15.112:0/2814138161 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2441657e0 con 0x7fb2440694c0 2023-01-24T03:30:29.474 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.455+0000 7fb241ffb700 1 -- 172.21.15.112:0/2814138161 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fb23c017020 con 0x7fb2440694c0 2023-01-24T03:30:29.474 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.456+0000 7fb241ffb700 1 --2- 172.21.15.112:0/2814138161 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb2240611c0 0x7fb224063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:29.474 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.456+0000 7fb241ffb700 1 -- 172.21.15.112:0/2814138161 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fb23c022070 con 0x7fb2440694c0 2023-01-24T03:30:29.474 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.456+0000 7fb249c42700 1 --2- 172.21.15.112:0/2814138161 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb2240611c0 0x7fb224063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:29.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.456+0000 7fb249c42700 1 --2- 172.21.15.112:0/2814138161 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb2240611c0 0x7fb224063670 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb234001b70 tx=0x7fb234007d40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:29.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.458+0000 7fb25518fb80 1 Processor -- start 2023-01-24T03:30:29.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.458+0000 7fb25518fb80 1 -- start start 2023-01-24T03:30:29.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.458+0000 7fb25518fb80 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x557f9ef50ba0 0x557f9ef47090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:29.476 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.458+0000 7fb25518fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f9ef50f70 0x557f9ef0f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:29.476 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.458+0000 7fb25518fb80 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f9ef4f370 0x557f9ef42410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:29.476 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.458+0000 7fb25518fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f9ee990a0 con 0x557f9ef50f70 2023-01-24T03:30:29.476 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.458+0000 7fb25518fb80 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f9ee1d460 con 0x557f9ef4f370 2023-01-24T03:30:29.477 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.458+0000 7fb25518fb80 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x557f9ee66270 con 0x557f9ef50ba0 2023-01-24T03:30:29.477 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.458+0000 7fb249441700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f9ef50f70 0x557f9ef0f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:29.477 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.458+0000 7fb249441700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f9ef50f70 0x557f9ef0f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50222/0 (socket says 172.21.15.112:50222) 2023-01-24T03:30:29.477 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.458+0000 7fb249441700 1 -- 172.21.15.112:0/3305904015 learned_addr learned my addr 172.21.15.112:0/3305904015 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:29.478 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.458+0000 7fb249c42700 1 --2- 172.21.15.112:0/3305904015 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x557f9ef50ba0 0x557f9ef47090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:29.478 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.458+0000 7fb24a443700 1 --2- 172.21.15.112:0/3305904015 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f9ef4f370 0x557f9ef42410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:29.478 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.459+0000 7fb249441700 1 -- 172.21.15.112:0/3305904015 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x557f9ef50ba0 msgr2=0x557f9ef47090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:29.478 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.459+0000 7fb249441700 1 --2- 172.21.15.112:0/3305904015 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x557f9ef50ba0 0x557f9ef47090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:29.479 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.459+0000 7fb249441700 1 -- 172.21.15.112:0/3305904015 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f9ef4f370 msgr2=0x557f9ef42410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:29.479 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.459+0000 7fb249441700 1 --2- 172.21.15.112:0/3305904015 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f9ef4f370 0x557f9ef42410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:29.479 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.459+0000 7fb249441700 1 -- 172.21.15.112:0/3305904015 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557f9ecfb270 con 0x557f9ef50f70 2023-01-24T03:30:29.479 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.459+0000 7fb24a443700 1 --2- 172.21.15.112:0/3305904015 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f9ef4f370 0x557f9ef42410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:30:29.480 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.459+0000 7fb249441700 1 --2- 172.21.15.112:0/3305904015 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f9ef50f70 0x557f9ef0f790 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb238009810 tx=0x7fb238008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f85aa0502a3f59a8 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:29.480 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.459+0000 7fb22affd700 1 -- 172.21.15.112:0/3305904015 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23800c070 con 0x557f9ef50f70 2023-01-24T03:30:29.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.459+0000 7fb22affd700 1 -- 172.21.15.112:0/3305904015 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb238013410 con 0x557f9ef50f70 2023-01-24T03:30:29.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.459+0000 7fb22affd700 1 -- 172.21.15.112:0/3305904015 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2380137b0 con 0x557f9ef50f70 2023-01-24T03:30:29.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.459+0000 7fb25518fb80 1 -- 172.21.15.112:0/3305904015 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f9ef50f70 msgr2=0x557f9ef0f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:29.485 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.459+0000 7fb25518fb80 1 --2- 172.21.15.112:0/3305904015 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f9ef50f70 0x557f9ef0f790 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb238009810 tx=0x7fb238008040 comp rx=0 tx=0).stop 2023-01-24T03:30:29.485 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.459+0000 7fb25518fb80 1 -- 172.21.15.112:0/3305904015 shutdown_connections 2023-01-24T03:30:29.485 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.459+0000 7fb25518fb80 1 --2- 172.21.15.112:0/3305904015 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f9ef4f370 0x557f9ef42410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:29.485 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.459+0000 7fb25518fb80 1 --2- 172.21.15.112:0/3305904015 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x557f9ef50ba0 0x557f9ef47090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:29.485 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.459+0000 7fb25518fb80 1 --2- 172.21.15.112:0/3305904015 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f9ef50f70 0x557f9ef0f790 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:29.486 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.459+0000 7fb25518fb80 1 -- 172.21.15.112:0/3305904015 >> 172.21.15.112:0/3305904015 conn(0x557f9ef0f080 msgr2=0x557f9ef0e130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:29.486 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.460+0000 7fb25518fb80 1 -- 172.21.15.112:0/3305904015 shutdown_connections 2023-01-24T03:30:29.486 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.460+0000 7fb25518fb80 1 -- 172.21.15.112:0/3305904015 wait complete. 2023-01-24T03:30:29.486 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.460+0000 7fb25518fb80 1 -- start start 2023-01-24T03:30:29.486 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.460+0000 7fb25518fb80 1 Processor -- start 2023-01-24T03:30:29.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.461+0000 7fb25518fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f9ef4f370 0x557f9ef49770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:29.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.461+0000 7fb25518fb80 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f9ef50ba0 0x557f9ef52f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:29.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.461+0000 7fb25518fb80 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x557f9ef50f70 0x557f9ef4a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:29.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.461+0000 7fb25518fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f9ee5fae0 con 0x557f9ef4f370 2023-01-24T03:30:29.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.461+0000 7fb25518fb80 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f9ece3e30 con 0x557f9ef50ba0 2023-01-24T03:30:29.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.461+0000 7fb25518fb80 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x557f9ef57590 con 0x557f9ef50f70 2023-01-24T03:30:29.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.461+0000 7fb25518fb80 10 client.0 _ll_register_callbacks cb 0x557f9eed1cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-01-24T03:30:29.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.461+0000 7fb249c42700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f9ef4f370 0x557f9ef49770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:29.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.461+0000 7fb249c42700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f9ef4f370 0x557f9ef49770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50224/0 (socket says 172.21.15.112:50224) 2023-01-24T03:30:29.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.461+0000 7fb249c42700 1 -- 172.21.15.112:0/141409927 learned_addr learned my addr 172.21.15.112:0/141409927 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:29.489 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.461+0000 7fb24a443700 1 --2- 172.21.15.112:0/141409927 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x557f9ef50f70 0x557f9ef4a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:29.489 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.461+0000 7fb249441700 1 --2- 172.21.15.112:0/141409927 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f9ef50ba0 0x557f9ef52f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:29.489 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.461+0000 7fb249c42700 1 -- 172.21.15.112:0/141409927 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x557f9ef50f70 msgr2=0x557f9ef4a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:29.489 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.461+0000 7fb249c42700 1 --2- 172.21.15.112:0/141409927 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x557f9ef50f70 0x557f9ef4a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:29.489 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.461+0000 7fb249c42700 1 -- 172.21.15.112:0/141409927 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f9ef50ba0 msgr2=0x557f9ef52f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:29.490 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.461+0000 7fb249c42700 1 --2- 172.21.15.112:0/141409927 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f9ef50ba0 0x557f9ef52f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:29.490 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.461+0000 7fb249c42700 1 -- 172.21.15.112:0/141409927 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb238009470 con 0x557f9ef4f370 2023-01-24T03:30:29.490 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.461+0000 7fb249c42700 1 --2- 172.21.15.112:0/141409927 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f9ef4f370 0x557f9ef49770 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb234016750 tx=0x7fb234011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b101192bba354bd3 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:29.490 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.461+0000 7fb21ffff700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2023-01-24T03:30:29.490 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.462+0000 7fb21ffff700 1 -- 172.21.15.112:0/141409927 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23400c700 con 0x557f9ef4f370 2023-01-24T03:30:29.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.462+0000 7fb21ffff700 1 -- 172.21.15.112:0/141409927 <== mon.0 v2:172.21.15.112:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fb234021440 con 0x557f9ef4f370 2023-01-24T03:30:29.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.462+0000 7fb25518fb80 1 -- 172.21.15.112:0/141409927 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557f9ef035d0 con 0x557f9ef4f370 2023-01-24T03:30:29.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.462+0000 7fb21ffff700 1 -- 172.21.15.112:0/141409927 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb234021810 con 0x557f9ef4f370 2023-01-24T03:30:29.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.462+0000 7fb25518fb80 10 client.4446 fetch_fsmap learned FSMap version 7 2023-01-24T03:30:29.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.462+0000 7fb25518fb80 1 -- 172.21.15.112:0/141409927 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x557f9ec9dfe0 con 0x557f9ef4f370 2023-01-24T03:30:29.492 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.462+0000 7fb21ffff700 1 -- 172.21.15.112:0/141409927 <== mon.0 v2:172.21.15.112:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x557f9ef035d0 con 0x557f9ef4f370 2023-01-24T03:30:29.492 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.462+0000 7fb21ffff700 1 -- 172.21.15.112:0/141409927 <== mon.0 v2:172.21.15.112:3300/0 5 ==== fsmap(e 7) v1 ==== 4198+0+0 (secure 0 0 0) 0x7fb234022a50 con 0x557f9ef4f370 2023-01-24T03:30:29.492 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.462+0000 7fb25518fb80 10 client.4446 fetch_fsmap finished waiting for FSMap version 7 2023-01-24T03:30:29.492 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.462+0000 7fb25518fb80 10 client.4446 resolve_mds: resolved name 'd' to daemon mds.d 2023-01-24T03:30:29.492 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.462+0000 7fb25518fb80 20 client.4446 populate_metadata read hostname 'smithi112' 2023-01-24T03:30:29.493 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.462+0000 7fb25518fb80 1 --2- 172.21.15.112:0/141409927 >> [v2:172.21.15.142:6833/2762621058,v1:172.21.15.142:6835/2762621058] conn(0x557f9ef4a930 0x557f9ef47de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:29.493 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.462+0000 7fb25518fb80 4 client.4446 mds_command: new command op to 4293 tid=0[{"prefix": "get_command_descriptions"}] 2023-01-24T03:30:29.493 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.462+0000 7fb25518fb80 1 -- 172.21.15.112:0/141409927 --> [v2:172.21.15.142:6833/2762621058,v1:172.21.15.142:6835/2762621058] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557f9ee04480 con 0x557f9ef4a930 2023-01-24T03:30:29.493 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.462+0000 7fb249441700 1 -- 172.21.15.112:0/141409927 >> [v2:172.21.15.142:6833/2762621058,v1:172.21.15.142:6835/2762621058] conn(0x557f9ef4a930 msgr2=0x557f9ef47de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 19 2023-01-24T03:30:29.493 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.462+0000 7fb249441700 1 -- 172.21.15.112:0/141409927 >> [v2:172.21.15.142:6833/2762621058,v1:172.21.15.142:6835/2762621058] conn(0x557f9ef4a930 msgr2=0x557f9ef47de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-01-24T03:30:29.494 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.462+0000 7fb249441700 1 --2- 172.21.15.112:0/141409927 >> [v2:172.21.15.142:6833/2762621058,v1:172.21.15.142:6835/2762621058] conn(0x557f9ef4a930 0x557f9ef47de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2023-01-24T03:30:29.494 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.462+0000 7fb249441700 1 --2- 172.21.15.112:0/141409927 >> [v2:172.21.15.142:6833/2762621058,v1:172.21.15.142:6835/2762621058] conn(0x557f9ef4a930 0x557f9ef47de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-01-24T03:30:29.665 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.663+0000 7fb249441700 1 --2- 172.21.15.112:0/141409927 >> [v2:172.21.15.142:6833/2762621058,v1:172.21.15.142:6835/2762621058] conn(0x557f9ef4a930 0x557f9ef47de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:29.666 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.663+0000 7fb249441700 1 --2- 172.21.15.112:0/141409927 >> [v2:172.21.15.142:6833/2762621058,v1:172.21.15.142:6835/2762621058] conn(0x557f9ef4a930 0x557f9ef47de0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1315649476a765a7 server_cookie=8155b9a4c0c92742 in_seq=0 out_seq=0 2023-01-24T03:30:29.666 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:29.663+0000 7fb21ffff700 10 client.4446 ms_handle_connect on v2:172.21.15.142:6833/2762621058 2023-01-24T03:30:31.994 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:31.991+0000 7fb21ffff700 1 -- 172.21.15.112:0/141409927 <== mds.0 v2:172.21.15.142:6833/2762621058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17090 (crc 0 0 0) 0x557f9ee04480 con 0x557f9ef4a930 2023-01-24T03:30:31.994 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:31.991+0000 7fb21ffff700 10 client.4446 handle_command_reply: tid=0 2023-01-24T03:30:31.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:31.991+0000 7fb25518fb80 1 client.4446 shutdown 2023-01-24T03:30:31.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:31.992+0000 7fb25518fb80 1 -- 172.21.15.112:0/141409927 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f9ef4f370 msgr2=0x557f9ef49770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:31.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:31.992+0000 7fb25518fb80 1 --2- 172.21.15.112:0/141409927 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f9ef4f370 0x557f9ef49770 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb234016750 tx=0x7fb234011000 comp rx=0 tx=0).stop 2023-01-24T03:30:31.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:31.992+0000 7fb25518fb80 1 -- 172.21.15.112:0/141409927 shutdown_connections 2023-01-24T03:30:31.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:31.992+0000 7fb25518fb80 1 --2- 172.21.15.112:0/141409927 >> [v2:172.21.15.142:6833/2762621058,v1:172.21.15.142:6835/2762621058] conn(0x557f9ef4a930 0x557f9ef47de0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:31.996 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:31.992+0000 7fb25518fb80 1 --2- 172.21.15.112:0/141409927 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f9ef4f370 0x557f9ef49770 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:31.996 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:31.992+0000 7fb25518fb80 1 --2- 172.21.15.112:0/141409927 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x557f9ef50f70 0x557f9ef4a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:31.996 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:31.992+0000 7fb21ffff700 0 client.4446 ms_handle_reset on v2:172.21.15.142:6833/2762621058 2023-01-24T03:30:31.996 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:31.992+0000 7fb25518fb80 1 --2- 172.21.15.112:0/141409927 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f9ef50ba0 0x557f9ef52f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:31.996 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:31.992+0000 7fb25518fb80 1 -- 172.21.15.112:0/141409927 >> 172.21.15.112:0/141409927 conn(0x557f9ef0f080 msgr2=0x557f9ef0f450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:31.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:31.992+0000 7fb25518fb80 1 -- 172.21.15.112:0/141409927 shutdown_connections 2023-01-24T03:30:31.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:31.992+0000 7fb25518fb80 1 -- 172.21.15.112:0/141409927 wait complete. 2023-01-24T03:30:31.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:31.992+0000 7fb25518fb80 20 client.4446 trim_cache size 0 max 16384 2023-01-24T03:30:32.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.010+0000 7fb25518fb80 1 Processor -- start 2023-01-24T03:30:32.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.011+0000 7fb25518fb80 1 -- start start 2023-01-24T03:30:32.015 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.011+0000 7fb25518fb80 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f9ef4f370 0x557f9ef38e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:32.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.011+0000 7fb25518fb80 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x557f9ef50ba0 0x557f9ef47de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:32.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.011+0000 7fb25518fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f9ef50f70 0x557f9ef56ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:32.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.011+0000 7fb25518fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f9ef03430 con 0x557f9ef50f70 2023-01-24T03:30:32.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.011+0000 7fb25518fb80 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f9ef48990 con 0x557f9ef4f370 2023-01-24T03:30:32.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.011+0000 7fb25518fb80 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x557f9ef4ad90 con 0x557f9ef50ba0 2023-01-24T03:30:32.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.011+0000 7fb24a443700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f9ef50f70 0x557f9ef56ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:32.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.011+0000 7fb24a443700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f9ef50f70 0x557f9ef56ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50238/0 (socket says 172.21.15.112:50238) 2023-01-24T03:30:32.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.012+0000 7fb24a443700 1 -- 172.21.15.112:0/3140406652 learned_addr learned my addr 172.21.15.112:0/3140406652 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:32.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.012+0000 7fb249441700 1 --2- 172.21.15.112:0/3140406652 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x557f9ef50ba0 0x557f9ef47de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:32.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.012+0000 7fb249c42700 1 --2- 172.21.15.112:0/3140406652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f9ef4f370 0x557f9ef38e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:32.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.012+0000 7fb24a443700 1 -- 172.21.15.112:0/3140406652 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x557f9ef50ba0 msgr2=0x557f9ef47de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:32.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.012+0000 7fb24a443700 1 --2- 172.21.15.112:0/3140406652 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x557f9ef50ba0 0x557f9ef47de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:32.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.012+0000 7fb24a443700 1 -- 172.21.15.112:0/3140406652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f9ef4f370 msgr2=0x557f9ef38e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:32.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.012+0000 7fb24a443700 1 --2- 172.21.15.112:0/3140406652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f9ef4f370 0x557f9ef38e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:32.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.012+0000 7fb24a443700 1 -- 172.21.15.112:0/3140406652 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557f9ef3fc10 con 0x557f9ef50f70 2023-01-24T03:30:32.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.012+0000 7fb24a443700 1 --2- 172.21.15.112:0/3140406652 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f9ef50f70 0x557f9ef56ad0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb23c0530c0 tx=0x7fb23c01e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdc38cbaeb8d2ee5 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:32.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.012+0000 7fb22b7fe700 1 -- 172.21.15.112:0/3140406652 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23c051500 con 0x557f9ef50f70 2023-01-24T03:30:32.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.012+0000 7fb22b7fe700 1 -- 172.21.15.112:0/3140406652 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb23c051760 con 0x557f9ef50f70 2023-01-24T03:30:32.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.012+0000 7fb22b7fe700 1 -- 172.21.15.112:0/3140406652 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23c052ac0 con 0x557f9ef50f70 2023-01-24T03:30:32.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.013+0000 7fb25518fb80 1 -- 172.21.15.112:0/3140406652 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f9ef50f70 msgr2=0x557f9ef56ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:32.020 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.013+0000 7fb25518fb80 1 --2- 172.21.15.112:0/3140406652 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f9ef50f70 0x557f9ef56ad0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb23c0530c0 tx=0x7fb23c01e040 comp rx=0 tx=0).stop 2023-01-24T03:30:32.020 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.013+0000 7fb25518fb80 1 -- 172.21.15.112:0/3140406652 shutdown_connections 2023-01-24T03:30:32.020 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.013+0000 7fb25518fb80 1 --2- 172.21.15.112:0/3140406652 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f9ef50f70 0x557f9ef56ad0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:32.020 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.013+0000 7fb25518fb80 1 --2- 172.21.15.112:0/3140406652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f9ef4f370 0x557f9ef38e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:32.021 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.013+0000 7fb25518fb80 1 --2- 172.21.15.112:0/3140406652 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x557f9ef50ba0 0x557f9ef47de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:32.021 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.013+0000 7fb25518fb80 1 -- 172.21.15.112:0/3140406652 >> 172.21.15.112:0/3140406652 conn(0x557f9ef0f080 msgr2=0x557f9ef4f740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:32.021 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.013+0000 7fb25518fb80 1 -- 172.21.15.112:0/3140406652 shutdown_connections 2023-01-24T03:30:32.021 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.013+0000 7fb25518fb80 1 -- 172.21.15.112:0/3140406652 wait complete. 2023-01-24T03:30:32.021 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.014+0000 7fb25518fb80 1 -- start start 2023-01-24T03:30:32.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.014+0000 7fb25518fb80 1 Processor -- start 2023-01-24T03:30:32.026 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.015+0000 7fb25518fb80 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x557f9ef4f370 0x557f9ef56df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:32.026 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.015+0000 7fb25518fb80 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f9ef50ba0 0x557f9ef0ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:32.026 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.015+0000 7fb25518fb80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f9ef50f70 0x557f9efaf880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:32.026 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.015+0000 7fb25518fb80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f9ef57330 con 0x557f9ef50f70 2023-01-24T03:30:32.027 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.015+0000 7fb25518fb80 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f9ef0c010 con 0x557f9ef50ba0 2023-01-24T03:30:32.027 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.015+0000 7fb25518fb80 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x557f9efb93c0 con 0x557f9ef4f370 2023-01-24T03:30:32.027 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.015+0000 7fb25518fb80 10 client.0 _ll_register_callbacks cb 0x557f9eed1cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-01-24T03:30:32.027 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.015+0000 7fb24a443700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f9ef50f70 0x557f9efaf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:32.027 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.015+0000 7fb249441700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f9ef50ba0 0x557f9ef0ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:32.028 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.015+0000 7fb249c42700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x557f9ef4f370 0x557f9ef56df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:32.028 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.015+0000 7fb249c42700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x557f9ef4f370 0x557f9ef56df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.112:37542/0 (socket says 172.21.15.112:37542) 2023-01-24T03:30:32.028 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.015+0000 7fb249441700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f9ef50ba0 0x557f9ef0ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.112:42736/0 (socket says 172.21.15.112:42736) 2023-01-24T03:30:32.028 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.015+0000 7fb24a443700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f9ef50f70 0x557f9efaf880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50244/0 (socket says 172.21.15.112:50244) 2023-01-24T03:30:32.028 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.015+0000 7fb249c42700 1 -- 172.21.15.112:0/3380683982 learned_addr learned my addr 172.21.15.112:0/3380683982 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:32.029 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.015+0000 7fb24a443700 1 -- 172.21.15.112:0/3380683982 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x557f9ef4f370 msgr2=0x557f9ef56df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:32.029 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.015+0000 7fb24a443700 1 --2- 172.21.15.112:0/3380683982 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x557f9ef4f370 0x557f9ef56df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:32.029 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.015+0000 7fb24a443700 1 -- 172.21.15.112:0/3380683982 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f9ef50ba0 msgr2=0x557f9ef0ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:32.029 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.015+0000 7fb24a443700 1 --2- 172.21.15.112:0/3380683982 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f9ef50ba0 0x557f9ef0ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:32.029 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.015+0000 7fb24a443700 1 -- 172.21.15.112:0/3380683982 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2441657e0 con 0x557f9ef50f70 2023-01-24T03:30:32.030 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.015+0000 7fb24a443700 1 --2- 172.21.15.112:0/3380683982 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f9ef50f70 0x557f9efaf880 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb23c079000 tx=0x7fb23c066000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79a64db9c3da6acb server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:32.030 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.015+0000 7fb21ffff700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2023-01-24T03:30:32.030 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.016+0000 7fb21ffff700 1 -- 172.21.15.112:0/3380683982 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23c0528b0 con 0x557f9ef50f70 2023-01-24T03:30:32.030 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.016+0000 7fb21ffff700 1 -- 172.21.15.112:0/3380683982 <== mon.0 v2:172.21.15.112:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fb23c052a10 con 0x557f9ef50f70 2023-01-24T03:30:32.030 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.016+0000 7fb25518fb80 1 -- 172.21.15.112:0/3380683982 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb238013410 con 0x557f9ef50f70 2023-01-24T03:30:32.031 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.016+0000 7fb21ffff700 1 -- 172.21.15.112:0/3380683982 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb23c06ad80 con 0x557f9ef50f70 2023-01-24T03:30:32.031 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.016+0000 7fb25518fb80 10 client.4452 fetch_fsmap learned FSMap version 7 2023-01-24T03:30:32.031 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.016+0000 7fb25518fb80 1 -- 172.21.15.112:0/3380683982 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x557f9ecc5f80 con 0x557f9ef50f70 2023-01-24T03:30:32.031 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.016+0000 7fb21ffff700 1 -- 172.21.15.112:0/3380683982 <== mon.0 v2:172.21.15.112:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb23c073020 con 0x557f9ef50f70 2023-01-24T03:30:32.031 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.016+0000 7fb21ffff700 1 -- 172.21.15.112:0/3380683982 <== mon.0 v2:172.21.15.112:3300/0 5 ==== fsmap(e 7) v1 ==== 4198+0+0 (secure 0 0 0) 0x7fb23c04f1b0 con 0x557f9ef50f70 2023-01-24T03:30:32.032 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.016+0000 7fb25518fb80 10 client.4452 fetch_fsmap finished waiting for FSMap version 7 2023-01-24T03:30:32.032 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.016+0000 7fb25518fb80 10 client.4452 resolve_mds: resolved name 'd' to daemon mds.d 2023-01-24T03:30:32.032 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.016+0000 7fb25518fb80 20 client.4452 populate_metadata read hostname 'smithi112' 2023-01-24T03:30:32.032 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.016+0000 7fb25518fb80 1 --2- 172.21.15.112:0/3380683982 >> [v2:172.21.15.142:6833/2762621058,v1:172.21.15.142:6835/2762621058] conn(0x557f9efb0070 0x557f9ef3b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:32.032 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.016+0000 7fb25518fb80 4 client.4452 mds_command: new command op to 4293 tid=0[{"prefix": "status"}] 2023-01-24T03:30:32.033 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.016+0000 7fb25518fb80 1 -- 172.21.15.112:0/3380683982 --> [v2:172.21.15.142:6833/2762621058,v1:172.21.15.142:6835/2762621058] -- command(tid 0: {"prefix": "status"}) v1 -- 0x557f9ee27ac0 con 0x557f9efb0070 2023-01-24T03:30:32.033 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.016+0000 7fb249c42700 1 --2- 172.21.15.112:0/3380683982 >> [v2:172.21.15.142:6833/2762621058,v1:172.21.15.142:6835/2762621058] conn(0x557f9efb0070 0x557f9ef3b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:32.033 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.017+0000 7fb21ffff700 10 client.4452 ms_handle_connect on v2:172.21.15.142:6833/2762621058 2023-01-24T03:30:32.033 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.017+0000 7fb249c42700 1 --2- 172.21.15.112:0/3380683982 >> [v2:172.21.15.142:6833/2762621058,v1:172.21.15.142:6835/2762621058] conn(0x557f9efb0070 0x557f9ef3b7a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e19260a91bf0de5 server_cookie=d33921d5cad7b303 in_seq=0 out_seq=0 2023-01-24T03:30:32.033 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.017+0000 7fb21ffff700 1 -- 172.21.15.112:0/3380683982 <== mds.0 v2:172.21.15.142:6833/2762621058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x557f9ee27ac0 con 0x557f9efb0070 2023-01-24T03:30:32.034 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.017+0000 7fb21ffff700 10 client.4452 handle_command_reply: tid=0 2023-01-24T03:30:32.034 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.017+0000 7fb25518fb80 1 client.4452 shutdown 2023-01-24T03:30:32.034 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.017+0000 7fb25518fb80 1 -- 172.21.15.112:0/3380683982 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f9ef50f70 msgr2=0x557f9efaf880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:32.034 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.017+0000 7fb25518fb80 1 --2- 172.21.15.112:0/3380683982 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f9ef50f70 0x557f9efaf880 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb23c079000 tx=0x7fb23c066000 comp rx=0 tx=0).stop 2023-01-24T03:30:32.034 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.017+0000 7fb25518fb80 1 -- 172.21.15.112:0/3380683982 shutdown_connections 2023-01-24T03:30:32.035 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.017+0000 7fb25518fb80 1 --2- 172.21.15.112:0/3380683982 >> [v2:172.21.15.142:6833/2762621058,v1:172.21.15.142:6835/2762621058] conn(0x557f9efb0070 0x557f9ef3b7a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:32.035 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.017+0000 7fb25518fb80 1 --2- 172.21.15.112:0/3380683982 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557f9ef50f70 0x557f9efaf880 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:32.035 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.017+0000 7fb21ffff700 0 client.4452 ms_handle_reset on v2:172.21.15.142:6833/2762621058 2023-01-24T03:30:32.035 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.017+0000 7fb25518fb80 1 --2- 172.21.15.112:0/3380683982 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x557f9ef4f370 0x557f9ef56df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:32.035 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.017+0000 7fb25518fb80 1 --2- 172.21.15.112:0/3380683982 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x557f9ef50ba0 0x557f9ef0ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:32.035 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.017+0000 7fb25518fb80 1 -- 172.21.15.112:0/3380683982 >> 172.21.15.112:0/3380683982 conn(0x557f9ef0f080 msgr2=0x557f9ef47db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:32.036 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.017+0000 7fb25518fb80 1 -- 172.21.15.112:0/3380683982 shutdown_connections 2023-01-24T03:30:32.036 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.017+0000 7fb25518fb80 1 -- 172.21.15.112:0/3380683982 wait complete. 2023-01-24T03:30:32.036 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.017+0000 7fb25518fb80 20 client.4452 trim_cache size 0 max 16384 2023-01-24T03:30:32.036 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-01-24T03:30:32.037 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_fsid": "054b9ac9-b7b0-40e5-b932-6c8a955c5a72", 2023-01-24T03:30:32.037 INFO:teuthology.orchestra.run.smithi112.stdout: "whoami": 0, 2023-01-24T03:30:32.037 INFO:teuthology.orchestra.run.smithi112.stdout: "id": 4293, 2023-01-24T03:30:32.037 INFO:teuthology.orchestra.run.smithi112.stdout: "want_state": "up:active", 2023-01-24T03:30:32.038 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "up:active", 2023-01-24T03:30:32.038 INFO:teuthology.orchestra.run.smithi112.stdout: "fs_name": "cephfs", 2023-01-24T03:30:32.038 INFO:teuthology.orchestra.run.smithi112.stdout: "rank_uptime": 11.044246792999999, 2023-01-24T03:30:32.038 INFO:teuthology.orchestra.run.smithi112.stdout: "mdsmap_epoch": 6, 2023-01-24T03:30:32.038 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap_epoch": 24, 2023-01-24T03:30:32.038 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap_epoch_barrier": 24, 2023-01-24T03:30:32.039 INFO:teuthology.orchestra.run.smithi112.stdout: "uptime": 15.850146533 2023-01-24T03:30:32.039 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-01-24T03:30:32.040 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.018+0000 7fb24bea6700 1 -- 172.21.15.112:0/2814138161 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb2240611c0 msgr2=0x7fb224063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:32.040 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.018+0000 7fb24bea6700 1 --2- 172.21.15.112:0/2814138161 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb2240611c0 0x7fb224063670 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb234001b70 tx=0x7fb234007d40 comp rx=0 tx=0).stop 2023-01-24T03:30:32.041 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.018+0000 7fb24bea6700 1 -- 172.21.15.112:0/2814138161 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2440694c0 msgr2=0x7fb2440caf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:32.041 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.018+0000 7fb24bea6700 1 --2- 172.21.15.112:0/2814138161 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2440694c0 0x7fb2440caf60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb23c009130 tx=0x7fb23c016000 comp rx=0 tx=0).stop 2023-01-24T03:30:32.041 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.018+0000 7fb24bea6700 1 -- 172.21.15.112:0/2814138161 shutdown_connections 2023-01-24T03:30:32.041 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.018+0000 7fb24bea6700 1 --2- 172.21.15.112:0/2814138161 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb2240611c0 0x7fb224063670 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:32.041 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.018+0000 7fb24bea6700 1 --2- 172.21.15.112:0/2814138161 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2440694c0 0x7fb2440caf60 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:32.042 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.018+0000 7fb24bea6700 1 --2- 172.21.15.112:0/2814138161 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb24405e420 0x7fb244169500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:32.042 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.018+0000 7fb24bea6700 1 --2- 172.21.15.112:0/2814138161 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb244068d20 0x7fb2440caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:32.042 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.018+0000 7fb24bea6700 1 -- 172.21.15.112:0/2814138161 >> 172.21.15.112:0/2814138161 conn(0x7fb24405d830 msgr2=0x7fb24414f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:32.042 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.018+0000 7fb24bea6700 1 -- 172.21.15.112:0/2814138161 shutdown_connections 2023-01-24T03:30:32.042 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.018+0000 7fb24bea6700 1 -- 172.21.15.112:0/2814138161 wait complete. 2023-01-24T03:30:32.043 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-01-24T03:30:32.161 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.159+0000 7f5f609da700 1 Processor -- start 2023-01-24T03:30:32.161 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.159+0000 7f5f609da700 1 -- start start 2023-01-24T03:30:32.164 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.160+0000 7f5f609da700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5f5c067d90 0x7f5f5c068160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:32.164 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.160+0000 7f5f609da700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f5f5c0c3f30 0x7f5f5c0c2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:32.164 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.160+0000 7f5f609da700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f5c0686a0 con 0x7f5f5c155bf0 2023-01-24T03:30:32.164 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.160+0000 7f5f609da700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f5c156070 con 0x7f5f5c0c3f30 2023-01-24T03:30:32.164 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.160+0000 7f5f609da700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5c0c5300 con 0x7f5f5c067d90 2023-01-24T03:30:32.165 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.160+0000 7f5f5bfff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f5f5c0c3f30 0x7f5f5c0c2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:32.165 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.160+0000 7f5f53fff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f5f5c155bf0 0x7f5f5c0c5790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:59468/0 (socket says 172.21.15.112:59468) 2023-01-24T03:30:32.165 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.160+0000 7f5f5bfff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f5f5c0c3f30 0x7f5f5c0c2c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50252/0 (socket says 172.21.15.112:50252) 2023-01-24T03:30:32.165 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.160+0000 7f5f53fff700 1 -- 172.21.15.112:0/3401207147 learned_addr learned my addr 172.21.15.112:0/3401207147 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:32.165 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.160+0000 7f5f5b7fe700 1 --2- 172.21.15.112:0/3401207147 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5f5c067d90 0x7f5f5c068160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:32.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.160+0000 7f5f5bfff700 1 -- 172.21.15.112:0/3401207147 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5f5c067d90 msgr2=0x7f5f5c068160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:32.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.160+0000 7f5f5bfff700 1 --2- 172.21.15.112:0/3401207147 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5f5c067d90 0x7f5f5c068160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:32.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.160+0000 7f5f5bfff700 1 -- 172.21.15.112:0/3401207147 >> v1:172.21.15.112:6789/0 conn(0x7f5f5c155bf0 legacy=0x7f5f5c0c5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:32.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.160+0000 7f5f5bfff700 1 -- 172.21.15.112:0/3401207147 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f5c0c5ea0 con 0x7f5f5c0c3f30 2023-01-24T03:30:32.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.160+0000 7f5f5affd700 1 -- 172.21.15.112:0/3401207147 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2751467658 0 0) 0x7f5f5c0686a0 con 0x7f5f5c155bf0 2023-01-24T03:30:32.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.160+0000 7f5f5bfff700 1 --2- 172.21.15.112:0/3401207147 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5f5c0c3f30 0x7f5f5c0c2c40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5f54009e60 tx=0x7f5f54009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12758641f9ace9aa server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:32.167 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.161+0000 7f5f5affd700 1 -- 172.21.15.112:0/3401207147 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5400e070 con 0x7f5f5c0c3f30 2023-01-24T03:30:32.167 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.161+0000 7f5f5affd700 1 -- 172.21.15.112:0/3401207147 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f5400bcc0 con 0x7f5f5c0c3f30 2023-01-24T03:30:32.167 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.161+0000 7f5f5affd700 1 -- 172.21.15.112:0/3401207147 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f54012680 con 0x7f5f5c0c3f30 2023-01-24T03:30:32.167 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.161+0000 7f5f609da700 1 -- 172.21.15.112:0/3401207147 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5f5c0c3f30 msgr2=0x7f5f5c0c2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:32.167 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.161+0000 7f5f609da700 1 --2- 172.21.15.112:0/3401207147 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5f5c0c3f30 0x7f5f5c0c2c40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5f54009e60 tx=0x7f5f54009780 comp rx=0 tx=0).stop 2023-01-24T03:30:32.168 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.161+0000 7f5f609da700 1 -- 172.21.15.112:0/3401207147 shutdown_connections 2023-01-24T03:30:32.168 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.161+0000 7f5f609da700 1 --2- 172.21.15.112:0/3401207147 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5f5c0c3f30 0x7f5f5c0c2c40 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:32.168 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.161+0000 7f5f609da700 1 --2- 172.21.15.112:0/3401207147 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5f5c067d90 0x7f5f5c068160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:32.168 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.161+0000 7f5f609da700 1 -- 172.21.15.112:0/3401207147 >> 172.21.15.112:0/3401207147 conn(0x7f5f5c0ca150 msgr2=0x7f5f5c0c6120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:32.168 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.161+0000 7f5f609da700 1 -- 172.21.15.112:0/3401207147 shutdown_connections 2023-01-24T03:30:32.169 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.161+0000 7f5f609da700 1 -- 172.21.15.112:0/3401207147 wait complete. 2023-01-24T03:30:32.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.162+0000 7f5f609da700 1 Processor -- start 2023-01-24T03:30:32.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.162+0000 7f5f609da700 1 -- start start 2023-01-24T03:30:32.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.163+0000 7f5f609da700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5f5c067d90 0x7f5f5c157510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:32.172 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.163+0000 7f5f609da700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5f5c0c3f30 0x7f5f5c157a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:32.172 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.163+0000 7f5f609da700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5f5c0c0cd0 0x7f5f5c0c1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:32.172 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.163+0000 7f5f609da700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5c0bfc60 con 0x7f5f5c0c0cd0 2023-01-24T03:30:32.172 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.163+0000 7f5f609da700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5c0c04e0 con 0x7f5f5c0c3f30 2023-01-24T03:30:32.172 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.163+0000 7f5f609da700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5c1659c0 con 0x7f5f5c067d90 2023-01-24T03:30:32.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.163+0000 7f5f5bfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5f5c0c0cd0 0x7f5f5c0c1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:32.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.163+0000 7f5f5bfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5f5c0c0cd0 0x7f5f5c0c1120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50258/0 (socket says 172.21.15.112:50258) 2023-01-24T03:30:32.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.163+0000 7f5f5bfff700 1 -- 172.21.15.112:0/3923248423 learned_addr learned my addr 172.21.15.112:0/3923248423 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:32.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.163+0000 7f5f53fff700 1 --2- 172.21.15.112:0/3923248423 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5f5c0c3f30 0x7f5f5c157a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:32.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.163+0000 7f5f5b7fe700 1 --2- 172.21.15.112:0/3923248423 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5f5c067d90 0x7f5f5c157510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:32.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.163+0000 7f5f5bfff700 1 -- 172.21.15.112:0/3923248423 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5f5c067d90 msgr2=0x7f5f5c157510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:32.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.163+0000 7f5f5bfff700 1 --2- 172.21.15.112:0/3923248423 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5f5c067d90 0x7f5f5c157510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:32.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.163+0000 7f5f5bfff700 1 -- 172.21.15.112:0/3923248423 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5f5c0c3f30 msgr2=0x7f5f5c157a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:32.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.163+0000 7f5f5bfff700 1 --2- 172.21.15.112:0/3923248423 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5f5c0c3f30 0x7f5f5c157a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:32.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.163+0000 7f5f5bfff700 1 -- 172.21.15.112:0/3923248423 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f54009a20 con 0x7f5f5c0c0cd0 2023-01-24T03:30:32.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.164+0000 7f5f5bfff700 1 --2- 172.21.15.112:0/3923248423 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5f5c0c0cd0 0x7f5f5c0c1120 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5f5400b6c0 tx=0x7f5f54019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:32.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.164+0000 7f5f537fe700 1 -- 172.21.15.112:0/3923248423 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f54027470 con 0x7f5f5c0c0cd0 2023-01-24T03:30:32.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.164+0000 7f5f537fe700 1 -- 172.21.15.112:0/3923248423 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f540275f0 con 0x7f5f5c0c0cd0 2023-01-24T03:30:32.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.164+0000 7f5f609da700 1 -- 172.21.15.112:0/3923248423 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f5c165c20 con 0x7f5f5c0c0cd0 2023-01-24T03:30:32.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.164+0000 7f5f537fe700 1 -- 172.21.15.112:0/3923248423 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f540279b0 con 0x7f5f5c0c0cd0 2023-01-24T03:30:32.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.164+0000 7f5f609da700 1 -- 172.21.15.112:0/3923248423 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f5c0bfe90 con 0x7f5f5c0c0cd0 2023-01-24T03:30:32.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.165+0000 7f5f609da700 1 -- 172.21.15.112:0/3923248423 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f5c0c09f0 con 0x7f5f5c0c0cd0 2023-01-24T03:30:32.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.165+0000 7f5f537fe700 1 -- 172.21.15.112:0/3923248423 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f5f5401d420 con 0x7f5f5c0c0cd0 2023-01-24T03:30:32.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.165+0000 7f5f537fe700 1 --2- 172.21.15.112:0/3923248423 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5f400611c0 0x7f5f40063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:32.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.165+0000 7f5f537fe700 1 -- 172.21.15.112:0/3923248423 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f5f54080770 con 0x7f5f5c0c0cd0 2023-01-24T03:30:32.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.165+0000 7f5f5b7fe700 1 --2- 172.21.15.112:0/3923248423 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5f400611c0 0x7f5f40063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:32.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.166+0000 7f5f5b7fe700 1 --2- 172.21.15.112:0/3923248423 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5f400611c0 0x7f5f40063670 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c00bbc0 tx=0x7f5f4c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:32.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.169+0000 7f5f537fe700 1 -- 172.21.15.112:0/3923248423 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f5f5c0c09f0 con 0x7f5f5c0c0cd0 2023-01-24T03:30:32.190 INFO:tasks.ceph.mds.d.smithi142.stderr:2023-01-24T03:30:32.188+0000 7f7053643700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-01-24T03:30:32.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.395+0000 7f5f609da700 1 -- 172.21.15.112:0/3923248423 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f5f5c0c09f0 con 0x7f5f5c0c0cd0 2023-01-24T03:30:32.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.395+0000 7f5f537fe700 1 -- 172.21.15.112:0/3923248423 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+88 (secure 0 0 0) 0x7f5f5c0c09f0 con 0x7f5f5c0c0cd0 2023-01-24T03:30:32.398 INFO:teuthology.orchestra.run.smithi112.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-01-24T03:30:32.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.397+0000 7f5f609da700 1 -- 172.21.15.112:0/3923248423 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5f400611c0 msgr2=0x7f5f40063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:32.400 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.397+0000 7f5f609da700 1 --2- 172.21.15.112:0/3923248423 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5f400611c0 0x7f5f40063670 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c00bbc0 tx=0x7f5f4c008040 comp rx=0 tx=0).stop 2023-01-24T03:30:32.400 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.397+0000 7f5f609da700 1 -- 172.21.15.112:0/3923248423 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5f5c0c0cd0 msgr2=0x7f5f5c0c1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:32.400 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.397+0000 7f5f609da700 1 --2- 172.21.15.112:0/3923248423 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5f5c0c0cd0 0x7f5f5c0c1120 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5f5400b6c0 tx=0x7f5f54019000 comp rx=0 tx=0).stop 2023-01-24T03:30:32.400 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.397+0000 7f5f609da700 1 -- 172.21.15.112:0/3923248423 shutdown_connections 2023-01-24T03:30:32.400 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.397+0000 7f5f609da700 1 --2- 172.21.15.112:0/3923248423 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5f400611c0 0x7f5f40063670 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:32.401 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.397+0000 7f5f609da700 1 --2- 172.21.15.112:0/3923248423 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5f5c0c0cd0 0x7f5f5c0c1120 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:32.401 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.397+0000 7f5f609da700 1 --2- 172.21.15.112:0/3923248423 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5f5c067d90 0x7f5f5c157510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:32.401 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.397+0000 7f5f609da700 1 --2- 172.21.15.112:0/3923248423 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5f5c0c3f30 0x7f5f5c157a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:32.401 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.398+0000 7f5f609da700 1 -- 172.21.15.112:0/3923248423 >> 172.21.15.112:0/3923248423 conn(0x7f5f5c0ca150 msgr2=0x7f5f5c0c6120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:32.401 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.398+0000 7f5f609da700 1 -- 172.21.15.112:0/3923248423 shutdown_connections 2023-01-24T03:30:32.402 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:32.398+0000 7f5f609da700 1 -- 172.21.15.112:0/3923248423 wait complete. 2023-01-24T03:30:32.411 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-01-24T03:30:32.411 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-01-24T03:30:32.412 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi112.front.sepia.ceph.com 2023-01-24T03:30:32.412 INFO:tasks.cephfs.mount:self.client.name = client.admin 2023-01-24T03:30:32.412 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-01-24T03:30:32.413 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-01-24T03:30:32.413 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-01-24T03:30:32.413 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-01-24T03:30:32.414 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2023-01-24T03:30:32.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-01-24T03:30:32.414 DEBUG:teuthology.orchestra.run.smithi112:> ip addr 2023-01-24T03:30:32.429 INFO:teuthology.orchestra.run.smithi112.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-01-24T03:30:32.429 INFO:teuthology.orchestra.run.smithi112.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-01-24T03:30:32.430 INFO:teuthology.orchestra.run.smithi112.stdout: inet 127.0.0.1/8 scope host lo 2023-01-24T03:30:32.430 INFO:teuthology.orchestra.run.smithi112.stdout: valid_lft forever preferred_lft forever 2023-01-24T03:30:32.430 INFO:teuthology.orchestra.run.smithi112.stdout: inet6 ::1/128 scope host 2023-01-24T03:30:32.430 INFO:teuthology.orchestra.run.smithi112.stdout: valid_lft forever preferred_lft forever 2023-01-24T03:30:32.431 INFO:teuthology.orchestra.run.smithi112.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-01-24T03:30:32.431 INFO:teuthology.orchestra.run.smithi112.stdout: link/ether 0c:c4:7a:d8:a5:2c brd ff:ff:ff:ff:ff:ff 2023-01-24T03:30:32.431 INFO:teuthology.orchestra.run.smithi112.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-01-24T03:30:32.431 INFO:teuthology.orchestra.run.smithi112.stdout: link/ether 0c:c4:7a:d8:a5:2d brd ff:ff:ff:ff:ff:ff 2023-01-24T03:30:32.431 INFO:teuthology.orchestra.run.smithi112.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-01-24T03:30:32.432 INFO:teuthology.orchestra.run.smithi112.stdout: link/ether 0c:c4:7a:8f:cb:68 brd ff:ff:ff:ff:ff:ff 2023-01-24T03:30:32.432 INFO:teuthology.orchestra.run.smithi112.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-01-24T03:30:32.432 INFO:teuthology.orchestra.run.smithi112.stdout: link/ether 0c:c4:7a:8f:cb:69 brd ff:ff:ff:ff:ff:ff 2023-01-24T03:30:32.432 INFO:teuthology.orchestra.run.smithi112.stdout: inet 172.21.15.112/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-01-24T03:30:32.433 INFO:teuthology.orchestra.run.smithi112.stdout: valid_lft 42606sec preferred_lft 42606sec 2023-01-24T03:30:32.433 INFO:teuthology.orchestra.run.smithi112.stdout: inet6 fe80::ec4:7aff:fe8f:cb69/64 scope link 2023-01-24T03:30:32.433 INFO:teuthology.orchestra.run.smithi112.stdout: valid_lft forever preferred_lft forever 2023-01-24T03:30:32.433 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-01-24T03:30:32.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-01-24T03:30:32.434 DEBUG:teuthology.orchestra.run.smithi112:> (cd / && exec bash -c ' 2023-01-24T03:30:32.435 DEBUG:teuthology.orchestra.run.smithi112:> set -e 2023-01-24T03:30:32.435 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip link add name ceph-brx type bridge 2023-01-24T03:30:32.435 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip addr flush dev ceph-brx 2023-01-24T03:30:32.436 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip link set ceph-brx up 2023-01-24T03:30:32.436 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-01-24T03:30:32.437 DEBUG:teuthology.orchestra.run.smithi112:> ') 2023-01-24T03:30:32.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-01-24T03:30:32.584 DEBUG:teuthology.orchestra.run.smithi112:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-01-24T03:30:32.609 INFO:teuthology.orchestra.run.smithi112.stdout:1 2023-01-24T03:30:32.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-01-24T03:30:32.610 DEBUG:teuthology.orchestra.run.smithi112:> route 2023-01-24T03:30:32.691 INFO:teuthology.orchestra.run.smithi112.stdout:Kernel IP routing table 2023-01-24T03:30:32.691 INFO:teuthology.orchestra.run.smithi112.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-01-24T03:30:32.691 INFO:teuthology.orchestra.run.smithi112.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-01-24T03:30:32.692 INFO:teuthology.orchestra.run.smithi112.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-01-24T03:30:32.692 INFO:teuthology.orchestra.run.smithi112.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-01-24T03:30:32.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-01-24T03:30:32.693 DEBUG:teuthology.orchestra.run.smithi112:> (cd / && exec bash -c ' 2023-01-24T03:30:32.693 DEBUG:teuthology.orchestra.run.smithi112:> set -e 2023-01-24T03:30:32.694 DEBUG:teuthology.orchestra.run.smithi112:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-01-24T03:30:32.695 DEBUG:teuthology.orchestra.run.smithi112:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-01-24T03:30:32.695 DEBUG:teuthology.orchestra.run.smithi112:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-01-24T03:30:32.695 DEBUG:teuthology.orchestra.run.smithi112:> ') 2023-01-24T03:30:32.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-01-24T03:30:32.915 DEBUG:teuthology.orchestra.run.smithi112:> ip netns list 2023-01-24T03:30:32.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-01-24T03:30:32.930 DEBUG:teuthology.orchestra.run.smithi112:> ip netns list-id 2023-01-24T03:30:32.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-01-24T03:30:32.984 DEBUG:teuthology.orchestra.run.smithi112:> (cd / && exec bash -c ' 2023-01-24T03:30:32.985 DEBUG:teuthology.orchestra.run.smithi112:> set -e 2023-01-24T03:30:32.985 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-01-24T03:30:32.985 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2023-01-24T03:30:32.986 DEBUG:teuthology.orchestra.run.smithi112:> ') 2023-01-24T03:30:33.071 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2023-01-24T03:30:33.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-01-24T03:30:33.072 DEBUG:teuthology.orchestra.run.smithi112:> (cd / && exec bash -c ' 2023-01-24T03:30:33.072 DEBUG:teuthology.orchestra.run.smithi112:> set -e 2023-01-24T03:30:33.072 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2023-01-24T03:30:33.073 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-01-24T03:30:33.073 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2023-01-24T03:30:33.074 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2023-01-24T03:30:33.074 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2023-01-24T03:30:33.074 DEBUG:teuthology.orchestra.run.smithi112:> ') 2023-01-24T03:30:33.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-01-24T03:30:33.249 DEBUG:teuthology.orchestra.run.smithi112:> (cd / && exec bash -c ' 2023-01-24T03:30:33.249 DEBUG:teuthology.orchestra.run.smithi112:> set -e 2023-01-24T03:30:33.250 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip link set brx.0 up 2023-01-24T03:30:33.250 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip link set dev brx.0 master ceph-brx 2023-01-24T03:30:33.250 DEBUG:teuthology.orchestra.run.smithi112:> ') 2023-01-24T03:30:33.340 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2023-01-24T03:30:33.341 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-01-24T03:30:33.341 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2023-01-24T03:30:33.355 INFO:teuthology.orchestra.run.smithi112.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2023-01-24T03:30:33.355 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-01-24T03:30:33.356 DEBUG:teuthology.orchestra.run.smithi112:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2023-01-24T03:30:33.409 DEBUG:teuthology.orchestra.run.smithi112:> sudo modprobe fuse 2023-01-24T03:30:33.473 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-01-24T03:30:33.474 DEBUG:teuthology.orchestra.run.smithi112:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-01-24T03:30:33.536 INFO:teuthology.orchestra.run.smithi112.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-01-24T03:30:33.537 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-01-24T03:30:33.538 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-01-24T03:30:33.539 DEBUG:teuthology.orchestra.run.smithi112:> ls /sys/fs/fuse/connections 2023-01-24T03:30:33.593 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-01-24T03:30:33.594 DEBUG:teuthology.orchestra.run.smithi112:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2023-01-24T03:30:33.636 DEBUG:teuthology.orchestra.run.smithi112:> sudo modprobe fuse 2023-01-24T03:30:33.659 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-01-24T03:30:33.660 DEBUG:teuthology.orchestra.run.smithi112:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-01-24T03:30:33.721 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi112.stderr:2023-01-24T03:30:33.719+0000 7f58c8d95380 -1 init, newargv = 0x560d4b7c9260 newargc=15 2023-01-24T03:30:33.724 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi112.stderr:ceph-fuse[90415]: starting ceph client 2023-01-24T03:30:33.725 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-01-24T03:30:33.726 INFO:teuthology.orchestra.run.smithi112.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-01-24T03:30:33.726 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-01-24T03:30:33.727 DEBUG:teuthology.orchestra.run.smithi112:> ls /sys/fs/fuse/connections 2023-01-24T03:30:33.728 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi112.stderr:ceph-fuse[90415]: starting fuse 2023-01-24T03:30:33.741 INFO:teuthology.orchestra.run.smithi112.stdout:47 2023-01-24T03:30:33.741 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [47] 2023-01-24T03:30:33.741 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-01-24T03:30:33.742 DEBUG:teuthology.orchestra.run.smithi112:> import glob 2023-01-24T03:30:33.742 DEBUG:teuthology.orchestra.run.smithi112:> import re 2023-01-24T03:30:33.742 DEBUG:teuthology.orchestra.run.smithi112:> import os 2023-01-24T03:30:33.743 DEBUG:teuthology.orchestra.run.smithi112:> import subprocess 2023-01-24T03:30:33.743 DEBUG:teuthology.orchestra.run.smithi112:> 2023-01-24T03:30:33.743 DEBUG:teuthology.orchestra.run.smithi112:> def _find_admin_socket(client_name): 2023-01-24T03:30:33.744 DEBUG:teuthology.orchestra.run.smithi112:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2023-01-24T03:30:33.744 DEBUG:teuthology.orchestra.run.smithi112:> files = glob.glob(asok_path) 2023-01-24T03:30:33.744 DEBUG:teuthology.orchestra.run.smithi112:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2023-01-24T03:30:33.745 DEBUG:teuthology.orchestra.run.smithi112:> 2023-01-24T03:30:33.745 DEBUG:teuthology.orchestra.run.smithi112:> # Given a non-glob path, it better be there 2023-01-24T03:30:33.745 DEBUG:teuthology.orchestra.run.smithi112:> if "*" not in asok_path: 2023-01-24T03:30:33.746 DEBUG:teuthology.orchestra.run.smithi112:> assert(len(files) == 1) 2023-01-24T03:30:33.746 DEBUG:teuthology.orchestra.run.smithi112:> return files[0] 2023-01-24T03:30:33.747 DEBUG:teuthology.orchestra.run.smithi112:> 2023-01-24T03:30:33.747 DEBUG:teuthology.orchestra.run.smithi112:> for f in files: 2023-01-24T03:30:33.747 DEBUG:teuthology.orchestra.run.smithi112:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-01-24T03:30:33.748 DEBUG:teuthology.orchestra.run.smithi112:> if os.path.exists("/proc/{0}".format(pid)): 2023-01-24T03:30:33.748 DEBUG:teuthology.orchestra.run.smithi112:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-01-24T03:30:33.748 DEBUG:teuthology.orchestra.run.smithi112:> contents = proc_f.read() 2023-01-24T03:30:33.749 DEBUG:teuthology.orchestra.run.smithi112:> if mountpoint in contents: 2023-01-24T03:30:33.749 DEBUG:teuthology.orchestra.run.smithi112:> return f 2023-01-24T03:30:33.749 DEBUG:teuthology.orchestra.run.smithi112:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-01-24T03:30:33.750 DEBUG:teuthology.orchestra.run.smithi112:> 2023-01-24T03:30:33.750 DEBUG:teuthology.orchestra.run.smithi112:> print(_find_admin_socket("client.admin")) 2023-01-24T03:30:33.750 DEBUG:teuthology.orchestra.run.smithi112:> ' 2023-01-24T03:30:33.859 INFO:teuthology.orchestra.run.smithi112.stdout:/var/run/ceph/ceph-client.admin.90415.asok 2023-01-24T03:30:34.041 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.90415.asok 2023-01-24T03:30:34.042 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-01-24T03:30:34.042 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.90415.asok status 2023-01-24T03:30:34.189 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-01-24T03:30:34.190 INFO:teuthology.orchestra.run.smithi112.stdout: "metadata": { 2023-01-24T03:30:34.190 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_sha1": "8387d427d91a301da19bd82e090b96e1348bfadb", 2023-01-24T03:30:34.190 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_version": "ceph version 17.2.5-701-g8387d427 (8387d427d91a301da19bd82e090b96e1348bfadb) quincy (stable)", 2023-01-24T03:30:34.190 INFO:teuthology.orchestra.run.smithi112.stdout: "entity_id": "admin", 2023-01-24T03:30:34.190 INFO:teuthology.orchestra.run.smithi112.stdout: "hostname": "smithi112", 2023-01-24T03:30:34.191 INFO:teuthology.orchestra.run.smithi112.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2023-01-24T03:30:34.191 INFO:teuthology.orchestra.run.smithi112.stdout: "pid": "90415", 2023-01-24T03:30:34.191 INFO:teuthology.orchestra.run.smithi112.stdout: "root": "/" 2023-01-24T03:30:34.191 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-01-24T03:30:34.191 INFO:teuthology.orchestra.run.smithi112.stdout: "dentry_count": 0, 2023-01-24T03:30:34.192 INFO:teuthology.orchestra.run.smithi112.stdout: "dentry_pinned_count": 0, 2023-01-24T03:30:34.192 INFO:teuthology.orchestra.run.smithi112.stdout: "id": 4470, 2023-01-24T03:30:34.192 INFO:teuthology.orchestra.run.smithi112.stdout: "inst": { 2023-01-24T03:30:34.192 INFO:teuthology.orchestra.run.smithi112.stdout: "name": { 2023-01-24T03:30:34.192 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "client", 2023-01-24T03:30:34.193 INFO:teuthology.orchestra.run.smithi112.stdout: "num": 4470 2023-01-24T03:30:34.193 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-01-24T03:30:34.193 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": { 2023-01-24T03:30:34.193 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2023-01-24T03:30:34.193 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "192.168.0.1:0", 2023-01-24T03:30:34.194 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 2968948095 2023-01-24T03:30:34.194 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-01-24T03:30:34.194 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-01-24T03:30:34.194 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": { 2023-01-24T03:30:34.194 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "v1", 2023-01-24T03:30:34.195 INFO:teuthology.orchestra.run.smithi112.stdout: "addr": "192.168.0.1:0", 2023-01-24T03:30:34.195 INFO:teuthology.orchestra.run.smithi112.stdout: "nonce": 2968948095 2023-01-24T03:30:34.195 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-01-24T03:30:34.195 INFO:teuthology.orchestra.run.smithi112.stdout: "inst_str": "client.4470 192.168.0.1:0/2968948095", 2023-01-24T03:30:34.195 INFO:teuthology.orchestra.run.smithi112.stdout: "addr_str": "192.168.0.1:0/2968948095", 2023-01-24T03:30:34.196 INFO:teuthology.orchestra.run.smithi112.stdout: "inode_count": 1, 2023-01-24T03:30:34.196 INFO:teuthology.orchestra.run.smithi112.stdout: "mds_epoch": 6, 2023-01-24T03:30:34.196 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_epoch": 24, 2023-01-24T03:30:34.196 INFO:teuthology.orchestra.run.smithi112.stdout: "osd_epoch_barrier": 0, 2023-01-24T03:30:34.196 INFO:teuthology.orchestra.run.smithi112.stdout: "blocklisted": false, 2023-01-24T03:30:34.196 INFO:teuthology.orchestra.run.smithi112.stdout: "fs_name": "cephfs" 2023-01-24T03:30:34.197 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-01-24T03:30:34.199 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-01-24T03:30:34.199 DEBUG:teuthology.orchestra.run.smithi112:> stat --file-system '--printf=%T 2023-01-24T03:30:34.199 DEBUG:teuthology.orchestra.run.smithi112:> ' -- /home/ubuntu/cephtest/mnt.admin 2023-01-24T03:30:34.214 INFO:teuthology.orchestra.run.smithi112.stdout:fuseblk 2023-01-24T03:30:34.214 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2023-01-24T03:30:34.215 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-01-24T03:30:34.215 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2023-01-24T03:30:34.279 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-01-24T03:30:34.280 DEBUG:teuthology.orchestra.run.smithi112:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2023-01-24T03:30:34.380 INFO:teuthology.orchestra.run.smithi112.stdout:# file: . 2023-01-24T03:30:34.381 INFO:teuthology.orchestra.run.smithi112.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2023-01-24T03:30:34.381 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:30:34.381 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi112.front.sepia.ceph.com... 2023-01-24T03:30:34.382 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-01-24T03:30:34.382 DEBUG:teuthology.orchestra.run.smithi112:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2023-01-24T03:30:34.416 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi112.stderr:ceph-fuse[90415]: fuse finished with error 0 and tester_r 0 2023-01-24T03:30:34.419 INFO:teuthology.orchestra.run:waiting for 900 2023-01-24T03:30:34.512 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi112.front.sepia.ceph.com 2023-01-24T03:30:34.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-01-24T03:30:34.513 DEBUG:teuthology.orchestra.run.smithi112:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2023-01-24T03:30:34.529 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T03:30:34.679 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.677+0000 7f589c591700 1 Processor -- start 2023-01-24T03:30:34.680 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.677+0000 7f589c591700 1 -- start start 2023-01-24T03:30:34.682 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.678+0000 7f589c591700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f589414d5d0 0x7f5894153400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:34.683 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.678+0000 7f589c591700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58941566c0 con 0x7f58940c3080 2023-01-24T03:30:34.683 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.678+0000 7f589c591700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58941569a0 con 0x7f58940c15b0 2023-01-24T03:30:34.683 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.678+0000 7f589c591700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f589405e650 con 0x7f589414d5d0 2023-01-24T03:30:34.683 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.678+0000 7f589a32d700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f58940c3080 0x7f589405df40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:59498/0 (socket says 172.21.15.112:59498) 2023-01-24T03:30:34.684 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.678+0000 7f589a32d700 1 -- 172.21.15.112:0/672463264 learned_addr learned my addr 172.21.15.112:0/672463264 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:34.684 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.678+0000 7f589ab2e700 1 --2- 172.21.15.112:0/672463264 >> v2:172.21.15.112:3300/0 conn(0x7f589414d5d0 0x7f5894153400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:34.684 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.679+0000 7f589932b700 1 -- 172.21.15.112:0/672463264 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1409569975 0 0) 0x7f58941566c0 con 0x7f58940c3080 2023-01-24T03:30:34.684 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.679+0000 7f589932b700 1 -- 172.21.15.112:0/672463264 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58800033f0 con 0x7f58940c3080 2023-01-24T03:30:34.684 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.679+0000 7f589ab2e700 1 -- 172.21.15.112:0/672463264 >> v1:172.21.15.142:6789/0 conn(0x7f58940c15b0 legacy=0x7f589414cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:34.685 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.679+0000 7f589ab2e700 1 -- 172.21.15.112:0/672463264 >> v1:172.21.15.112:6789/0 conn(0x7f58940c3080 legacy=0x7f589405df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:34.685 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.679+0000 7f589ab2e700 1 -- 172.21.15.112:0/672463264 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5894156c80 con 0x7f589414d5d0 2023-01-24T03:30:34.685 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.679+0000 7f589ab2e700 1 --2- 172.21.15.112:0/672463264 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f589414d5d0 0x7f5894153400 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f588c009760 tx=0x7f588c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff21b9fe01f09542 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:34.685 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.679+0000 7f589932b700 1 -- 172.21.15.112:0/672463264 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f588c011630 con 0x7f589414d5d0 2023-01-24T03:30:34.685 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.679+0000 7f589932b700 1 -- 172.21.15.112:0/672463264 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f588c011790 con 0x7f589414d5d0 2023-01-24T03:30:34.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.679+0000 7f589932b700 1 -- 172.21.15.112:0/672463264 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f588c011b10 con 0x7f589414d5d0 2023-01-24T03:30:34.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.679+0000 7f589c591700 1 -- 172.21.15.112:0/672463264 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f589414d5d0 msgr2=0x7f5894153400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:34.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.679+0000 7f589c591700 1 --2- 172.21.15.112:0/672463264 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f589414d5d0 0x7f5894153400 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f588c009760 tx=0x7f588c009060 comp rx=0 tx=0).stop 2023-01-24T03:30:34.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.680+0000 7f589c591700 1 -- 172.21.15.112:0/672463264 shutdown_connections 2023-01-24T03:30:34.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.680+0000 7f589c591700 1 --2- 172.21.15.112:0/672463264 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f589414d5d0 0x7f5894153400 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:34.687 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.680+0000 7f589c591700 1 -- 172.21.15.112:0/672463264 >> 172.21.15.112:0/672463264 conn(0x7f589406daf0 msgr2=0x7f589405d830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:34.687 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.680+0000 7f589c591700 1 -- 172.21.15.112:0/672463264 shutdown_connections 2023-01-24T03:30:34.687 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.680+0000 7f589c591700 1 -- 172.21.15.112:0/672463264 wait complete. 2023-01-24T03:30:34.687 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.681+0000 7f589c591700 1 Processor -- start 2023-01-24T03:30:34.687 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.681+0000 7f589c591700 1 -- start start 2023-01-24T03:30:34.690 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.681+0000 7f589c591700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58940c15b0 0x7f5894147b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:34.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.681+0000 7f589c591700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f58940c3080 0x7f58940bdac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:34.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.681+0000 7f589c591700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f589414d5d0 0x7f58940be000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:34.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.681+0000 7f589c591700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58941630e0 con 0x7f58940c3080 2023-01-24T03:30:34.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.681+0000 7f589c591700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58941633c0 con 0x7f58940c15b0 2023-01-24T03:30:34.692 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.681+0000 7f589c591700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58941636a0 con 0x7f589414d5d0 2023-01-24T03:30:34.692 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.681+0000 7f5899b2c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f58940c3080 0x7f58940bdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:34.692 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.681+0000 7f5899b2c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f58940c3080 0x7f58940bdac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50286/0 (socket says 172.21.15.112:50286) 2023-01-24T03:30:34.692 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.681+0000 7f5899b2c700 1 -- 172.21.15.112:0/2130792814 learned_addr learned my addr 172.21.15.112:0/2130792814 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:34.692 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.682+0000 7f589ab2e700 1 --2- 172.21.15.112:0/2130792814 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f589414d5d0 0x7f58940be000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:34.693 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.682+0000 7f589a32d700 1 --2- 172.21.15.112:0/2130792814 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58940c15b0 0x7f5894147b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:34.693 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.682+0000 7f5899b2c700 1 -- 172.21.15.112:0/2130792814 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f589414d5d0 msgr2=0x7f58940be000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:34.693 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.682+0000 7f5899b2c700 1 --2- 172.21.15.112:0/2130792814 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f589414d5d0 0x7f58940be000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:34.693 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.682+0000 7f5899b2c700 1 -- 172.21.15.112:0/2130792814 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58940c15b0 msgr2=0x7f5894147b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:34.693 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.682+0000 7f5899b2c700 1 --2- 172.21.15.112:0/2130792814 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58940c15b0 0x7f5894147b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:34.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.682+0000 7f5899b2c700 1 -- 172.21.15.112:0/2130792814 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f588c0092a0 con 0x7f58940c3080 2023-01-24T03:30:34.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.682+0000 7f5899b2c700 1 --2- 172.21.15.112:0/2130792814 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f58940c3080 0x7f58940bdac0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f588800b490 tx=0x7f5888006ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:34.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.682+0000 7f58927fc700 1 -- 172.21.15.112:0/2130792814 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f588800f510 con 0x7f58940c3080 2023-01-24T03:30:34.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.682+0000 7f58927fc700 1 -- 172.21.15.112:0/2130792814 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f588800f670 con 0x7f58940c3080 2023-01-24T03:30:34.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.682+0000 7f589c591700 1 -- 172.21.15.112:0/2130792814 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5894163980 con 0x7f58940c3080 2023-01-24T03:30:34.695 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.682+0000 7f58927fc700 1 -- 172.21.15.112:0/2130792814 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f588800fa30 con 0x7f58940c3080 2023-01-24T03:30:34.695 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.682+0000 7f589c591700 1 -- 172.21.15.112:0/2130792814 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58941499d0 con 0x7f58940c3080 2023-01-24T03:30:34.695 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.683+0000 7f589c591700 1 -- 172.21.15.112:0/2130792814 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f589404bb80 con 0x7f58940c3080 2023-01-24T03:30:34.695 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.683+0000 7f58927fc700 1 -- 172.21.15.112:0/2130792814 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f588800f070 con 0x7f58940c3080 2023-01-24T03:30:34.695 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.684+0000 7f58927fc700 1 --2- 172.21.15.112:0/2130792814 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f58740611f0 0x7f58740636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:34.696 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.684+0000 7f58927fc700 1 -- 172.21.15.112:0/2130792814 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f588807d0f0 con 0x7f58940c3080 2023-01-24T03:30:34.696 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.684+0000 7f589a32d700 1 --2- 172.21.15.112:0/2130792814 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f58740611f0 0x7f58740636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:34.696 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.684+0000 7f589a32d700 1 --2- 172.21.15.112:0/2130792814 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f58740611f0 0x7f58740636a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5884005ad0 tx=0x7f5884009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:34.696 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.688+0000 7f58927fc700 1 -- 172.21.15.112:0/2130792814 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f588804cfb0 con 0x7f58940c3080 2023-01-24T03:30:34.870 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.869+0000 7f589c591700 1 -- 172.21.15.112:0/2130792814 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58940c0ef0 con 0x7f58940c3080 2023-01-24T03:30:34.871 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.869+0000 7f58927fc700 1 -- 172.21.15.112:0/2130792814 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18755 (secure 0 0 0) 0x7f58880505d0 con 0x7f58940c3080 2023-01-24T03:30:34.872 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:30:34.872 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":24,"fsid":"054b9ac9-b7b0-40e5-b932-6c8a955c5a72","created":"2023-01-24T03:30:03.798533+0000","modified":"2023-01-24T03:30:25.373340+0000","last_up_change":"2023-01-24T03:30:13.945316+0000","last_in_change":"2023-01-24T03:30:06.198433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-01-24T03:30:14.931166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-01-24T03:30:15.023730+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-01-24T03:30:19.060597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-01-24T03:30:19.943686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-01-24T03:30:22.999036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"fe6ad231-a39a-42eb-9a3a-49d008dd35e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6809","nonce":84512},{"type":"v1","addr":"172.21.15.112:6811","nonce":84512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":84512},{"type":"v1","addr":"172.21.15.112:6817","nonce":84512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6825","nonce":84512},{"type":"v1","addr":"172.21.15.112:6828","nonce":84512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6819","nonce":84512},{"type":"v1","addr":"172.21.15.112:6822","nonce":84512}]},"public_addr":"172.21.15.112:6811/84512","cluster_addr":"172.21.15.112:6817/84512","heartbeat_back_addr":"172.21.15.112:6828/84512","heartbeat_front_addr":"172.21.15.112:6822/84512","state":["exists","up"]},{"osd":1,"uuid":"8526a1d2-d540-4021-9766-c4bbb30317ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":84513},{"type":"v1","addr":"172.21.15.112:6812","nonce":84513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6815","nonce":84513},{"type":"v1","addr":"172.21.15.112:6818","nonce":84513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6827","nonce":84513},{"type":"v1","addr":"172.21.15.112:6830","nonce":84513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6821","nonce":84513},{"type":"v1","addr":"172.21.15.112:6824","nonce":84513}]},"public_addr":"172.21.15.112:6812/84513","cluster_addr":"172.21.15.112:6818/84513","heartbeat_back_addr":"172.21.15.112:6830/84513","heartbeat_front_addr":"172.21.15.112:6824/84513","state":["exists","up"]},{"osd":2,"uuid":"041ef8e4-4858-432d-9476-cbdb3a0dc47c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":84511},{"type":"v1","addr":"172.21.15.112:6813","nonce":84511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":84511},{"type":"v1","addr":"172.21.15.112:6820","nonce":84511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6829","nonce":84511},{"type":"v1","addr":"172.21.15.112:6831","nonce":84511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6823","nonce":84511},{"type":"v1","addr":"172.21.15.112:6826","nonce":84511}]},"public_addr":"172.21.15.112:6813/84511","cluster_addr":"172.21.15.112:6820/84511","heartbeat_back_addr":"172.21.15.112:6831/84511","heartbeat_front_addr":"172.21.15.112:6826/84511","state":["exists","up"]},{"osd":3,"uuid":"987587cf-1357-4a70-ad85-67b189c67729","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":84510},{"type":"v1","addr":"172.21.15.112:6801","nonce":84510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":84510},{"type":"v1","addr":"172.21.15.112:6803","nonce":84510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":84510},{"type":"v1","addr":"172.21.15.112:6807","nonce":84510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":84510},{"type":"v1","addr":"172.21.15.112:6805","nonce":84510}]},"public_addr":"172.21.15.112:6801/84510","cluster_addr":"172.21.15.112:6803/84510","heartbeat_back_addr":"172.21.15.112:6807/84510","heartbeat_front_addr":"172.21.15.112:6805/84510","state":["exists","up"]},{"osd":4,"uuid":"8503512b-43db-40b8-8b2b-c86cf2e54e17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":84614},{"type":"v1","addr":"172.21.15.142:6801","nonce":84614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":84614},{"type":"v1","addr":"172.21.15.142:6803","nonce":84614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":84614},{"type":"v1","addr":"172.21.15.142:6807","nonce":84614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":84614},{"type":"v1","addr":"172.21.15.142:6805","nonce":84614}]},"public_addr":"172.21.15.142:6801/84614","cluster_addr":"172.21.15.142:6803/84614","heartbeat_back_addr":"172.21.15.142:6807/84614","heartbeat_front_addr":"172.21.15.142:6805/84614","state":["exists","up"]},{"osd":5,"uuid":"d3bc0030-8b08-4645-9ec2-1f8a4e29ef36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":84638},{"type":"v1","addr":"172.21.15.142:6817","nonce":84638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":84638},{"type":"v1","addr":"172.21.15.142:6819","nonce":84638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":84638},{"type":"v1","addr":"172.21.15.142:6823","nonce":84638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":84638},{"type":"v1","addr":"172.21.15.142:6821","nonce":84638}]},"public_addr":"172.21.15.142:6817/84638","cluster_addr":"172.21.15.142:6819/84638","heartbeat_back_addr":"172.21.15.142:6823/84638","heartbeat_front_addr":"172.21.15.142:6821/84638","state":["exists","up"]},{"osd":6,"uuid":"c7649e3b-832d-4e6f-8866-b15e5d814476","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":84645},{"type":"v1","addr":"172.21.15.142:6825","nonce":84645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":84645},{"type":"v1","addr":"172.21.15.142:6827","nonce":84645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":84645},{"type":"v1","addr":"172.21.15.142:6831","nonce":84645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":84645},{"type":"v1","addr":"172.21.15.142:6829","nonce":84645}]},"public_addr":"172.21.15.142:6825/84645","cluster_addr":"172.21.15.142:6827/84645","heartbeat_back_addr":"172.21.15.142:6831/84645","heartbeat_front_addr":"172.21.15.142:6829/84645","state":["exists","up"]},{"osd":7,"uuid":"2ea5b3bf-884e-4b20-b572-1443efdc7477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":84615},{"type":"v1","addr":"172.21.15.142:6809","nonce":84615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":84615},{"type":"v1","addr":"172.21.15.142:6811","nonce":84615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":84615},{"type":"v1","addr":"172.21.15.142:6815","nonce":84615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":84615},{"type":"v1","addr":"172.21.15.142:6813","nonce":84615}]},"public_addr":"172.21.15.142:6809/84615","cluster_addr":"172.21.15.142:6811/84615","heartbeat_back_addr":"172.21.15.142:6815/84615","heartbeat_front_addr":"172.21.15.142:6813/84615","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.370027+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.335683+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.350667+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.350794+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.391119+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.422933+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.400439+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.354018+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T03:30:34.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.871+0000 7f589c591700 1 -- 172.21.15.112:0/2130792814 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f58740611f0 msgr2=0x7f58740636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:34.874 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.871+0000 7f589c591700 1 --2- 172.21.15.112:0/2130792814 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f58740611f0 0x7f58740636a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5884005ad0 tx=0x7f5884009040 comp rx=0 tx=0).stop 2023-01-24T03:30:34.874 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.871+0000 7f589c591700 1 -- 172.21.15.112:0/2130792814 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f58940c3080 msgr2=0x7f58940bdac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:34.874 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.871+0000 7f589c591700 1 --2- 172.21.15.112:0/2130792814 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f58940c3080 0x7f58940bdac0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f588800b490 tx=0x7f5888006ef0 comp rx=0 tx=0).stop 2023-01-24T03:30:34.874 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.871+0000 7f589c591700 1 -- 172.21.15.112:0/2130792814 shutdown_connections 2023-01-24T03:30:34.874 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.871+0000 7f589c591700 1 --2- 172.21.15.112:0/2130792814 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f58740611f0 0x7f58740636a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:34.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.871+0000 7f589c591700 1 --2- 172.21.15.112:0/2130792814 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f589414d5d0 0x7f58940be000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:34.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.871+0000 7f589c591700 1 --2- 172.21.15.112:0/2130792814 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f58940c15b0 0x7f5894147b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:34.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.871+0000 7f589c591700 1 --2- 172.21.15.112:0/2130792814 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f58940c3080 0x7f58940bdac0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:34.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.871+0000 7f589c591700 1 -- 172.21.15.112:0/2130792814 >> 172.21.15.112:0/2130792814 conn(0x7f589406daf0 msgr2=0x7f5894153400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:34.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.871+0000 7f589c591700 1 -- 172.21.15.112:0/2130792814 shutdown_connections 2023-01-24T03:30:34.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.871+0000 7f589c591700 1 -- 172.21.15.112:0/2130792814 wait complete. 2023-01-24T03:30:34.885 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T03:30:34.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.995+0000 7f174ea76700 1 Processor -- start 2023-01-24T03:30:34.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.995+0000 7f174ea76700 1 -- start start 2023-01-24T03:30:35.000 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.996+0000 7f174ea76700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f17480c5680 0x7f1748156730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:35.000 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.996+0000 7f174ea76700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1748155e20 0x7f1748156c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:35.001 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.996+0000 7f174ea76700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17480c3d10 con 0x7f17480c5a50 2023-01-24T03:30:35.001 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.996+0000 7f174ea76700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17480c4eb0 con 0x7f17480c5680 2023-01-24T03:30:35.001 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.996+0000 7f174ea76700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17480c2e20 con 0x7f1748155e20 2023-01-24T03:30:35.001 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.996+0000 7f174da74700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f17480c5680 0x7f1748156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:35.001 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.996+0000 7f174da74700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f17480c5680 0x7f1748156730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50292/0 (socket says 172.21.15.112:50292) 2023-01-24T03:30:35.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.996+0000 7f174da74700 1 -- 172.21.15.112:0/3497233868 learned_addr learned my addr 172.21.15.112:0/3497233868 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:35.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.996+0000 7f174d273700 1 --2- 172.21.15.112:0/3497233868 >> v2:172.21.15.142:3300/0 conn(0x7f1748155e20 0x7f1748156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:35.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.996+0000 7f174da74700 1 -- 172.21.15.112:0/3497233868 >> v2:172.21.15.142:3300/0 conn(0x7f1748155e20 msgr2=0x7f1748156c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:35.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.996+0000 7f174da74700 1 --2- 172.21.15.112:0/3497233868 >> v2:172.21.15.142:3300/0 conn(0x7f1748155e20 0x7f1748156c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:35.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.996+0000 7f174da74700 1 -- 172.21.15.112:0/3497233868 >> v1:172.21.15.142:6789/0 conn(0x7f17480c5a50 legacy=0x7f17480cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:35.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.996+0000 7f174da74700 1 -- 172.21.15.112:0/3497233868 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17480c3100 con 0x7f17480c5680 2023-01-24T03:30:35.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.996+0000 7f174d273700 1 --2- 172.21.15.112:0/3497233868 >> v2:172.21.15.142:3300/0 conn(0x7f1748155e20 0x7f1748156c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-01-24T03:30:35.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.996+0000 7f174da74700 1 --2- 172.21.15.112:0/3497233868 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17480c5680 0x7f1748156730 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1748152dc0 tx=0x7f17440090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=874c927d4b1aa575 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:35.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.997+0000 7f174ca72700 1 -- 172.21.15.112:0/3497233868 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1744011630 con 0x7f17480c5680 2023-01-24T03:30:35.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.997+0000 7f174ca72700 1 -- 172.21.15.112:0/3497233868 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17440117b0 con 0x7f17480c5680 2023-01-24T03:30:35.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.997+0000 7f174ca72700 1 -- 172.21.15.112:0/3497233868 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1744011b70 con 0x7f17480c5680 2023-01-24T03:30:35.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.997+0000 7f174ea76700 1 -- 172.21.15.112:0/3497233868 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17480c5680 msgr2=0x7f1748156730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:35.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.997+0000 7f174ea76700 1 --2- 172.21.15.112:0/3497233868 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17480c5680 0x7f1748156730 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1748152dc0 tx=0x7f17440090f0 comp rx=0 tx=0).stop 2023-01-24T03:30:35.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.997+0000 7f174ea76700 1 -- 172.21.15.112:0/3497233868 shutdown_connections 2023-01-24T03:30:35.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.997+0000 7f174ea76700 1 --2- 172.21.15.112:0/3497233868 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17480c5680 0x7f1748156730 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:35.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.997+0000 7f174ea76700 1 --2- 172.21.15.112:0/3497233868 >> v2:172.21.15.142:3300/0 conn(0x7f1748155e20 0x7f1748156c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:35.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.997+0000 7f174ea76700 1 -- 172.21.15.112:0/3497233868 >> 172.21.15.112:0/3497233868 conn(0x7f17480685b0 msgr2=0x7f17480689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:35.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.997+0000 7f174ea76700 1 -- 172.21.15.112:0/3497233868 shutdown_connections 2023-01-24T03:30:35.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.997+0000 7f174ea76700 1 -- 172.21.15.112:0/3497233868 wait complete. 2023-01-24T03:30:35.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.998+0000 7f174ea76700 1 Processor -- start 2023-01-24T03:30:35.006 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.998+0000 7f174ea76700 1 -- start start 2023-01-24T03:30:35.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.999+0000 7f174ea76700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f17480c5680 0x7f174814abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:35.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.999+0000 7f174ea76700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17480c5a50 0x7f174814b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:35.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.999+0000 7f174ea76700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1748155e20 0x7f17481689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:35.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.999+0000 7f174ea76700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17480c8b30 con 0x7f17480c5a50 2023-01-24T03:30:35.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.999+0000 7f174ea76700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1748169580 con 0x7f17480c5680 2023-01-24T03:30:35.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.999+0000 7f174ea76700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17480c8dd0 con 0x7f1748155e20 2023-01-24T03:30:35.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.999+0000 7f174d273700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17480c5a50 0x7f174814b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:35.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.999+0000 7f174d273700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17480c5a50 0x7f174814b110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50296/0 (socket says 172.21.15.112:50296) 2023-01-24T03:30:35.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.999+0000 7f174d273700 1 -- 172.21.15.112:0/3385893939 learned_addr learned my addr 172.21.15.112:0/3385893939 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:35.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.999+0000 7f174da74700 1 --2- 172.21.15.112:0/3385893939 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f17480c5680 0x7f174814abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:35.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.999+0000 7f174e275700 1 --2- 172.21.15.112:0/3385893939 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1748155e20 0x7f17481689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:35.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.999+0000 7f174d273700 1 -- 172.21.15.112:0/3385893939 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1748155e20 msgr2=0x7f17481689e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:35.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.999+0000 7f174d273700 1 --2- 172.21.15.112:0/3385893939 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1748155e20 0x7f17481689e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:35.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.999+0000 7f174d273700 1 -- 172.21.15.112:0/3385893939 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f17480c5680 msgr2=0x7f174814abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:35.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.999+0000 7f174d273700 1 --2- 172.21.15.112:0/3385893939 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f17480c5680 0x7f174814abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:35.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:34.999+0000 7f174d273700 1 -- 172.21.15.112:0/3385893939 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1744009370 con 0x7f17480c5a50 2023-01-24T03:30:35.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.000+0000 7f174d273700 1 --2- 172.21.15.112:0/3385893939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17480c5a50 0x7f174814b110 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f173800c880 tx=0x7f1738008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:35.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.000+0000 7f1735ffb700 1 -- 172.21.15.112:0/3385893939 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1738011600 con 0x7f17480c5a50 2023-01-24T03:30:35.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.000+0000 7f1735ffb700 1 -- 172.21.15.112:0/3385893939 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1738011760 con 0x7f17480c5a50 2023-01-24T03:30:35.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.000+0000 7f174ea76700 1 -- 172.21.15.112:0/3385893939 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17480c90b0 con 0x7f17480c5a50 2023-01-24T03:30:35.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.000+0000 7f1735ffb700 1 -- 172.21.15.112:0/3385893939 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1738011b20 con 0x7f17480c5a50 2023-01-24T03:30:35.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.000+0000 7f174ea76700 1 -- 172.21.15.112:0/3385893939 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17480a9e30 con 0x7f17480c5a50 2023-01-24T03:30:35.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.001+0000 7f174ea76700 1 -- 172.21.15.112:0/3385893939 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f174805ea20 con 0x7f17480c5a50 2023-01-24T03:30:35.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.001+0000 7f1735ffb700 1 -- 172.21.15.112:0/3385893939 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f1738021070 con 0x7f17480c5a50 2023-01-24T03:30:35.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.001+0000 7f1735ffb700 1 --2- 172.21.15.112:0/3385893939 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f17280611c0 0x7f1728063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:35.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.002+0000 7f1735ffb700 1 -- 172.21.15.112:0/3385893939 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f173807d4d0 con 0x7f17480c5a50 2023-01-24T03:30:35.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.002+0000 7f174da74700 1 --2- 172.21.15.112:0/3385893939 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f17280611c0 0x7f1728063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:35.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.002+0000 7f174da74700 1 --2- 172.21.15.112:0/3385893939 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f17280611c0 0x7f1728063670 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1748152dc0 tx=0x7f1744014000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:35.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.006+0000 7f1735ffb700 1 -- 172.21.15.112:0/3385893939 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f173804d310 con 0x7f17480c5a50 2023-01-24T03:30:35.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.184+0000 7f174ea76700 1 -- 172.21.15.112:0/3385893939 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17480c86e0 con 0x7f17480c5a50 2023-01-24T03:30:35.187 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.185+0000 7f1735ffb700 1 -- 172.21.15.112:0/3385893939 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18755 (secure 0 0 0) 0x7f1738015090 con 0x7f17480c5a50 2023-01-24T03:30:35.187 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:30:35.188 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":24,"fsid":"054b9ac9-b7b0-40e5-b932-6c8a955c5a72","created":"2023-01-24T03:30:03.798533+0000","modified":"2023-01-24T03:30:25.373340+0000","last_up_change":"2023-01-24T03:30:13.945316+0000","last_in_change":"2023-01-24T03:30:06.198433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-01-24T03:30:14.931166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-01-24T03:30:15.023730+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-01-24T03:30:19.060597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-01-24T03:30:19.943686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-01-24T03:30:22.999036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"fe6ad231-a39a-42eb-9a3a-49d008dd35e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6809","nonce":84512},{"type":"v1","addr":"172.21.15.112:6811","nonce":84512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":84512},{"type":"v1","addr":"172.21.15.112:6817","nonce":84512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6825","nonce":84512},{"type":"v1","addr":"172.21.15.112:6828","nonce":84512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6819","nonce":84512},{"type":"v1","addr":"172.21.15.112:6822","nonce":84512}]},"public_addr":"172.21.15.112:6811/84512","cluster_addr":"172.21.15.112:6817/84512","heartbeat_back_addr":"172.21.15.112:6828/84512","heartbeat_front_addr":"172.21.15.112:6822/84512","state":["exists","up"]},{"osd":1,"uuid":"8526a1d2-d540-4021-9766-c4bbb30317ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":84513},{"type":"v1","addr":"172.21.15.112:6812","nonce":84513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6815","nonce":84513},{"type":"v1","addr":"172.21.15.112:6818","nonce":84513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6827","nonce":84513},{"type":"v1","addr":"172.21.15.112:6830","nonce":84513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6821","nonce":84513},{"type":"v1","addr":"172.21.15.112:6824","nonce":84513}]},"public_addr":"172.21.15.112:6812/84513","cluster_addr":"172.21.15.112:6818/84513","heartbeat_back_addr":"172.21.15.112:6830/84513","heartbeat_front_addr":"172.21.15.112:6824/84513","state":["exists","up"]},{"osd":2,"uuid":"041ef8e4-4858-432d-9476-cbdb3a0dc47c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":84511},{"type":"v1","addr":"172.21.15.112:6813","nonce":84511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":84511},{"type":"v1","addr":"172.21.15.112:6820","nonce":84511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6829","nonce":84511},{"type":"v1","addr":"172.21.15.112:6831","nonce":84511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6823","nonce":84511},{"type":"v1","addr":"172.21.15.112:6826","nonce":84511}]},"public_addr":"172.21.15.112:6813/84511","cluster_addr":"172.21.15.112:6820/84511","heartbeat_back_addr":"172.21.15.112:6831/84511","heartbeat_front_addr":"172.21.15.112:6826/84511","state":["exists","up"]},{"osd":3,"uuid":"987587cf-1357-4a70-ad85-67b189c67729","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":84510},{"type":"v1","addr":"172.21.15.112:6801","nonce":84510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":84510},{"type":"v1","addr":"172.21.15.112:6803","nonce":84510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":84510},{"type":"v1","addr":"172.21.15.112:6807","nonce":84510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":84510},{"type":"v1","addr":"172.21.15.112:6805","nonce":84510}]},"public_addr":"172.21.15.112:6801/84510","cluster_addr":"172.21.15.112:6803/84510","heartbeat_back_addr":"172.21.15.112:6807/84510","heartbeat_front_addr":"172.21.15.112:6805/84510","state":["exists","up"]},{"osd":4,"uuid":"8503512b-43db-40b8-8b2b-c86cf2e54e17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":84614},{"type":"v1","addr":"172.21.15.142:6801","nonce":84614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":84614},{"type":"v1","addr":"172.21.15.142:6803","nonce":84614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":84614},{"type":"v1","addr":"172.21.15.142:6807","nonce":84614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":84614},{"type":"v1","addr":"172.21.15.142:6805","nonce":84614}]},"public_addr":"172.21.15.142:6801/84614","cluster_addr":"172.21.15.142:6803/84614","heartbeat_back_addr":"172.21.15.142:6807/84614","heartbeat_front_addr":"172.21.15.142:6805/84614","state":["exists","up"]},{"osd":5,"uuid":"d3bc0030-8b08-4645-9ec2-1f8a4e29ef36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":84638},{"type":"v1","addr":"172.21.15.142:6817","nonce":84638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":84638},{"type":"v1","addr":"172.21.15.142:6819","nonce":84638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":84638},{"type":"v1","addr":"172.21.15.142:6823","nonce":84638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":84638},{"type":"v1","addr":"172.21.15.142:6821","nonce":84638}]},"public_addr":"172.21.15.142:6817/84638","cluster_addr":"172.21.15.142:6819/84638","heartbeat_back_addr":"172.21.15.142:6823/84638","heartbeat_front_addr":"172.21.15.142:6821/84638","state":["exists","up"]},{"osd":6,"uuid":"c7649e3b-832d-4e6f-8866-b15e5d814476","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":84645},{"type":"v1","addr":"172.21.15.142:6825","nonce":84645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":84645},{"type":"v1","addr":"172.21.15.142:6827","nonce":84645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":84645},{"type":"v1","addr":"172.21.15.142:6831","nonce":84645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":84645},{"type":"v1","addr":"172.21.15.142:6829","nonce":84645}]},"public_addr":"172.21.15.142:6825/84645","cluster_addr":"172.21.15.142:6827/84645","heartbeat_back_addr":"172.21.15.142:6831/84645","heartbeat_front_addr":"172.21.15.142:6829/84645","state":["exists","up"]},{"osd":7,"uuid":"2ea5b3bf-884e-4b20-b572-1443efdc7477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":84615},{"type":"v1","addr":"172.21.15.142:6809","nonce":84615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":84615},{"type":"v1","addr":"172.21.15.142:6811","nonce":84615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":84615},{"type":"v1","addr":"172.21.15.142:6815","nonce":84615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":84615},{"type":"v1","addr":"172.21.15.142:6813","nonce":84615}]},"public_addr":"172.21.15.142:6809/84615","cluster_addr":"172.21.15.142:6811/84615","heartbeat_back_addr":"172.21.15.142:6815/84615","heartbeat_front_addr":"172.21.15.142:6813/84615","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.370027+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.335683+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.350667+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.350794+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.391119+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.422933+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.400439+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.354018+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T03:30:35.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.187+0000 7f174ea76700 1 -- 172.21.15.112:0/3385893939 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f17280611c0 msgr2=0x7f1728063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:35.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.187+0000 7f174ea76700 1 --2- 172.21.15.112:0/3385893939 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f17280611c0 0x7f1728063670 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1748152dc0 tx=0x7f1744014000 comp rx=0 tx=0).stop 2023-01-24T03:30:35.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.187+0000 7f174ea76700 1 -- 172.21.15.112:0/3385893939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17480c5a50 msgr2=0x7f174814b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:35.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.187+0000 7f174ea76700 1 --2- 172.21.15.112:0/3385893939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17480c5a50 0x7f174814b110 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f173800c880 tx=0x7f1738008040 comp rx=0 tx=0).stop 2023-01-24T03:30:35.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.187+0000 7f174ea76700 1 -- 172.21.15.112:0/3385893939 shutdown_connections 2023-01-24T03:30:35.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.187+0000 7f174ea76700 1 --2- 172.21.15.112:0/3385893939 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f17280611c0 0x7f1728063670 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:35.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.187+0000 7f174ea76700 1 --2- 172.21.15.112:0/3385893939 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1748155e20 0x7f17481689e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:35.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.187+0000 7f174ea76700 1 --2- 172.21.15.112:0/3385893939 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f17480c5680 0x7f174814abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:35.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.187+0000 7f174ea76700 1 --2- 172.21.15.112:0/3385893939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f17480c5a50 0x7f174814b110 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:35.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.187+0000 7f174ea76700 1 -- 172.21.15.112:0/3385893939 >> 172.21.15.112:0/3385893939 conn(0x7f17480685b0 msgr2=0x7f174814e470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:35.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.187+0000 7f174ea76700 1 -- 172.21.15.112:0/3385893939 shutdown_connections 2023-01-24T03:30:35.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.187+0000 7f174ea76700 1 -- 172.21.15.112:0/3385893939 wait complete. 2023-01-24T03:30:35.201 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-01-24T03:30:35.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.315+0000 7fe4d3083700 1 Processor -- start 2023-01-24T03:30:35.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.315+0000 7fe4d3083700 1 -- start start 2023-01-24T03:30:35.324 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.316+0000 7fe4d3083700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe4cc0694c0 0x7fe4cc069890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:35.324 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.316+0000 7fe4d3083700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4cc155ab0 con 0x7fe4cc068d20 2023-01-24T03:30:35.324 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.316+0000 7fe4d3083700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4cc0c70e0 con 0x7fe4cc05e420 2023-01-24T03:30:35.324 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.316+0000 7fe4d3083700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4cc155550 con 0x7fe4cc0694c0 2023-01-24T03:30:35.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.316+0000 7fe4d1620700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fe4cc068d20 0x7fe4cc0ae9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:59526/0 (socket says 172.21.15.112:59526) 2023-01-24T03:30:35.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.316+0000 7fe4d1620700 1 -- 172.21.15.112:0/1631104954 learned_addr learned my addr 172.21.15.112:0/1631104954 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:35.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.316+0000 7fe4d0e1f700 1 --2- 172.21.15.112:0/1631104954 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe4cc0694c0 0x7fe4cc069890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:35.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.316+0000 7fe4cb7fe700 1 -- 172.21.15.112:0/1631104954 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2483212483 0 0) 0x7fe4cc155ab0 con 0x7fe4cc068d20 2023-01-24T03:30:35.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.316+0000 7fe4cb7fe700 1 -- 172.21.15.112:0/1631104954 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4b80033f0 con 0x7fe4cc068d20 2023-01-24T03:30:35.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.316+0000 7fe4cb7fe700 1 -- 172.21.15.112:0/1631104954 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3269082007 0 0) 0x7fe4b80033f0 con 0x7fe4cc068d20 2023-01-24T03:30:35.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.316+0000 7fe4cb7fe700 1 -- 172.21.15.112:0/1631104954 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe4cc0694c0 msgr2=0x7fe4cc069890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:35.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.316+0000 7fe4cb7fe700 1 --2- 172.21.15.112:0/1631104954 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe4cc0694c0 0x7fe4cc069890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:35.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.316+0000 7fe4cb7fe700 1 -- 172.21.15.112:0/1631104954 >> v1:172.21.15.142:6789/0 conn(0x7fe4cc05e420 legacy=0x7fe4cc0c68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:35.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.316+0000 7fe4cb7fe700 1 -- 172.21.15.112:0/1631104954 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4cc0c73c0 con 0x7fe4cc068d20 2023-01-24T03:30:35.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.316+0000 7fe4cb7fe700 1 -- 172.21.15.112:0/1631104954 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7fe4c40017f0 con 0x7fe4cc068d20 2023-01-24T03:30:35.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.316+0000 7fe4cb7fe700 1 -- 172.21.15.112:0/1631104954 >> v1:172.21.15.112:6789/0 conn(0x7fe4cc068d20 legacy=0x7fe4cc0ae9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:35.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.316+0000 7fe4cb7fe700 1 --2- 172.21.15.112:0/1631104954 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe4b8003910 0x7fe4b8003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:35.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.316+0000 7fe4cb7fe700 1 --2- 172.21.15.112:0/1631104954 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4b8004690 0x7fe4b8004b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:35.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.316+0000 7fe4cb7fe700 1 --2- 172.21.15.112:0/1631104954 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe4cc0694c0 0x7fe4b8005040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:35.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.316+0000 7fe4cb7fe700 1 -- 172.21.15.112:0/1631104954 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b800b990 con 0x7fe4b8003910 2023-01-24T03:30:35.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.316+0000 7fe4cb7fe700 1 -- 172.21.15.112:0/1631104954 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b800bad0 con 0x7fe4b8004690 2023-01-24T03:30:35.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.316+0000 7fe4cb7fe700 1 -- 172.21.15.112:0/1631104954 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b800bc90 con 0x7fe4cc0694c0 2023-01-24T03:30:35.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.316+0000 7fe4d0e1f700 1 --2- 172.21.15.112:0/1631104954 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe4b8003910 0x7fe4b8003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:35.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.317+0000 7fe4cbfff700 1 --2- 172.21.15.112:0/1631104954 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4b8004690 0x7fe4b8004b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:35.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.317+0000 7fe4d1620700 1 --2- 172.21.15.112:0/1631104954 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe4cc0694c0 0x7fe4b8005040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:35.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.317+0000 7fe4d0e1f700 1 -- 172.21.15.112:0/1631104954 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe4cc0694c0 msgr2=0x7fe4b8005040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:35.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.317+0000 7fe4d0e1f700 1 --2- 172.21.15.112:0/1631104954 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe4cc0694c0 0x7fe4b8005040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:35.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.317+0000 7fe4d0e1f700 1 -- 172.21.15.112:0/1631104954 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4b8004690 msgr2=0x7fe4b8004b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:35.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.317+0000 7fe4d0e1f700 1 --2- 172.21.15.112:0/1631104954 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4b8004690 0x7fe4b8004b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:35.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.317+0000 7fe4d0e1f700 1 -- 172.21.15.112:0/1631104954 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b80033f0 con 0x7fe4b8003910 2023-01-24T03:30:35.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.317+0000 7fe4cbfff700 1 --2- 172.21.15.112:0/1631104954 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4b8004690 0x7fe4b8004b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:30:35.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.317+0000 7fe4d0e1f700 1 --2- 172.21.15.112:0/1631104954 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe4b8003910 0x7fe4b8003ce0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc009a80 tx=0x7fe4bc009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=412905f1d3e795be server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:35.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.317+0000 7fe4cb7fe700 1 -- 172.21.15.112:0/1631104954 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc009430 con 0x7fe4b8003910 2023-01-24T03:30:35.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.317+0000 7fe4cb7fe700 1 -- 172.21.15.112:0/1631104954 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4bc00e640 con 0x7fe4b8003910 2023-01-24T03:30:35.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.317+0000 7fe4cb7fe700 1 -- 172.21.15.112:0/1631104954 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc00ea00 con 0x7fe4b8003910 2023-01-24T03:30:35.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.317+0000 7fe4d3083700 1 -- 172.21.15.112:0/1631104954 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe4b8003910 msgr2=0x7fe4b8003ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:35.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.317+0000 7fe4d3083700 1 --2- 172.21.15.112:0/1631104954 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe4b8003910 0x7fe4b8003ce0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc009a80 tx=0x7fe4bc009060 comp rx=0 tx=0).stop 2023-01-24T03:30:35.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.317+0000 7fe4d1620700 1 -- 172.21.15.112:0/1631104954 reap_dead start 2023-01-24T03:30:35.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.317+0000 7fe4d3083700 1 -- 172.21.15.112:0/1631104954 shutdown_connections 2023-01-24T03:30:35.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.317+0000 7fe4d3083700 1 -- 172.21.15.112:0/1631104954 >> 172.21.15.112:0/1631104954 conn(0x7fe4cc05d830 msgr2=0x7fe4cc05dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:35.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.317+0000 7fe4d3083700 1 -- 172.21.15.112:0/1631104954 shutdown_connections 2023-01-24T03:30:35.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.318+0000 7fe4d3083700 1 -- 172.21.15.112:0/1631104954 wait complete. 2023-01-24T03:30:35.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.318+0000 7fe4d3083700 1 Processor -- start 2023-01-24T03:30:35.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.319+0000 7fe4d3083700 1 -- start start 2023-01-24T03:30:35.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.319+0000 7fe4d3083700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe4b8003910 0x7fe4cc0c3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:35.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.319+0000 7fe4d3083700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe4cc0af1a0 0x7fe4cc0c4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:35.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.319+0000 7fe4d3083700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4cc0c4c90 0x7fe4cc0c50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:35.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.319+0000 7fe4d3083700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4cc1568e0 con 0x7fe4cc0af1a0 2023-01-24T03:30:35.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.319+0000 7fe4d3083700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4cc151a20 con 0x7fe4cc0c4c90 2023-01-24T03:30:35.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.319+0000 7fe4d3083700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4cc151d00 con 0x7fe4b8003910 2023-01-24T03:30:35.335 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.319+0000 7fe4cbfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe4cc0af1a0 0x7fe4cc0c4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:35.335 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.319+0000 7fe4cbfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe4cc0af1a0 0x7fe4cc0c4310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50316/0 (socket says 172.21.15.112:50316) 2023-01-24T03:30:35.335 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.319+0000 7fe4cbfff700 1 -- 172.21.15.112:0/73591874 learned_addr learned my addr 172.21.15.112:0/73591874 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:35.335 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.320+0000 7fe4d0e1f700 1 --2- 172.21.15.112:0/73591874 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe4b8003910 0x7fe4cc0c3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:35.335 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.320+0000 7fe4d1620700 1 --2- 172.21.15.112:0/73591874 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4cc0c4c90 0x7fe4cc0c50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:35.336 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.320+0000 7fe4cbfff700 1 -- 172.21.15.112:0/73591874 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe4b8003910 msgr2=0x7fe4cc0c3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:35.336 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.320+0000 7fe4cbfff700 1 --2- 172.21.15.112:0/73591874 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe4b8003910 0x7fe4cc0c3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:35.336 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.320+0000 7fe4cbfff700 1 -- 172.21.15.112:0/73591874 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4cc0c4c90 msgr2=0x7fe4cc0c50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:35.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.320+0000 7fe4cbfff700 1 --2- 172.21.15.112:0/73591874 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4cc0c4c90 0x7fe4cc0c50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:35.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.320+0000 7fe4cbfff700 1 -- 172.21.15.112:0/73591874 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4cc151fe0 con 0x7fe4cc0af1a0 2023-01-24T03:30:35.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.320+0000 7fe4cbfff700 1 --2- 172.21.15.112:0/73591874 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe4cc0af1a0 0x7fe4cc0c4310 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe4c0007720 tx=0x7fe4c0009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:35.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.320+0000 7fe4c8ff9700 1 -- 172.21.15.112:0/73591874 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c0010520 con 0x7fe4cc0af1a0 2023-01-24T03:30:35.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.320+0000 7fe4c8ff9700 1 -- 172.21.15.112:0/73591874 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4c00106a0 con 0x7fe4cc0af1a0 2023-01-24T03:30:35.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.320+0000 7fe4d3083700 1 -- 172.21.15.112:0/73591874 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4cc1522c0 con 0x7fe4cc0af1a0 2023-01-24T03:30:35.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.320+0000 7fe4c8ff9700 1 -- 172.21.15.112:0/73591874 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c0010a60 con 0x7fe4cc0af1a0 2023-01-24T03:30:35.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.320+0000 7fe4d3083700 1 -- 172.21.15.112:0/73591874 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4cc0c5620 con 0x7fe4cc0af1a0 2023-01-24T03:30:35.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.321+0000 7fe4d3083700 1 -- 172.21.15.112:0/73591874 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4cc06f3a0 con 0x7fe4cc0af1a0 2023-01-24T03:30:35.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.321+0000 7fe4c8ff9700 1 -- 172.21.15.112:0/73591874 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fe4c001b030 con 0x7fe4cc0af1a0 2023-01-24T03:30:35.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.322+0000 7fe4c8ff9700 1 --2- 172.21.15.112:0/73591874 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fe4ac0611f0 0x7fe4ac0636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:35.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.322+0000 7fe4c8ff9700 1 -- 172.21.15.112:0/73591874 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fe4c007d070 con 0x7fe4cc0af1a0 2023-01-24T03:30:35.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.322+0000 7fe4d0e1f700 1 --2- 172.21.15.112:0/73591874 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fe4ac0611f0 0x7fe4ac0636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:35.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.322+0000 7fe4d0e1f700 1 --2- 172.21.15.112:0/73591874 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fe4ac0611f0 0x7fe4ac0636a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc006d50 tx=0x7fe4bc01a000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:35.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.326+0000 7fe4c8ff9700 1 -- 172.21.15.112:0/73591874 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fe4c004c1d0 con 0x7fe4cc0af1a0 2023-01-24T03:30:35.548 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.547+0000 7fe4d3083700 1 -- 172.21.15.112:0/73591874 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe4cc0cca90 con 0x7fe4cc0af1a0 2023-01-24T03:30:35.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.596+0000 7fe4c8ff9700 1 -- 172.21.15.112:0/73591874 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v8) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe4c00190b0 con 0x7fe4cc0af1a0 2023-01-24T03:30:35.600 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.598+0000 7fe4d3083700 1 -- 172.21.15.112:0/73591874 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fe4ac0611f0 msgr2=0x7fe4ac0636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:35.600 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.598+0000 7fe4d3083700 1 --2- 172.21.15.112:0/73591874 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fe4ac0611f0 0x7fe4ac0636a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc006d50 tx=0x7fe4bc01a000 comp rx=0 tx=0).stop 2023-01-24T03:30:35.600 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.598+0000 7fe4d3083700 1 -- 172.21.15.112:0/73591874 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe4cc0af1a0 msgr2=0x7fe4cc0c4310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:35.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.598+0000 7fe4d3083700 1 --2- 172.21.15.112:0/73591874 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe4cc0af1a0 0x7fe4cc0c4310 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe4c0007720 tx=0x7fe4c0009040 comp rx=0 tx=0).stop 2023-01-24T03:30:35.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.598+0000 7fe4d3083700 1 -- 172.21.15.112:0/73591874 shutdown_connections 2023-01-24T03:30:35.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.598+0000 7fe4d3083700 1 --2- 172.21.15.112:0/73591874 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fe4ac0611f0 0x7fe4ac0636a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:35.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.598+0000 7fe4d3083700 1 --2- 172.21.15.112:0/73591874 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe4cc0c4c90 0x7fe4cc0c50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:35.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.598+0000 7fe4d3083700 1 --2- 172.21.15.112:0/73591874 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe4b8003910 0x7fe4cc0c3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:35.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.598+0000 7fe4d3083700 1 --2- 172.21.15.112:0/73591874 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe4cc0af1a0 0x7fe4cc0c4310 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:35.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.598+0000 7fe4d3083700 1 -- 172.21.15.112:0/73591874 >> 172.21.15.112:0/73591874 conn(0x7fe4cc05d830 msgr2=0x7fe4cc14f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:35.603 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.598+0000 7fe4d3083700 1 -- 172.21.15.112:0/73591874 shutdown_connections 2023-01-24T03:30:35.603 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.598+0000 7fe4d3083700 1 -- 172.21.15.112:0/73591874 wait complete. 2023-01-24T03:30:35.611 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-01-24T03:30:35.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.724+0000 7f3bcda0e700 1 Processor -- start 2023-01-24T03:30:35.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.724+0000 7f3bcda0e700 1 -- start start 2023-01-24T03:30:35.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.725+0000 7f3bcda0e700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f3bc80694c0 0x7f3bc8069890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:35.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.725+0000 7f3bcda0e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f3bc805e420 0x7f3bc8155f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:35.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.725+0000 7f3bcda0e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bc8155ab0 con 0x7f3bc8068d20 2023-01-24T03:30:35.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.725+0000 7f3bcda0e700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bc8155550 con 0x7f3bc80694c0 2023-01-24T03:30:35.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.725+0000 7f3bcda0e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bc8156480 con 0x7f3bc805e420 2023-01-24T03:30:35.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.725+0000 7f3bc6ffd700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f3bc80694c0 0x7f3bc8069890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:35.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.725+0000 7f3bc77fe700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f3bc8068d20 0x7f3bc80c68d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:59548/0 (socket says 172.21.15.112:59548) 2023-01-24T03:30:35.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.725+0000 7f3bc77fe700 1 -- 172.21.15.112:0/2225356487 learned_addr learned my addr 172.21.15.112:0/2225356487 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:35.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.725+0000 7f3bc67fc700 1 --2- 172.21.15.112:0/2225356487 >> v2:172.21.15.142:3300/0 conn(0x7f3bc805e420 0x7f3bc8155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:35.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.725+0000 7f3bc5ffb700 1 -- 172.21.15.112:0/2225356487 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2678082561 0 0) 0x7f3bc8155ab0 con 0x7f3bc8068d20 2023-01-24T03:30:35.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.725+0000 7f3bc5ffb700 1 -- 172.21.15.112:0/2225356487 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ba80033f0 con 0x7f3bc8068d20 2023-01-24T03:30:35.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.725+0000 7f3bc6ffd700 1 -- 172.21.15.112:0/2225356487 >> v2:172.21.15.142:3300/0 conn(0x7f3bc805e420 msgr2=0x7f3bc8155f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:35.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.725+0000 7f3bc6ffd700 1 --2- 172.21.15.112:0/2225356487 >> v2:172.21.15.142:3300/0 conn(0x7f3bc805e420 0x7f3bc8155f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:35.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.725+0000 7f3bc6ffd700 1 -- 172.21.15.112:0/2225356487 >> v1:172.21.15.112:6789/0 conn(0x7f3bc8068d20 legacy=0x7f3bc80c68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:35.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.725+0000 7f3bc6ffd700 1 -- 172.21.15.112:0/2225356487 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bc80c70e0 con 0x7f3bc80694c0 2023-01-24T03:30:35.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.726+0000 7f3bc6ffd700 1 --2- 172.21.15.112:0/2225356487 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3bc80694c0 0x7f3bc8069890 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3bc80ae770 tx=0x7f3bb8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f154938509f37d20 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:35.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.726+0000 7f3bc5ffb700 1 -- 172.21.15.112:0/2225356487 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb800e070 con 0x7f3bc80694c0 2023-01-24T03:30:35.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.726+0000 7f3bc5ffb700 1 -- 172.21.15.112:0/2225356487 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bb800bb60 con 0x7f3bc80694c0 2023-01-24T03:30:35.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.726+0000 7f3bc5ffb700 1 -- 172.21.15.112:0/2225356487 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb8012430 con 0x7f3bc80694c0 2023-01-24T03:30:35.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.726+0000 7f3bcda0e700 1 -- 172.21.15.112:0/2225356487 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3bc80694c0 msgr2=0x7f3bc8069890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:35.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.726+0000 7f3bcda0e700 1 --2- 172.21.15.112:0/2225356487 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3bc80694c0 0x7f3bc8069890 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3bc80ae770 tx=0x7f3bb8009780 comp rx=0 tx=0).stop 2023-01-24T03:30:35.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.726+0000 7f3bcda0e700 1 -- 172.21.15.112:0/2225356487 shutdown_connections 2023-01-24T03:30:35.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.726+0000 7f3bcda0e700 1 --2- 172.21.15.112:0/2225356487 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3bc80694c0 0x7f3bc8069890 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:35.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.726+0000 7f3bcda0e700 1 --2- 172.21.15.112:0/2225356487 >> v2:172.21.15.142:3300/0 conn(0x7f3bc805e420 0x7f3bc8155f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:35.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.726+0000 7f3bcda0e700 1 -- 172.21.15.112:0/2225356487 >> 172.21.15.112:0/2225356487 conn(0x7f3bc805d830 msgr2=0x7f3bc805dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:35.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.726+0000 7f3bcda0e700 1 -- 172.21.15.112:0/2225356487 shutdown_connections 2023-01-24T03:30:35.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.726+0000 7f3bcda0e700 1 -- 172.21.15.112:0/2225356487 wait complete. 2023-01-24T03:30:35.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.727+0000 7f3bcda0e700 1 Processor -- start 2023-01-24T03:30:35.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.727+0000 7f3bcda0e700 1 -- start start 2023-01-24T03:30:35.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.728+0000 7f3bcda0e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3bc805e420 0x7f3bc8152580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:35.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.728+0000 7f3bcda0e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3bc8068d20 0x7f3bc80c09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:35.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.728+0000 7f3bcda0e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3bc80694c0 0x7f3bc814df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:35.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.728+0000 7f3bcda0e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc8165a50 con 0x7f3bc8068d20 2023-01-24T03:30:35.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.728+0000 7f3bcda0e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc80c2300 con 0x7f3bc805e420 2023-01-24T03:30:35.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.728+0000 7f3bcda0e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc80c1de0 con 0x7f3bc80694c0 2023-01-24T03:30:35.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.728+0000 7f3bc67fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3bc8068d20 0x7f3bc80c09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:35.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.728+0000 7f3bc67fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3bc8068d20 0x7f3bc80c09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50332/0 (socket says 172.21.15.112:50332) 2023-01-24T03:30:35.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.728+0000 7f3bc67fc700 1 -- 172.21.15.112:0/2247950950 learned_addr learned my addr 172.21.15.112:0/2247950950 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:35.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.728+0000 7f3bc6ffd700 1 --2- 172.21.15.112:0/2247950950 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3bc805e420 0x7f3bc8152580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:35.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.728+0000 7f3bc77fe700 1 --2- 172.21.15.112:0/2247950950 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3bc80694c0 0x7f3bc814df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:35.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.728+0000 7f3bc67fc700 1 -- 172.21.15.112:0/2247950950 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3bc80694c0 msgr2=0x7f3bc814df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:35.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.729+0000 7f3bc67fc700 1 --2- 172.21.15.112:0/2247950950 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3bc80694c0 0x7f3bc814df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:35.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.729+0000 7f3bc67fc700 1 -- 172.21.15.112:0/2247950950 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3bc805e420 msgr2=0x7f3bc8152580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:35.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.729+0000 7f3bc67fc700 1 --2- 172.21.15.112:0/2247950950 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3bc805e420 0x7f3bc8152580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:35.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.729+0000 7f3bc67fc700 1 -- 172.21.15.112:0/2247950950 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb8009a20 con 0x7f3bc8068d20 2023-01-24T03:30:35.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.729+0000 7f3bc6ffd700 1 --2- 172.21.15.112:0/2247950950 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3bc805e420 0x7f3bc8152580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-01-24T03:30:35.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.729+0000 7f3bc77fe700 1 --2- 172.21.15.112:0/2247950950 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3bc80694c0 0x7f3bc814df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-01-24T03:30:35.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.729+0000 7f3bc67fc700 1 --2- 172.21.15.112:0/2247950950 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3bc8068d20 0x7f3bc80c09d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3bb000d860 tx=0x7f3bb0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:35.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.729+0000 7f3bbeffd700 1 -- 172.21.15.112:0/2247950950 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb0014620 con 0x7f3bc8068d20 2023-01-24T03:30:35.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.729+0000 7f3bbeffd700 1 -- 172.21.15.112:0/2247950950 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bb0014780 con 0x7f3bc8068d20 2023-01-24T03:30:35.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.729+0000 7f3bbeffd700 1 -- 172.21.15.112:0/2247950950 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb0014b20 con 0x7f3bc8068d20 2023-01-24T03:30:35.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.729+0000 7f3bcda0e700 1 -- 172.21.15.112:0/2247950950 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bc8152ac0 con 0x7f3bc8068d20 2023-01-24T03:30:35.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.729+0000 7f3bcda0e700 1 -- 172.21.15.112:0/2247950950 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bc80c2040 con 0x7f3bc8068d20 2023-01-24T03:30:35.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.730+0000 7f3bcda0e700 1 -- 172.21.15.112:0/2247950950 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bc806f3a0 con 0x7f3bc8068d20 2023-01-24T03:30:35.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.730+0000 7f3bbeffd700 1 -- 172.21.15.112:0/2247950950 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f3bb0024070 con 0x7f3bc8068d20 2023-01-24T03:30:35.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.730+0000 7f3bbeffd700 1 --2- 172.21.15.112:0/2247950950 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3bac0611c0 0x7f3bac063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:35.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.731+0000 7f3bbeffd700 1 -- 172.21.15.112:0/2247950950 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f3bb007de70 con 0x7f3bc8068d20 2023-01-24T03:30:35.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.731+0000 7f3bc6ffd700 1 --2- 172.21.15.112:0/2247950950 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3bac0611c0 0x7f3bac063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:35.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.731+0000 7f3bc6ffd700 1 --2- 172.21.15.112:0/2247950950 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3bac0611c0 0x7f3bac063670 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3bb8009fd0 tx=0x7f3bb8019000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:35.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.735+0000 7f3bbeffd700 1 -- 172.21.15.112:0/2247950950 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f3bb004dcb0 con 0x7f3bc8068d20 2023-01-24T03:30:35.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:35.954+0000 7f3bcda0e700 1 -- 172.21.15.112:0/2247950950 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f3bc8156800 con 0x7f3bc8068d20 2023-01-24T03:30:36.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.600+0000 7f3bbeffd700 1 -- 172.21.15.112:0/2247950950 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3bb00512d0 con 0x7f3bc8068d20 2023-01-24T03:30:36.605 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.602+0000 7f3bcda0e700 1 -- 172.21.15.112:0/2247950950 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3bac0611c0 msgr2=0x7f3bac063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:36.605 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.602+0000 7f3bcda0e700 1 --2- 172.21.15.112:0/2247950950 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3bac0611c0 0x7f3bac063670 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3bb8009fd0 tx=0x7f3bb8019000 comp rx=0 tx=0).stop 2023-01-24T03:30:36.605 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.602+0000 7f3bcda0e700 1 -- 172.21.15.112:0/2247950950 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3bc8068d20 msgr2=0x7f3bc80c09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:36.605 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.602+0000 7f3bcda0e700 1 --2- 172.21.15.112:0/2247950950 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3bc8068d20 0x7f3bc80c09d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3bb000d860 tx=0x7f3bb0008040 comp rx=0 tx=0).stop 2023-01-24T03:30:36.605 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.603+0000 7f3bcda0e700 1 -- 172.21.15.112:0/2247950950 shutdown_connections 2023-01-24T03:30:36.606 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.603+0000 7f3bcda0e700 1 --2- 172.21.15.112:0/2247950950 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3bac0611c0 0x7f3bac063670 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:36.606 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.603+0000 7f3bcda0e700 1 --2- 172.21.15.112:0/2247950950 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3bc80694c0 0x7f3bc814df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:36.606 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.603+0000 7f3bcda0e700 1 --2- 172.21.15.112:0/2247950950 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3bc805e420 0x7f3bc8152580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:36.606 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.603+0000 7f3bcda0e700 1 --2- 172.21.15.112:0/2247950950 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3bc8068d20 0x7f3bc80c09d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:36.606 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.603+0000 7f3bcda0e700 1 -- 172.21.15.112:0/2247950950 >> 172.21.15.112:0/2247950950 conn(0x7f3bc805d830 msgr2=0x7f3bc8153e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:36.607 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.603+0000 7f3bcda0e700 1 -- 172.21.15.112:0/2247950950 shutdown_connections 2023-01-24T03:30:36.607 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.603+0000 7f3bcda0e700 1 -- 172.21.15.112:0/2247950950 wait complete. 2023-01-24T03:30:36.619 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-01-24T03:30:36.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.735+0000 7f6a3e02a700 1 Processor -- start 2023-01-24T03:30:36.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.735+0000 7f6a3e02a700 1 -- start start 2023-01-24T03:30:36.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.736+0000 7f6a3e02a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6a380694c0 0x7f6a38069890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:36.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.736+0000 7f6a3e02a700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f6a38068d20 0x7f6a38155f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:36.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.736+0000 7f6a3e02a700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a38155ab0 con 0x7f6a3805e420 2023-01-24T03:30:36.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.736+0000 7f6a3e02a700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a38155550 con 0x7f6a38068d20 2023-01-24T03:30:36.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.736+0000 7f6a3e02a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a38156480 con 0x7f6a380694c0 2023-01-24T03:30:36.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.736+0000 7f6a37fff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f6a38068d20 0x7f6a38155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:36.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.736+0000 7f6a37fff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f6a38068d20 0x7f6a38155f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50340/0 (socket says 172.21.15.112:50340) 2023-01-24T03:30:36.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.736+0000 7f6a37fff700 1 -- 172.21.15.112:0/3548642822 learned_addr learned my addr 172.21.15.112:0/3548642822 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:36.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.736+0000 7f6a377fe700 1 --2- 172.21.15.112:0/3548642822 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6a380694c0 0x7f6a38069890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:36.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.736+0000 7f6a37fff700 1 -- 172.21.15.112:0/3548642822 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6a380694c0 msgr2=0x7f6a38069890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:36.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.736+0000 7f6a37fff700 1 --2- 172.21.15.112:0/3548642822 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6a380694c0 0x7f6a38069890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:36.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.736+0000 7f6a37fff700 1 -- 172.21.15.112:0/3548642822 >> v1:172.21.15.142:6789/0 conn(0x7f6a3805e420 legacy=0x7f6a380c68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:36.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.737+0000 7f6a37fff700 1 -- 172.21.15.112:0/3548642822 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a380c70e0 con 0x7f6a38068d20 2023-01-24T03:30:36.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.737+0000 7f6a37fff700 1 --2- 172.21.15.112:0/3548642822 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a38068d20 0x7f6a38155f40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6a30006fd0 tx=0x7f6a300090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba99d3e6a7e9425f server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:36.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.737+0000 7f6a367fc700 1 -- 172.21.15.112:0/3548642822 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a30011630 con 0x7f6a38068d20 2023-01-24T03:30:36.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.737+0000 7f6a367fc700 1 -- 172.21.15.112:0/3548642822 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a30011790 con 0x7f6a38068d20 2023-01-24T03:30:36.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.737+0000 7f6a367fc700 1 -- 172.21.15.112:0/3548642822 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a30011b50 con 0x7f6a38068d20 2023-01-24T03:30:36.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.737+0000 7f6a3e02a700 1 -- 172.21.15.112:0/3548642822 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a38068d20 msgr2=0x7f6a38155f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:36.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.737+0000 7f6a3e02a700 1 --2- 172.21.15.112:0/3548642822 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a38068d20 0x7f6a38155f40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6a30006fd0 tx=0x7f6a300090f0 comp rx=0 tx=0).stop 2023-01-24T03:30:36.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.737+0000 7f6a3e02a700 1 -- 172.21.15.112:0/3548642822 shutdown_connections 2023-01-24T03:30:36.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.737+0000 7f6a3e02a700 1 --2- 172.21.15.112:0/3548642822 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a38068d20 0x7f6a38155f40 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:36.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.737+0000 7f6a3e02a700 1 --2- 172.21.15.112:0/3548642822 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6a380694c0 0x7f6a38069890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:36.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.737+0000 7f6a3e02a700 1 -- 172.21.15.112:0/3548642822 >> 172.21.15.112:0/3548642822 conn(0x7f6a3805d830 msgr2=0x7f6a3805dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:36.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.737+0000 7f6a3e02a700 1 -- 172.21.15.112:0/3548642822 shutdown_connections 2023-01-24T03:30:36.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.737+0000 7f6a3e02a700 1 -- 172.21.15.112:0/3548642822 wait complete. 2023-01-24T03:30:36.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.739+0000 7f6a3e02a700 1 Processor -- start 2023-01-24T03:30:36.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.739+0000 7f6a3e02a700 1 -- start start 2023-01-24T03:30:36.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.739+0000 7f6a3e02a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6a3805e420 0x7f6a380c8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:36.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.739+0000 7f6a3e02a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a38068d20 0x7f6a38150350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:36.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.739+0000 7f6a3e02a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a380694c0 0x7f6a38150890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:36.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.739+0000 7f6a3e02a700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a381522b0 con 0x7f6a38068d20 2023-01-24T03:30:36.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.739+0000 7f6a3e02a700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a38152df0 con 0x7f6a380694c0 2023-01-24T03:30:36.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.739+0000 7f6a3e02a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a38158760 con 0x7f6a3805e420 2023-01-24T03:30:36.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.739+0000 7f6a36ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a38068d20 0x7f6a38150350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:36.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.739+0000 7f6a36ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a38068d20 0x7f6a38150350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50344/0 (socket says 172.21.15.112:50344) 2023-01-24T03:30:36.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.739+0000 7f6a36ffd700 1 -- 172.21.15.112:0/3828355167 learned_addr learned my addr 172.21.15.112:0/3828355167 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:36.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.739+0000 7f6a377fe700 1 --2- 172.21.15.112:0/3828355167 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6a3805e420 0x7f6a380c8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:36.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.739+0000 7f6a37fff700 1 --2- 172.21.15.112:0/3828355167 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a380694c0 0x7f6a38150890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:36.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.740+0000 7f6a36ffd700 1 -- 172.21.15.112:0/3828355167 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6a3805e420 msgr2=0x7f6a380c8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:36.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.740+0000 7f6a36ffd700 1 --2- 172.21.15.112:0/3828355167 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6a3805e420 0x7f6a380c8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:36.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.740+0000 7f6a36ffd700 1 -- 172.21.15.112:0/3828355167 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a380694c0 msgr2=0x7f6a38150890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:36.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.740+0000 7f6a36ffd700 1 --2- 172.21.15.112:0/3828355167 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a380694c0 0x7f6a38150890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:36.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.740+0000 7f6a36ffd700 1 -- 172.21.15.112:0/3828355167 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a30009370 con 0x7f6a38068d20 2023-01-24T03:30:36.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.740+0000 7f6a37fff700 1 --2- 172.21.15.112:0/3828355167 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a380694c0 0x7f6a38150890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:30:36.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.740+0000 7f6a36ffd700 1 --2- 172.21.15.112:0/3828355167 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a38068d20 0x7f6a38150350 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c00c440 tx=0x7f6a2c007d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:36.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.740+0000 7f6a1f7fe700 1 -- 172.21.15.112:0/3828355167 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c00ce40 con 0x7f6a38068d20 2023-01-24T03:30:36.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.740+0000 7f6a1f7fe700 1 -- 172.21.15.112:0/3828355167 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a2c011590 con 0x7f6a38068d20 2023-01-24T03:30:36.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.740+0000 7f6a3e02a700 1 -- 172.21.15.112:0/3828355167 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a381589d0 con 0x7f6a38068d20 2023-01-24T03:30:36.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.740+0000 7f6a1f7fe700 1 -- 172.21.15.112:0/3828355167 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c011950 con 0x7f6a38068d20 2023-01-24T03:30:36.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.740+0000 7f6a3e02a700 1 -- 172.21.15.112:0/3828355167 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a381527a0 con 0x7f6a38068d20 2023-01-24T03:30:36.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.741+0000 7f6a3e02a700 1 -- 172.21.15.112:0/3828355167 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a3806f3a0 con 0x7f6a38068d20 2023-01-24T03:30:36.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.741+0000 7f6a1f7fe700 1 -- 172.21.15.112:0/3828355167 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f6a2c019030 con 0x7f6a38068d20 2023-01-24T03:30:36.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.741+0000 7f6a1f7fe700 1 --2- 172.21.15.112:0/3828355167 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f6a180611c0 0x7f6a18063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:36.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.742+0000 7f6a1f7fe700 1 -- 172.21.15.112:0/3828355167 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f6a2c07c960 con 0x7f6a38068d20 2023-01-24T03:30:36.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.742+0000 7f6a377fe700 1 --2- 172.21.15.112:0/3828355167 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f6a180611c0 0x7f6a18063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:36.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.742+0000 7f6a377fe700 1 --2- 172.21.15.112:0/3828355167 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f6a180611c0 0x7f6a18063670 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6a280075a0 tx=0x7f6a280097e0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:36.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.745+0000 7f6a1f7fe700 1 -- 172.21.15.112:0/3828355167 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f6a2c04c740 con 0x7f6a38068d20 2023-01-24T03:30:36.974 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:36.971+0000 7f6a3e02a700 1 -- 172.21.15.112:0/3828355167 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f6a38152f30 con 0x7f6a38068d20 2023-01-24T03:30:37.608 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.604+0000 7f6a1f7fe700 1 -- 172.21.15.112:0/3828355167 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v10) v1 ==== 115+0+0 (secure 0 0 0) 0x7f6a2c04fd60 con 0x7f6a38068d20 2023-01-24T03:30:37.611 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.607+0000 7f6a3e02a700 1 -- 172.21.15.112:0/3828355167 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f6a180611c0 msgr2=0x7f6a18063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:37.611 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.607+0000 7f6a3e02a700 1 --2- 172.21.15.112:0/3828355167 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f6a180611c0 0x7f6a18063670 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6a280075a0 tx=0x7f6a280097e0 comp rx=0 tx=0).stop 2023-01-24T03:30:37.611 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.607+0000 7f6a3e02a700 1 -- 172.21.15.112:0/3828355167 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a38068d20 msgr2=0x7f6a38150350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:37.611 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.607+0000 7f6a3e02a700 1 --2- 172.21.15.112:0/3828355167 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a38068d20 0x7f6a38150350 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c00c440 tx=0x7f6a2c007d70 comp rx=0 tx=0).stop 2023-01-24T03:30:37.612 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.607+0000 7f6a3e02a700 1 -- 172.21.15.112:0/3828355167 shutdown_connections 2023-01-24T03:30:37.612 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.607+0000 7f6a3e02a700 1 --2- 172.21.15.112:0/3828355167 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f6a180611c0 0x7f6a18063670 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:37.612 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.607+0000 7f6a3e02a700 1 --2- 172.21.15.112:0/3828355167 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6a380694c0 0x7f6a38150890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:37.613 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.607+0000 7f6a3e02a700 1 --2- 172.21.15.112:0/3828355167 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6a3805e420 0x7f6a380c8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:37.613 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.607+0000 7f6a3e02a700 1 --2- 172.21.15.112:0/3828355167 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a38068d20 0x7f6a38150350 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:37.613 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.607+0000 7f6a3e02a700 1 -- 172.21.15.112:0/3828355167 >> 172.21.15.112:0/3828355167 conn(0x7f6a3805d830 msgr2=0x7f6a38153df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:37.614 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.608+0000 7f6a3e02a700 1 -- 172.21.15.112:0/3828355167 shutdown_connections 2023-01-24T03:30:37.614 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.608+0000 7f6a3e02a700 1 -- 172.21.15.112:0/3828355167 wait complete. 2023-01-24T03:30:37.625 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:30:37.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.741+0000 7f0464e68700 1 Processor -- start 2023-01-24T03:30:37.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.741+0000 7f0464e68700 1 -- start start 2023-01-24T03:30:37.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.742+0000 7f0464e68700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f04600c0bb0 0x7f04600c5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:37.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.742+0000 7f0464e68700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f04600c4fb0 0x7f04600c5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:37.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.742+0000 7f0464e68700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0460153e30 con 0x7f046006c4e0 2023-01-24T03:30:37.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.742+0000 7f0464e68700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0460152fd0 con 0x7f04600c4fb0 2023-01-24T03:30:37.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.742+0000 7f0464e68700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04601532b0 con 0x7f04600c0bb0 2023-01-24T03:30:37.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.742+0000 7f045ffff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f04600c4fb0 0x7f04600c5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:37.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.742+0000 7f045ffff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f04600c4fb0 0x7f04600c5bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50352/0 (socket says 172.21.15.112:50352) 2023-01-24T03:30:37.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.742+0000 7f045ffff700 1 -- 172.21.15.112:0/1598920539 learned_addr learned my addr 172.21.15.112:0/1598920539 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:37.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.742+0000 7f045f7fe700 1 --2- 172.21.15.112:0/1598920539 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f04600c0bb0 0x7f04600c5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:37.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.742+0000 7f045ffff700 1 -- 172.21.15.112:0/1598920539 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f04600c0bb0 msgr2=0x7f04600c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:37.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.742+0000 7f045ffff700 1 --2- 172.21.15.112:0/1598920539 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f04600c0bb0 0x7f04600c5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:37.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.742+0000 7f045ffff700 1 -- 172.21.15.112:0/1598920539 >> v1:172.21.15.142:6789/0 conn(0x7f046006c4e0 legacy=0x7f04600c48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:37.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.742+0000 7f045ffff700 1 -- 172.21.15.112:0/1598920539 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0460153590 con 0x7f04600c4fb0 2023-01-24T03:30:37.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.742+0000 7f045f7fe700 1 --2- 172.21.15.112:0/1598920539 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f04600c0bb0 0x7f04600c5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-01-24T03:30:37.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.742+0000 7f045ffff700 1 --2- 172.21.15.112:0/1598920539 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04600c4fb0 0x7f04600c5bb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0458006fd0 tx=0x7f04580090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c6b66ab8c984f0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:37.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.742+0000 7f045e7fc700 1 -- 172.21.15.112:0/1598920539 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0458011630 con 0x7f04600c4fb0 2023-01-24T03:30:37.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.742+0000 7f045e7fc700 1 -- 172.21.15.112:0/1598920539 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0458011790 con 0x7f04600c4fb0 2023-01-24T03:30:37.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.742+0000 7f045e7fc700 1 -- 172.21.15.112:0/1598920539 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0458011b50 con 0x7f04600c4fb0 2023-01-24T03:30:37.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.743+0000 7f0464e68700 1 -- 172.21.15.112:0/1598920539 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04600c4fb0 msgr2=0x7f04600c5bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:37.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.743+0000 7f0464e68700 1 --2- 172.21.15.112:0/1598920539 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04600c4fb0 0x7f04600c5bb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0458006fd0 tx=0x7f04580090f0 comp rx=0 tx=0).stop 2023-01-24T03:30:37.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.743+0000 7f0464e68700 1 -- 172.21.15.112:0/1598920539 shutdown_connections 2023-01-24T03:30:37.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.743+0000 7f0464e68700 1 --2- 172.21.15.112:0/1598920539 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04600c4fb0 0x7f04600c5bb0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:37.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.743+0000 7f0464e68700 1 --2- 172.21.15.112:0/1598920539 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f04600c0bb0 0x7f04600c5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:37.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.743+0000 7f0464e68700 1 -- 172.21.15.112:0/1598920539 >> 172.21.15.112:0/1598920539 conn(0x7f04600ca150 msgr2=0x7f04600c6120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:37.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.743+0000 7f0464e68700 1 -- 172.21.15.112:0/1598920539 shutdown_connections 2023-01-24T03:30:37.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.743+0000 7f0464e68700 1 -- 172.21.15.112:0/1598920539 wait complete. 2023-01-24T03:30:37.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.744+0000 7f0464e68700 1 Processor -- start 2023-01-24T03:30:37.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.744+0000 7f0464e68700 1 -- start start 2023-01-24T03:30:37.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.745+0000 7f0464e68700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f046006c4e0 0x7f046014c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:37.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.745+0000 7f0464e68700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f04600c0bb0 0x7f04600bf880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:37.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.745+0000 7f0464e68700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04600c4fb0 0x7f04600bfdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:37.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.745+0000 7f0464e68700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0460166380 con 0x7f04600c4fb0 2023-01-24T03:30:37.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.745+0000 7f0464e68700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0460069880 con 0x7f046006c4e0 2023-01-24T03:30:37.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.745+0000 7f0464e68700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0460068980 con 0x7f04600c0bb0 2023-01-24T03:30:37.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.745+0000 7f045ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04600c4fb0 0x7f04600bfdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:37.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.745+0000 7f045ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04600c4fb0 0x7f04600bfdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50360/0 (socket says 172.21.15.112:50360) 2023-01-24T03:30:37.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.745+0000 7f045ffff700 1 -- 172.21.15.112:0/1731526523 learned_addr learned my addr 172.21.15.112:0/1731526523 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:37.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.745+0000 7f045f7fe700 1 --2- 172.21.15.112:0/1731526523 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f046006c4e0 0x7f046014c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:37.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.745+0000 7f045effd700 1 --2- 172.21.15.112:0/1731526523 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f04600c0bb0 0x7f04600bf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:37.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.745+0000 7f045ffff700 1 -- 172.21.15.112:0/1731526523 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f04600c0bb0 msgr2=0x7f04600bf880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:37.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.745+0000 7f045ffff700 1 --2- 172.21.15.112:0/1731526523 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f04600c0bb0 0x7f04600bf880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:37.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.745+0000 7f045ffff700 1 -- 172.21.15.112:0/1731526523 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f046006c4e0 msgr2=0x7f046014c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:37.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.745+0000 7f045ffff700 1 --2- 172.21.15.112:0/1731526523 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f046006c4e0 0x7f046014c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:37.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.745+0000 7f045ffff700 1 -- 172.21.15.112:0/1731526523 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0458009370 con 0x7f04600c4fb0 2023-01-24T03:30:37.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.745+0000 7f045ffff700 1 --2- 172.21.15.112:0/1731526523 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04600c4fb0 0x7f04600bfdc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f04600677b0 tx=0x7f0458016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:37.759 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.745+0000 7f04477fe700 1 -- 172.21.15.112:0/1731526523 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045801be20 con 0x7f04600c4fb0 2023-01-24T03:30:37.759 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.745+0000 7f04477fe700 1 -- 172.21.15.112:0/1731526523 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f045801c450 con 0x7f04600c4fb0 2023-01-24T03:30:37.759 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.745+0000 7f04477fe700 1 -- 172.21.15.112:0/1731526523 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045801c810 con 0x7f04600c4fb0 2023-01-24T03:30:37.759 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.745+0000 7f0464e68700 1 -- 172.21.15.112:0/1731526523 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04600c03a0 con 0x7f04600c4fb0 2023-01-24T03:30:37.759 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.746+0000 7f0464e68700 1 -- 172.21.15.112:0/1731526523 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04601659e0 con 0x7f04600c4fb0 2023-01-24T03:30:37.760 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.746+0000 7f0464e68700 1 -- 172.21.15.112:0/1731526523 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f046005ea20 con 0x7f04600c4fb0 2023-01-24T03:30:37.760 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.747+0000 7f04477fe700 1 -- 172.21.15.112:0/1731526523 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f045802d070 con 0x7f04600c4fb0 2023-01-24T03:30:37.760 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.747+0000 7f04477fe700 1 --2- 172.21.15.112:0/1731526523 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f04400611f0 0x7f04400636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:37.760 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.747+0000 7f04477fe700 1 -- 172.21.15.112:0/1731526523 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f045807efb0 con 0x7f04600c4fb0 2023-01-24T03:30:37.760 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.747+0000 7f045f7fe700 1 --2- 172.21.15.112:0/1731526523 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f04400611f0 0x7f04400636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:37.761 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.748+0000 7f045f7fe700 1 --2- 172.21.15.112:0/1731526523 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f04400611f0 0x7f04400636a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f045000d6e0 tx=0x7f0450007850 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:37.761 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.751+0000 7f04477fe700 1 -- 172.21.15.112:0/1731526523 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f045804edf0 con 0x7f04600c4fb0 2023-01-24T03:30:37.972 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.969+0000 7f0464e68700 1 -- 172.21.15.112:0/1731526523 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0460153870 con 0x7f04600c4fb0 2023-01-24T03:30:37.973 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.970+0000 7f04477fe700 1 -- 172.21.15.112:0/1731526523 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6114 (secure 0 0 0) 0x7f0458052410 con 0x7f04600c4fb0 2023-01-24T03:30:37.973 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:30:37.974 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/869267413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":869267413},{"type":"v1","addr":"172.21.15.142:6834","nonce":869267413}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:30:37.600288+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.142:6835/2762621058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2762621058},{"type":"v1","addr":"172.21.15.142:6835","nonce":2762621058}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:30:37.975 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.972+0000 7f0464e68700 1 -- 172.21.15.112:0/1731526523 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f04400611f0 msgr2=0x7f04400636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:37.975 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.972+0000 7f0464e68700 1 --2- 172.21.15.112:0/1731526523 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f04400611f0 0x7f04400636a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f045000d6e0 tx=0x7f0450007850 comp rx=0 tx=0).stop 2023-01-24T03:30:37.976 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.972+0000 7f0464e68700 1 -- 172.21.15.112:0/1731526523 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04600c4fb0 msgr2=0x7f04600bfdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:37.976 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.972+0000 7f0464e68700 1 --2- 172.21.15.112:0/1731526523 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04600c4fb0 0x7f04600bfdc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f04600677b0 tx=0x7f0458016000 comp rx=0 tx=0).stop 2023-01-24T03:30:37.976 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.972+0000 7f0464e68700 1 -- 172.21.15.112:0/1731526523 shutdown_connections 2023-01-24T03:30:37.976 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.972+0000 7f0464e68700 1 --2- 172.21.15.112:0/1731526523 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f04400611f0 0x7f04400636a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:37.977 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.972+0000 7f0464e68700 1 --2- 172.21.15.112:0/1731526523 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04600c4fb0 0x7f04600bfdc0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:37.977 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.972+0000 7f0464e68700 1 --2- 172.21.15.112:0/1731526523 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f046006c4e0 0x7f046014c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:37.977 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.972+0000 7f0464e68700 1 --2- 172.21.15.112:0/1731526523 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f04600c0bb0 0x7f04600bf880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:37.977 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.972+0000 7f0464e68700 1 -- 172.21.15.112:0/1731526523 >> 172.21.15.112:0/1731526523 conn(0x7f04600ca150 msgr2=0x7f04601515d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:37.977 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.972+0000 7f0464e68700 1 -- 172.21.15.112:0/1731526523 shutdown_connections 2023-01-24T03:30:37.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:37.972+0000 7f0464e68700 1 -- 172.21.15.112:0/1731526523 wait complete. 2023-01-24T03:30:37.978 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 10 2023-01-24T03:30:37.987 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T03:30:38.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.096+0000 7f6158d0e700 1 Processor -- start 2023-01-24T03:30:38.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.096+0000 7f6158d0e700 1 -- start start 2023-01-24T03:30:38.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.096+0000 7f6158d0e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f61540c4ea0 0x7f61540c1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:38.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.097+0000 7f6158d0e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6154153270 con 0x7f615406cf50 2023-01-24T03:30:38.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.097+0000 7f6158d0e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61541534e0 con 0x7f61540c3560 2023-01-24T03:30:38.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.097+0000 7f6158d0e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6154153e80 con 0x7f61540c4ea0 2023-01-24T03:30:38.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.097+0000 7f615259c700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f615406cf50 0x7f61540c2e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:59584/0 (socket says 172.21.15.112:59584) 2023-01-24T03:30:38.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.097+0000 7f615259c700 1 -- 172.21.15.112:0/316507585 learned_addr learned my addr 172.21.15.112:0/316507585 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:38.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.097+0000 7f615159a700 1 -- 172.21.15.112:0/316507585 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 864306306 0 0) 0x7f6154153270 con 0x7f615406cf50 2023-01-24T03:30:38.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.097+0000 7f615159a700 1 -- 172.21.15.112:0/316507585 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61380033f0 con 0x7f615406cf50 2023-01-24T03:30:38.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.097+0000 7f6152d9d700 1 --2- 172.21.15.112:0/316507585 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f61540c4ea0 0x7f61540c1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:38.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.097+0000 7f615159a700 1 -- 172.21.15.112:0/316507585 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3686359455 0 0) 0x7f61380033f0 con 0x7f615406cf50 2023-01-24T03:30:38.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.097+0000 7f615159a700 1 -- 172.21.15.112:0/316507585 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f61540c4ea0 msgr2=0x7f61540c1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:38.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.097+0000 7f615159a700 1 --2- 172.21.15.112:0/316507585 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f61540c4ea0 0x7f61540c1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:38.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.097+0000 7f615159a700 1 -- 172.21.15.112:0/316507585 >> v1:172.21.15.142:6789/0 conn(0x7f61540c3560 legacy=0x7f61540c4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:38.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.097+0000 7f615159a700 1 -- 172.21.15.112:0/316507585 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61541537c0 con 0x7f615406cf50 2023-01-24T03:30:38.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.097+0000 7f615159a700 1 -- 172.21.15.112:0/316507585 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7f614c001470 con 0x7f615406cf50 2023-01-24T03:30:38.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.097+0000 7f615159a700 1 -- 172.21.15.112:0/316507585 >> v1:172.21.15.112:6789/0 conn(0x7f615406cf50 legacy=0x7f61540c2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:38.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.097+0000 7f615159a700 1 --2- 172.21.15.112:0/316507585 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6138003910 0x7f6138003d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:38.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.097+0000 7f615159a700 1 --2- 172.21.15.112:0/316507585 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6138004690 0x7f6138004b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:38.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.097+0000 7f615159a700 1 --2- 172.21.15.112:0/316507585 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f61540c4ea0 0x7f6138005060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:38.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.097+0000 7f615159a700 1 -- 172.21.15.112:0/316507585 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6154153e80 con 0x7f6138004690 2023-01-24T03:30:38.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.097+0000 7f615159a700 1 -- 172.21.15.112:0/316507585 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613800b9e0 con 0x7f6138003910 2023-01-24T03:30:38.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.097+0000 7f615159a700 1 -- 172.21.15.112:0/316507585 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f613800bba0 con 0x7f61540c4ea0 2023-01-24T03:30:38.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.097+0000 7f6151d9b700 1 --2- 172.21.15.112:0/316507585 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6138004690 0x7f6138004b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:38.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.097+0000 7f615259c700 1 --2- 172.21.15.112:0/316507585 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6138003910 0x7f6138003d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:38.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.097+0000 7f6152d9d700 1 --2- 172.21.15.112:0/316507585 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f61540c4ea0 0x7f6138005060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:38.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.098+0000 7f6151d9b700 1 -- 172.21.15.112:0/316507585 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f61540c4ea0 msgr2=0x7f6138005060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:38.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.098+0000 7f6151d9b700 1 --2- 172.21.15.112:0/316507585 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f61540c4ea0 0x7f6138005060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:38.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.098+0000 7f6151d9b700 1 -- 172.21.15.112:0/316507585 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6138003910 msgr2=0x7f6138003d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:38.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.098+0000 7f6151d9b700 1 --2- 172.21.15.112:0/316507585 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6138003910 0x7f6138003d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:38.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.098+0000 7f6151d9b700 1 -- 172.21.15.112:0/316507585 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61380033f0 con 0x7f6138004690 2023-01-24T03:30:38.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.098+0000 7f6152d9d700 1 --2- 172.21.15.112:0/316507585 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f61540c4ea0 0x7f6138005060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:30:38.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.098+0000 7f6151d9b700 1 --2- 172.21.15.112:0/316507585 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6138004690 0x7f6138004b00 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6148001700 tx=0x7f6148007bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53df9c9e2bc97c4d server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:38.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.098+0000 7f615159a700 1 -- 172.21.15.112:0/316507585 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6148012420 con 0x7f6138004690 2023-01-24T03:30:38.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.098+0000 7f615159a700 1 -- 172.21.15.112:0/316507585 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61480125a0 con 0x7f6138004690 2023-01-24T03:30:38.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.098+0000 7f615159a700 1 -- 172.21.15.112:0/316507585 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6148012960 con 0x7f6138004690 2023-01-24T03:30:38.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.098+0000 7f6158d0e700 1 -- 172.21.15.112:0/316507585 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6138004690 msgr2=0x7f6138004b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:38.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.098+0000 7f6158d0e700 1 --2- 172.21.15.112:0/316507585 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6138004690 0x7f6138004b00 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6148001700 tx=0x7f6148007bb0 comp rx=0 tx=0).stop 2023-01-24T03:30:38.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.098+0000 7f6152d9d700 1 -- 172.21.15.112:0/316507585 reap_dead start 2023-01-24T03:30:38.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.098+0000 7f6158d0e700 1 -- 172.21.15.112:0/316507585 shutdown_connections 2023-01-24T03:30:38.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.098+0000 7f6158d0e700 1 -- 172.21.15.112:0/316507585 >> 172.21.15.112:0/316507585 conn(0x7f61540c9820 msgr2=0x7f61540c9c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:38.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.098+0000 7f6158d0e700 1 -- 172.21.15.112:0/316507585 shutdown_connections 2023-01-24T03:30:38.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.099+0000 7f6158d0e700 1 -- 172.21.15.112:0/316507585 wait complete. 2023-01-24T03:30:38.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.099+0000 7f6158d0e700 1 Processor -- start 2023-01-24T03:30:38.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.100+0000 7f6158d0e700 1 -- start start 2023-01-24T03:30:38.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.100+0000 7f6158d0e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6138004690 0x7f6154147170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:38.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.100+0000 7f6158d0e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f61540c7990 0x7f61540c6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:38.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.100+0000 7f6158d0e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f61540685c0 0x7f6154068a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:38.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.100+0000 7f6158d0e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6154146df0 con 0x7f61540c7990 2023-01-24T03:30:38.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.100+0000 7f6158d0e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6154068130 con 0x7f61540685c0 2023-01-24T03:30:38.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.100+0000 7f6158d0e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61541467b0 con 0x7f6138004690 2023-01-24T03:30:38.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.100+0000 7f6151d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f61540c7990 0x7f61540c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:38.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.100+0000 7f6151d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f61540c7990 0x7f61540c6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50378/0 (socket says 172.21.15.112:50378) 2023-01-24T03:30:38.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.100+0000 7f6151d9b700 1 -- 172.21.15.112:0/349228849 learned_addr learned my addr 172.21.15.112:0/349228849 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:38.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.100+0000 7f615259c700 1 --2- 172.21.15.112:0/349228849 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6138004690 0x7f6154147170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:38.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.100+0000 7f6152d9d700 1 --2- 172.21.15.112:0/349228849 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f61540685c0 0x7f6154068a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:38.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.101+0000 7f6151d9b700 1 -- 172.21.15.112:0/349228849 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6138004690 msgr2=0x7f6154147170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:38.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.101+0000 7f6151d9b700 1 --2- 172.21.15.112:0/349228849 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6138004690 0x7f6154147170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:38.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.101+0000 7f6151d9b700 1 -- 172.21.15.112:0/349228849 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f61540685c0 msgr2=0x7f6154068a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:38.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.101+0000 7f6151d9b700 1 --2- 172.21.15.112:0/349228849 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f61540685c0 0x7f6154068a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:38.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.101+0000 7f6151d9b700 1 -- 172.21.15.112:0/349228849 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6154148030 con 0x7f61540c7990 2023-01-24T03:30:38.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.101+0000 7f6152d9d700 1 --2- 172.21.15.112:0/349228849 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f61540685c0 0x7f6154068a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:30:38.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.101+0000 7f6151d9b700 1 --2- 172.21.15.112:0/349228849 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f61540c7990 0x7f61540c6460 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6148001cf0 tx=0x7f6148015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:38.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.101+0000 7f61427fc700 1 -- 172.21.15.112:0/349228849 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6148012420 con 0x7f61540c7990 2023-01-24T03:30:38.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.101+0000 7f61427fc700 1 -- 172.21.15.112:0/349228849 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6148012cd0 con 0x7f61540c7990 2023-01-24T03:30:38.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.101+0000 7f6158d0e700 1 -- 172.21.15.112:0/349228849 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6154068f70 con 0x7f61540c7990 2023-01-24T03:30:38.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.101+0000 7f61427fc700 1 -- 172.21.15.112:0/349228849 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6148012960 con 0x7f61540c7990 2023-01-24T03:30:38.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.101+0000 7f6158d0e700 1 -- 172.21.15.112:0/349228849 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6154147d90 con 0x7f61540c7990 2023-01-24T03:30:38.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.102+0000 7f6158d0e700 1 -- 172.21.15.112:0/349228849 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61540682e0 con 0x7f61540c7990 2023-01-24T03:30:38.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.102+0000 7f61427fc700 1 -- 172.21.15.112:0/349228849 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f6148010050 con 0x7f61540c7990 2023-01-24T03:30:38.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.103+0000 7f61427fc700 1 --2- 172.21.15.112:0/349228849 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f61300611f0 0x7f61300636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:38.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.103+0000 7f615259c700 1 --2- 172.21.15.112:0/349228849 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f61300611f0 0x7f61300636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:38.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.103+0000 7f61427fc700 1 -- 172.21.15.112:0/349228849 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f614807ed30 con 0x7f61540c7990 2023-01-24T03:30:38.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.103+0000 7f615259c700 1 --2- 172.21.15.112:0/349228849 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f61300611f0 0x7f61300636a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f614c0015a0 tx=0x7f614c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:38.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.107+0000 7f61427fc700 1 -- 172.21.15.112:0/349228849 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f61540682e0 con 0x7f61540c7990 2023-01-24T03:30:38.287 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.284+0000 7f6158d0e700 1 -- 172.21.15.112:0/349228849 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61540682a0 con 0x7f61540c7990 2023-01-24T03:30:38.287 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.285+0000 7f61427fc700 1 -- 172.21.15.112:0/349228849 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18755 (secure 0 0 0) 0x7f6148017090 con 0x7f61540c7990 2023-01-24T03:30:38.288 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:30:38.288 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":24,"fsid":"054b9ac9-b7b0-40e5-b932-6c8a955c5a72","created":"2023-01-24T03:30:03.798533+0000","modified":"2023-01-24T03:30:25.373340+0000","last_up_change":"2023-01-24T03:30:13.945316+0000","last_in_change":"2023-01-24T03:30:06.198433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-01-24T03:30:14.931166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-01-24T03:30:15.023730+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-01-24T03:30:19.060597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-01-24T03:30:19.943686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-01-24T03:30:22.999036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"fe6ad231-a39a-42eb-9a3a-49d008dd35e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6809","nonce":84512},{"type":"v1","addr":"172.21.15.112:6811","nonce":84512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":84512},{"type":"v1","addr":"172.21.15.112:6817","nonce":84512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6825","nonce":84512},{"type":"v1","addr":"172.21.15.112:6828","nonce":84512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6819","nonce":84512},{"type":"v1","addr":"172.21.15.112:6822","nonce":84512}]},"public_addr":"172.21.15.112:6811/84512","cluster_addr":"172.21.15.112:6817/84512","heartbeat_back_addr":"172.21.15.112:6828/84512","heartbeat_front_addr":"172.21.15.112:6822/84512","state":["exists","up"]},{"osd":1,"uuid":"8526a1d2-d540-4021-9766-c4bbb30317ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":84513},{"type":"v1","addr":"172.21.15.112:6812","nonce":84513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6815","nonce":84513},{"type":"v1","addr":"172.21.15.112:6818","nonce":84513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6827","nonce":84513},{"type":"v1","addr":"172.21.15.112:6830","nonce":84513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6821","nonce":84513},{"type":"v1","addr":"172.21.15.112:6824","nonce":84513}]},"public_addr":"172.21.15.112:6812/84513","cluster_addr":"172.21.15.112:6818/84513","heartbeat_back_addr":"172.21.15.112:6830/84513","heartbeat_front_addr":"172.21.15.112:6824/84513","state":["exists","up"]},{"osd":2,"uuid":"041ef8e4-4858-432d-9476-cbdb3a0dc47c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":84511},{"type":"v1","addr":"172.21.15.112:6813","nonce":84511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":84511},{"type":"v1","addr":"172.21.15.112:6820","nonce":84511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6829","nonce":84511},{"type":"v1","addr":"172.21.15.112:6831","nonce":84511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6823","nonce":84511},{"type":"v1","addr":"172.21.15.112:6826","nonce":84511}]},"public_addr":"172.21.15.112:6813/84511","cluster_addr":"172.21.15.112:6820/84511","heartbeat_back_addr":"172.21.15.112:6831/84511","heartbeat_front_addr":"172.21.15.112:6826/84511","state":["exists","up"]},{"osd":3,"uuid":"987587cf-1357-4a70-ad85-67b189c67729","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":84510},{"type":"v1","addr":"172.21.15.112:6801","nonce":84510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":84510},{"type":"v1","addr":"172.21.15.112:6803","nonce":84510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":84510},{"type":"v1","addr":"172.21.15.112:6807","nonce":84510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":84510},{"type":"v1","addr":"172.21.15.112:6805","nonce":84510}]},"public_addr":"172.21.15.112:6801/84510","cluster_addr":"172.21.15.112:6803/84510","heartbeat_back_addr":"172.21.15.112:6807/84510","heartbeat_front_addr":"172.21.15.112:6805/84510","state":["exists","up"]},{"osd":4,"uuid":"8503512b-43db-40b8-8b2b-c86cf2e54e17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":84614},{"type":"v1","addr":"172.21.15.142:6801","nonce":84614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":84614},{"type":"v1","addr":"172.21.15.142:6803","nonce":84614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":84614},{"type":"v1","addr":"172.21.15.142:6807","nonce":84614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":84614},{"type":"v1","addr":"172.21.15.142:6805","nonce":84614}]},"public_addr":"172.21.15.142:6801/84614","cluster_addr":"172.21.15.142:6803/84614","heartbeat_back_addr":"172.21.15.142:6807/84614","heartbeat_front_addr":"172.21.15.142:6805/84614","state":["exists","up"]},{"osd":5,"uuid":"d3bc0030-8b08-4645-9ec2-1f8a4e29ef36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":84638},{"type":"v1","addr":"172.21.15.142:6817","nonce":84638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":84638},{"type":"v1","addr":"172.21.15.142:6819","nonce":84638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":84638},{"type":"v1","addr":"172.21.15.142:6823","nonce":84638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":84638},{"type":"v1","addr":"172.21.15.142:6821","nonce":84638}]},"public_addr":"172.21.15.142:6817/84638","cluster_addr":"172.21.15.142:6819/84638","heartbeat_back_addr":"172.21.15.142:6823/84638","heartbeat_front_addr":"172.21.15.142:6821/84638","state":["exists","up"]},{"osd":6,"uuid":"c7649e3b-832d-4e6f-8866-b15e5d814476","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":84645},{"type":"v1","addr":"172.21.15.142:6825","nonce":84645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":84645},{"type":"v1","addr":"172.21.15.142:6827","nonce":84645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":84645},{"type":"v1","addr":"172.21.15.142:6831","nonce":84645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":84645},{"type":"v1","addr":"172.21.15.142:6829","nonce":84645}]},"public_addr":"172.21.15.142:6825/84645","cluster_addr":"172.21.15.142:6827/84645","heartbeat_back_addr":"172.21.15.142:6831/84645","heartbeat_front_addr":"172.21.15.142:6829/84645","state":["exists","up"]},{"osd":7,"uuid":"2ea5b3bf-884e-4b20-b572-1443efdc7477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":84615},{"type":"v1","addr":"172.21.15.142:6809","nonce":84615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":84615},{"type":"v1","addr":"172.21.15.142:6811","nonce":84615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":84615},{"type":"v1","addr":"172.21.15.142:6815","nonce":84615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":84615},{"type":"v1","addr":"172.21.15.142:6813","nonce":84615}]},"public_addr":"172.21.15.142:6809/84615","cluster_addr":"172.21.15.142:6811/84615","heartbeat_back_addr":"172.21.15.142:6815/84615","heartbeat_front_addr":"172.21.15.142:6813/84615","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.370027+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.335683+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.350667+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.350794+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.391119+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.422933+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.400439+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.354018+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T03:30:38.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.287+0000 7f6158d0e700 1 -- 172.21.15.112:0/349228849 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f61300611f0 msgr2=0x7f61300636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:38.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.287+0000 7f6158d0e700 1 --2- 172.21.15.112:0/349228849 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f61300611f0 0x7f61300636a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f614c0015a0 tx=0x7f614c009040 comp rx=0 tx=0).stop 2023-01-24T03:30:38.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.287+0000 7f6158d0e700 1 -- 172.21.15.112:0/349228849 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f61540c7990 msgr2=0x7f61540c6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:38.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.287+0000 7f6158d0e700 1 --2- 172.21.15.112:0/349228849 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f61540c7990 0x7f61540c6460 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6148001cf0 tx=0x7f6148015000 comp rx=0 tx=0).stop 2023-01-24T03:30:38.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.287+0000 7f6158d0e700 1 -- 172.21.15.112:0/349228849 shutdown_connections 2023-01-24T03:30:38.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.287+0000 7f6158d0e700 1 --2- 172.21.15.112:0/349228849 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f61300611f0 0x7f61300636a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:38.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.287+0000 7f6158d0e700 1 --2- 172.21.15.112:0/349228849 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f61540685c0 0x7f6154068a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:38.292 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.287+0000 7f6158d0e700 1 --2- 172.21.15.112:0/349228849 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6138004690 0x7f6154147170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:38.292 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.287+0000 7f6158d0e700 1 --2- 172.21.15.112:0/349228849 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f61540c7990 0x7f61540c6460 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:38.292 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.287+0000 7f6158d0e700 1 -- 172.21.15.112:0/349228849 >> 172.21.15.112:0/349228849 conn(0x7f61540c9820 msgr2=0x7f6154150e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:38.292 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.287+0000 7f6158d0e700 1 -- 172.21.15.112:0/349228849 shutdown_connections 2023-01-24T03:30:38.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.287+0000 7f6158d0e700 1 -- 172.21.15.112:0/349228849 wait complete. 2023-01-24T03:30:38.303 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-01-24T03:30:38.303 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-24T03:30:38.415 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.412+0000 7fcba61a9700 1 Processor -- start 2023-01-24T03:30:38.416 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.412+0000 7fcba61a9700 1 -- start start 2023-01-24T03:30:38.420 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.413+0000 7fcba61a9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcba00b0aa0 0x7fcba0155da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:38.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.413+0000 7fcba61a9700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fcba00ca3e0 0x7fcba00ca840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:38.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.413+0000 7fcba61a9700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcba00b0f40 con 0x7fcba00c7e40 2023-01-24T03:30:38.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.413+0000 7fcba61a9700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcba00cad80 con 0x7fcba00ca3e0 2023-01-24T03:30:38.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.413+0000 7fcba61a9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcba00b1170 con 0x7fcba00b0aa0 2023-01-24T03:30:38.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.413+0000 7fcb9f7fe700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fcba00c7e40 0x7fcba00ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:59604/0 (socket says 172.21.15.112:59604) 2023-01-24T03:30:38.422 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.413+0000 7fcb9ffff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fcba00ca3e0 0x7fcba00ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:38.422 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.413+0000 7fcb9f7fe700 1 -- 172.21.15.112:0/2645080720 learned_addr learned my addr 172.21.15.112:0/2645080720 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:38.422 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.414+0000 7fcb9effd700 1 --2- 172.21.15.112:0/2645080720 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcba00b0aa0 0x7fcba0155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:38.422 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.414+0000 7fcb9ffff700 1 -- 172.21.15.112:0/2645080720 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcba00b0aa0 msgr2=0x7fcba0155da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:38.422 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.414+0000 7fcb9ffff700 1 --2- 172.21.15.112:0/2645080720 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcba00b0aa0 0x7fcba0155da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:38.423 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.414+0000 7fcb9ffff700 1 -- 172.21.15.112:0/2645080720 >> v1:172.21.15.112:6789/0 conn(0x7fcba00c7e40 legacy=0x7fcba00ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:38.423 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.414+0000 7fcb9e7fc700 1 -- 172.21.15.112:0/2645080720 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1123248598 0 0) 0x7fcba00b0f40 con 0x7fcba00c7e40 2023-01-24T03:30:38.423 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.414+0000 7fcb9ffff700 1 -- 172.21.15.112:0/2645080720 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba00b1450 con 0x7fcba00ca3e0 2023-01-24T03:30:38.423 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.414+0000 7fcb9effd700 1 --2- 172.21.15.112:0/2645080720 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcba00b0aa0 0x7fcba0155da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-01-24T03:30:38.423 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.414+0000 7fcb9ffff700 1 --2- 172.21.15.112:0/2645080720 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcba00ca3e0 0x7fcba00ca840 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcb98009e60 tx=0x7fcb98009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebadf8c7269b8268 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:38.424 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.414+0000 7fcb9e7fc700 1 -- 172.21.15.112:0/2645080720 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9800e070 con 0x7fcba00ca3e0 2023-01-24T03:30:38.424 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.414+0000 7fcb9e7fc700 1 -- 172.21.15.112:0/2645080720 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb9800bc30 con 0x7fcba00ca3e0 2023-01-24T03:30:38.424 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.414+0000 7fcb9e7fc700 1 -- 172.21.15.112:0/2645080720 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb98012430 con 0x7fcba00ca3e0 2023-01-24T03:30:38.424 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.414+0000 7fcba61a9700 1 -- 172.21.15.112:0/2645080720 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcba00ca3e0 msgr2=0x7fcba00ca840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:38.424 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.414+0000 7fcba61a9700 1 --2- 172.21.15.112:0/2645080720 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcba00ca3e0 0x7fcba00ca840 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcb98009e60 tx=0x7fcb98009780 comp rx=0 tx=0).stop 2023-01-24T03:30:38.425 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.414+0000 7fcba61a9700 1 -- 172.21.15.112:0/2645080720 shutdown_connections 2023-01-24T03:30:38.425 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.414+0000 7fcba61a9700 1 --2- 172.21.15.112:0/2645080720 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcba00ca3e0 0x7fcba00ca840 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:38.425 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.414+0000 7fcba61a9700 1 --2- 172.21.15.112:0/2645080720 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcba00b0aa0 0x7fcba0155da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:38.425 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.414+0000 7fcba61a9700 1 -- 172.21.15.112:0/2645080720 >> 172.21.15.112:0/2645080720 conn(0x7fcba005e2c0 msgr2=0x7fcba00cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:38.425 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.414+0000 7fcba61a9700 1 -- 172.21.15.112:0/2645080720 shutdown_connections 2023-01-24T03:30:38.426 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.414+0000 7fcba61a9700 1 -- 172.21.15.112:0/2645080720 wait complete. 2023-01-24T03:30:38.426 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.416+0000 7fcba61a9700 1 Processor -- start 2023-01-24T03:30:38.426 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.416+0000 7fcba61a9700 1 -- start start 2023-01-24T03:30:38.426 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.416+0000 7fcba61a9700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcba00b0aa0 0x7fcba01552d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:38.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.416+0000 7fcba61a9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcba00c7e40 0x7fcba015acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:38.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.416+0000 7fcba61a9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcba00ca3e0 0x7fcba01652d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:38.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.416+0000 7fcba61a9700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba00c20a0 con 0x7fcba00c7e40 2023-01-24T03:30:38.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.416+0000 7fcba61a9700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba0154940 con 0x7fcba00b0aa0 2023-01-24T03:30:38.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.416+0000 7fcba61a9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcba00c01f0 con 0x7fcba00ca3e0 2023-01-24T03:30:38.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.416+0000 7fcb9effd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcba00c7e40 0x7fcba015acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:38.428 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.416+0000 7fcb9effd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcba00c7e40 0x7fcba015acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50394/0 (socket says 172.21.15.112:50394) 2023-01-24T03:30:38.428 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.416+0000 7fcb9effd700 1 -- 172.21.15.112:0/1286169041 learned_addr learned my addr 172.21.15.112:0/1286169041 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:38.428 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.417+0000 7fcb9effd700 1 -- 172.21.15.112:0/1286169041 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcba00ca3e0 msgr2=0x7fcba01652d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:38.428 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.417+0000 7fcb9effd700 1 --2- 172.21.15.112:0/1286169041 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcba00ca3e0 0x7fcba01652d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:38.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.417+0000 7fcb9effd700 1 -- 172.21.15.112:0/1286169041 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcba00b0aa0 msgr2=0x7fcba01552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:38.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.417+0000 7fcb9effd700 1 --2- 172.21.15.112:0/1286169041 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcba00b0aa0 0x7fcba01552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:38.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.417+0000 7fcb9effd700 1 -- 172.21.15.112:0/1286169041 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb98009a20 con 0x7fcba00c7e40 2023-01-24T03:30:38.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.417+0000 7fcb9effd700 1 --2- 172.21.15.112:0/1286169041 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcba00c7e40 0x7fcba015acf0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fcb9400c880 tx=0x7fcb94008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:38.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.417+0000 7fcb877fe700 1 -- 172.21.15.112:0/1286169041 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb94011600 con 0x7fcba00c7e40 2023-01-24T03:30:38.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.417+0000 7fcb877fe700 1 -- 172.21.15.112:0/1286169041 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb94011760 con 0x7fcba00c7e40 2023-01-24T03:30:38.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.417+0000 7fcba61a9700 1 -- 172.21.15.112:0/1286169041 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcba00c0710 con 0x7fcba00c7e40 2023-01-24T03:30:38.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.417+0000 7fcb877fe700 1 -- 172.21.15.112:0/1286169041 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb94011b00 con 0x7fcba00c7e40 2023-01-24T03:30:38.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.417+0000 7fcba61a9700 1 -- 172.21.15.112:0/1286169041 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcba0154660 con 0x7fcba00c7e40 2023-01-24T03:30:38.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.418+0000 7fcba61a9700 1 -- 172.21.15.112:0/1286169041 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcba004bb80 con 0x7fcba00c7e40 2023-01-24T03:30:38.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.418+0000 7fcb877fe700 1 -- 172.21.15.112:0/1286169041 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fcb94021070 con 0x7fcba00c7e40 2023-01-24T03:30:38.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.418+0000 7fcb877fe700 1 --2- 172.21.15.112:0/1286169041 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fcb800611f0 0x7fcb800636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:38.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.419+0000 7fcb877fe700 1 -- 172.21.15.112:0/1286169041 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fcb9407d5d0 con 0x7fcba00c7e40 2023-01-24T03:30:38.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.419+0000 7fcb9f7fe700 1 --2- 172.21.15.112:0/1286169041 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fcb800611f0 0x7fcb800636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:38.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.419+0000 7fcb9f7fe700 1 --2- 172.21.15.112:0/1286169041 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fcb800611f0 0x7fcb800636a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcb900011e0 tx=0x7fcb90007850 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:38.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.422+0000 7fcb877fe700 1 -- 172.21.15.112:0/1286169041 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fcb9404d410 con 0x7fcba00c7e40 2023-01-24T03:30:38.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.596+0000 7fcba61a9700 1 -- 172.21.15.112:0/1286169041 --> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcba00c5df0 con 0x7fcb800611f0 2023-01-24T03:30:38.609 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.607+0000 7fcb877fe700 1 -- 172.21.15.112:0/1286169041 <== mgr.4104 v2:172.21.15.112:6832/84276 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+365098 (secure 0 0 0) 0x7fcba00c5df0 con 0x7fcb800611f0 2023-01-24T03:30:38.614 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:30:38.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.610+0000 7fcba61a9700 1 -- 172.21.15.112:0/1286169041 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fcb800611f0 msgr2=0x7fcb800636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:38.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.610+0000 7fcba61a9700 1 --2- 172.21.15.112:0/1286169041 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fcb800611f0 0x7fcb800636a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcb900011e0 tx=0x7fcb90007850 comp rx=0 tx=0).stop 2023-01-24T03:30:38.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.610+0000 7fcba61a9700 1 -- 172.21.15.112:0/1286169041 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcba00c7e40 msgr2=0x7fcba015acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:38.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.610+0000 7fcba61a9700 1 --2- 172.21.15.112:0/1286169041 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcba00c7e40 0x7fcba015acf0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fcb9400c880 tx=0x7fcb94008040 comp rx=0 tx=0).stop 2023-01-24T03:30:38.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.610+0000 7fcba61a9700 1 -- 172.21.15.112:0/1286169041 shutdown_connections 2023-01-24T03:30:38.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.610+0000 7fcba61a9700 1 --2- 172.21.15.112:0/1286169041 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fcb800611f0 0x7fcb800636a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:38.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.610+0000 7fcba61a9700 1 --2- 172.21.15.112:0/1286169041 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcba00ca3e0 0x7fcba01652d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:38.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.610+0000 7fcba61a9700 1 --2- 172.21.15.112:0/1286169041 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcba00b0aa0 0x7fcba01552d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:38.619 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.610+0000 7fcba61a9700 1 --2- 172.21.15.112:0/1286169041 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcba00c7e40 0x7fcba015acf0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:38.619 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.610+0000 7fcba61a9700 1 -- 172.21.15.112:0/1286169041 >> 172.21.15.112:0/1286169041 conn(0x7fcba005e2c0 msgr2=0x7fcba0068660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:38.619 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.610+0000 7fcba61a9700 1 -- 172.21.15.112:0/1286169041 shutdown_connections 2023-01-24T03:30:38.619 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.610+0000 7fcba61a9700 1 -- 172.21.15.112:0/1286169041 wait complete. 2023-01-24T03:30:38.619 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2023-01-24T03:30:38.625 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2023-01-24T03:30:36.998984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":593126,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":205,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":69880,"kb_used_data":2168,"kb_used_omap":0,"kb_used_meta":67648,"kb_avail":754904840,"statfs":{"total":773094113280,"available":773022556160,"internally_reserved":0,"allocated":2220032,"data_stored":2018861,"data_compressed":64574,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":69271552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2758,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.623747"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993289+0000","last_change":"2023-01-24T03:30:20.986308+0000","last_active":"2023-01-24T03:30:24.993289+0000","last_peered":"2023-01-24T03:30:24.993289+0000","last_clean":"2023-01-24T03:30:24.993289+0000","last_became_active":"2023-01-24T03:30:20.986025+0000","last_became_peered":"2023-01-24T03:30:20.986025+0000","last_unstale":"2023-01-24T03:30:24.993289+0000","last_undegraded":"2023-01-24T03:30:24.993289+0000","last_fullsized":"2023-01-24T03:30:24.993289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:47:58.616698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.006202+0000","last_change":"2023-01-24T03:30:20.994478+0000","last_active":"2023-01-24T03:30:25.006202+0000","last_peered":"2023-01-24T03:30:25.006202+0000","last_clean":"2023-01-24T03:30:25.006202+0000","last_became_active":"2023-01-24T03:30:20.994207+0000","last_became_peered":"2023-01-24T03:30:20.994207+0000","last_unstale":"2023-01-24T03:30:25.006202+0000","last_undegraded":"2023-01-24T03:30:25.006202+0000","last_fullsized":"2023-01-24T03:30:25.006202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:02:53.230231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993177+0000","last_change":"2023-01-24T03:30:20.989651+0000","last_active":"2023-01-24T03:30:24.993177+0000","last_peered":"2023-01-24T03:30:24.993177+0000","last_clean":"2023-01-24T03:30:24.993177+0000","last_became_active":"2023-01-24T03:30:20.987585+0000","last_became_peered":"2023-01-24T03:30:20.987585+0000","last_unstale":"2023-01-24T03:30:24.993177+0000","last_undegraded":"2023-01-24T03:30:24.993177+0000","last_fullsized":"2023-01-24T03:30:24.993177+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:36:02.177544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990511+0000","last_change":"2023-01-24T03:30:20.993601+0000","last_active":"2023-01-24T03:30:24.990511+0000","last_peered":"2023-01-24T03:30:24.990511+0000","last_clean":"2023-01-24T03:30:24.990511+0000","last_became_active":"2023-01-24T03:30:20.993418+0000","last_became_peered":"2023-01-24T03:30:20.993418+0000","last_unstale":"2023-01-24T03:30:24.990511+0000","last_undegraded":"2023-01-24T03:30:24.990511+0000","last_fullsized":"2023-01-24T03:30:24.990511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:47:50.260497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996430+0000","last_change":"2023-01-24T03:30:20.991605+0000","last_active":"2023-01-24T03:30:24.996430+0000","last_peered":"2023-01-24T03:30:24.996430+0000","last_clean":"2023-01-24T03:30:24.996430+0000","last_became_active":"2023-01-24T03:30:20.991361+0000","last_became_peered":"2023-01-24T03:30:20.991361+0000","last_unstale":"2023-01-24T03:30:24.996430+0000","last_undegraded":"2023-01-24T03:30:24.996430+0000","last_fullsized":"2023-01-24T03:30:24.996430+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:31:25.418430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993288+0000","last_change":"2023-01-24T03:30:20.985058+0000","last_active":"2023-01-24T03:30:24.993288+0000","last_peered":"2023-01-24T03:30:24.993288+0000","last_clean":"2023-01-24T03:30:24.993288+0000","last_became_active":"2023-01-24T03:30:20.984754+0000","last_became_peered":"2023-01-24T03:30:20.984754+0000","last_unstale":"2023-01-24T03:30:24.993288+0000","last_undegraded":"2023-01-24T03:30:24.993288+0000","last_fullsized":"2023-01-24T03:30:24.993288+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:12:03.225109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996279+0000","last_change":"2023-01-24T03:30:21.000890+0000","last_active":"2023-01-24T03:30:24.996279+0000","last_peered":"2023-01-24T03:30:24.996279+0000","last_clean":"2023-01-24T03:30:24.996279+0000","last_became_active":"2023-01-24T03:30:21.000438+0000","last_became_peered":"2023-01-24T03:30:21.000438+0000","last_unstale":"2023-01-24T03:30:24.996279+0000","last_undegraded":"2023-01-24T03:30:24.996279+0000","last_fullsized":"2023-01-24T03:30:24.996279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:17:12.999664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.004717+0000","last_change":"2023-01-24T03:30:20.986753+0000","last_active":"2023-01-24T03:30:25.004717+0000","last_peered":"2023-01-24T03:30:25.004717+0000","last_clean":"2023-01-24T03:30:25.004717+0000","last_became_active":"2023-01-24T03:30:20.986524+0000","last_became_peered":"2023-01-24T03:30:20.986524+0000","last_unstale":"2023-01-24T03:30:25.004717+0000","last_undegraded":"2023-01-24T03:30:25.004717+0000","last_fullsized":"2023-01-24T03:30:25.004717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:24:55.699379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992350+0000","last_change":"2023-01-24T03:30:20.991136+0000","last_active":"2023-01-24T03:30:24.992350+0000","last_peered":"2023-01-24T03:30:24.992350+0000","last_clean":"2023-01-24T03:30:24.992350+0000","last_became_active":"2023-01-24T03:30:20.990925+0000","last_became_peered":"2023-01-24T03:30:20.990925+0000","last_unstale":"2023-01-24T03:30:24.992350+0000","last_undegraded":"2023-01-24T03:30:24.992350+0000","last_fullsized":"2023-01-24T03:30:24.992350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:45:37.485199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.006401+0000","last_change":"2023-01-24T03:30:20.995730+0000","last_active":"2023-01-24T03:30:25.006401+0000","last_peered":"2023-01-24T03:30:25.006401+0000","last_clean":"2023-01-24T03:30:25.006401+0000","last_became_active":"2023-01-24T03:30:20.994880+0000","last_became_peered":"2023-01-24T03:30:20.994880+0000","last_unstale":"2023-01-24T03:30:25.006401+0000","last_undegraded":"2023-01-24T03:30:25.006401+0000","last_fullsized":"2023-01-24T03:30:25.006401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:07:21.546386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993144+0000","last_change":"2023-01-24T03:30:20.993328+0000","last_active":"2023-01-24T03:30:24.993144+0000","last_peered":"2023-01-24T03:30:24.993144+0000","last_clean":"2023-01-24T03:30:24.993144+0000","last_became_active":"2023-01-24T03:30:20.992730+0000","last_became_peered":"2023-01-24T03:30:20.992730+0000","last_unstale":"2023-01-24T03:30:24.993144+0000","last_undegraded":"2023-01-24T03:30:24.993144+0000","last_fullsized":"2023-01-24T03:30:24.993144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:43:39.048550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.994605+0000","last_change":"2023-01-24T03:30:20.984993+0000","last_active":"2023-01-24T03:30:24.994605+0000","last_peered":"2023-01-24T03:30:24.994605+0000","last_clean":"2023-01-24T03:30:24.994605+0000","last_became_active":"2023-01-24T03:30:20.984540+0000","last_became_peered":"2023-01-24T03:30:20.984540+0000","last_unstale":"2023-01-24T03:30:24.994605+0000","last_undegraded":"2023-01-24T03:30:24.994605+0000","last_fullsized":"2023-01-24T03:30:24.994605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:16:39.993359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.995936+0000","last_change":"2023-01-24T03:30:20.986567+0000","last_active":"2023-01-24T03:30:24.995936+0000","last_peered":"2023-01-24T03:30:24.995936+0000","last_clean":"2023-01-24T03:30:24.995936+0000","last_became_active":"2023-01-24T03:30:20.986224+0000","last_became_peered":"2023-01-24T03:30:20.986224+0000","last_unstale":"2023-01-24T03:30:24.995936+0000","last_undegraded":"2023-01-24T03:30:24.995936+0000","last_fullsized":"2023-01-24T03:30:24.995936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:09:14.692354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005087+0000","last_change":"2023-01-24T03:30:20.988843+0000","last_active":"2023-01-24T03:30:25.005087+0000","last_peered":"2023-01-24T03:30:25.005087+0000","last_clean":"2023-01-24T03:30:25.005087+0000","last_became_active":"2023-01-24T03:30:20.988486+0000","last_became_peered":"2023-01-24T03:30:20.988486+0000","last_unstale":"2023-01-24T03:30:25.005087+0000","last_undegraded":"2023-01-24T03:30:25.005087+0000","last_fullsized":"2023-01-24T03:30:25.005087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:18:20.538844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996110+0000","last_change":"2023-01-24T03:30:21.000738+0000","last_active":"2023-01-24T03:30:24.996110+0000","last_peered":"2023-01-24T03:30:24.996110+0000","last_clean":"2023-01-24T03:30:24.996110+0000","last_became_active":"2023-01-24T03:30:21.000458+0000","last_became_peered":"2023-01-24T03:30:21.000458+0000","last_unstale":"2023-01-24T03:30:24.996110+0000","last_undegraded":"2023-01-24T03:30:24.996110+0000","last_fullsized":"2023-01-24T03:30:24.996110+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:21:42.876426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992383+0000","last_change":"2023-01-24T03:30:20.985062+0000","last_active":"2023-01-24T03:30:24.992383+0000","last_peered":"2023-01-24T03:30:24.992383+0000","last_clean":"2023-01-24T03:30:24.992383+0000","last_became_active":"2023-01-24T03:30:20.984759+0000","last_became_peered":"2023-01-24T03:30:20.984759+0000","last_unstale":"2023-01-24T03:30:24.992383+0000","last_undegraded":"2023-01-24T03:30:24.992383+0000","last_fullsized":"2023-01-24T03:30:24.992383+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:38:34.615748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992439+0000","last_change":"2023-01-24T03:30:20.989080+0000","last_active":"2023-01-24T03:30:24.992439+0000","last_peered":"2023-01-24T03:30:24.992439+0000","last_clean":"2023-01-24T03:30:24.992439+0000","last_became_active":"2023-01-24T03:30:20.988823+0000","last_became_peered":"2023-01-24T03:30:20.988823+0000","last_unstale":"2023-01-24T03:30:24.992439+0000","last_undegraded":"2023-01-24T03:30:24.992439+0000","last_fullsized":"2023-01-24T03:30:24.992439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:22:21.807370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993008+0000","last_change":"2023-01-24T03:30:20.984222+0000","last_active":"2023-01-24T03:30:24.993008+0000","last_peered":"2023-01-24T03:30:24.993008+0000","last_clean":"2023-01-24T03:30:24.993008+0000","last_became_active":"2023-01-24T03:30:20.984002+0000","last_became_peered":"2023-01-24T03:30:20.984002+0000","last_unstale":"2023-01-24T03:30:24.993008+0000","last_undegraded":"2023-01-24T03:30:24.993008+0000","last_fullsized":"2023-01-24T03:30:24.993008+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:54:24.005486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.997598+0000","last_change":"2023-01-24T03:30:20.988197+0000","last_active":"2023-01-24T03:30:24.997598+0000","last_peered":"2023-01-24T03:30:24.997598+0000","last_clean":"2023-01-24T03:30:24.997598+0000","last_became_active":"2023-01-24T03:30:20.987941+0000","last_became_peered":"2023-01-24T03:30:20.987941+0000","last_unstale":"2023-01-24T03:30:24.997598+0000","last_undegraded":"2023-01-24T03:30:24.997598+0000","last_fullsized":"2023-01-24T03:30:24.997598+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T15:08:29.659964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992957+0000","last_change":"2023-01-24T03:30:20.988701+0000","last_active":"2023-01-24T03:30:24.992957+0000","last_peered":"2023-01-24T03:30:24.992957+0000","last_clean":"2023-01-24T03:30:24.992957+0000","last_became_active":"2023-01-24T03:30:20.988476+0000","last_became_peered":"2023-01-24T03:30:20.988476+0000","last_unstale":"2023-01-24T03:30:24.992957+0000","last_undegraded":"2023-01-24T03:30:24.992957+0000","last_fullsized":"2023-01-24T03:30:24.992957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:33:29.463194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996546+0000","last_change":"2023-01-24T03:30:21.001125+0000","last_active":"2023-01-24T03:30:24.996546+0000","last_peered":"2023-01-24T03:30:24.996546+0000","last_clean":"2023-01-24T03:30:24.996546+0000","last_became_active":"2023-01-24T03:30:21.000873+0000","last_became_peered":"2023-01-24T03:30:21.000873+0000","last_unstale":"2023-01-24T03:30:24.996546+0000","last_undegraded":"2023-01-24T03:30:24.996546+0000","last_fullsized":"2023-01-24T03:30:24.996546+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:22:16.136875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993432+0000","last_change":"2023-01-24T03:30:20.992639+0000","last_active":"2023-01-24T03:30:24.993432+0000","last_peered":"2023-01-24T03:30:24.993432+0000","last_clean":"2023-01-24T03:30:24.993432+0000","last_became_active":"2023-01-24T03:30:20.992386+0000","last_became_peered":"2023-01-24T03:30:20.992386+0000","last_unstale":"2023-01-24T03:30:24.993432+0000","last_undegraded":"2023-01-24T03:30:24.993432+0000","last_fullsized":"2023-01-24T03:30:24.993432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:29:17.912198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992608+0000","last_change":"2023-01-24T03:30:20.994066+0000","last_active":"2023-01-24T03:30:24.992608+0000","last_peered":"2023-01-24T03:30:24.992608+0000","last_clean":"2023-01-24T03:30:24.992608+0000","last_became_active":"2023-01-24T03:30:20.993820+0000","last_became_peered":"2023-01-24T03:30:20.993820+0000","last_unstale":"2023-01-24T03:30:24.992608+0000","last_undegraded":"2023-01-24T03:30:24.992608+0000","last_fullsized":"2023-01-24T03:30:24.992608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:31:23.874058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996188+0000","last_change":"2023-01-24T03:30:21.001652+0000","last_active":"2023-01-24T03:30:24.996188+0000","last_peered":"2023-01-24T03:30:24.996188+0000","last_clean":"2023-01-24T03:30:24.996188+0000","last_became_active":"2023-01-24T03:30:21.001400+0000","last_became_peered":"2023-01-24T03:30:21.001400+0000","last_unstale":"2023-01-24T03:30:24.996188+0000","last_undegraded":"2023-01-24T03:30:24.996188+0000","last_fullsized":"2023-01-24T03:30:24.996188+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:55:53.249816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002243+0000","last_change":"2023-01-24T03:30:20.998304+0000","last_active":"2023-01-24T03:30:25.002243+0000","last_peered":"2023-01-24T03:30:25.002243+0000","last_clean":"2023-01-24T03:30:25.002243+0000","last_became_active":"2023-01-24T03:30:20.996288+0000","last_became_peered":"2023-01-24T03:30:20.996288+0000","last_unstale":"2023-01-24T03:30:25.002243+0000","last_undegraded":"2023-01-24T03:30:25.002243+0000","last_fullsized":"2023-01-24T03:30:25.002243+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:11:08.120197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002638+0000","last_change":"2023-01-24T03:30:20.996489+0000","last_active":"2023-01-24T03:30:25.002638+0000","last_peered":"2023-01-24T03:30:25.002638+0000","last_clean":"2023-01-24T03:30:25.002638+0000","last_became_active":"2023-01-24T03:30:20.994371+0000","last_became_peered":"2023-01-24T03:30:20.994371+0000","last_unstale":"2023-01-24T03:30:25.002638+0000","last_undegraded":"2023-01-24T03:30:25.002638+0000","last_fullsized":"2023-01-24T03:30:25.002638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:52:21.369928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005744+0000","last_change":"2023-01-24T03:30:20.997753+0000","last_active":"2023-01-24T03:30:25.005744+0000","last_peered":"2023-01-24T03:30:25.005744+0000","last_clean":"2023-01-24T03:30:25.005744+0000","last_became_active":"2023-01-24T03:30:20.997475+0000","last_became_peered":"2023-01-24T03:30:20.997475+0000","last_unstale":"2023-01-24T03:30:25.005744+0000","last_undegraded":"2023-01-24T03:30:25.005744+0000","last_fullsized":"2023-01-24T03:30:25.005744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:43:16.199843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.013267+0000","last_change":"2023-01-24T03:30:25.004944+0000","last_active":"2023-01-24T03:30:25.013267+0000","last_peered":"2023-01-24T03:30:25.013267+0000","last_clean":"2023-01-24T03:30:25.013267+0000","last_became_active":"2023-01-24T03:30:25.004590+0000","last_became_peered":"2023-01-24T03:30:25.004590+0000","last_unstale":"2023-01-24T03:30:25.013267+0000","last_undegraded":"2023-01-24T03:30:25.013267+0000","last_fullsized":"2023-01-24T03:30:25.013267+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:04:11.054225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2,0],"acting":[4,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"24'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.380450+0000","last_change":"2023-01-24T03:30:20.607211+0000","last_active":"2023-01-24T03:30:25.380450+0000","last_peered":"2023-01-24T03:30:25.380450+0000","last_clean":"2023-01-24T03:30:25.380450+0000","last_became_active":"2023-01-24T03:30:20.606855+0000","last_became_peered":"2023-01-24T03:30:20.606855+0000","last_unstale":"2023-01-24T03:30:25.380450+0000","last_undegraded":"2023-01-24T03:30:25.380450+0000","last_fullsized":"2023-01-24T03:30:25.380450+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:42:29.300089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991276+0000","last_change":"2023-01-24T03:30:20.993128+0000","last_active":"2023-01-24T03:30:24.991276+0000","last_peered":"2023-01-24T03:30:24.991276+0000","last_clean":"2023-01-24T03:30:24.991276+0000","last_became_active":"2023-01-24T03:30:20.992967+0000","last_became_peered":"2023-01-24T03:30:20.992967+0000","last_unstale":"2023-01-24T03:30:24.991276+0000","last_undegraded":"2023-01-24T03:30:24.991276+0000","last_fullsized":"2023-01-24T03:30:24.991276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:09:59.170218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.008814+0000","last_change":"2023-01-24T03:30:25.006569+0000","last_active":"2023-01-24T03:30:25.008814+0000","last_peered":"2023-01-24T03:30:25.008814+0000","last_clean":"2023-01-24T03:30:25.008814+0000","last_became_active":"2023-01-24T03:30:25.006199+0000","last_became_peered":"2023-01-24T03:30:25.006199+0000","last_unstale":"2023-01-24T03:30:25.008814+0000","last_undegraded":"2023-01-24T03:30:25.008814+0000","last_fullsized":"2023-01-24T03:30:25.008814+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:31:46.839485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,7],"acting":[6,4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.001971+0000","last_change":"2023-01-24T03:30:20.612738+0000","last_active":"2023-01-24T03:30:25.001971+0000","last_peered":"2023-01-24T03:30:25.001971+0000","last_clean":"2023-01-24T03:30:25.001971+0000","last_became_active":"2023-01-24T03:30:20.612375+0000","last_became_peered":"2023-01-24T03:30:20.612375+0000","last_unstale":"2023-01-24T03:30:25.001971+0000","last_undegraded":"2023-01-24T03:30:25.001971+0000","last_fullsized":"2023-01-24T03:30:25.001971+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:17:43.251945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993741+0000","last_change":"2023-01-24T03:30:20.996368+0000","last_active":"2023-01-24T03:30:24.993741+0000","last_peered":"2023-01-24T03:30:24.993741+0000","last_clean":"2023-01-24T03:30:24.993741+0000","last_became_active":"2023-01-24T03:30:20.996051+0000","last_became_peered":"2023-01-24T03:30:20.996051+0000","last_unstale":"2023-01-24T03:30:24.993741+0000","last_undegraded":"2023-01-24T03:30:24.993741+0000","last_fullsized":"2023-01-24T03:30:24.993741+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:53:28.193862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.014921+0000","last_change":"2023-01-24T03:30:25.008528+0000","last_active":"2023-01-24T03:30:25.014921+0000","last_peered":"2023-01-24T03:30:25.014921+0000","last_clean":"2023-01-24T03:30:25.014921+0000","last_became_active":"2023-01-24T03:30:25.008146+0000","last_became_peered":"2023-01-24T03:30:25.008146+0000","last_unstale":"2023-01-24T03:30:25.014921+0000","last_undegraded":"2023-01-24T03:30:25.014921+0000","last_fullsized":"2023-01-24T03:30:25.014921+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:17:11.457359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1,2],"acting":[0,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991341+0000","last_change":"2023-01-24T03:30:20.608932+0000","last_active":"2023-01-24T03:30:24.991341+0000","last_peered":"2023-01-24T03:30:24.991341+0000","last_clean":"2023-01-24T03:30:24.991341+0000","last_became_active":"2023-01-24T03:30:20.608135+0000","last_became_peered":"2023-01-24T03:30:20.608135+0000","last_unstale":"2023-01-24T03:30:24.991341+0000","last_undegraded":"2023-01-24T03:30:24.991341+0000","last_fullsized":"2023-01-24T03:30:24.991341+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:38:30.012279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990753+0000","last_change":"2023-01-24T03:30:20.986490+0000","last_active":"2023-01-24T03:30:24.990753+0000","last_peered":"2023-01-24T03:30:24.990753+0000","last_clean":"2023-01-24T03:30:24.990753+0000","last_became_active":"2023-01-24T03:30:20.986245+0000","last_became_peered":"2023-01-24T03:30:20.986245+0000","last_unstale":"2023-01-24T03:30:24.990753+0000","last_undegraded":"2023-01-24T03:30:24.990753+0000","last_fullsized":"2023-01-24T03:30:24.990753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:38:30.801032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.012186+0000","last_change":"2023-01-24T03:30:25.008221+0000","last_active":"2023-01-24T03:30:25.012186+0000","last_peered":"2023-01-24T03:30:25.012186+0000","last_clean":"2023-01-24T03:30:25.012186+0000","last_became_active":"2023-01-24T03:30:25.007855+0000","last_became_peered":"2023-01-24T03:30:25.007855+0000","last_unstale":"2023-01-24T03:30:25.012186+0000","last_undegraded":"2023-01-24T03:30:25.012186+0000","last_fullsized":"2023-01-24T03:30:25.012186+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:45:15.317067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,1],"acting":[6,4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992445+0000","last_change":"2023-01-24T03:30:20.607385+0000","last_active":"2023-01-24T03:30:24.992445+0000","last_peered":"2023-01-24T03:30:24.992445+0000","last_clean":"2023-01-24T03:30:24.992445+0000","last_became_active":"2023-01-24T03:30:20.607123+0000","last_became_peered":"2023-01-24T03:30:20.607123+0000","last_unstale":"2023-01-24T03:30:24.992445+0000","last_undegraded":"2023-01-24T03:30:24.992445+0000","last_fullsized":"2023-01-24T03:30:24.992445+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T15:24:00.474758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991258+0000","last_change":"2023-01-24T03:30:20.994124+0000","last_active":"2023-01-24T03:30:24.991258+0000","last_peered":"2023-01-24T03:30:24.991258+0000","last_clean":"2023-01-24T03:30:24.991258+0000","last_became_active":"2023-01-24T03:30:20.993876+0000","last_became_peered":"2023-01-24T03:30:20.993876+0000","last_unstale":"2023-01-24T03:30:24.991258+0000","last_undegraded":"2023-01-24T03:30:24.991258+0000","last_fullsized":"2023-01-24T03:30:24.991258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:07:26.252741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.011535+0000","last_change":"2023-01-24T03:30:25.006312+0000","last_active":"2023-01-24T03:30:25.011535+0000","last_peered":"2023-01-24T03:30:25.011535+0000","last_clean":"2023-01-24T03:30:25.011535+0000","last_became_active":"2023-01-24T03:30:25.005946+0000","last_became_peered":"2023-01-24T03:30:25.005946+0000","last_unstale":"2023-01-24T03:30:25.011535+0000","last_undegraded":"2023-01-24T03:30:25.011535+0000","last_fullsized":"2023-01-24T03:30:25.011535+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:35:02.760190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6,0],"acting":[7,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.006326+0000","last_change":"2023-01-24T03:30:20.610825+0000","last_active":"2023-01-24T03:30:25.006326+0000","last_peered":"2023-01-24T03:30:25.006326+0000","last_clean":"2023-01-24T03:30:25.006326+0000","last_became_active":"2023-01-24T03:30:20.610540+0000","last_became_peered":"2023-01-24T03:30:20.610540+0000","last_unstale":"2023-01-24T03:30:25.006326+0000","last_undegraded":"2023-01-24T03:30:25.006326+0000","last_fullsized":"2023-01-24T03:30:25.006326+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:23:15.279283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990818+0000","last_change":"2023-01-24T03:30:20.994605+0000","last_active":"2023-01-24T03:30:24.990818+0000","last_peered":"2023-01-24T03:30:24.990818+0000","last_clean":"2023-01-24T03:30:24.990818+0000","last_became_active":"2023-01-24T03:30:20.994356+0000","last_became_peered":"2023-01-24T03:30:20.994356+0000","last_unstale":"2023-01-24T03:30:24.990818+0000","last_undegraded":"2023-01-24T03:30:24.990818+0000","last_fullsized":"2023-01-24T03:30:24.990818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:06:08.797922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.018595+0000","last_change":"2023-01-24T03:30:25.012301+0000","last_active":"2023-01-24T03:30:25.018595+0000","last_peered":"2023-01-24T03:30:25.018595+0000","last_clean":"2023-01-24T03:30:25.018595+0000","last_became_active":"2023-01-24T03:30:25.011891+0000","last_became_peered":"2023-01-24T03:30:25.011891+0000","last_unstale":"2023-01-24T03:30:25.018595+0000","last_undegraded":"2023-01-24T03:30:25.018595+0000","last_fullsized":"2023-01-24T03:30:25.018595+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:15:46.012675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,5],"acting":[3,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"24'3","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.986137+0000","last_change":"2023-01-24T03:30:20.609828+0000","last_active":"2023-01-24T03:30:25.986137+0000","last_peered":"2023-01-24T03:30:25.986137+0000","last_clean":"2023-01-24T03:30:25.986137+0000","last_became_active":"2023-01-24T03:30:20.609492+0000","last_became_peered":"2023-01-24T03:30:20.609492+0000","last_unstale":"2023-01-24T03:30:25.986137+0000","last_undegraded":"2023-01-24T03:30:25.986137+0000","last_fullsized":"2023-01-24T03:30:25.986137+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:22:53.430711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002553+0000","last_change":"2023-01-24T03:30:20.999747+0000","last_active":"2023-01-24T03:30:25.002553+0000","last_peered":"2023-01-24T03:30:25.002553+0000","last_clean":"2023-01-24T03:30:25.002553+0000","last_became_active":"2023-01-24T03:30:20.999003+0000","last_became_peered":"2023-01-24T03:30:20.999003+0000","last_unstale":"2023-01-24T03:30:25.002553+0000","last_undegraded":"2023-01-24T03:30:25.002553+0000","last_fullsized":"2023-01-24T03:30:25.002553+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:19:22.205251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016573+0000","last_change":"2023-01-24T03:30:25.010659+0000","last_active":"2023-01-24T03:30:25.016573+0000","last_peered":"2023-01-24T03:30:25.016573+0000","last_clean":"2023-01-24T03:30:25.016573+0000","last_became_active":"2023-01-24T03:30:25.010237+0000","last_became_peered":"2023-01-24T03:30:25.010237+0000","last_unstale":"2023-01-24T03:30:25.016573+0000","last_undegraded":"2023-01-24T03:30:25.016573+0000","last_fullsized":"2023-01-24T03:30:25.016573+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:01:53.501070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3,7],"acting":[1,5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"24'7","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.988114+0000","last_change":"2023-01-24T03:30:20.610682+0000","last_active":"2023-01-24T03:30:25.988114+0000","last_peered":"2023-01-24T03:30:25.988114+0000","last_clean":"2023-01-24T03:30:25.988114+0000","last_became_active":"2023-01-24T03:30:20.610463+0000","last_became_peered":"2023-01-24T03:30:20.610463+0000","last_unstale":"2023-01-24T03:30:25.988114+0000","last_undegraded":"2023-01-24T03:30:25.988114+0000","last_fullsized":"2023-01-24T03:30:25.988114+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:25:00.447596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1988,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992845+0000","last_change":"2023-01-24T03:30:20.987326+0000","last_active":"2023-01-24T03:30:24.992845+0000","last_peered":"2023-01-24T03:30:24.992845+0000","last_clean":"2023-01-24T03:30:24.992845+0000","last_became_active":"2023-01-24T03:30:20.987042+0000","last_became_peered":"2023-01-24T03:30:20.987042+0000","last_unstale":"2023-01-24T03:30:24.992845+0000","last_undegraded":"2023-01-24T03:30:24.992845+0000","last_fullsized":"2023-01-24T03:30:24.992845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:57:43.257624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.020022+0000","last_change":"2023-01-24T03:30:25.012004+0000","last_active":"2023-01-24T03:30:25.020022+0000","last_peered":"2023-01-24T03:30:25.020022+0000","last_clean":"2023-01-24T03:30:25.020022+0000","last_became_active":"2023-01-24T03:30:25.007410+0000","last_became_peered":"2023-01-24T03:30:25.007410+0000","last_unstale":"2023-01-24T03:30:25.020022+0000","last_undegraded":"2023-01-24T03:30:25.020022+0000","last_fullsized":"2023-01-24T03:30:25.020022+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:34:21.681011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3,0],"acting":[5,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"24'2","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985201+0000","last_change":"2023-01-24T03:30:20.613026+0000","last_active":"2023-01-24T03:30:25.985201+0000","last_peered":"2023-01-24T03:30:25.985201+0000","last_clean":"2023-01-24T03:30:25.985201+0000","last_became_active":"2023-01-24T03:30:20.612605+0000","last_became_peered":"2023-01-24T03:30:20.612605+0000","last_unstale":"2023-01-24T03:30:25.985201+0000","last_undegraded":"2023-01-24T03:30:25.985201+0000","last_fullsized":"2023-01-24T03:30:25.985201+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:45:08.016740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991113+0000","last_change":"2023-01-24T03:30:20.991929+0000","last_active":"2023-01-24T03:30:24.991113+0000","last_peered":"2023-01-24T03:30:24.991113+0000","last_clean":"2023-01-24T03:30:24.991113+0000","last_became_active":"2023-01-24T03:30:20.991693+0000","last_became_peered":"2023-01-24T03:30:20.991693+0000","last_unstale":"2023-01-24T03:30:24.991113+0000","last_undegraded":"2023-01-24T03:30:24.991113+0000","last_fullsized":"2023-01-24T03:30:24.991113+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:55:51.637568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.018472+0000","last_change":"2023-01-24T03:30:25.010572+0000","last_active":"2023-01-24T03:30:25.018472+0000","last_peered":"2023-01-24T03:30:25.018472+0000","last_clean":"2023-01-24T03:30:25.018472+0000","last_became_active":"2023-01-24T03:30:25.010193+0000","last_became_peered":"2023-01-24T03:30:25.010193+0000","last_unstale":"2023-01-24T03:30:25.018472+0000","last_undegraded":"2023-01-24T03:30:25.018472+0000","last_fullsized":"2023-01-24T03:30:25.018472+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:25:39.676179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,1],"acting":[3,7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993579+0000","last_change":"2023-01-24T03:30:20.612047+0000","last_active":"2023-01-24T03:30:24.993579+0000","last_peered":"2023-01-24T03:30:24.993579+0000","last_clean":"2023-01-24T03:30:24.993579+0000","last_became_active":"2023-01-24T03:30:20.611798+0000","last_became_peered":"2023-01-24T03:30:20.611798+0000","last_unstale":"2023-01-24T03:30:24.993579+0000","last_undegraded":"2023-01-24T03:30:24.993579+0000","last_fullsized":"2023-01-24T03:30:24.993579+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:01:41.731689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005194+0000","last_change":"2023-01-24T03:30:20.996432+0000","last_active":"2023-01-24T03:30:25.005194+0000","last_peered":"2023-01-24T03:30:25.005194+0000","last_clean":"2023-01-24T03:30:25.005194+0000","last_became_active":"2023-01-24T03:30:20.996173+0000","last_became_peered":"2023-01-24T03:30:20.996173+0000","last_unstale":"2023-01-24T03:30:25.005194+0000","last_undegraded":"2023-01-24T03:30:25.005194+0000","last_fullsized":"2023-01-24T03:30:25.005194+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:13:31.596587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.018507+0000","last_change":"2023-01-24T03:30:25.008479+0000","last_active":"2023-01-24T03:30:25.018507+0000","last_peered":"2023-01-24T03:30:25.018507+0000","last_clean":"2023-01-24T03:30:25.018507+0000","last_became_active":"2023-01-24T03:30:25.007858+0000","last_became_peered":"2023-01-24T03:30:25.007858+0000","last_unstale":"2023-01-24T03:30:25.018507+0000","last_undegraded":"2023-01-24T03:30:25.018507+0000","last_fullsized":"2023-01-24T03:30:25.018507+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:10:38.425874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,0],"acting":[3,1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996134+0000","last_change":"2023-01-24T03:30:20.612848+0000","last_active":"2023-01-24T03:30:24.996134+0000","last_peered":"2023-01-24T03:30:24.996134+0000","last_clean":"2023-01-24T03:30:24.996134+0000","last_became_active":"2023-01-24T03:30:20.612600+0000","last_became_peered":"2023-01-24T03:30:20.612600+0000","last_unstale":"2023-01-24T03:30:24.996134+0000","last_undegraded":"2023-01-24T03:30:24.996134+0000","last_fullsized":"2023-01-24T03:30:24.996134+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:46:45.634472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993883+0000","last_change":"2023-01-24T03:30:20.991133+0000","last_active":"2023-01-24T03:30:24.993883+0000","last_peered":"2023-01-24T03:30:24.993883+0000","last_clean":"2023-01-24T03:30:24.993883+0000","last_became_active":"2023-01-24T03:30:20.990856+0000","last_became_peered":"2023-01-24T03:30:20.990856+0000","last_unstale":"2023-01-24T03:30:24.993883+0000","last_undegraded":"2023-01-24T03:30:24.993883+0000","last_fullsized":"2023-01-24T03:30:24.993883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:27:32.163224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.022452+0000","last_change":"2023-01-24T03:30:25.017038+0000","last_active":"2023-01-24T03:30:25.022452+0000","last_peered":"2023-01-24T03:30:25.022452+0000","last_clean":"2023-01-24T03:30:25.022452+0000","last_became_active":"2023-01-24T03:30:25.016684+0000","last_became_peered":"2023-01-24T03:30:25.016684+0000","last_unstale":"2023-01-24T03:30:25.022452+0000","last_undegraded":"2023-01-24T03:30:25.022452+0000","last_fullsized":"2023-01-24T03:30:25.022452+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:57:30.028834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1,6],"acting":[5,3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"24'8","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.988793+0000","last_change":"2023-01-24T03:30:20.604617+0000","last_active":"2023-01-24T03:30:25.988793+0000","last_peered":"2023-01-24T03:30:25.988793+0000","last_clean":"2023-01-24T03:30:25.988793+0000","last_became_active":"2023-01-24T03:30:20.604445+0000","last_became_peered":"2023-01-24T03:30:20.604445+0000","last_unstale":"2023-01-24T03:30:25.988793+0000","last_undegraded":"2023-01-24T03:30:25.988793+0000","last_fullsized":"2023-01-24T03:30:25.988793+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:06:51.460895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993017+0000","last_change":"2023-01-24T03:30:20.995872+0000","last_active":"2023-01-24T03:30:24.993017+0000","last_peered":"2023-01-24T03:30:24.993017+0000","last_clean":"2023-01-24T03:30:24.993017+0000","last_became_active":"2023-01-24T03:30:20.995599+0000","last_became_peered":"2023-01-24T03:30:20.995599+0000","last_unstale":"2023-01-24T03:30:24.993017+0000","last_undegraded":"2023-01-24T03:30:24.993017+0000","last_fullsized":"2023-01-24T03:30:24.993017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:53:54.356899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.020130+0000","last_change":"2023-01-24T03:30:25.016101+0000","last_active":"2023-01-24T03:30:25.020130+0000","last_peered":"2023-01-24T03:30:25.020130+0000","last_clean":"2023-01-24T03:30:25.020130+0000","last_became_active":"2023-01-24T03:30:25.015705+0000","last_became_peered":"2023-01-24T03:30:25.015705+0000","last_unstale":"2023-01-24T03:30:25.020130+0000","last_undegraded":"2023-01-24T03:30:25.020130+0000","last_fullsized":"2023-01-24T03:30:25.020130+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:54:37.798191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3,4],"acting":[7,6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"24'1","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.446876+0000","last_change":"2023-01-24T03:30:20.610632+0000","last_active":"2023-01-24T03:30:25.446876+0000","last_peered":"2023-01-24T03:30:25.446876+0000","last_clean":"2023-01-24T03:30:25.446876+0000","last_became_active":"2023-01-24T03:30:20.610367+0000","last_became_peered":"2023-01-24T03:30:20.610367+0000","last_unstale":"2023-01-24T03:30:25.446876+0000","last_undegraded":"2023-01-24T03:30:25.446876+0000","last_fullsized":"2023-01-24T03:30:25.446876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:26:05.938178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.004623+0000","last_change":"2023-01-24T03:30:20.996493+0000","last_active":"2023-01-24T03:30:25.004623+0000","last_peered":"2023-01-24T03:30:25.004623+0000","last_clean":"2023-01-24T03:30:25.004623+0000","last_became_active":"2023-01-24T03:30:20.996229+0000","last_became_peered":"2023-01-24T03:30:20.996229+0000","last_unstale":"2023-01-24T03:30:25.004623+0000","last_undegraded":"2023-01-24T03:30:25.004623+0000","last_fullsized":"2023-01-24T03:30:25.004623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:23:53.642522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016195+0000","last_change":"2023-01-24T03:30:25.009215+0000","last_active":"2023-01-24T03:30:25.016195+0000","last_peered":"2023-01-24T03:30:25.016195+0000","last_clean":"2023-01-24T03:30:25.016195+0000","last_became_active":"2023-01-24T03:30:25.008213+0000","last_became_peered":"2023-01-24T03:30:25.008213+0000","last_unstale":"2023-01-24T03:30:25.016195+0000","last_undegraded":"2023-01-24T03:30:25.016195+0000","last_fullsized":"2023-01-24T03:30:25.016195+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:48:37.517172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,5],"acting":[2,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"24'2","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985223+0000","last_change":"2023-01-24T03:30:20.612158+0000","last_active":"2023-01-24T03:30:25.985223+0000","last_peered":"2023-01-24T03:30:25.985223+0000","last_clean":"2023-01-24T03:30:25.985223+0000","last_became_active":"2023-01-24T03:30:20.611931+0000","last_became_peered":"2023-01-24T03:30:20.611931+0000","last_unstale":"2023-01-24T03:30:25.985223+0000","last_undegraded":"2023-01-24T03:30:25.985223+0000","last_fullsized":"2023-01-24T03:30:25.985223+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:30:37.532390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992500+0000","last_change":"2023-01-24T03:30:20.991582+0000","last_active":"2023-01-24T03:30:24.992500+0000","last_peered":"2023-01-24T03:30:24.992500+0000","last_clean":"2023-01-24T03:30:24.992500+0000","last_became_active":"2023-01-24T03:30:20.991335+0000","last_became_peered":"2023-01-24T03:30:20.991335+0000","last_unstale":"2023-01-24T03:30:24.992500+0000","last_undegraded":"2023-01-24T03:30:24.992500+0000","last_fullsized":"2023-01-24T03:30:24.992500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:45:20.786712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016224+0000","last_change":"2023-01-24T03:30:25.008058+0000","last_active":"2023-01-24T03:30:25.016224+0000","last_peered":"2023-01-24T03:30:25.016224+0000","last_clean":"2023-01-24T03:30:25.016224+0000","last_became_active":"2023-01-24T03:30:25.007711+0000","last_became_peered":"2023-01-24T03:30:25.007711+0000","last_unstale":"2023-01-24T03:30:25.016224+0000","last_undegraded":"2023-01-24T03:30:25.016224+0000","last_fullsized":"2023-01-24T03:30:25.016224+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:26:56.270674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5,6],"acting":[2,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"24'1","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.447163+0000","last_change":"2023-01-24T03:30:20.610495+0000","last_active":"2023-01-24T03:30:25.447163+0000","last_peered":"2023-01-24T03:30:25.447163+0000","last_clean":"2023-01-24T03:30:25.447163+0000","last_became_active":"2023-01-24T03:30:20.610035+0000","last_became_peered":"2023-01-24T03:30:20.610035+0000","last_unstale":"2023-01-24T03:30:25.447163+0000","last_undegraded":"2023-01-24T03:30:25.447163+0000","last_fullsized":"2023-01-24T03:30:25.447163+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:25:30.944275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.989631+0000","last_change":"2023-01-24T03:30:20.994362+0000","last_active":"2023-01-24T03:30:24.989631+0000","last_peered":"2023-01-24T03:30:24.989631+0000","last_clean":"2023-01-24T03:30:24.989631+0000","last_became_active":"2023-01-24T03:30:20.994101+0000","last_became_peered":"2023-01-24T03:30:20.994101+0000","last_unstale":"2023-01-24T03:30:24.989631+0000","last_undegraded":"2023-01-24T03:30:24.989631+0000","last_fullsized":"2023-01-24T03:30:24.989631+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:55:12.837646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.015999+0000","last_change":"2023-01-24T03:30:25.006995+0000","last_active":"2023-01-24T03:30:25.015999+0000","last_peered":"2023-01-24T03:30:25.015999+0000","last_clean":"2023-01-24T03:30:25.015999+0000","last_became_active":"2023-01-24T03:30:25.006473+0000","last_became_peered":"2023-01-24T03:30:25.006473+0000","last_unstale":"2023-01-24T03:30:25.015999+0000","last_undegraded":"2023-01-24T03:30:25.015999+0000","last_fullsized":"2023-01-24T03:30:25.015999+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:48:16.320162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3,6],"acting":[2,4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005342+0000","last_change":"2023-01-24T03:30:20.606526+0000","last_active":"2023-01-24T03:30:25.005342+0000","last_peered":"2023-01-24T03:30:25.005342+0000","last_clean":"2023-01-24T03:30:25.005342+0000","last_became_active":"2023-01-24T03:30:20.606301+0000","last_became_peered":"2023-01-24T03:30:20.606301+0000","last_unstale":"2023-01-24T03:30:25.005342+0000","last_undegraded":"2023-01-24T03:30:25.005342+0000","last_fullsized":"2023-01-24T03:30:25.005342+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:43:28.702475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992428+0000","last_change":"2023-01-24T03:30:20.993379+0000","last_active":"2023-01-24T03:30:24.992428+0000","last_peered":"2023-01-24T03:30:24.992428+0000","last_clean":"2023-01-24T03:30:24.992428+0000","last_became_active":"2023-01-24T03:30:20.993151+0000","last_became_peered":"2023-01-24T03:30:20.993151+0000","last_unstale":"2023-01-24T03:30:24.992428+0000","last_undegraded":"2023-01-24T03:30:24.992428+0000","last_fullsized":"2023-01-24T03:30:24.992428+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:00:45.845082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.020912+0000","last_change":"2023-01-24T03:30:25.013701+0000","last_active":"2023-01-24T03:30:25.020912+0000","last_peered":"2023-01-24T03:30:25.020912+0000","last_clean":"2023-01-24T03:30:25.020912+0000","last_became_active":"2023-01-24T03:30:25.013290+0000","last_became_peered":"2023-01-24T03:30:25.013290+0000","last_unstale":"2023-01-24T03:30:25.020912+0000","last_undegraded":"2023-01-24T03:30:25.020912+0000","last_fullsized":"2023-01-24T03:30:25.020912+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:32:17.340583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5,6],"acting":[2,7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"24'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985860+0000","last_change":"2023-01-24T03:30:20.609656+0000","last_active":"2023-01-24T03:30:25.985860+0000","last_peered":"2023-01-24T03:30:25.985860+0000","last_clean":"2023-01-24T03:30:25.985860+0000","last_became_active":"2023-01-24T03:30:20.609374+0000","last_became_peered":"2023-01-24T03:30:20.609374+0000","last_unstale":"2023-01-24T03:30:25.985860+0000","last_undegraded":"2023-01-24T03:30:25.985860+0000","last_fullsized":"2023-01-24T03:30:25.985860+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:58:12.653746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992726+0000","last_change":"2023-01-24T03:30:20.986089+0000","last_active":"2023-01-24T03:30:24.992726+0000","last_peered":"2023-01-24T03:30:24.992726+0000","last_clean":"2023-01-24T03:30:24.992726+0000","last_became_active":"2023-01-24T03:30:20.985857+0000","last_became_peered":"2023-01-24T03:30:20.985857+0000","last_unstale":"2023-01-24T03:30:24.992726+0000","last_undegraded":"2023-01-24T03:30:24.992726+0000","last_fullsized":"2023-01-24T03:30:24.992726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:00:57.338328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016603+0000","last_change":"2023-01-24T03:30:25.009962+0000","last_active":"2023-01-24T03:30:25.016603+0000","last_peered":"2023-01-24T03:30:25.016603+0000","last_clean":"2023-01-24T03:30:25.016603+0000","last_became_active":"2023-01-24T03:30:25.009623+0000","last_became_peered":"2023-01-24T03:30:25.009623+0000","last_unstale":"2023-01-24T03:30:25.016603+0000","last_undegraded":"2023-01-24T03:30:25.016603+0000","last_fullsized":"2023-01-24T03:30:25.016603+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:24:42.958531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0,6],"acting":[1,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.989699+0000","last_change":"2023-01-24T03:30:20.608114+0000","last_active":"2023-01-24T03:30:24.989699+0000","last_peered":"2023-01-24T03:30:24.989699+0000","last_clean":"2023-01-24T03:30:24.989699+0000","last_became_active":"2023-01-24T03:30:20.607816+0000","last_became_peered":"2023-01-24T03:30:20.607816+0000","last_unstale":"2023-01-24T03:30:24.989699+0000","last_undegraded":"2023-01-24T03:30:24.989699+0000","last_fullsized":"2023-01-24T03:30:24.989699+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:40:00.784496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992908+0000","last_change":"2023-01-24T03:30:20.982589+0000","last_active":"2023-01-24T03:30:24.992908+0000","last_peered":"2023-01-24T03:30:24.992908+0000","last_clean":"2023-01-24T03:30:24.992908+0000","last_became_active":"2023-01-24T03:30:20.979779+0000","last_became_peered":"2023-01-24T03:30:20.979779+0000","last_unstale":"2023-01-24T03:30:24.992908+0000","last_undegraded":"2023-01-24T03:30:24.992908+0000","last_fullsized":"2023-01-24T03:30:24.992908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:01:18.491363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.019938+0000","last_change":"2023-01-24T03:30:25.013327+0000","last_active":"2023-01-24T03:30:25.019938+0000","last_peered":"2023-01-24T03:30:25.019938+0000","last_clean":"2023-01-24T03:30:25.019938+0000","last_became_active":"2023-01-24T03:30:25.011303+0000","last_became_peered":"2023-01-24T03:30:25.011303+0000","last_unstale":"2023-01-24T03:30:25.019938+0000","last_undegraded":"2023-01-24T03:30:25.019938+0000","last_fullsized":"2023-01-24T03:30:25.019938+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:14:43.169287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6,2],"acting":[5,4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992390+0000","last_change":"2023-01-24T03:30:20.602369+0000","last_active":"2023-01-24T03:30:24.992390+0000","last_peered":"2023-01-24T03:30:24.992390+0000","last_clean":"2023-01-24T03:30:24.992390+0000","last_became_active":"2023-01-24T03:30:20.601996+0000","last_became_peered":"2023-01-24T03:30:20.601996+0000","last_unstale":"2023-01-24T03:30:24.992390+0000","last_undegraded":"2023-01-24T03:30:24.992390+0000","last_fullsized":"2023-01-24T03:30:24.992390+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:33:55.328080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993109+0000","last_change":"2023-01-24T03:30:20.988738+0000","last_active":"2023-01-24T03:30:24.993109+0000","last_peered":"2023-01-24T03:30:24.993109+0000","last_clean":"2023-01-24T03:30:24.993109+0000","last_became_active":"2023-01-24T03:30:20.987190+0000","last_became_peered":"2023-01-24T03:30:20.987190+0000","last_unstale":"2023-01-24T03:30:24.993109+0000","last_undegraded":"2023-01-24T03:30:24.993109+0000","last_fullsized":"2023-01-24T03:30:24.993109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:32:05.689674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.013303+0000","last_change":"2023-01-24T03:30:25.008401+0000","last_active":"2023-01-24T03:30:25.013303+0000","last_peered":"2023-01-24T03:30:25.013303+0000","last_clean":"2023-01-24T03:30:25.013303+0000","last_became_active":"2023-01-24T03:30:25.007973+0000","last_became_peered":"2023-01-24T03:30:25.007973+0000","last_unstale":"2023-01-24T03:30:25.013303+0000","last_undegraded":"2023-01-24T03:30:25.013303+0000","last_fullsized":"2023-01-24T03:30:25.013303+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:36:22.339295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,2],"acting":[4,5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991180+0000","last_change":"2023-01-24T03:30:20.608145+0000","last_active":"2023-01-24T03:30:24.991180+0000","last_peered":"2023-01-24T03:30:24.991180+0000","last_clean":"2023-01-24T03:30:24.991180+0000","last_became_active":"2023-01-24T03:30:20.607165+0000","last_became_peered":"2023-01-24T03:30:20.607165+0000","last_unstale":"2023-01-24T03:30:24.991180+0000","last_undegraded":"2023-01-24T03:30:24.991180+0000","last_fullsized":"2023-01-24T03:30:24.991180+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:58:56.490482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002416+0000","last_change":"2023-01-24T03:30:20.998777+0000","last_active":"2023-01-24T03:30:25.002416+0000","last_peered":"2023-01-24T03:30:25.002416+0000","last_clean":"2023-01-24T03:30:25.002416+0000","last_became_active":"2023-01-24T03:30:20.998519+0000","last_became_peered":"2023-01-24T03:30:20.998519+0000","last_unstale":"2023-01-24T03:30:25.002416+0000","last_undegraded":"2023-01-24T03:30:25.002416+0000","last_fullsized":"2023-01-24T03:30:25.002416+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:55:55.320556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.012455+0000","last_change":"2023-01-24T03:30:25.008154+0000","last_active":"2023-01-24T03:30:25.012455+0000","last_peered":"2023-01-24T03:30:25.012455+0000","last_clean":"2023-01-24T03:30:25.012455+0000","last_became_active":"2023-01-24T03:30:25.007797+0000","last_became_peered":"2023-01-24T03:30:25.007797+0000","last_unstale":"2023-01-24T03:30:25.012455+0000","last_undegraded":"2023-01-24T03:30:25.012455+0000","last_fullsized":"2023-01-24T03:30:25.012455+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:29:05.153733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7,0],"acting":[4,3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"24'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985835+0000","last_change":"2023-01-24T03:30:20.610690+0000","last_active":"2023-01-24T03:30:25.985835+0000","last_peered":"2023-01-24T03:30:25.985835+0000","last_clean":"2023-01-24T03:30:25.985835+0000","last_became_active":"2023-01-24T03:30:20.610409+0000","last_became_peered":"2023-01-24T03:30:20.610409+0000","last_unstale":"2023-01-24T03:30:25.985835+0000","last_undegraded":"2023-01-24T03:30:25.985835+0000","last_fullsized":"2023-01-24T03:30:25.985835+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:43:37.740181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993033+0000","last_change":"2023-01-24T03:30:16.085885+0000","last_active":"2023-01-24T03:30:24.993033+0000","last_peered":"2023-01-24T03:30:24.993033+0000","last_clean":"2023-01-24T03:30:24.993033+0000","last_became_active":"2023-01-24T03:30:16.085525+0000","last_became_peered":"2023-01-24T03:30:16.085525+0000","last_unstale":"2023-01-24T03:30:24.993033+0000","last_undegraded":"2023-01-24T03:30:24.993033+0000","last_fullsized":"2023-01-24T03:30:24.993033+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:57:36.499644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991099+0000","last_change":"2023-01-24T03:30:20.996370+0000","last_active":"2023-01-24T03:30:24.991099+0000","last_peered":"2023-01-24T03:30:24.991099+0000","last_clean":"2023-01-24T03:30:24.991099+0000","last_became_active":"2023-01-24T03:30:20.996122+0000","last_became_peered":"2023-01-24T03:30:20.996122+0000","last_unstale":"2023-01-24T03:30:24.991099+0000","last_undegraded":"2023-01-24T03:30:24.991099+0000","last_fullsized":"2023-01-24T03:30:24.991099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:13:06.913474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.014170+0000","last_change":"2023-01-24T03:30:25.007262+0000","last_active":"2023-01-24T03:30:25.014170+0000","last_peered":"2023-01-24T03:30:25.014170+0000","last_clean":"2023-01-24T03:30:25.014170+0000","last_became_active":"2023-01-24T03:30:25.006862+0000","last_became_peered":"2023-01-24T03:30:25.006862+0000","last_unstale":"2023-01-24T03:30:25.014170+0000","last_undegraded":"2023-01-24T03:30:25.014170+0000","last_fullsized":"2023-01-24T03:30:25.014170+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:42:05.268862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,0],"acting":[3,1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991165+0000","last_change":"2023-01-24T03:30:16.057240+0000","last_active":"2023-01-24T03:30:24.991165+0000","last_peered":"2023-01-24T03:30:24.991165+0000","last_clean":"2023-01-24T03:30:24.991165+0000","last_became_active":"2023-01-24T03:30:16.056941+0000","last_became_peered":"2023-01-24T03:30:16.056941+0000","last_unstale":"2023-01-24T03:30:24.991165+0000","last_undegraded":"2023-01-24T03:30:24.991165+0000","last_fullsized":"2023-01-24T03:30:24.991165+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:22:26.168527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"24'1","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.447279+0000","last_change":"2023-01-24T03:30:20.606130+0000","last_active":"2023-01-24T03:30:25.447279+0000","last_peered":"2023-01-24T03:30:25.447279+0000","last_clean":"2023-01-24T03:30:25.447279+0000","last_became_active":"2023-01-24T03:30:20.605899+0000","last_became_peered":"2023-01-24T03:30:20.605899+0000","last_unstale":"2023-01-24T03:30:25.447279+0000","last_undegraded":"2023-01-24T03:30:25.447279+0000","last_fullsized":"2023-01-24T03:30:25.447279+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:30:47.956569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992411+0000","last_change":"2023-01-24T03:30:20.988906+0000","last_active":"2023-01-24T03:30:24.992411+0000","last_peered":"2023-01-24T03:30:24.992411+0000","last_clean":"2023-01-24T03:30:24.992411+0000","last_became_active":"2023-01-24T03:30:20.988688+0000","last_became_peered":"2023-01-24T03:30:20.988688+0000","last_unstale":"2023-01-24T03:30:24.992411+0000","last_undegraded":"2023-01-24T03:30:24.992411+0000","last_fullsized":"2023-01-24T03:30:24.992411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:25:43.365167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.019966+0000","last_change":"2023-01-24T03:30:25.013834+0000","last_active":"2023-01-24T03:30:25.019966+0000","last_peered":"2023-01-24T03:30:25.019966+0000","last_clean":"2023-01-24T03:30:25.019966+0000","last_became_active":"2023-01-24T03:30:25.011774+0000","last_became_peered":"2023-01-24T03:30:25.011774+0000","last_unstale":"2023-01-24T03:30:25.019966+0000","last_undegraded":"2023-01-24T03:30:25.019966+0000","last_fullsized":"2023-01-24T03:30:25.019966+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:02:50.158049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0,4],"acting":[5,1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002878+0000","last_change":"2023-01-24T03:30:16.051822+0000","last_active":"2023-01-24T03:30:25.002878+0000","last_peered":"2023-01-24T03:30:25.002878+0000","last_clean":"2023-01-24T03:30:25.002878+0000","last_became_active":"2023-01-24T03:30:16.051525+0000","last_became_peered":"2023-01-24T03:30:16.051525+0000","last_unstale":"2023-01-24T03:30:25.002878+0000","last_undegraded":"2023-01-24T03:30:25.002878+0000","last_fullsized":"2023-01-24T03:30:25.002878+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:37:30.802424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"16'117","reported_seq":166,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996317+0000","last_change":"2023-01-24T03:30:16.054907+0000","last_active":"2023-01-24T03:30:24.996317+0000","last_peered":"2023-01-24T03:30:24.996317+0000","last_clean":"2023-01-24T03:30:24.996317+0000","last_became_active":"2023-01-24T03:30:16.054071+0000","last_became_peered":"2023-01-24T03:30:16.054071+0000","last_unstale":"2023-01-24T03:30:24.996317+0000","last_undegraded":"2023-01-24T03:30:24.996317+0000","last_fullsized":"2023-01-24T03:30:24.996317+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:31:26.790766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992331+0000","last_change":"2023-01-24T03:30:20.607914+0000","last_active":"2023-01-24T03:30:24.992331+0000","last_peered":"2023-01-24T03:30:24.992331+0000","last_clean":"2023-01-24T03:30:24.992331+0000","last_became_active":"2023-01-24T03:30:20.607663+0000","last_became_peered":"2023-01-24T03:30:20.607663+0000","last_unstale":"2023-01-24T03:30:24.992331+0000","last_undegraded":"2023-01-24T03:30:24.992331+0000","last_fullsized":"2023-01-24T03:30:24.992331+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:32:24.271747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993825+0000","last_change":"2023-01-24T03:30:20.998105+0000","last_active":"2023-01-24T03:30:24.993825+0000","last_peered":"2023-01-24T03:30:24.993825+0000","last_clean":"2023-01-24T03:30:24.993825+0000","last_became_active":"2023-01-24T03:30:20.997871+0000","last_became_peered":"2023-01-24T03:30:20.997871+0000","last_unstale":"2023-01-24T03:30:24.993825+0000","last_undegraded":"2023-01-24T03:30:24.993825+0000","last_fullsized":"2023-01-24T03:30:24.993825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:47:42.740428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016190+0000","last_change":"2023-01-24T03:30:25.008079+0000","last_active":"2023-01-24T03:30:25.016190+0000","last_peered":"2023-01-24T03:30:25.016190+0000","last_clean":"2023-01-24T03:30:25.016190+0000","last_became_active":"2023-01-24T03:30:25.007596+0000","last_became_peered":"2023-01-24T03:30:25.007596+0000","last_unstale":"2023-01-24T03:30:25.016190+0000","last_undegraded":"2023-01-24T03:30:25.016190+0000","last_fullsized":"2023-01-24T03:30:25.016190+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:30:57.179885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1,7],"acting":[2,5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996872+0000","last_change":"2023-01-24T03:30:16.272445+0000","last_active":"2023-01-24T03:30:24.996872+0000","last_peered":"2023-01-24T03:30:24.996872+0000","last_clean":"2023-01-24T03:30:24.996872+0000","last_became_active":"2023-01-24T03:30:16.272184+0000","last_became_peered":"2023-01-24T03:30:16.272184+0000","last_unstale":"2023-01-24T03:30:24.996872+0000","last_undegraded":"2023-01-24T03:30:24.996872+0000","last_fullsized":"2023-01-24T03:30:24.996872+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:17:05.555068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002328+0000","last_change":"2023-01-24T03:30:20.612832+0000","last_active":"2023-01-24T03:30:25.002328+0000","last_peered":"2023-01-24T03:30:25.002328+0000","last_clean":"2023-01-24T03:30:25.002328+0000","last_became_active":"2023-01-24T03:30:20.612569+0000","last_became_peered":"2023-01-24T03:30:20.612569+0000","last_unstale":"2023-01-24T03:30:25.002328+0000","last_undegraded":"2023-01-24T03:30:25.002328+0000","last_fullsized":"2023-01-24T03:30:25.002328+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:37:40.534015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002330+0000","last_change":"2023-01-24T03:30:20.998171+0000","last_active":"2023-01-24T03:30:25.002330+0000","last_peered":"2023-01-24T03:30:25.002330+0000","last_clean":"2023-01-24T03:30:25.002330+0000","last_became_active":"2023-01-24T03:30:20.994887+0000","last_became_peered":"2023-01-24T03:30:20.994887+0000","last_unstale":"2023-01-24T03:30:25.002330+0000","last_undegraded":"2023-01-24T03:30:25.002330+0000","last_fullsized":"2023-01-24T03:30:25.002330+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:57:40.909930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.020081+0000","last_change":"2023-01-24T03:30:25.014948+0000","last_active":"2023-01-24T03:30:25.020081+0000","last_peered":"2023-01-24T03:30:25.020081+0000","last_clean":"2023-01-24T03:30:25.020081+0000","last_became_active":"2023-01-24T03:30:25.014572+0000","last_became_peered":"2023-01-24T03:30:25.014572+0000","last_unstale":"2023-01-24T03:30:25.020081+0000","last_undegraded":"2023-01-24T03:30:25.020081+0000","last_fullsized":"2023-01-24T03:30:25.020081+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:59:13.108074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5,3],"acting":[7,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996112+0000","last_change":"2023-01-24T03:30:16.272574+0000","last_active":"2023-01-24T03:30:24.996112+0000","last_peered":"2023-01-24T03:30:24.996112+0000","last_clean":"2023-01-24T03:30:24.996112+0000","last_became_active":"2023-01-24T03:30:16.272252+0000","last_became_peered":"2023-01-24T03:30:16.272252+0000","last_unstale":"2023-01-24T03:30:24.996112+0000","last_undegraded":"2023-01-24T03:30:24.996112+0000","last_fullsized":"2023-01-24T03:30:24.996112+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:12:08.393585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991181+0000","last_change":"2023-01-24T03:30:20.606878+0000","last_active":"2023-01-24T03:30:24.991181+0000","last_peered":"2023-01-24T03:30:24.991181+0000","last_clean":"2023-01-24T03:30:24.991181+0000","last_became_active":"2023-01-24T03:30:20.606442+0000","last_became_peered":"2023-01-24T03:30:20.606442+0000","last_unstale":"2023-01-24T03:30:24.991181+0000","last_undegraded":"2023-01-24T03:30:24.991181+0000","last_fullsized":"2023-01-24T03:30:24.991181+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:26:16.916080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990646+0000","last_change":"2023-01-24T03:30:20.990840+0000","last_active":"2023-01-24T03:30:24.990646+0000","last_peered":"2023-01-24T03:30:24.990646+0000","last_clean":"2023-01-24T03:30:24.990646+0000","last_became_active":"2023-01-24T03:30:20.990615+0000","last_became_peered":"2023-01-24T03:30:20.990615+0000","last_unstale":"2023-01-24T03:30:24.990646+0000","last_undegraded":"2023-01-24T03:30:24.990646+0000","last_fullsized":"2023-01-24T03:30:24.990646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T15:06:37.670127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.013106+0000","last_change":"2023-01-24T03:30:25.006375+0000","last_active":"2023-01-24T03:30:25.013106+0000","last_peered":"2023-01-24T03:30:25.013106+0000","last_clean":"2023-01-24T03:30:25.013106+0000","last_became_active":"2023-01-24T03:30:25.006027+0000","last_became_peered":"2023-01-24T03:30:25.006027+0000","last_unstale":"2023-01-24T03:30:25.013106+0000","last_undegraded":"2023-01-24T03:30:25.013106+0000","last_fullsized":"2023-01-24T03:30:25.013106+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:51:29.481545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,6],"acting":[0,1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005695+0000","last_change":"2023-01-24T03:30:16.272475+0000","last_active":"2023-01-24T03:30:25.005695+0000","last_peered":"2023-01-24T03:30:25.005695+0000","last_clean":"2023-01-24T03:30:25.005695+0000","last_became_active":"2023-01-24T03:30:16.272140+0000","last_became_peered":"2023-01-24T03:30:16.272140+0000","last_unstale":"2023-01-24T03:30:25.005695+0000","last_undegraded":"2023-01-24T03:30:25.005695+0000","last_fullsized":"2023-01-24T03:30:25.005695+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:48:16.300876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992670+0000","last_change":"2023-01-24T03:30:20.609391+0000","last_active":"2023-01-24T03:30:24.992670+0000","last_peered":"2023-01-24T03:30:24.992670+0000","last_clean":"2023-01-24T03:30:24.992670+0000","last_became_active":"2023-01-24T03:30:20.609124+0000","last_became_peered":"2023-01-24T03:30:20.609124+0000","last_unstale":"2023-01-24T03:30:24.992670+0000","last_undegraded":"2023-01-24T03:30:24.992670+0000","last_fullsized":"2023-01-24T03:30:24.992670+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:36:49.616649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.001859+0000","last_change":"2023-01-24T03:30:20.996985+0000","last_active":"2023-01-24T03:30:25.001859+0000","last_peered":"2023-01-24T03:30:25.001859+0000","last_clean":"2023-01-24T03:30:25.001859+0000","last_became_active":"2023-01-24T03:30:20.996612+0000","last_became_peered":"2023-01-24T03:30:20.996612+0000","last_unstale":"2023-01-24T03:30:25.001859+0000","last_undegraded":"2023-01-24T03:30:25.001859+0000","last_fullsized":"2023-01-24T03:30:25.001859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:10:50.378902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.012218+0000","last_change":"2023-01-24T03:30:25.008155+0000","last_active":"2023-01-24T03:30:25.012218+0000","last_peered":"2023-01-24T03:30:25.012218+0000","last_clean":"2023-01-24T03:30:25.012218+0000","last_became_active":"2023-01-24T03:30:25.007674+0000","last_became_peered":"2023-01-24T03:30:25.007674+0000","last_unstale":"2023-01-24T03:30:25.012218+0000","last_undegraded":"2023-01-24T03:30:25.012218+0000","last_fullsized":"2023-01-24T03:30:25.012218+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:55:46.448808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,2],"acting":[6,0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990827+0000","last_change":"2023-01-24T03:30:16.272372+0000","last_active":"2023-01-24T03:30:24.990827+0000","last_peered":"2023-01-24T03:30:24.990827+0000","last_clean":"2023-01-24T03:30:24.990827+0000","last_became_active":"2023-01-24T03:30:16.272104+0000","last_became_peered":"2023-01-24T03:30:16.272104+0000","last_unstale":"2023-01-24T03:30:24.990827+0000","last_undegraded":"2023-01-24T03:30:24.990827+0000","last_fullsized":"2023-01-24T03:30:24.990827+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:15:10.865596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002114+0000","last_change":"2023-01-24T03:30:20.612284+0000","last_active":"2023-01-24T03:30:25.002114+0000","last_peered":"2023-01-24T03:30:25.002114+0000","last_clean":"2023-01-24T03:30:25.002114+0000","last_became_active":"2023-01-24T03:30:20.612029+0000","last_became_peered":"2023-01-24T03:30:20.612029+0000","last_unstale":"2023-01-24T03:30:25.002114+0000","last_undegraded":"2023-01-24T03:30:25.002114+0000","last_fullsized":"2023-01-24T03:30:25.002114+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:18:58.755974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992643+0000","last_change":"2023-01-24T03:30:20.989834+0000","last_active":"2023-01-24T03:30:24.992643+0000","last_peered":"2023-01-24T03:30:24.992643+0000","last_clean":"2023-01-24T03:30:24.992643+0000","last_became_active":"2023-01-24T03:30:20.989645+0000","last_became_peered":"2023-01-24T03:30:20.989645+0000","last_unstale":"2023-01-24T03:30:24.992643+0000","last_undegraded":"2023-01-24T03:30:24.992643+0000","last_fullsized":"2023-01-24T03:30:24.992643+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:42:06.477043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016259+0000","last_change":"2023-01-24T03:30:25.011952+0000","last_active":"2023-01-24T03:30:25.016259+0000","last_peered":"2023-01-24T03:30:25.016259+0000","last_clean":"2023-01-24T03:30:25.016259+0000","last_became_active":"2023-01-24T03:30:25.011612+0000","last_became_peered":"2023-01-24T03:30:25.011612+0000","last_unstale":"2023-01-24T03:30:25.016259+0000","last_undegraded":"2023-01-24T03:30:25.016259+0000","last_fullsized":"2023-01-24T03:30:25.016259+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:51:52.027285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6,5],"acting":[0,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990404+0000","last_change":"2023-01-24T03:30:16.048979+0000","last_active":"2023-01-24T03:30:24.990404+0000","last_peered":"2023-01-24T03:30:24.990404+0000","last_clean":"2023-01-24T03:30:24.990404+0000","last_became_active":"2023-01-24T03:30:16.048641+0000","last_became_peered":"2023-01-24T03:30:16.048641+0000","last_unstale":"2023-01-24T03:30:24.990404+0000","last_undegraded":"2023-01-24T03:30:24.990404+0000","last_fullsized":"2023-01-24T03:30:24.990404+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T15:11:26.211024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005087+0000","last_change":"2023-01-24T03:30:20.606906+0000","last_active":"2023-01-24T03:30:25.005087+0000","last_peered":"2023-01-24T03:30:25.005087+0000","last_clean":"2023-01-24T03:30:25.005087+0000","last_became_active":"2023-01-24T03:30:20.606640+0000","last_became_peered":"2023-01-24T03:30:20.606640+0000","last_unstale":"2023-01-24T03:30:25.005087+0000","last_undegraded":"2023-01-24T03:30:25.005087+0000","last_fullsized":"2023-01-24T03:30:25.005087+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:21:09.008653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002357+0000","last_change":"2023-01-24T03:30:20.999886+0000","last_active":"2023-01-24T03:30:25.002357+0000","last_peered":"2023-01-24T03:30:25.002357+0000","last_clean":"2023-01-24T03:30:25.002357+0000","last_became_active":"2023-01-24T03:30:20.999394+0000","last_became_peered":"2023-01-24T03:30:20.999394+0000","last_unstale":"2023-01-24T03:30:25.002357+0000","last_undegraded":"2023-01-24T03:30:25.002357+0000","last_fullsized":"2023-01-24T03:30:25.002357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:30:22.395611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016356+0000","last_change":"2023-01-24T03:30:25.006190+0000","last_active":"2023-01-24T03:30:25.016356+0000","last_peered":"2023-01-24T03:30:25.016356+0000","last_clean":"2023-01-24T03:30:25.016356+0000","last_became_active":"2023-01-24T03:30:25.005825+0000","last_became_peered":"2023-01-24T03:30:25.005825+0000","last_unstale":"2023-01-24T03:30:25.016356+0000","last_undegraded":"2023-01-24T03:30:25.016356+0000","last_fullsized":"2023-01-24T03:30:25.016356+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:51:50.888938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,0],"acting":[1,3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"24'1","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.446961+0000","last_change":"2023-01-24T03:30:20.611079+0000","last_active":"2023-01-24T03:30:25.446961+0000","last_peered":"2023-01-24T03:30:25.446961+0000","last_clean":"2023-01-24T03:30:25.446961+0000","last_became_active":"2023-01-24T03:30:20.610841+0000","last_became_peered":"2023-01-24T03:30:20.610841+0000","last_unstale":"2023-01-24T03:30:25.446961+0000","last_undegraded":"2023-01-24T03:30:25.446961+0000","last_fullsized":"2023-01-24T03:30:25.446961+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:34:24.922661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.994693+0000","last_change":"2023-01-24T03:30:20.991418+0000","last_active":"2023-01-24T03:30:24.994693+0000","last_peered":"2023-01-24T03:30:24.994693+0000","last_clean":"2023-01-24T03:30:24.994693+0000","last_became_active":"2023-01-24T03:30:20.991167+0000","last_became_peered":"2023-01-24T03:30:20.991167+0000","last_unstale":"2023-01-24T03:30:24.994693+0000","last_undegraded":"2023-01-24T03:30:24.994693+0000","last_fullsized":"2023-01-24T03:30:24.994693+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:51:52.736691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.013519+0000","last_change":"2023-01-24T03:30:25.010394+0000","last_active":"2023-01-24T03:30:25.013519+0000","last_peered":"2023-01-24T03:30:25.013519+0000","last_clean":"2023-01-24T03:30:25.013519+0000","last_became_active":"2023-01-24T03:30:25.010044+0000","last_became_peered":"2023-01-24T03:30:25.010044+0000","last_unstale":"2023-01-24T03:30:25.013519+0000","last_undegraded":"2023-01-24T03:30:25.013519+0000","last_fullsized":"2023-01-24T03:30:25.013519+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:08:55.250184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1,6],"acting":[7,4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"24'2","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.984877+0000","last_change":"2023-01-24T03:30:20.607538+0000","last_active":"2023-01-24T03:30:25.984877+0000","last_peered":"2023-01-24T03:30:25.984877+0000","last_clean":"2023-01-24T03:30:25.984877+0000","last_became_active":"2023-01-24T03:30:20.607285+0000","last_became_peered":"2023-01-24T03:30:20.607285+0000","last_unstale":"2023-01-24T03:30:25.984877+0000","last_undegraded":"2023-01-24T03:30:25.984877+0000","last_fullsized":"2023-01-24T03:30:25.984877+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:42:21.836603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992857+0000","last_change":"2023-01-24T03:30:20.986791+0000","last_active":"2023-01-24T03:30:24.992857+0000","last_peered":"2023-01-24T03:30:24.992857+0000","last_clean":"2023-01-24T03:30:24.992857+0000","last_became_active":"2023-01-24T03:30:20.986516+0000","last_became_peered":"2023-01-24T03:30:20.986516+0000","last_unstale":"2023-01-24T03:30:24.992857+0000","last_undegraded":"2023-01-24T03:30:24.992857+0000","last_fullsized":"2023-01-24T03:30:24.992857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:53:19.191166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.015383+0000","last_change":"2023-01-24T03:30:25.010751+0000","last_active":"2023-01-24T03:30:25.015383+0000","last_peered":"2023-01-24T03:30:25.015383+0000","last_clean":"2023-01-24T03:30:25.015383+0000","last_became_active":"2023-01-24T03:30:25.010332+0000","last_became_peered":"2023-01-24T03:30:25.010332+0000","last_unstale":"2023-01-24T03:30:25.015383+0000","last_undegraded":"2023-01-24T03:30:25.015383+0000","last_fullsized":"2023-01-24T03:30:25.015383+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:48:30.190712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7,4],"acting":[5,0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"24'2","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985166+0000","last_change":"2023-01-24T03:30:20.608509+0000","last_active":"2023-01-24T03:30:25.985166+0000","last_peered":"2023-01-24T03:30:25.985166+0000","last_clean":"2023-01-24T03:30:25.985166+0000","last_became_active":"2023-01-24T03:30:20.608313+0000","last_became_peered":"2023-01-24T03:30:20.608313+0000","last_unstale":"2023-01-24T03:30:25.985166+0000","last_undegraded":"2023-01-24T03:30:25.985166+0000","last_fullsized":"2023-01-24T03:30:25.985166+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:31:52.253588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992448+0000","last_change":"2023-01-24T03:30:20.992334+0000","last_active":"2023-01-24T03:30:24.992448+0000","last_peered":"2023-01-24T03:30:24.992448+0000","last_clean":"2023-01-24T03:30:24.992448+0000","last_became_active":"2023-01-24T03:30:20.992027+0000","last_became_peered":"2023-01-24T03:30:20.992027+0000","last_unstale":"2023-01-24T03:30:24.992448+0000","last_undegraded":"2023-01-24T03:30:24.992448+0000","last_fullsized":"2023-01-24T03:30:24.992448+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:57:56.985507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.012177+0000","last_change":"2023-01-24T03:30:25.004167+0000","last_active":"2023-01-24T03:30:25.012177+0000","last_peered":"2023-01-24T03:30:25.012177+0000","last_clean":"2023-01-24T03:30:25.012177+0000","last_became_active":"2023-01-24T03:30:25.003752+0000","last_became_peered":"2023-01-24T03:30:25.003752+0000","last_unstale":"2023-01-24T03:30:25.012177+0000","last_undegraded":"2023-01-24T03:30:25.012177+0000","last_fullsized":"2023-01-24T03:30:25.012177+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:19:16.285529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"24'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.448123+0000","last_change":"2023-01-24T03:30:20.603542+0000","last_active":"2023-01-24T03:30:25.448123+0000","last_peered":"2023-01-24T03:30:25.448123+0000","last_clean":"2023-01-24T03:30:25.448123+0000","last_became_active":"2023-01-24T03:30:20.603287+0000","last_became_peered":"2023-01-24T03:30:20.603287+0000","last_unstale":"2023-01-24T03:30:25.448123+0000","last_undegraded":"2023-01-24T03:30:25.448123+0000","last_fullsized":"2023-01-24T03:30:25.448123+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:03:56.865763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991098+0000","last_change":"2023-01-24T03:30:20.989519+0000","last_active":"2023-01-24T03:30:24.991098+0000","last_peered":"2023-01-24T03:30:24.991098+0000","last_clean":"2023-01-24T03:30:24.991098+0000","last_became_active":"2023-01-24T03:30:20.989286+0000","last_became_peered":"2023-01-24T03:30:20.989286+0000","last_unstale":"2023-01-24T03:30:24.991098+0000","last_undegraded":"2023-01-24T03:30:24.991098+0000","last_fullsized":"2023-01-24T03:30:24.991098+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:17:30.829327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016619+0000","last_change":"2023-01-24T03:30:25.010262+0000","last_active":"2023-01-24T03:30:25.016619+0000","last_peered":"2023-01-24T03:30:25.016619+0000","last_clean":"2023-01-24T03:30:25.016619+0000","last_became_active":"2023-01-24T03:30:25.009930+0000","last_became_peered":"2023-01-24T03:30:25.009930+0000","last_unstale":"2023-01-24T03:30:25.016619+0000","last_undegraded":"2023-01-24T03:30:25.016619+0000","last_fullsized":"2023-01-24T03:30:25.016619+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:36:17.807046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7,2],"acting":[1,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"24'4","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985911+0000","last_change":"2023-01-24T03:30:20.607334+0000","last_active":"2023-01-24T03:30:25.985911+0000","last_peered":"2023-01-24T03:30:25.985911+0000","last_clean":"2023-01-24T03:30:25.985911+0000","last_became_active":"2023-01-24T03:30:20.607079+0000","last_became_peered":"2023-01-24T03:30:20.607079+0000","last_unstale":"2023-01-24T03:30:25.985911+0000","last_undegraded":"2023-01-24T03:30:25.985911+0000","last_fullsized":"2023-01-24T03:30:25.985911+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:05:18.886532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992554+0000","last_change":"2023-01-24T03:30:20.987567+0000","last_active":"2023-01-24T03:30:24.992554+0000","last_peered":"2023-01-24T03:30:24.992554+0000","last_clean":"2023-01-24T03:30:24.992554+0000","last_became_active":"2023-01-24T03:30:20.987058+0000","last_became_peered":"2023-01-24T03:30:20.987058+0000","last_unstale":"2023-01-24T03:30:24.992554+0000","last_undegraded":"2023-01-24T03:30:24.992554+0000","last_fullsized":"2023-01-24T03:30:24.992554+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:58:30.391733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.989538+0000","last_change":"2023-01-24T03:30:20.986393+0000","last_active":"2023-01-24T03:30:24.989538+0000","last_peered":"2023-01-24T03:30:24.989538+0000","last_clean":"2023-01-24T03:30:24.989538+0000","last_became_active":"2023-01-24T03:30:20.986089+0000","last_became_peered":"2023-01-24T03:30:20.986089+0000","last_unstale":"2023-01-24T03:30:24.989538+0000","last_undegraded":"2023-01-24T03:30:24.989538+0000","last_fullsized":"2023-01-24T03:30:24.989538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:44:21.992291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990902+0000","last_change":"2023-01-24T03:30:20.991468+0000","last_active":"2023-01-24T03:30:24.990902+0000","last_peered":"2023-01-24T03:30:24.990902+0000","last_clean":"2023-01-24T03:30:24.990902+0000","last_became_active":"2023-01-24T03:30:20.991227+0000","last_became_peered":"2023-01-24T03:30:20.991227+0000","last_unstale":"2023-01-24T03:30:24.990902+0000","last_undegraded":"2023-01-24T03:30:24.990902+0000","last_fullsized":"2023-01-24T03:30:24.990902+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:27:25.806340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992602+0000","last_change":"2023-01-24T03:30:20.989824+0000","last_active":"2023-01-24T03:30:24.992602+0000","last_peered":"2023-01-24T03:30:24.992602+0000","last_clean":"2023-01-24T03:30:24.992602+0000","last_became_active":"2023-01-24T03:30:20.989577+0000","last_became_peered":"2023-01-24T03:30:20.989577+0000","last_unstale":"2023-01-24T03:30:24.992602+0000","last_undegraded":"2023-01-24T03:30:24.992602+0000","last_fullsized":"2023-01-24T03:30:24.992602+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:48:17.948459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992452+0000","last_change":"2023-01-24T03:30:20.996266+0000","last_active":"2023-01-24T03:30:24.992452+0000","last_peered":"2023-01-24T03:30:24.992452+0000","last_clean":"2023-01-24T03:30:24.992452+0000","last_became_active":"2023-01-24T03:30:20.995610+0000","last_became_peered":"2023-01-24T03:30:20.995610+0000","last_unstale":"2023-01-24T03:30:24.992452+0000","last_undegraded":"2023-01-24T03:30:24.992452+0000","last_fullsized":"2023-01-24T03:30:24.992452+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:24:43.022530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.001943+0000","last_change":"2023-01-24T03:30:20.997081+0000","last_active":"2023-01-24T03:30:25.001943+0000","last_peered":"2023-01-24T03:30:25.001943+0000","last_clean":"2023-01-24T03:30:25.001943+0000","last_became_active":"2023-01-24T03:30:20.996837+0000","last_became_peered":"2023-01-24T03:30:20.996837+0000","last_unstale":"2023-01-24T03:30:25.001943+0000","last_undegraded":"2023-01-24T03:30:25.001943+0000","last_fullsized":"2023-01-24T03:30:25.001943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:42:06.134817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2758,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40,"ondisk_log_size":40,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64574,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9180,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":8832,"kb_avail":94362660,"statfs":{"total":96636764160,"available":96627363840,"internally_reserved":0,"allocated":348160,"data_stored":696554,"data_compressed":32287,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":9043968},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8576,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94363264,"statfs":{"total":96636764160,"available":96627982336,"internally_reserved":0,"allocated":253952,"data_stored":104434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9152,"kb_used_data":312,"kb_used_omap":0,"kb_used_meta":8832,"kb_avail":94362688,"statfs":{"total":96636764160,"available":96627392512,"internally_reserved":0,"allocated":319488,"data_stored":694802,"data_compressed":32287,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":9043968},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28000000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8640,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94363200,"statfs":{"total":96636764160,"available":96627916800,"internally_reserved":0,"allocated":253952,"data_stored":104382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8512,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94363328,"statfs":{"total":96636764160,"available":96628047872,"internally_reserved":0,"allocated":253952,"data_stored":104434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8580,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94363260,"statfs":{"total":96636764160,"available":96627978240,"internally_reserved":0,"allocated":258048,"data_stored":104524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.311}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8660,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94363180,"statfs":{"total":96636764160,"available":96627896320,"internally_reserved":0,"allocated":274432,"data_stored":106026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8580,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94363260,"statfs":{"total":96636764160,"available":96627978240,"internally_reserved":0,"allocated":258048,"data_stored":103705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32287,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32287,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-01-24T03:30:38.632 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-24T03:30:38.777 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.774+0000 7f04e6873700 1 Processor -- start 2023-01-24T03:30:38.777 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.774+0000 7f04e6873700 1 -- start start 2023-01-24T03:30:38.778 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.775+0000 7f04e6873700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f04e00c7e40 0x7f04e00b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:38.778 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.775+0000 7f04e6873700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f04e00ca3e0 0x7f04e00ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:38.783 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.775+0000 7f04e6873700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04e0156240 con 0x7f04e0155da0 2023-01-24T03:30:38.784 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.775+0000 7f04e6873700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04e00b1170 con 0x7f04e00c7e40 2023-01-24T03:30:38.784 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.775+0000 7f04e6873700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04e00b1450 con 0x7f04e00ca3e0 2023-01-24T03:30:38.784 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.775+0000 7f04dffff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f04e00c7e40 0x7f04e00b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:38.784 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.775+0000 7f04df7fe700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f04e0155da0 0x7f04e00ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:59620/0 (socket says 172.21.15.112:59620) 2023-01-24T03:30:38.785 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.775+0000 7f04df7fe700 1 -- 172.21.15.112:0/1591174532 learned_addr learned my addr 172.21.15.112:0/1591174532 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:38.785 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.775+0000 7f04dffff700 1 -- 172.21.15.112:0/1591174532 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f04e00ca3e0 msgr2=0x7f04e00ca7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:38.785 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.775+0000 7f04dffff700 1 --2- 172.21.15.112:0/1591174532 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f04e00ca3e0 0x7f04e00ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:38.786 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.775+0000 7f04dffff700 1 -- 172.21.15.112:0/1591174532 >> v1:172.21.15.112:6789/0 conn(0x7f04e0155da0 legacy=0x7f04e00ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:38.786 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.775+0000 7f04dffff700 1 -- 172.21.15.112:0/1591174532 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e00b1730 con 0x7f04e00c7e40 2023-01-24T03:30:38.786 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.775+0000 7f04deffd700 1 -- 172.21.15.112:0/1591174532 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1306437055 0 0) 0x7f04e0156240 con 0x7f04e0155da0 2023-01-24T03:30:38.786 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.775+0000 7f04dffff700 1 --2- 172.21.15.112:0/1591174532 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04e00c7e40 0x7f04e00b0aa0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f04d0009760 tx=0x7f04d0009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=862ad30ab261b5b3 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:38.787 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.776+0000 7f04deffd700 1 -- 172.21.15.112:0/1591174532 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d0011630 con 0x7f04e00c7e40 2023-01-24T03:30:38.787 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.776+0000 7f04deffd700 1 -- 172.21.15.112:0/1591174532 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04d0011790 con 0x7f04e00c7e40 2023-01-24T03:30:38.787 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.776+0000 7f04deffd700 1 -- 172.21.15.112:0/1591174532 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d0011b30 con 0x7f04e00c7e40 2023-01-24T03:30:38.787 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.776+0000 7f04e6873700 1 -- 172.21.15.112:0/1591174532 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04e00c7e40 msgr2=0x7f04e00b0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:38.787 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.776+0000 7f04e6873700 1 --2- 172.21.15.112:0/1591174532 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04e00c7e40 0x7f04e00b0aa0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f04d0009760 tx=0x7f04d0009060 comp rx=0 tx=0).stop 2023-01-24T03:30:38.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.776+0000 7f04e6873700 1 -- 172.21.15.112:0/1591174532 shutdown_connections 2023-01-24T03:30:38.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.776+0000 7f04e6873700 1 --2- 172.21.15.112:0/1591174532 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f04e00ca3e0 0x7f04e00ca7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:38.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.776+0000 7f04e6873700 1 --2- 172.21.15.112:0/1591174532 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04e00c7e40 0x7f04e00b0aa0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:38.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.776+0000 7f04e6873700 1 -- 172.21.15.112:0/1591174532 >> 172.21.15.112:0/1591174532 conn(0x7f04e005e2c0 msgr2=0x7f04e00cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:38.789 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.776+0000 7f04e6873700 1 -- 172.21.15.112:0/1591174532 shutdown_connections 2023-01-24T03:30:38.789 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.776+0000 7f04e6873700 1 -- 172.21.15.112:0/1591174532 wait complete. 2023-01-24T03:30:38.789 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.777+0000 7f04e6873700 1 Processor -- start 2023-01-24T03:30:38.789 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.777+0000 7f04e6873700 1 -- start start 2023-01-24T03:30:38.790 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.778+0000 7f04e6873700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04e00c7e40 0x7f04e0068550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:38.790 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.778+0000 7f04e6873700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f04e00ca3e0 0x7f04e0148f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:38.790 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.778+0000 7f04e6873700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f04e0155da0 0x7f04e014b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:38.790 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.778+0000 7f04e6873700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e00c6160 con 0x7f04e00c7e40 2023-01-24T03:30:38.791 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.778+0000 7f04e6873700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e0150370 con 0x7f04e0155da0 2023-01-24T03:30:38.791 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.778+0000 7f04e6873700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04e0152680 con 0x7f04e00ca3e0 2023-01-24T03:30:38.791 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.778+0000 7f04dffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04e00c7e40 0x7f04e0068550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:38.791 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.778+0000 7f04dffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04e00c7e40 0x7f04e0068550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50404/0 (socket says 172.21.15.112:50404) 2023-01-24T03:30:38.791 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.778+0000 7f04dffff700 1 -- 172.21.15.112:0/3406577740 learned_addr learned my addr 172.21.15.112:0/3406577740 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:38.792 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.778+0000 7f04df7fe700 1 --2- 172.21.15.112:0/3406577740 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f04e00ca3e0 0x7f04e0148f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:38.792 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.778+0000 7f04e4e10700 1 --2- 172.21.15.112:0/3406577740 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f04e0155da0 0x7f04e014b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:38.792 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.778+0000 7f04dffff700 1 -- 172.21.15.112:0/3406577740 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f04e00ca3e0 msgr2=0x7f04e0148f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:38.792 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.778+0000 7f04dffff700 1 --2- 172.21.15.112:0/3406577740 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f04e00ca3e0 0x7f04e0148f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:38.792 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.778+0000 7f04dffff700 1 -- 172.21.15.112:0/3406577740 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f04e0155da0 msgr2=0x7f04e014b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:38.793 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.778+0000 7f04dffff700 1 --2- 172.21.15.112:0/3406577740 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f04e0155da0 0x7f04e014b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:38.793 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.778+0000 7f04dffff700 1 -- 172.21.15.112:0/3406577740 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d00092a0 con 0x7f04e00c7e40 2023-01-24T03:30:38.793 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.778+0000 7f04e4e10700 1 --2- 172.21.15.112:0/3406577740 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f04e0155da0 0x7f04e014b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:30:38.793 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.779+0000 7f04dffff700 1 --2- 172.21.15.112:0/3406577740 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04e00c7e40 0x7f04e0068550 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f04e0089340 tx=0x7f04d0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:38.794 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.779+0000 7f04c7fff700 1 -- 172.21.15.112:0/3406577740 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d0011900 con 0x7f04e00c7e40 2023-01-24T03:30:38.794 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.779+0000 7f04c7fff700 1 -- 172.21.15.112:0/3406577740 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04d001f070 con 0x7f04e00c7e40 2023-01-24T03:30:38.794 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.779+0000 7f04e6873700 1 -- 172.21.15.112:0/3406577740 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04e01528f0 con 0x7f04e00c7e40 2023-01-24T03:30:38.794 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.779+0000 7f04c7fff700 1 -- 172.21.15.112:0/3406577740 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d001b430 con 0x7f04e00c7e40 2023-01-24T03:30:38.794 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.779+0000 7f04e6873700 1 -- 172.21.15.112:0/3406577740 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04e0150000 con 0x7f04e00c7e40 2023-01-24T03:30:38.795 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.780+0000 7f04e6873700 1 -- 172.21.15.112:0/3406577740 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04e004bb80 con 0x7f04e00c7e40 2023-01-24T03:30:38.795 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.780+0000 7f04c7fff700 1 -- 172.21.15.112:0/3406577740 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f04d000f050 con 0x7f04e00c7e40 2023-01-24T03:30:38.795 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.780+0000 7f04c7fff700 1 --2- 172.21.15.112:0/3406577740 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f04c00611f0 0x7f04c00636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:38.796 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.781+0000 7f04c7fff700 1 -- 172.21.15.112:0/3406577740 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f04d007ebb0 con 0x7f04e00c7e40 2023-01-24T03:30:38.796 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.781+0000 7f04df7fe700 1 --2- 172.21.15.112:0/3406577740 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f04c00611f0 0x7f04c00636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:38.796 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.781+0000 7f04df7fe700 1 --2- 172.21.15.112:0/3406577740 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f04c00611f0 0x7f04c00636a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f04e00b0e60 tx=0x7f04d4007bb0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:38.797 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.785+0000 7f04c7fff700 1 -- 172.21.15.112:0/3406577740 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f04d0085080 con 0x7f04e00c7e40 2023-01-24T03:30:38.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.960+0000 7f04e6873700 1 -- 172.21.15.112:0/3406577740 --> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f04e0068d20 con 0x7f04c00611f0 2023-01-24T03:30:38.972 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.969+0000 7f04c7fff700 1 -- 172.21.15.112:0/3406577740 <== mgr.4104 v2:172.21.15.112:6832/84276 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+365098 (secure 0 0 0) 0x7f04e0068d20 con 0x7f04c00611f0 2023-01-24T03:30:38.977 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:30:38.980 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.973+0000 7f04e6873700 1 -- 172.21.15.112:0/3406577740 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f04c00611f0 msgr2=0x7f04c00636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:38.980 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.973+0000 7f04e6873700 1 --2- 172.21.15.112:0/3406577740 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f04c00611f0 0x7f04c00636a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f04e00b0e60 tx=0x7f04d4007bb0 comp rx=0 tx=0).stop 2023-01-24T03:30:38.980 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.973+0000 7f04e6873700 1 -- 172.21.15.112:0/3406577740 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04e00c7e40 msgr2=0x7f04e0068550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:38.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.973+0000 7f04e6873700 1 --2- 172.21.15.112:0/3406577740 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04e00c7e40 0x7f04e0068550 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f04e0089340 tx=0x7f04d0017000 comp rx=0 tx=0).stop 2023-01-24T03:30:38.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.973+0000 7f04e6873700 1 -- 172.21.15.112:0/3406577740 shutdown_connections 2023-01-24T03:30:38.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.973+0000 7f04e6873700 1 --2- 172.21.15.112:0/3406577740 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f04c00611f0 0x7f04c00636a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:38.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.973+0000 7f04e6873700 1 --2- 172.21.15.112:0/3406577740 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f04e00c7e40 0x7f04e0068550 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:38.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.973+0000 7f04e6873700 1 --2- 172.21.15.112:0/3406577740 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f04e0155da0 0x7f04e014b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:38.982 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.973+0000 7f04e6873700 1 --2- 172.21.15.112:0/3406577740 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f04e00ca3e0 0x7f04e0148f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:38.982 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.973+0000 7f04e6873700 1 -- 172.21.15.112:0/3406577740 >> 172.21.15.112:0/3406577740 conn(0x7f04e005e2c0 msgr2=0x7f04e00ae9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:38.982 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.973+0000 7f04e6873700 1 -- 172.21.15.112:0/3406577740 shutdown_connections 2023-01-24T03:30:38.982 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:38.973+0000 7f04e6873700 1 -- 172.21.15.112:0/3406577740 wait complete. 2023-01-24T03:30:38.982 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2023-01-24T03:30:38.988 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2023-01-24T03:30:36.998984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":593126,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":205,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":69880,"kb_used_data":2168,"kb_used_omap":0,"kb_used_meta":67648,"kb_avail":754904840,"statfs":{"total":773094113280,"available":773022556160,"internally_reserved":0,"allocated":2220032,"data_stored":2018861,"data_compressed":64574,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":69271552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2758,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.623747"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993289+0000","last_change":"2023-01-24T03:30:20.986308+0000","last_active":"2023-01-24T03:30:24.993289+0000","last_peered":"2023-01-24T03:30:24.993289+0000","last_clean":"2023-01-24T03:30:24.993289+0000","last_became_active":"2023-01-24T03:30:20.986025+0000","last_became_peered":"2023-01-24T03:30:20.986025+0000","last_unstale":"2023-01-24T03:30:24.993289+0000","last_undegraded":"2023-01-24T03:30:24.993289+0000","last_fullsized":"2023-01-24T03:30:24.993289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:47:58.616698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.006202+0000","last_change":"2023-01-24T03:30:20.994478+0000","last_active":"2023-01-24T03:30:25.006202+0000","last_peered":"2023-01-24T03:30:25.006202+0000","last_clean":"2023-01-24T03:30:25.006202+0000","last_became_active":"2023-01-24T03:30:20.994207+0000","last_became_peered":"2023-01-24T03:30:20.994207+0000","last_unstale":"2023-01-24T03:30:25.006202+0000","last_undegraded":"2023-01-24T03:30:25.006202+0000","last_fullsized":"2023-01-24T03:30:25.006202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:02:53.230231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993177+0000","last_change":"2023-01-24T03:30:20.989651+0000","last_active":"2023-01-24T03:30:24.993177+0000","last_peered":"2023-01-24T03:30:24.993177+0000","last_clean":"2023-01-24T03:30:24.993177+0000","last_became_active":"2023-01-24T03:30:20.987585+0000","last_became_peered":"2023-01-24T03:30:20.987585+0000","last_unstale":"2023-01-24T03:30:24.993177+0000","last_undegraded":"2023-01-24T03:30:24.993177+0000","last_fullsized":"2023-01-24T03:30:24.993177+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:36:02.177544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990511+0000","last_change":"2023-01-24T03:30:20.993601+0000","last_active":"2023-01-24T03:30:24.990511+0000","last_peered":"2023-01-24T03:30:24.990511+0000","last_clean":"2023-01-24T03:30:24.990511+0000","last_became_active":"2023-01-24T03:30:20.993418+0000","last_became_peered":"2023-01-24T03:30:20.993418+0000","last_unstale":"2023-01-24T03:30:24.990511+0000","last_undegraded":"2023-01-24T03:30:24.990511+0000","last_fullsized":"2023-01-24T03:30:24.990511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:47:50.260497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996430+0000","last_change":"2023-01-24T03:30:20.991605+0000","last_active":"2023-01-24T03:30:24.996430+0000","last_peered":"2023-01-24T03:30:24.996430+0000","last_clean":"2023-01-24T03:30:24.996430+0000","last_became_active":"2023-01-24T03:30:20.991361+0000","last_became_peered":"2023-01-24T03:30:20.991361+0000","last_unstale":"2023-01-24T03:30:24.996430+0000","last_undegraded":"2023-01-24T03:30:24.996430+0000","last_fullsized":"2023-01-24T03:30:24.996430+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:31:25.418430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993288+0000","last_change":"2023-01-24T03:30:20.985058+0000","last_active":"2023-01-24T03:30:24.993288+0000","last_peered":"2023-01-24T03:30:24.993288+0000","last_clean":"2023-01-24T03:30:24.993288+0000","last_became_active":"2023-01-24T03:30:20.984754+0000","last_became_peered":"2023-01-24T03:30:20.984754+0000","last_unstale":"2023-01-24T03:30:24.993288+0000","last_undegraded":"2023-01-24T03:30:24.993288+0000","last_fullsized":"2023-01-24T03:30:24.993288+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:12:03.225109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996279+0000","last_change":"2023-01-24T03:30:21.000890+0000","last_active":"2023-01-24T03:30:24.996279+0000","last_peered":"2023-01-24T03:30:24.996279+0000","last_clean":"2023-01-24T03:30:24.996279+0000","last_became_active":"2023-01-24T03:30:21.000438+0000","last_became_peered":"2023-01-24T03:30:21.000438+0000","last_unstale":"2023-01-24T03:30:24.996279+0000","last_undegraded":"2023-01-24T03:30:24.996279+0000","last_fullsized":"2023-01-24T03:30:24.996279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:17:12.999664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.004717+0000","last_change":"2023-01-24T03:30:20.986753+0000","last_active":"2023-01-24T03:30:25.004717+0000","last_peered":"2023-01-24T03:30:25.004717+0000","last_clean":"2023-01-24T03:30:25.004717+0000","last_became_active":"2023-01-24T03:30:20.986524+0000","last_became_peered":"2023-01-24T03:30:20.986524+0000","last_unstale":"2023-01-24T03:30:25.004717+0000","last_undegraded":"2023-01-24T03:30:25.004717+0000","last_fullsized":"2023-01-24T03:30:25.004717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:24:55.699379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992350+0000","last_change":"2023-01-24T03:30:20.991136+0000","last_active":"2023-01-24T03:30:24.992350+0000","last_peered":"2023-01-24T03:30:24.992350+0000","last_clean":"2023-01-24T03:30:24.992350+0000","last_became_active":"2023-01-24T03:30:20.990925+0000","last_became_peered":"2023-01-24T03:30:20.990925+0000","last_unstale":"2023-01-24T03:30:24.992350+0000","last_undegraded":"2023-01-24T03:30:24.992350+0000","last_fullsized":"2023-01-24T03:30:24.992350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:45:37.485199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.006401+0000","last_change":"2023-01-24T03:30:20.995730+0000","last_active":"2023-01-24T03:30:25.006401+0000","last_peered":"2023-01-24T03:30:25.006401+0000","last_clean":"2023-01-24T03:30:25.006401+0000","last_became_active":"2023-01-24T03:30:20.994880+0000","last_became_peered":"2023-01-24T03:30:20.994880+0000","last_unstale":"2023-01-24T03:30:25.006401+0000","last_undegraded":"2023-01-24T03:30:25.006401+0000","last_fullsized":"2023-01-24T03:30:25.006401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:07:21.546386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993144+0000","last_change":"2023-01-24T03:30:20.993328+0000","last_active":"2023-01-24T03:30:24.993144+0000","last_peered":"2023-01-24T03:30:24.993144+0000","last_clean":"2023-01-24T03:30:24.993144+0000","last_became_active":"2023-01-24T03:30:20.992730+0000","last_became_peered":"2023-01-24T03:30:20.992730+0000","last_unstale":"2023-01-24T03:30:24.993144+0000","last_undegraded":"2023-01-24T03:30:24.993144+0000","last_fullsized":"2023-01-24T03:30:24.993144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:43:39.048550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.994605+0000","last_change":"2023-01-24T03:30:20.984993+0000","last_active":"2023-01-24T03:30:24.994605+0000","last_peered":"2023-01-24T03:30:24.994605+0000","last_clean":"2023-01-24T03:30:24.994605+0000","last_became_active":"2023-01-24T03:30:20.984540+0000","last_became_peered":"2023-01-24T03:30:20.984540+0000","last_unstale":"2023-01-24T03:30:24.994605+0000","last_undegraded":"2023-01-24T03:30:24.994605+0000","last_fullsized":"2023-01-24T03:30:24.994605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:16:39.993359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.995936+0000","last_change":"2023-01-24T03:30:20.986567+0000","last_active":"2023-01-24T03:30:24.995936+0000","last_peered":"2023-01-24T03:30:24.995936+0000","last_clean":"2023-01-24T03:30:24.995936+0000","last_became_active":"2023-01-24T03:30:20.986224+0000","last_became_peered":"2023-01-24T03:30:20.986224+0000","last_unstale":"2023-01-24T03:30:24.995936+0000","last_undegraded":"2023-01-24T03:30:24.995936+0000","last_fullsized":"2023-01-24T03:30:24.995936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:09:14.692354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005087+0000","last_change":"2023-01-24T03:30:20.988843+0000","last_active":"2023-01-24T03:30:25.005087+0000","last_peered":"2023-01-24T03:30:25.005087+0000","last_clean":"2023-01-24T03:30:25.005087+0000","last_became_active":"2023-01-24T03:30:20.988486+0000","last_became_peered":"2023-01-24T03:30:20.988486+0000","last_unstale":"2023-01-24T03:30:25.005087+0000","last_undegraded":"2023-01-24T03:30:25.005087+0000","last_fullsized":"2023-01-24T03:30:25.005087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:18:20.538844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996110+0000","last_change":"2023-01-24T03:30:21.000738+0000","last_active":"2023-01-24T03:30:24.996110+0000","last_peered":"2023-01-24T03:30:24.996110+0000","last_clean":"2023-01-24T03:30:24.996110+0000","last_became_active":"2023-01-24T03:30:21.000458+0000","last_became_peered":"2023-01-24T03:30:21.000458+0000","last_unstale":"2023-01-24T03:30:24.996110+0000","last_undegraded":"2023-01-24T03:30:24.996110+0000","last_fullsized":"2023-01-24T03:30:24.996110+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:21:42.876426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992383+0000","last_change":"2023-01-24T03:30:20.985062+0000","last_active":"2023-01-24T03:30:24.992383+0000","last_peered":"2023-01-24T03:30:24.992383+0000","last_clean":"2023-01-24T03:30:24.992383+0000","last_became_active":"2023-01-24T03:30:20.984759+0000","last_became_peered":"2023-01-24T03:30:20.984759+0000","last_unstale":"2023-01-24T03:30:24.992383+0000","last_undegraded":"2023-01-24T03:30:24.992383+0000","last_fullsized":"2023-01-24T03:30:24.992383+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:38:34.615748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992439+0000","last_change":"2023-01-24T03:30:20.989080+0000","last_active":"2023-01-24T03:30:24.992439+0000","last_peered":"2023-01-24T03:30:24.992439+0000","last_clean":"2023-01-24T03:30:24.992439+0000","last_became_active":"2023-01-24T03:30:20.988823+0000","last_became_peered":"2023-01-24T03:30:20.988823+0000","last_unstale":"2023-01-24T03:30:24.992439+0000","last_undegraded":"2023-01-24T03:30:24.992439+0000","last_fullsized":"2023-01-24T03:30:24.992439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:22:21.807370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993008+0000","last_change":"2023-01-24T03:30:20.984222+0000","last_active":"2023-01-24T03:30:24.993008+0000","last_peered":"2023-01-24T03:30:24.993008+0000","last_clean":"2023-01-24T03:30:24.993008+0000","last_became_active":"2023-01-24T03:30:20.984002+0000","last_became_peered":"2023-01-24T03:30:20.984002+0000","last_unstale":"2023-01-24T03:30:24.993008+0000","last_undegraded":"2023-01-24T03:30:24.993008+0000","last_fullsized":"2023-01-24T03:30:24.993008+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:54:24.005486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.997598+0000","last_change":"2023-01-24T03:30:20.988197+0000","last_active":"2023-01-24T03:30:24.997598+0000","last_peered":"2023-01-24T03:30:24.997598+0000","last_clean":"2023-01-24T03:30:24.997598+0000","last_became_active":"2023-01-24T03:30:20.987941+0000","last_became_peered":"2023-01-24T03:30:20.987941+0000","last_unstale":"2023-01-24T03:30:24.997598+0000","last_undegraded":"2023-01-24T03:30:24.997598+0000","last_fullsized":"2023-01-24T03:30:24.997598+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T15:08:29.659964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992957+0000","last_change":"2023-01-24T03:30:20.988701+0000","last_active":"2023-01-24T03:30:24.992957+0000","last_peered":"2023-01-24T03:30:24.992957+0000","last_clean":"2023-01-24T03:30:24.992957+0000","last_became_active":"2023-01-24T03:30:20.988476+0000","last_became_peered":"2023-01-24T03:30:20.988476+0000","last_unstale":"2023-01-24T03:30:24.992957+0000","last_undegraded":"2023-01-24T03:30:24.992957+0000","last_fullsized":"2023-01-24T03:30:24.992957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:33:29.463194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996546+0000","last_change":"2023-01-24T03:30:21.001125+0000","last_active":"2023-01-24T03:30:24.996546+0000","last_peered":"2023-01-24T03:30:24.996546+0000","last_clean":"2023-01-24T03:30:24.996546+0000","last_became_active":"2023-01-24T03:30:21.000873+0000","last_became_peered":"2023-01-24T03:30:21.000873+0000","last_unstale":"2023-01-24T03:30:24.996546+0000","last_undegraded":"2023-01-24T03:30:24.996546+0000","last_fullsized":"2023-01-24T03:30:24.996546+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:22:16.136875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993432+0000","last_change":"2023-01-24T03:30:20.992639+0000","last_active":"2023-01-24T03:30:24.993432+0000","last_peered":"2023-01-24T03:30:24.993432+0000","last_clean":"2023-01-24T03:30:24.993432+0000","last_became_active":"2023-01-24T03:30:20.992386+0000","last_became_peered":"2023-01-24T03:30:20.992386+0000","last_unstale":"2023-01-24T03:30:24.993432+0000","last_undegraded":"2023-01-24T03:30:24.993432+0000","last_fullsized":"2023-01-24T03:30:24.993432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:29:17.912198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992608+0000","last_change":"2023-01-24T03:30:20.994066+0000","last_active":"2023-01-24T03:30:24.992608+0000","last_peered":"2023-01-24T03:30:24.992608+0000","last_clean":"2023-01-24T03:30:24.992608+0000","last_became_active":"2023-01-24T03:30:20.993820+0000","last_became_peered":"2023-01-24T03:30:20.993820+0000","last_unstale":"2023-01-24T03:30:24.992608+0000","last_undegraded":"2023-01-24T03:30:24.992608+0000","last_fullsized":"2023-01-24T03:30:24.992608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:31:23.874058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996188+0000","last_change":"2023-01-24T03:30:21.001652+0000","last_active":"2023-01-24T03:30:24.996188+0000","last_peered":"2023-01-24T03:30:24.996188+0000","last_clean":"2023-01-24T03:30:24.996188+0000","last_became_active":"2023-01-24T03:30:21.001400+0000","last_became_peered":"2023-01-24T03:30:21.001400+0000","last_unstale":"2023-01-24T03:30:24.996188+0000","last_undegraded":"2023-01-24T03:30:24.996188+0000","last_fullsized":"2023-01-24T03:30:24.996188+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:55:53.249816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002243+0000","last_change":"2023-01-24T03:30:20.998304+0000","last_active":"2023-01-24T03:30:25.002243+0000","last_peered":"2023-01-24T03:30:25.002243+0000","last_clean":"2023-01-24T03:30:25.002243+0000","last_became_active":"2023-01-24T03:30:20.996288+0000","last_became_peered":"2023-01-24T03:30:20.996288+0000","last_unstale":"2023-01-24T03:30:25.002243+0000","last_undegraded":"2023-01-24T03:30:25.002243+0000","last_fullsized":"2023-01-24T03:30:25.002243+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:11:08.120197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002638+0000","last_change":"2023-01-24T03:30:20.996489+0000","last_active":"2023-01-24T03:30:25.002638+0000","last_peered":"2023-01-24T03:30:25.002638+0000","last_clean":"2023-01-24T03:30:25.002638+0000","last_became_active":"2023-01-24T03:30:20.994371+0000","last_became_peered":"2023-01-24T03:30:20.994371+0000","last_unstale":"2023-01-24T03:30:25.002638+0000","last_undegraded":"2023-01-24T03:30:25.002638+0000","last_fullsized":"2023-01-24T03:30:25.002638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:52:21.369928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005744+0000","last_change":"2023-01-24T03:30:20.997753+0000","last_active":"2023-01-24T03:30:25.005744+0000","last_peered":"2023-01-24T03:30:25.005744+0000","last_clean":"2023-01-24T03:30:25.005744+0000","last_became_active":"2023-01-24T03:30:20.997475+0000","last_became_peered":"2023-01-24T03:30:20.997475+0000","last_unstale":"2023-01-24T03:30:25.005744+0000","last_undegraded":"2023-01-24T03:30:25.005744+0000","last_fullsized":"2023-01-24T03:30:25.005744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:43:16.199843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.013267+0000","last_change":"2023-01-24T03:30:25.004944+0000","last_active":"2023-01-24T03:30:25.013267+0000","last_peered":"2023-01-24T03:30:25.013267+0000","last_clean":"2023-01-24T03:30:25.013267+0000","last_became_active":"2023-01-24T03:30:25.004590+0000","last_became_peered":"2023-01-24T03:30:25.004590+0000","last_unstale":"2023-01-24T03:30:25.013267+0000","last_undegraded":"2023-01-24T03:30:25.013267+0000","last_fullsized":"2023-01-24T03:30:25.013267+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:04:11.054225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2,0],"acting":[4,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"24'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.380450+0000","last_change":"2023-01-24T03:30:20.607211+0000","last_active":"2023-01-24T03:30:25.380450+0000","last_peered":"2023-01-24T03:30:25.380450+0000","last_clean":"2023-01-24T03:30:25.380450+0000","last_became_active":"2023-01-24T03:30:20.606855+0000","last_became_peered":"2023-01-24T03:30:20.606855+0000","last_unstale":"2023-01-24T03:30:25.380450+0000","last_undegraded":"2023-01-24T03:30:25.380450+0000","last_fullsized":"2023-01-24T03:30:25.380450+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:42:29.300089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991276+0000","last_change":"2023-01-24T03:30:20.993128+0000","last_active":"2023-01-24T03:30:24.991276+0000","last_peered":"2023-01-24T03:30:24.991276+0000","last_clean":"2023-01-24T03:30:24.991276+0000","last_became_active":"2023-01-24T03:30:20.992967+0000","last_became_peered":"2023-01-24T03:30:20.992967+0000","last_unstale":"2023-01-24T03:30:24.991276+0000","last_undegraded":"2023-01-24T03:30:24.991276+0000","last_fullsized":"2023-01-24T03:30:24.991276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:09:59.170218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.008814+0000","last_change":"2023-01-24T03:30:25.006569+0000","last_active":"2023-01-24T03:30:25.008814+0000","last_peered":"2023-01-24T03:30:25.008814+0000","last_clean":"2023-01-24T03:30:25.008814+0000","last_became_active":"2023-01-24T03:30:25.006199+0000","last_became_peered":"2023-01-24T03:30:25.006199+0000","last_unstale":"2023-01-24T03:30:25.008814+0000","last_undegraded":"2023-01-24T03:30:25.008814+0000","last_fullsized":"2023-01-24T03:30:25.008814+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:31:46.839485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,7],"acting":[6,4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.001971+0000","last_change":"2023-01-24T03:30:20.612738+0000","last_active":"2023-01-24T03:30:25.001971+0000","last_peered":"2023-01-24T03:30:25.001971+0000","last_clean":"2023-01-24T03:30:25.001971+0000","last_became_active":"2023-01-24T03:30:20.612375+0000","last_became_peered":"2023-01-24T03:30:20.612375+0000","last_unstale":"2023-01-24T03:30:25.001971+0000","last_undegraded":"2023-01-24T03:30:25.001971+0000","last_fullsized":"2023-01-24T03:30:25.001971+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:17:43.251945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993741+0000","last_change":"2023-01-24T03:30:20.996368+0000","last_active":"2023-01-24T03:30:24.993741+0000","last_peered":"2023-01-24T03:30:24.993741+0000","last_clean":"2023-01-24T03:30:24.993741+0000","last_became_active":"2023-01-24T03:30:20.996051+0000","last_became_peered":"2023-01-24T03:30:20.996051+0000","last_unstale":"2023-01-24T03:30:24.993741+0000","last_undegraded":"2023-01-24T03:30:24.993741+0000","last_fullsized":"2023-01-24T03:30:24.993741+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:53:28.193862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.014921+0000","last_change":"2023-01-24T03:30:25.008528+0000","last_active":"2023-01-24T03:30:25.014921+0000","last_peered":"2023-01-24T03:30:25.014921+0000","last_clean":"2023-01-24T03:30:25.014921+0000","last_became_active":"2023-01-24T03:30:25.008146+0000","last_became_peered":"2023-01-24T03:30:25.008146+0000","last_unstale":"2023-01-24T03:30:25.014921+0000","last_undegraded":"2023-01-24T03:30:25.014921+0000","last_fullsized":"2023-01-24T03:30:25.014921+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:17:11.457359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1,2],"acting":[0,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991341+0000","last_change":"2023-01-24T03:30:20.608932+0000","last_active":"2023-01-24T03:30:24.991341+0000","last_peered":"2023-01-24T03:30:24.991341+0000","last_clean":"2023-01-24T03:30:24.991341+0000","last_became_active":"2023-01-24T03:30:20.608135+0000","last_became_peered":"2023-01-24T03:30:20.608135+0000","last_unstale":"2023-01-24T03:30:24.991341+0000","last_undegraded":"2023-01-24T03:30:24.991341+0000","last_fullsized":"2023-01-24T03:30:24.991341+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:38:30.012279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990753+0000","last_change":"2023-01-24T03:30:20.986490+0000","last_active":"2023-01-24T03:30:24.990753+0000","last_peered":"2023-01-24T03:30:24.990753+0000","last_clean":"2023-01-24T03:30:24.990753+0000","last_became_active":"2023-01-24T03:30:20.986245+0000","last_became_peered":"2023-01-24T03:30:20.986245+0000","last_unstale":"2023-01-24T03:30:24.990753+0000","last_undegraded":"2023-01-24T03:30:24.990753+0000","last_fullsized":"2023-01-24T03:30:24.990753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:38:30.801032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.012186+0000","last_change":"2023-01-24T03:30:25.008221+0000","last_active":"2023-01-24T03:30:25.012186+0000","last_peered":"2023-01-24T03:30:25.012186+0000","last_clean":"2023-01-24T03:30:25.012186+0000","last_became_active":"2023-01-24T03:30:25.007855+0000","last_became_peered":"2023-01-24T03:30:25.007855+0000","last_unstale":"2023-01-24T03:30:25.012186+0000","last_undegraded":"2023-01-24T03:30:25.012186+0000","last_fullsized":"2023-01-24T03:30:25.012186+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:45:15.317067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,1],"acting":[6,4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992445+0000","last_change":"2023-01-24T03:30:20.607385+0000","last_active":"2023-01-24T03:30:24.992445+0000","last_peered":"2023-01-24T03:30:24.992445+0000","last_clean":"2023-01-24T03:30:24.992445+0000","last_became_active":"2023-01-24T03:30:20.607123+0000","last_became_peered":"2023-01-24T03:30:20.607123+0000","last_unstale":"2023-01-24T03:30:24.992445+0000","last_undegraded":"2023-01-24T03:30:24.992445+0000","last_fullsized":"2023-01-24T03:30:24.992445+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T15:24:00.474758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991258+0000","last_change":"2023-01-24T03:30:20.994124+0000","last_active":"2023-01-24T03:30:24.991258+0000","last_peered":"2023-01-24T03:30:24.991258+0000","last_clean":"2023-01-24T03:30:24.991258+0000","last_became_active":"2023-01-24T03:30:20.993876+0000","last_became_peered":"2023-01-24T03:30:20.993876+0000","last_unstale":"2023-01-24T03:30:24.991258+0000","last_undegraded":"2023-01-24T03:30:24.991258+0000","last_fullsized":"2023-01-24T03:30:24.991258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:07:26.252741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.011535+0000","last_change":"2023-01-24T03:30:25.006312+0000","last_active":"2023-01-24T03:30:25.011535+0000","last_peered":"2023-01-24T03:30:25.011535+0000","last_clean":"2023-01-24T03:30:25.011535+0000","last_became_active":"2023-01-24T03:30:25.005946+0000","last_became_peered":"2023-01-24T03:30:25.005946+0000","last_unstale":"2023-01-24T03:30:25.011535+0000","last_undegraded":"2023-01-24T03:30:25.011535+0000","last_fullsized":"2023-01-24T03:30:25.011535+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:35:02.760190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6,0],"acting":[7,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.006326+0000","last_change":"2023-01-24T03:30:20.610825+0000","last_active":"2023-01-24T03:30:25.006326+0000","last_peered":"2023-01-24T03:30:25.006326+0000","last_clean":"2023-01-24T03:30:25.006326+0000","last_became_active":"2023-01-24T03:30:20.610540+0000","last_became_peered":"2023-01-24T03:30:20.610540+0000","last_unstale":"2023-01-24T03:30:25.006326+0000","last_undegraded":"2023-01-24T03:30:25.006326+0000","last_fullsized":"2023-01-24T03:30:25.006326+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:23:15.279283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990818+0000","last_change":"2023-01-24T03:30:20.994605+0000","last_active":"2023-01-24T03:30:24.990818+0000","last_peered":"2023-01-24T03:30:24.990818+0000","last_clean":"2023-01-24T03:30:24.990818+0000","last_became_active":"2023-01-24T03:30:20.994356+0000","last_became_peered":"2023-01-24T03:30:20.994356+0000","last_unstale":"2023-01-24T03:30:24.990818+0000","last_undegraded":"2023-01-24T03:30:24.990818+0000","last_fullsized":"2023-01-24T03:30:24.990818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:06:08.797922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.018595+0000","last_change":"2023-01-24T03:30:25.012301+0000","last_active":"2023-01-24T03:30:25.018595+0000","last_peered":"2023-01-24T03:30:25.018595+0000","last_clean":"2023-01-24T03:30:25.018595+0000","last_became_active":"2023-01-24T03:30:25.011891+0000","last_became_peered":"2023-01-24T03:30:25.011891+0000","last_unstale":"2023-01-24T03:30:25.018595+0000","last_undegraded":"2023-01-24T03:30:25.018595+0000","last_fullsized":"2023-01-24T03:30:25.018595+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:15:46.012675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,5],"acting":[3,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"24'3","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.986137+0000","last_change":"2023-01-24T03:30:20.609828+0000","last_active":"2023-01-24T03:30:25.986137+0000","last_peered":"2023-01-24T03:30:25.986137+0000","last_clean":"2023-01-24T03:30:25.986137+0000","last_became_active":"2023-01-24T03:30:20.609492+0000","last_became_peered":"2023-01-24T03:30:20.609492+0000","last_unstale":"2023-01-24T03:30:25.986137+0000","last_undegraded":"2023-01-24T03:30:25.986137+0000","last_fullsized":"2023-01-24T03:30:25.986137+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:22:53.430711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002553+0000","last_change":"2023-01-24T03:30:20.999747+0000","last_active":"2023-01-24T03:30:25.002553+0000","last_peered":"2023-01-24T03:30:25.002553+0000","last_clean":"2023-01-24T03:30:25.002553+0000","last_became_active":"2023-01-24T03:30:20.999003+0000","last_became_peered":"2023-01-24T03:30:20.999003+0000","last_unstale":"2023-01-24T03:30:25.002553+0000","last_undegraded":"2023-01-24T03:30:25.002553+0000","last_fullsized":"2023-01-24T03:30:25.002553+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:19:22.205251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016573+0000","last_change":"2023-01-24T03:30:25.010659+0000","last_active":"2023-01-24T03:30:25.016573+0000","last_peered":"2023-01-24T03:30:25.016573+0000","last_clean":"2023-01-24T03:30:25.016573+0000","last_became_active":"2023-01-24T03:30:25.010237+0000","last_became_peered":"2023-01-24T03:30:25.010237+0000","last_unstale":"2023-01-24T03:30:25.016573+0000","last_undegraded":"2023-01-24T03:30:25.016573+0000","last_fullsized":"2023-01-24T03:30:25.016573+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:01:53.501070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3,7],"acting":[1,5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"24'7","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.988114+0000","last_change":"2023-01-24T03:30:20.610682+0000","last_active":"2023-01-24T03:30:25.988114+0000","last_peered":"2023-01-24T03:30:25.988114+0000","last_clean":"2023-01-24T03:30:25.988114+0000","last_became_active":"2023-01-24T03:30:20.610463+0000","last_became_peered":"2023-01-24T03:30:20.610463+0000","last_unstale":"2023-01-24T03:30:25.988114+0000","last_undegraded":"2023-01-24T03:30:25.988114+0000","last_fullsized":"2023-01-24T03:30:25.988114+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:25:00.447596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1988,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992845+0000","last_change":"2023-01-24T03:30:20.987326+0000","last_active":"2023-01-24T03:30:24.992845+0000","last_peered":"2023-01-24T03:30:24.992845+0000","last_clean":"2023-01-24T03:30:24.992845+0000","last_became_active":"2023-01-24T03:30:20.987042+0000","last_became_peered":"2023-01-24T03:30:20.987042+0000","last_unstale":"2023-01-24T03:30:24.992845+0000","last_undegraded":"2023-01-24T03:30:24.992845+0000","last_fullsized":"2023-01-24T03:30:24.992845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:57:43.257624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.020022+0000","last_change":"2023-01-24T03:30:25.012004+0000","last_active":"2023-01-24T03:30:25.020022+0000","last_peered":"2023-01-24T03:30:25.020022+0000","last_clean":"2023-01-24T03:30:25.020022+0000","last_became_active":"2023-01-24T03:30:25.007410+0000","last_became_peered":"2023-01-24T03:30:25.007410+0000","last_unstale":"2023-01-24T03:30:25.020022+0000","last_undegraded":"2023-01-24T03:30:25.020022+0000","last_fullsized":"2023-01-24T03:30:25.020022+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:34:21.681011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3,0],"acting":[5,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"24'2","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985201+0000","last_change":"2023-01-24T03:30:20.613026+0000","last_active":"2023-01-24T03:30:25.985201+0000","last_peered":"2023-01-24T03:30:25.985201+0000","last_clean":"2023-01-24T03:30:25.985201+0000","last_became_active":"2023-01-24T03:30:20.612605+0000","last_became_peered":"2023-01-24T03:30:20.612605+0000","last_unstale":"2023-01-24T03:30:25.985201+0000","last_undegraded":"2023-01-24T03:30:25.985201+0000","last_fullsized":"2023-01-24T03:30:25.985201+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:45:08.016740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991113+0000","last_change":"2023-01-24T03:30:20.991929+0000","last_active":"2023-01-24T03:30:24.991113+0000","last_peered":"2023-01-24T03:30:24.991113+0000","last_clean":"2023-01-24T03:30:24.991113+0000","last_became_active":"2023-01-24T03:30:20.991693+0000","last_became_peered":"2023-01-24T03:30:20.991693+0000","last_unstale":"2023-01-24T03:30:24.991113+0000","last_undegraded":"2023-01-24T03:30:24.991113+0000","last_fullsized":"2023-01-24T03:30:24.991113+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:55:51.637568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.018472+0000","last_change":"2023-01-24T03:30:25.010572+0000","last_active":"2023-01-24T03:30:25.018472+0000","last_peered":"2023-01-24T03:30:25.018472+0000","last_clean":"2023-01-24T03:30:25.018472+0000","last_became_active":"2023-01-24T03:30:25.010193+0000","last_became_peered":"2023-01-24T03:30:25.010193+0000","last_unstale":"2023-01-24T03:30:25.018472+0000","last_undegraded":"2023-01-24T03:30:25.018472+0000","last_fullsized":"2023-01-24T03:30:25.018472+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:25:39.676179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,1],"acting":[3,7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993579+0000","last_change":"2023-01-24T03:30:20.612047+0000","last_active":"2023-01-24T03:30:24.993579+0000","last_peered":"2023-01-24T03:30:24.993579+0000","last_clean":"2023-01-24T03:30:24.993579+0000","last_became_active":"2023-01-24T03:30:20.611798+0000","last_became_peered":"2023-01-24T03:30:20.611798+0000","last_unstale":"2023-01-24T03:30:24.993579+0000","last_undegraded":"2023-01-24T03:30:24.993579+0000","last_fullsized":"2023-01-24T03:30:24.993579+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:01:41.731689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005194+0000","last_change":"2023-01-24T03:30:20.996432+0000","last_active":"2023-01-24T03:30:25.005194+0000","last_peered":"2023-01-24T03:30:25.005194+0000","last_clean":"2023-01-24T03:30:25.005194+0000","last_became_active":"2023-01-24T03:30:20.996173+0000","last_became_peered":"2023-01-24T03:30:20.996173+0000","last_unstale":"2023-01-24T03:30:25.005194+0000","last_undegraded":"2023-01-24T03:30:25.005194+0000","last_fullsized":"2023-01-24T03:30:25.005194+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:13:31.596587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.018507+0000","last_change":"2023-01-24T03:30:25.008479+0000","last_active":"2023-01-24T03:30:25.018507+0000","last_peered":"2023-01-24T03:30:25.018507+0000","last_clean":"2023-01-24T03:30:25.018507+0000","last_became_active":"2023-01-24T03:30:25.007858+0000","last_became_peered":"2023-01-24T03:30:25.007858+0000","last_unstale":"2023-01-24T03:30:25.018507+0000","last_undegraded":"2023-01-24T03:30:25.018507+0000","last_fullsized":"2023-01-24T03:30:25.018507+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:10:38.425874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,0],"acting":[3,1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996134+0000","last_change":"2023-01-24T03:30:20.612848+0000","last_active":"2023-01-24T03:30:24.996134+0000","last_peered":"2023-01-24T03:30:24.996134+0000","last_clean":"2023-01-24T03:30:24.996134+0000","last_became_active":"2023-01-24T03:30:20.612600+0000","last_became_peered":"2023-01-24T03:30:20.612600+0000","last_unstale":"2023-01-24T03:30:24.996134+0000","last_undegraded":"2023-01-24T03:30:24.996134+0000","last_fullsized":"2023-01-24T03:30:24.996134+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:46:45.634472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993883+0000","last_change":"2023-01-24T03:30:20.991133+0000","last_active":"2023-01-24T03:30:24.993883+0000","last_peered":"2023-01-24T03:30:24.993883+0000","last_clean":"2023-01-24T03:30:24.993883+0000","last_became_active":"2023-01-24T03:30:20.990856+0000","last_became_peered":"2023-01-24T03:30:20.990856+0000","last_unstale":"2023-01-24T03:30:24.993883+0000","last_undegraded":"2023-01-24T03:30:24.993883+0000","last_fullsized":"2023-01-24T03:30:24.993883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:27:32.163224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.022452+0000","last_change":"2023-01-24T03:30:25.017038+0000","last_active":"2023-01-24T03:30:25.022452+0000","last_peered":"2023-01-24T03:30:25.022452+0000","last_clean":"2023-01-24T03:30:25.022452+0000","last_became_active":"2023-01-24T03:30:25.016684+0000","last_became_peered":"2023-01-24T03:30:25.016684+0000","last_unstale":"2023-01-24T03:30:25.022452+0000","last_undegraded":"2023-01-24T03:30:25.022452+0000","last_fullsized":"2023-01-24T03:30:25.022452+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:57:30.028834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1,6],"acting":[5,3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"24'8","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.988793+0000","last_change":"2023-01-24T03:30:20.604617+0000","last_active":"2023-01-24T03:30:25.988793+0000","last_peered":"2023-01-24T03:30:25.988793+0000","last_clean":"2023-01-24T03:30:25.988793+0000","last_became_active":"2023-01-24T03:30:20.604445+0000","last_became_peered":"2023-01-24T03:30:20.604445+0000","last_unstale":"2023-01-24T03:30:25.988793+0000","last_undegraded":"2023-01-24T03:30:25.988793+0000","last_fullsized":"2023-01-24T03:30:25.988793+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:06:51.460895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993017+0000","last_change":"2023-01-24T03:30:20.995872+0000","last_active":"2023-01-24T03:30:24.993017+0000","last_peered":"2023-01-24T03:30:24.993017+0000","last_clean":"2023-01-24T03:30:24.993017+0000","last_became_active":"2023-01-24T03:30:20.995599+0000","last_became_peered":"2023-01-24T03:30:20.995599+0000","last_unstale":"2023-01-24T03:30:24.993017+0000","last_undegraded":"2023-01-24T03:30:24.993017+0000","last_fullsized":"2023-01-24T03:30:24.993017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:53:54.356899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.020130+0000","last_change":"2023-01-24T03:30:25.016101+0000","last_active":"2023-01-24T03:30:25.020130+0000","last_peered":"2023-01-24T03:30:25.020130+0000","last_clean":"2023-01-24T03:30:25.020130+0000","last_became_active":"2023-01-24T03:30:25.015705+0000","last_became_peered":"2023-01-24T03:30:25.015705+0000","last_unstale":"2023-01-24T03:30:25.020130+0000","last_undegraded":"2023-01-24T03:30:25.020130+0000","last_fullsized":"2023-01-24T03:30:25.020130+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:54:37.798191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3,4],"acting":[7,6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"24'1","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.446876+0000","last_change":"2023-01-24T03:30:20.610632+0000","last_active":"2023-01-24T03:30:25.446876+0000","last_peered":"2023-01-24T03:30:25.446876+0000","last_clean":"2023-01-24T03:30:25.446876+0000","last_became_active":"2023-01-24T03:30:20.610367+0000","last_became_peered":"2023-01-24T03:30:20.610367+0000","last_unstale":"2023-01-24T03:30:25.446876+0000","last_undegraded":"2023-01-24T03:30:25.446876+0000","last_fullsized":"2023-01-24T03:30:25.446876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:26:05.938178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.004623+0000","last_change":"2023-01-24T03:30:20.996493+0000","last_active":"2023-01-24T03:30:25.004623+0000","last_peered":"2023-01-24T03:30:25.004623+0000","last_clean":"2023-01-24T03:30:25.004623+0000","last_became_active":"2023-01-24T03:30:20.996229+0000","last_became_peered":"2023-01-24T03:30:20.996229+0000","last_unstale":"2023-01-24T03:30:25.004623+0000","last_undegraded":"2023-01-24T03:30:25.004623+0000","last_fullsized":"2023-01-24T03:30:25.004623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:23:53.642522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016195+0000","last_change":"2023-01-24T03:30:25.009215+0000","last_active":"2023-01-24T03:30:25.016195+0000","last_peered":"2023-01-24T03:30:25.016195+0000","last_clean":"2023-01-24T03:30:25.016195+0000","last_became_active":"2023-01-24T03:30:25.008213+0000","last_became_peered":"2023-01-24T03:30:25.008213+0000","last_unstale":"2023-01-24T03:30:25.016195+0000","last_undegraded":"2023-01-24T03:30:25.016195+0000","last_fullsized":"2023-01-24T03:30:25.016195+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:48:37.517172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,5],"acting":[2,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"24'2","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985223+0000","last_change":"2023-01-24T03:30:20.612158+0000","last_active":"2023-01-24T03:30:25.985223+0000","last_peered":"2023-01-24T03:30:25.985223+0000","last_clean":"2023-01-24T03:30:25.985223+0000","last_became_active":"2023-01-24T03:30:20.611931+0000","last_became_peered":"2023-01-24T03:30:20.611931+0000","last_unstale":"2023-01-24T03:30:25.985223+0000","last_undegraded":"2023-01-24T03:30:25.985223+0000","last_fullsized":"2023-01-24T03:30:25.985223+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:30:37.532390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992500+0000","last_change":"2023-01-24T03:30:20.991582+0000","last_active":"2023-01-24T03:30:24.992500+0000","last_peered":"2023-01-24T03:30:24.992500+0000","last_clean":"2023-01-24T03:30:24.992500+0000","last_became_active":"2023-01-24T03:30:20.991335+0000","last_became_peered":"2023-01-24T03:30:20.991335+0000","last_unstale":"2023-01-24T03:30:24.992500+0000","last_undegraded":"2023-01-24T03:30:24.992500+0000","last_fullsized":"2023-01-24T03:30:24.992500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:45:20.786712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016224+0000","last_change":"2023-01-24T03:30:25.008058+0000","last_active":"2023-01-24T03:30:25.016224+0000","last_peered":"2023-01-24T03:30:25.016224+0000","last_clean":"2023-01-24T03:30:25.016224+0000","last_became_active":"2023-01-24T03:30:25.007711+0000","last_became_peered":"2023-01-24T03:30:25.007711+0000","last_unstale":"2023-01-24T03:30:25.016224+0000","last_undegraded":"2023-01-24T03:30:25.016224+0000","last_fullsized":"2023-01-24T03:30:25.016224+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:26:56.270674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5,6],"acting":[2,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"24'1","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.447163+0000","last_change":"2023-01-24T03:30:20.610495+0000","last_active":"2023-01-24T03:30:25.447163+0000","last_peered":"2023-01-24T03:30:25.447163+0000","last_clean":"2023-01-24T03:30:25.447163+0000","last_became_active":"2023-01-24T03:30:20.610035+0000","last_became_peered":"2023-01-24T03:30:20.610035+0000","last_unstale":"2023-01-24T03:30:25.447163+0000","last_undegraded":"2023-01-24T03:30:25.447163+0000","last_fullsized":"2023-01-24T03:30:25.447163+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:25:30.944275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.989631+0000","last_change":"2023-01-24T03:30:20.994362+0000","last_active":"2023-01-24T03:30:24.989631+0000","last_peered":"2023-01-24T03:30:24.989631+0000","last_clean":"2023-01-24T03:30:24.989631+0000","last_became_active":"2023-01-24T03:30:20.994101+0000","last_became_peered":"2023-01-24T03:30:20.994101+0000","last_unstale":"2023-01-24T03:30:24.989631+0000","last_undegraded":"2023-01-24T03:30:24.989631+0000","last_fullsized":"2023-01-24T03:30:24.989631+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:55:12.837646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.015999+0000","last_change":"2023-01-24T03:30:25.006995+0000","last_active":"2023-01-24T03:30:25.015999+0000","last_peered":"2023-01-24T03:30:25.015999+0000","last_clean":"2023-01-24T03:30:25.015999+0000","last_became_active":"2023-01-24T03:30:25.006473+0000","last_became_peered":"2023-01-24T03:30:25.006473+0000","last_unstale":"2023-01-24T03:30:25.015999+0000","last_undegraded":"2023-01-24T03:30:25.015999+0000","last_fullsized":"2023-01-24T03:30:25.015999+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:48:16.320162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3,6],"acting":[2,4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005342+0000","last_change":"2023-01-24T03:30:20.606526+0000","last_active":"2023-01-24T03:30:25.005342+0000","last_peered":"2023-01-24T03:30:25.005342+0000","last_clean":"2023-01-24T03:30:25.005342+0000","last_became_active":"2023-01-24T03:30:20.606301+0000","last_became_peered":"2023-01-24T03:30:20.606301+0000","last_unstale":"2023-01-24T03:30:25.005342+0000","last_undegraded":"2023-01-24T03:30:25.005342+0000","last_fullsized":"2023-01-24T03:30:25.005342+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:43:28.702475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992428+0000","last_change":"2023-01-24T03:30:20.993379+0000","last_active":"2023-01-24T03:30:24.992428+0000","last_peered":"2023-01-24T03:30:24.992428+0000","last_clean":"2023-01-24T03:30:24.992428+0000","last_became_active":"2023-01-24T03:30:20.993151+0000","last_became_peered":"2023-01-24T03:30:20.993151+0000","last_unstale":"2023-01-24T03:30:24.992428+0000","last_undegraded":"2023-01-24T03:30:24.992428+0000","last_fullsized":"2023-01-24T03:30:24.992428+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:00:45.845082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.020912+0000","last_change":"2023-01-24T03:30:25.013701+0000","last_active":"2023-01-24T03:30:25.020912+0000","last_peered":"2023-01-24T03:30:25.020912+0000","last_clean":"2023-01-24T03:30:25.020912+0000","last_became_active":"2023-01-24T03:30:25.013290+0000","last_became_peered":"2023-01-24T03:30:25.013290+0000","last_unstale":"2023-01-24T03:30:25.020912+0000","last_undegraded":"2023-01-24T03:30:25.020912+0000","last_fullsized":"2023-01-24T03:30:25.020912+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:32:17.340583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5,6],"acting":[2,7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"24'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985860+0000","last_change":"2023-01-24T03:30:20.609656+0000","last_active":"2023-01-24T03:30:25.985860+0000","last_peered":"2023-01-24T03:30:25.985860+0000","last_clean":"2023-01-24T03:30:25.985860+0000","last_became_active":"2023-01-24T03:30:20.609374+0000","last_became_peered":"2023-01-24T03:30:20.609374+0000","last_unstale":"2023-01-24T03:30:25.985860+0000","last_undegraded":"2023-01-24T03:30:25.985860+0000","last_fullsized":"2023-01-24T03:30:25.985860+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:58:12.653746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992726+0000","last_change":"2023-01-24T03:30:20.986089+0000","last_active":"2023-01-24T03:30:24.992726+0000","last_peered":"2023-01-24T03:30:24.992726+0000","last_clean":"2023-01-24T03:30:24.992726+0000","last_became_active":"2023-01-24T03:30:20.985857+0000","last_became_peered":"2023-01-24T03:30:20.985857+0000","last_unstale":"2023-01-24T03:30:24.992726+0000","last_undegraded":"2023-01-24T03:30:24.992726+0000","last_fullsized":"2023-01-24T03:30:24.992726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:00:57.338328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016603+0000","last_change":"2023-01-24T03:30:25.009962+0000","last_active":"2023-01-24T03:30:25.016603+0000","last_peered":"2023-01-24T03:30:25.016603+0000","last_clean":"2023-01-24T03:30:25.016603+0000","last_became_active":"2023-01-24T03:30:25.009623+0000","last_became_peered":"2023-01-24T03:30:25.009623+0000","last_unstale":"2023-01-24T03:30:25.016603+0000","last_undegraded":"2023-01-24T03:30:25.016603+0000","last_fullsized":"2023-01-24T03:30:25.016603+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:24:42.958531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0,6],"acting":[1,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.989699+0000","last_change":"2023-01-24T03:30:20.608114+0000","last_active":"2023-01-24T03:30:24.989699+0000","last_peered":"2023-01-24T03:30:24.989699+0000","last_clean":"2023-01-24T03:30:24.989699+0000","last_became_active":"2023-01-24T03:30:20.607816+0000","last_became_peered":"2023-01-24T03:30:20.607816+0000","last_unstale":"2023-01-24T03:30:24.989699+0000","last_undegraded":"2023-01-24T03:30:24.989699+0000","last_fullsized":"2023-01-24T03:30:24.989699+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:40:00.784496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992908+0000","last_change":"2023-01-24T03:30:20.982589+0000","last_active":"2023-01-24T03:30:24.992908+0000","last_peered":"2023-01-24T03:30:24.992908+0000","last_clean":"2023-01-24T03:30:24.992908+0000","last_became_active":"2023-01-24T03:30:20.979779+0000","last_became_peered":"2023-01-24T03:30:20.979779+0000","last_unstale":"2023-01-24T03:30:24.992908+0000","last_undegraded":"2023-01-24T03:30:24.992908+0000","last_fullsized":"2023-01-24T03:30:24.992908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:01:18.491363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.019938+0000","last_change":"2023-01-24T03:30:25.013327+0000","last_active":"2023-01-24T03:30:25.019938+0000","last_peered":"2023-01-24T03:30:25.019938+0000","last_clean":"2023-01-24T03:30:25.019938+0000","last_became_active":"2023-01-24T03:30:25.011303+0000","last_became_peered":"2023-01-24T03:30:25.011303+0000","last_unstale":"2023-01-24T03:30:25.019938+0000","last_undegraded":"2023-01-24T03:30:25.019938+0000","last_fullsized":"2023-01-24T03:30:25.019938+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:14:43.169287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6,2],"acting":[5,4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992390+0000","last_change":"2023-01-24T03:30:20.602369+0000","last_active":"2023-01-24T03:30:24.992390+0000","last_peered":"2023-01-24T03:30:24.992390+0000","last_clean":"2023-01-24T03:30:24.992390+0000","last_became_active":"2023-01-24T03:30:20.601996+0000","last_became_peered":"2023-01-24T03:30:20.601996+0000","last_unstale":"2023-01-24T03:30:24.992390+0000","last_undegraded":"2023-01-24T03:30:24.992390+0000","last_fullsized":"2023-01-24T03:30:24.992390+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:33:55.328080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993109+0000","last_change":"2023-01-24T03:30:20.988738+0000","last_active":"2023-01-24T03:30:24.993109+0000","last_peered":"2023-01-24T03:30:24.993109+0000","last_clean":"2023-01-24T03:30:24.993109+0000","last_became_active":"2023-01-24T03:30:20.987190+0000","last_became_peered":"2023-01-24T03:30:20.987190+0000","last_unstale":"2023-01-24T03:30:24.993109+0000","last_undegraded":"2023-01-24T03:30:24.993109+0000","last_fullsized":"2023-01-24T03:30:24.993109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:32:05.689674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.013303+0000","last_change":"2023-01-24T03:30:25.008401+0000","last_active":"2023-01-24T03:30:25.013303+0000","last_peered":"2023-01-24T03:30:25.013303+0000","last_clean":"2023-01-24T03:30:25.013303+0000","last_became_active":"2023-01-24T03:30:25.007973+0000","last_became_peered":"2023-01-24T03:30:25.007973+0000","last_unstale":"2023-01-24T03:30:25.013303+0000","last_undegraded":"2023-01-24T03:30:25.013303+0000","last_fullsized":"2023-01-24T03:30:25.013303+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:36:22.339295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,2],"acting":[4,5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991180+0000","last_change":"2023-01-24T03:30:20.608145+0000","last_active":"2023-01-24T03:30:24.991180+0000","last_peered":"2023-01-24T03:30:24.991180+0000","last_clean":"2023-01-24T03:30:24.991180+0000","last_became_active":"2023-01-24T03:30:20.607165+0000","last_became_peered":"2023-01-24T03:30:20.607165+0000","last_unstale":"2023-01-24T03:30:24.991180+0000","last_undegraded":"2023-01-24T03:30:24.991180+0000","last_fullsized":"2023-01-24T03:30:24.991180+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:58:56.490482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002416+0000","last_change":"2023-01-24T03:30:20.998777+0000","last_active":"2023-01-24T03:30:25.002416+0000","last_peered":"2023-01-24T03:30:25.002416+0000","last_clean":"2023-01-24T03:30:25.002416+0000","last_became_active":"2023-01-24T03:30:20.998519+0000","last_became_peered":"2023-01-24T03:30:20.998519+0000","last_unstale":"2023-01-24T03:30:25.002416+0000","last_undegraded":"2023-01-24T03:30:25.002416+0000","last_fullsized":"2023-01-24T03:30:25.002416+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:55:55.320556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.012455+0000","last_change":"2023-01-24T03:30:25.008154+0000","last_active":"2023-01-24T03:30:25.012455+0000","last_peered":"2023-01-24T03:30:25.012455+0000","last_clean":"2023-01-24T03:30:25.012455+0000","last_became_active":"2023-01-24T03:30:25.007797+0000","last_became_peered":"2023-01-24T03:30:25.007797+0000","last_unstale":"2023-01-24T03:30:25.012455+0000","last_undegraded":"2023-01-24T03:30:25.012455+0000","last_fullsized":"2023-01-24T03:30:25.012455+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:29:05.153733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7,0],"acting":[4,3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"24'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985835+0000","last_change":"2023-01-24T03:30:20.610690+0000","last_active":"2023-01-24T03:30:25.985835+0000","last_peered":"2023-01-24T03:30:25.985835+0000","last_clean":"2023-01-24T03:30:25.985835+0000","last_became_active":"2023-01-24T03:30:20.610409+0000","last_became_peered":"2023-01-24T03:30:20.610409+0000","last_unstale":"2023-01-24T03:30:25.985835+0000","last_undegraded":"2023-01-24T03:30:25.985835+0000","last_fullsized":"2023-01-24T03:30:25.985835+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:43:37.740181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993033+0000","last_change":"2023-01-24T03:30:16.085885+0000","last_active":"2023-01-24T03:30:24.993033+0000","last_peered":"2023-01-24T03:30:24.993033+0000","last_clean":"2023-01-24T03:30:24.993033+0000","last_became_active":"2023-01-24T03:30:16.085525+0000","last_became_peered":"2023-01-24T03:30:16.085525+0000","last_unstale":"2023-01-24T03:30:24.993033+0000","last_undegraded":"2023-01-24T03:30:24.993033+0000","last_fullsized":"2023-01-24T03:30:24.993033+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:57:36.499644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991099+0000","last_change":"2023-01-24T03:30:20.996370+0000","last_active":"2023-01-24T03:30:24.991099+0000","last_peered":"2023-01-24T03:30:24.991099+0000","last_clean":"2023-01-24T03:30:24.991099+0000","last_became_active":"2023-01-24T03:30:20.996122+0000","last_became_peered":"2023-01-24T03:30:20.996122+0000","last_unstale":"2023-01-24T03:30:24.991099+0000","last_undegraded":"2023-01-24T03:30:24.991099+0000","last_fullsized":"2023-01-24T03:30:24.991099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:13:06.913474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.014170+0000","last_change":"2023-01-24T03:30:25.007262+0000","last_active":"2023-01-24T03:30:25.014170+0000","last_peered":"2023-01-24T03:30:25.014170+0000","last_clean":"2023-01-24T03:30:25.014170+0000","last_became_active":"2023-01-24T03:30:25.006862+0000","last_became_peered":"2023-01-24T03:30:25.006862+0000","last_unstale":"2023-01-24T03:30:25.014170+0000","last_undegraded":"2023-01-24T03:30:25.014170+0000","last_fullsized":"2023-01-24T03:30:25.014170+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:42:05.268862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,0],"acting":[3,1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991165+0000","last_change":"2023-01-24T03:30:16.057240+0000","last_active":"2023-01-24T03:30:24.991165+0000","last_peered":"2023-01-24T03:30:24.991165+0000","last_clean":"2023-01-24T03:30:24.991165+0000","last_became_active":"2023-01-24T03:30:16.056941+0000","last_became_peered":"2023-01-24T03:30:16.056941+0000","last_unstale":"2023-01-24T03:30:24.991165+0000","last_undegraded":"2023-01-24T03:30:24.991165+0000","last_fullsized":"2023-01-24T03:30:24.991165+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:22:26.168527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"24'1","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.447279+0000","last_change":"2023-01-24T03:30:20.606130+0000","last_active":"2023-01-24T03:30:25.447279+0000","last_peered":"2023-01-24T03:30:25.447279+0000","last_clean":"2023-01-24T03:30:25.447279+0000","last_became_active":"2023-01-24T03:30:20.605899+0000","last_became_peered":"2023-01-24T03:30:20.605899+0000","last_unstale":"2023-01-24T03:30:25.447279+0000","last_undegraded":"2023-01-24T03:30:25.447279+0000","last_fullsized":"2023-01-24T03:30:25.447279+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:30:47.956569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992411+0000","last_change":"2023-01-24T03:30:20.988906+0000","last_active":"2023-01-24T03:30:24.992411+0000","last_peered":"2023-01-24T03:30:24.992411+0000","last_clean":"2023-01-24T03:30:24.992411+0000","last_became_active":"2023-01-24T03:30:20.988688+0000","last_became_peered":"2023-01-24T03:30:20.988688+0000","last_unstale":"2023-01-24T03:30:24.992411+0000","last_undegraded":"2023-01-24T03:30:24.992411+0000","last_fullsized":"2023-01-24T03:30:24.992411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:25:43.365167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.019966+0000","last_change":"2023-01-24T03:30:25.013834+0000","last_active":"2023-01-24T03:30:25.019966+0000","last_peered":"2023-01-24T03:30:25.019966+0000","last_clean":"2023-01-24T03:30:25.019966+0000","last_became_active":"2023-01-24T03:30:25.011774+0000","last_became_peered":"2023-01-24T03:30:25.011774+0000","last_unstale":"2023-01-24T03:30:25.019966+0000","last_undegraded":"2023-01-24T03:30:25.019966+0000","last_fullsized":"2023-01-24T03:30:25.019966+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:02:50.158049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0,4],"acting":[5,1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002878+0000","last_change":"2023-01-24T03:30:16.051822+0000","last_active":"2023-01-24T03:30:25.002878+0000","last_peered":"2023-01-24T03:30:25.002878+0000","last_clean":"2023-01-24T03:30:25.002878+0000","last_became_active":"2023-01-24T03:30:16.051525+0000","last_became_peered":"2023-01-24T03:30:16.051525+0000","last_unstale":"2023-01-24T03:30:25.002878+0000","last_undegraded":"2023-01-24T03:30:25.002878+0000","last_fullsized":"2023-01-24T03:30:25.002878+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:37:30.802424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"16'117","reported_seq":166,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996317+0000","last_change":"2023-01-24T03:30:16.054907+0000","last_active":"2023-01-24T03:30:24.996317+0000","last_peered":"2023-01-24T03:30:24.996317+0000","last_clean":"2023-01-24T03:30:24.996317+0000","last_became_active":"2023-01-24T03:30:16.054071+0000","last_became_peered":"2023-01-24T03:30:16.054071+0000","last_unstale":"2023-01-24T03:30:24.996317+0000","last_undegraded":"2023-01-24T03:30:24.996317+0000","last_fullsized":"2023-01-24T03:30:24.996317+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:31:26.790766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992331+0000","last_change":"2023-01-24T03:30:20.607914+0000","last_active":"2023-01-24T03:30:24.992331+0000","last_peered":"2023-01-24T03:30:24.992331+0000","last_clean":"2023-01-24T03:30:24.992331+0000","last_became_active":"2023-01-24T03:30:20.607663+0000","last_became_peered":"2023-01-24T03:30:20.607663+0000","last_unstale":"2023-01-24T03:30:24.992331+0000","last_undegraded":"2023-01-24T03:30:24.992331+0000","last_fullsized":"2023-01-24T03:30:24.992331+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:32:24.271747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993825+0000","last_change":"2023-01-24T03:30:20.998105+0000","last_active":"2023-01-24T03:30:24.993825+0000","last_peered":"2023-01-24T03:30:24.993825+0000","last_clean":"2023-01-24T03:30:24.993825+0000","last_became_active":"2023-01-24T03:30:20.997871+0000","last_became_peered":"2023-01-24T03:30:20.997871+0000","last_unstale":"2023-01-24T03:30:24.993825+0000","last_undegraded":"2023-01-24T03:30:24.993825+0000","last_fullsized":"2023-01-24T03:30:24.993825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:47:42.740428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016190+0000","last_change":"2023-01-24T03:30:25.008079+0000","last_active":"2023-01-24T03:30:25.016190+0000","last_peered":"2023-01-24T03:30:25.016190+0000","last_clean":"2023-01-24T03:30:25.016190+0000","last_became_active":"2023-01-24T03:30:25.007596+0000","last_became_peered":"2023-01-24T03:30:25.007596+0000","last_unstale":"2023-01-24T03:30:25.016190+0000","last_undegraded":"2023-01-24T03:30:25.016190+0000","last_fullsized":"2023-01-24T03:30:25.016190+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:30:57.179885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1,7],"acting":[2,5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996872+0000","last_change":"2023-01-24T03:30:16.272445+0000","last_active":"2023-01-24T03:30:24.996872+0000","last_peered":"2023-01-24T03:30:24.996872+0000","last_clean":"2023-01-24T03:30:24.996872+0000","last_became_active":"2023-01-24T03:30:16.272184+0000","last_became_peered":"2023-01-24T03:30:16.272184+0000","last_unstale":"2023-01-24T03:30:24.996872+0000","last_undegraded":"2023-01-24T03:30:24.996872+0000","last_fullsized":"2023-01-24T03:30:24.996872+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:17:05.555068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002328+0000","last_change":"2023-01-24T03:30:20.612832+0000","last_active":"2023-01-24T03:30:25.002328+0000","last_peered":"2023-01-24T03:30:25.002328+0000","last_clean":"2023-01-24T03:30:25.002328+0000","last_became_active":"2023-01-24T03:30:20.612569+0000","last_became_peered":"2023-01-24T03:30:20.612569+0000","last_unstale":"2023-01-24T03:30:25.002328+0000","last_undegraded":"2023-01-24T03:30:25.002328+0000","last_fullsized":"2023-01-24T03:30:25.002328+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:37:40.534015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002330+0000","last_change":"2023-01-24T03:30:20.998171+0000","last_active":"2023-01-24T03:30:25.002330+0000","last_peered":"2023-01-24T03:30:25.002330+0000","last_clean":"2023-01-24T03:30:25.002330+0000","last_became_active":"2023-01-24T03:30:20.994887+0000","last_became_peered":"2023-01-24T03:30:20.994887+0000","last_unstale":"2023-01-24T03:30:25.002330+0000","last_undegraded":"2023-01-24T03:30:25.002330+0000","last_fullsized":"2023-01-24T03:30:25.002330+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:57:40.909930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.020081+0000","last_change":"2023-01-24T03:30:25.014948+0000","last_active":"2023-01-24T03:30:25.020081+0000","last_peered":"2023-01-24T03:30:25.020081+0000","last_clean":"2023-01-24T03:30:25.020081+0000","last_became_active":"2023-01-24T03:30:25.014572+0000","last_became_peered":"2023-01-24T03:30:25.014572+0000","last_unstale":"2023-01-24T03:30:25.020081+0000","last_undegraded":"2023-01-24T03:30:25.020081+0000","last_fullsized":"2023-01-24T03:30:25.020081+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:59:13.108074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5,3],"acting":[7,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996112+0000","last_change":"2023-01-24T03:30:16.272574+0000","last_active":"2023-01-24T03:30:24.996112+0000","last_peered":"2023-01-24T03:30:24.996112+0000","last_clean":"2023-01-24T03:30:24.996112+0000","last_became_active":"2023-01-24T03:30:16.272252+0000","last_became_peered":"2023-01-24T03:30:16.272252+0000","last_unstale":"2023-01-24T03:30:24.996112+0000","last_undegraded":"2023-01-24T03:30:24.996112+0000","last_fullsized":"2023-01-24T03:30:24.996112+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:12:08.393585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991181+0000","last_change":"2023-01-24T03:30:20.606878+0000","last_active":"2023-01-24T03:30:24.991181+0000","last_peered":"2023-01-24T03:30:24.991181+0000","last_clean":"2023-01-24T03:30:24.991181+0000","last_became_active":"2023-01-24T03:30:20.606442+0000","last_became_peered":"2023-01-24T03:30:20.606442+0000","last_unstale":"2023-01-24T03:30:24.991181+0000","last_undegraded":"2023-01-24T03:30:24.991181+0000","last_fullsized":"2023-01-24T03:30:24.991181+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:26:16.916080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990646+0000","last_change":"2023-01-24T03:30:20.990840+0000","last_active":"2023-01-24T03:30:24.990646+0000","last_peered":"2023-01-24T03:30:24.990646+0000","last_clean":"2023-01-24T03:30:24.990646+0000","last_became_active":"2023-01-24T03:30:20.990615+0000","last_became_peered":"2023-01-24T03:30:20.990615+0000","last_unstale":"2023-01-24T03:30:24.990646+0000","last_undegraded":"2023-01-24T03:30:24.990646+0000","last_fullsized":"2023-01-24T03:30:24.990646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T15:06:37.670127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.013106+0000","last_change":"2023-01-24T03:30:25.006375+0000","last_active":"2023-01-24T03:30:25.013106+0000","last_peered":"2023-01-24T03:30:25.013106+0000","last_clean":"2023-01-24T03:30:25.013106+0000","last_became_active":"2023-01-24T03:30:25.006027+0000","last_became_peered":"2023-01-24T03:30:25.006027+0000","last_unstale":"2023-01-24T03:30:25.013106+0000","last_undegraded":"2023-01-24T03:30:25.013106+0000","last_fullsized":"2023-01-24T03:30:25.013106+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:51:29.481545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,6],"acting":[0,1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005695+0000","last_change":"2023-01-24T03:30:16.272475+0000","last_active":"2023-01-24T03:30:25.005695+0000","last_peered":"2023-01-24T03:30:25.005695+0000","last_clean":"2023-01-24T03:30:25.005695+0000","last_became_active":"2023-01-24T03:30:16.272140+0000","last_became_peered":"2023-01-24T03:30:16.272140+0000","last_unstale":"2023-01-24T03:30:25.005695+0000","last_undegraded":"2023-01-24T03:30:25.005695+0000","last_fullsized":"2023-01-24T03:30:25.005695+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:48:16.300876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992670+0000","last_change":"2023-01-24T03:30:20.609391+0000","last_active":"2023-01-24T03:30:24.992670+0000","last_peered":"2023-01-24T03:30:24.992670+0000","last_clean":"2023-01-24T03:30:24.992670+0000","last_became_active":"2023-01-24T03:30:20.609124+0000","last_became_peered":"2023-01-24T03:30:20.609124+0000","last_unstale":"2023-01-24T03:30:24.992670+0000","last_undegraded":"2023-01-24T03:30:24.992670+0000","last_fullsized":"2023-01-24T03:30:24.992670+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:36:49.616649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.001859+0000","last_change":"2023-01-24T03:30:20.996985+0000","last_active":"2023-01-24T03:30:25.001859+0000","last_peered":"2023-01-24T03:30:25.001859+0000","last_clean":"2023-01-24T03:30:25.001859+0000","last_became_active":"2023-01-24T03:30:20.996612+0000","last_became_peered":"2023-01-24T03:30:20.996612+0000","last_unstale":"2023-01-24T03:30:25.001859+0000","last_undegraded":"2023-01-24T03:30:25.001859+0000","last_fullsized":"2023-01-24T03:30:25.001859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:10:50.378902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.012218+0000","last_change":"2023-01-24T03:30:25.008155+0000","last_active":"2023-01-24T03:30:25.012218+0000","last_peered":"2023-01-24T03:30:25.012218+0000","last_clean":"2023-01-24T03:30:25.012218+0000","last_became_active":"2023-01-24T03:30:25.007674+0000","last_became_peered":"2023-01-24T03:30:25.007674+0000","last_unstale":"2023-01-24T03:30:25.012218+0000","last_undegraded":"2023-01-24T03:30:25.012218+0000","last_fullsized":"2023-01-24T03:30:25.012218+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:55:46.448808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,2],"acting":[6,0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990827+0000","last_change":"2023-01-24T03:30:16.272372+0000","last_active":"2023-01-24T03:30:24.990827+0000","last_peered":"2023-01-24T03:30:24.990827+0000","last_clean":"2023-01-24T03:30:24.990827+0000","last_became_active":"2023-01-24T03:30:16.272104+0000","last_became_peered":"2023-01-24T03:30:16.272104+0000","last_unstale":"2023-01-24T03:30:24.990827+0000","last_undegraded":"2023-01-24T03:30:24.990827+0000","last_fullsized":"2023-01-24T03:30:24.990827+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:15:10.865596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002114+0000","last_change":"2023-01-24T03:30:20.612284+0000","last_active":"2023-01-24T03:30:25.002114+0000","last_peered":"2023-01-24T03:30:25.002114+0000","last_clean":"2023-01-24T03:30:25.002114+0000","last_became_active":"2023-01-24T03:30:20.612029+0000","last_became_peered":"2023-01-24T03:30:20.612029+0000","last_unstale":"2023-01-24T03:30:25.002114+0000","last_undegraded":"2023-01-24T03:30:25.002114+0000","last_fullsized":"2023-01-24T03:30:25.002114+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:18:58.755974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992643+0000","last_change":"2023-01-24T03:30:20.989834+0000","last_active":"2023-01-24T03:30:24.992643+0000","last_peered":"2023-01-24T03:30:24.992643+0000","last_clean":"2023-01-24T03:30:24.992643+0000","last_became_active":"2023-01-24T03:30:20.989645+0000","last_became_peered":"2023-01-24T03:30:20.989645+0000","last_unstale":"2023-01-24T03:30:24.992643+0000","last_undegraded":"2023-01-24T03:30:24.992643+0000","last_fullsized":"2023-01-24T03:30:24.992643+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:42:06.477043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016259+0000","last_change":"2023-01-24T03:30:25.011952+0000","last_active":"2023-01-24T03:30:25.016259+0000","last_peered":"2023-01-24T03:30:25.016259+0000","last_clean":"2023-01-24T03:30:25.016259+0000","last_became_active":"2023-01-24T03:30:25.011612+0000","last_became_peered":"2023-01-24T03:30:25.011612+0000","last_unstale":"2023-01-24T03:30:25.016259+0000","last_undegraded":"2023-01-24T03:30:25.016259+0000","last_fullsized":"2023-01-24T03:30:25.016259+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:51:52.027285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6,5],"acting":[0,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990404+0000","last_change":"2023-01-24T03:30:16.048979+0000","last_active":"2023-01-24T03:30:24.990404+0000","last_peered":"2023-01-24T03:30:24.990404+0000","last_clean":"2023-01-24T03:30:24.990404+0000","last_became_active":"2023-01-24T03:30:16.048641+0000","last_became_peered":"2023-01-24T03:30:16.048641+0000","last_unstale":"2023-01-24T03:30:24.990404+0000","last_undegraded":"2023-01-24T03:30:24.990404+0000","last_fullsized":"2023-01-24T03:30:24.990404+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T15:11:26.211024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005087+0000","last_change":"2023-01-24T03:30:20.606906+0000","last_active":"2023-01-24T03:30:25.005087+0000","last_peered":"2023-01-24T03:30:25.005087+0000","last_clean":"2023-01-24T03:30:25.005087+0000","last_became_active":"2023-01-24T03:30:20.606640+0000","last_became_peered":"2023-01-24T03:30:20.606640+0000","last_unstale":"2023-01-24T03:30:25.005087+0000","last_undegraded":"2023-01-24T03:30:25.005087+0000","last_fullsized":"2023-01-24T03:30:25.005087+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:21:09.008653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002357+0000","last_change":"2023-01-24T03:30:20.999886+0000","last_active":"2023-01-24T03:30:25.002357+0000","last_peered":"2023-01-24T03:30:25.002357+0000","last_clean":"2023-01-24T03:30:25.002357+0000","last_became_active":"2023-01-24T03:30:20.999394+0000","last_became_peered":"2023-01-24T03:30:20.999394+0000","last_unstale":"2023-01-24T03:30:25.002357+0000","last_undegraded":"2023-01-24T03:30:25.002357+0000","last_fullsized":"2023-01-24T03:30:25.002357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:30:22.395611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016356+0000","last_change":"2023-01-24T03:30:25.006190+0000","last_active":"2023-01-24T03:30:25.016356+0000","last_peered":"2023-01-24T03:30:25.016356+0000","last_clean":"2023-01-24T03:30:25.016356+0000","last_became_active":"2023-01-24T03:30:25.005825+0000","last_became_peered":"2023-01-24T03:30:25.005825+0000","last_unstale":"2023-01-24T03:30:25.016356+0000","last_undegraded":"2023-01-24T03:30:25.016356+0000","last_fullsized":"2023-01-24T03:30:25.016356+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:51:50.888938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,0],"acting":[1,3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"24'1","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.446961+0000","last_change":"2023-01-24T03:30:20.611079+0000","last_active":"2023-01-24T03:30:25.446961+0000","last_peered":"2023-01-24T03:30:25.446961+0000","last_clean":"2023-01-24T03:30:25.446961+0000","last_became_active":"2023-01-24T03:30:20.610841+0000","last_became_peered":"2023-01-24T03:30:20.610841+0000","last_unstale":"2023-01-24T03:30:25.446961+0000","last_undegraded":"2023-01-24T03:30:25.446961+0000","last_fullsized":"2023-01-24T03:30:25.446961+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:34:24.922661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.994693+0000","last_change":"2023-01-24T03:30:20.991418+0000","last_active":"2023-01-24T03:30:24.994693+0000","last_peered":"2023-01-24T03:30:24.994693+0000","last_clean":"2023-01-24T03:30:24.994693+0000","last_became_active":"2023-01-24T03:30:20.991167+0000","last_became_peered":"2023-01-24T03:30:20.991167+0000","last_unstale":"2023-01-24T03:30:24.994693+0000","last_undegraded":"2023-01-24T03:30:24.994693+0000","last_fullsized":"2023-01-24T03:30:24.994693+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:51:52.736691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.013519+0000","last_change":"2023-01-24T03:30:25.010394+0000","last_active":"2023-01-24T03:30:25.013519+0000","last_peered":"2023-01-24T03:30:25.013519+0000","last_clean":"2023-01-24T03:30:25.013519+0000","last_became_active":"2023-01-24T03:30:25.010044+0000","last_became_peered":"2023-01-24T03:30:25.010044+0000","last_unstale":"2023-01-24T03:30:25.013519+0000","last_undegraded":"2023-01-24T03:30:25.013519+0000","last_fullsized":"2023-01-24T03:30:25.013519+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:08:55.250184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1,6],"acting":[7,4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"24'2","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.984877+0000","last_change":"2023-01-24T03:30:20.607538+0000","last_active":"2023-01-24T03:30:25.984877+0000","last_peered":"2023-01-24T03:30:25.984877+0000","last_clean":"2023-01-24T03:30:25.984877+0000","last_became_active":"2023-01-24T03:30:20.607285+0000","last_became_peered":"2023-01-24T03:30:20.607285+0000","last_unstale":"2023-01-24T03:30:25.984877+0000","last_undegraded":"2023-01-24T03:30:25.984877+0000","last_fullsized":"2023-01-24T03:30:25.984877+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:42:21.836603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992857+0000","last_change":"2023-01-24T03:30:20.986791+0000","last_active":"2023-01-24T03:30:24.992857+0000","last_peered":"2023-01-24T03:30:24.992857+0000","last_clean":"2023-01-24T03:30:24.992857+0000","last_became_active":"2023-01-24T03:30:20.986516+0000","last_became_peered":"2023-01-24T03:30:20.986516+0000","last_unstale":"2023-01-24T03:30:24.992857+0000","last_undegraded":"2023-01-24T03:30:24.992857+0000","last_fullsized":"2023-01-24T03:30:24.992857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:53:19.191166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.015383+0000","last_change":"2023-01-24T03:30:25.010751+0000","last_active":"2023-01-24T03:30:25.015383+0000","last_peered":"2023-01-24T03:30:25.015383+0000","last_clean":"2023-01-24T03:30:25.015383+0000","last_became_active":"2023-01-24T03:30:25.010332+0000","last_became_peered":"2023-01-24T03:30:25.010332+0000","last_unstale":"2023-01-24T03:30:25.015383+0000","last_undegraded":"2023-01-24T03:30:25.015383+0000","last_fullsized":"2023-01-24T03:30:25.015383+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:48:30.190712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7,4],"acting":[5,0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"24'2","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985166+0000","last_change":"2023-01-24T03:30:20.608509+0000","last_active":"2023-01-24T03:30:25.985166+0000","last_peered":"2023-01-24T03:30:25.985166+0000","last_clean":"2023-01-24T03:30:25.985166+0000","last_became_active":"2023-01-24T03:30:20.608313+0000","last_became_peered":"2023-01-24T03:30:20.608313+0000","last_unstale":"2023-01-24T03:30:25.985166+0000","last_undegraded":"2023-01-24T03:30:25.985166+0000","last_fullsized":"2023-01-24T03:30:25.985166+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:31:52.253588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992448+0000","last_change":"2023-01-24T03:30:20.992334+0000","last_active":"2023-01-24T03:30:24.992448+0000","last_peered":"2023-01-24T03:30:24.992448+0000","last_clean":"2023-01-24T03:30:24.992448+0000","last_became_active":"2023-01-24T03:30:20.992027+0000","last_became_peered":"2023-01-24T03:30:20.992027+0000","last_unstale":"2023-01-24T03:30:24.992448+0000","last_undegraded":"2023-01-24T03:30:24.992448+0000","last_fullsized":"2023-01-24T03:30:24.992448+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:57:56.985507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.012177+0000","last_change":"2023-01-24T03:30:25.004167+0000","last_active":"2023-01-24T03:30:25.012177+0000","last_peered":"2023-01-24T03:30:25.012177+0000","last_clean":"2023-01-24T03:30:25.012177+0000","last_became_active":"2023-01-24T03:30:25.003752+0000","last_became_peered":"2023-01-24T03:30:25.003752+0000","last_unstale":"2023-01-24T03:30:25.012177+0000","last_undegraded":"2023-01-24T03:30:25.012177+0000","last_fullsized":"2023-01-24T03:30:25.012177+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:19:16.285529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"24'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.448123+0000","last_change":"2023-01-24T03:30:20.603542+0000","last_active":"2023-01-24T03:30:25.448123+0000","last_peered":"2023-01-24T03:30:25.448123+0000","last_clean":"2023-01-24T03:30:25.448123+0000","last_became_active":"2023-01-24T03:30:20.603287+0000","last_became_peered":"2023-01-24T03:30:20.603287+0000","last_unstale":"2023-01-24T03:30:25.448123+0000","last_undegraded":"2023-01-24T03:30:25.448123+0000","last_fullsized":"2023-01-24T03:30:25.448123+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:03:56.865763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991098+0000","last_change":"2023-01-24T03:30:20.989519+0000","last_active":"2023-01-24T03:30:24.991098+0000","last_peered":"2023-01-24T03:30:24.991098+0000","last_clean":"2023-01-24T03:30:24.991098+0000","last_became_active":"2023-01-24T03:30:20.989286+0000","last_became_peered":"2023-01-24T03:30:20.989286+0000","last_unstale":"2023-01-24T03:30:24.991098+0000","last_undegraded":"2023-01-24T03:30:24.991098+0000","last_fullsized":"2023-01-24T03:30:24.991098+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:17:30.829327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016619+0000","last_change":"2023-01-24T03:30:25.010262+0000","last_active":"2023-01-24T03:30:25.016619+0000","last_peered":"2023-01-24T03:30:25.016619+0000","last_clean":"2023-01-24T03:30:25.016619+0000","last_became_active":"2023-01-24T03:30:25.009930+0000","last_became_peered":"2023-01-24T03:30:25.009930+0000","last_unstale":"2023-01-24T03:30:25.016619+0000","last_undegraded":"2023-01-24T03:30:25.016619+0000","last_fullsized":"2023-01-24T03:30:25.016619+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:36:17.807046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7,2],"acting":[1,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"24'4","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985911+0000","last_change":"2023-01-24T03:30:20.607334+0000","last_active":"2023-01-24T03:30:25.985911+0000","last_peered":"2023-01-24T03:30:25.985911+0000","last_clean":"2023-01-24T03:30:25.985911+0000","last_became_active":"2023-01-24T03:30:20.607079+0000","last_became_peered":"2023-01-24T03:30:20.607079+0000","last_unstale":"2023-01-24T03:30:25.985911+0000","last_undegraded":"2023-01-24T03:30:25.985911+0000","last_fullsized":"2023-01-24T03:30:25.985911+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:05:18.886532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992554+0000","last_change":"2023-01-24T03:30:20.987567+0000","last_active":"2023-01-24T03:30:24.992554+0000","last_peered":"2023-01-24T03:30:24.992554+0000","last_clean":"2023-01-24T03:30:24.992554+0000","last_became_active":"2023-01-24T03:30:20.987058+0000","last_became_peered":"2023-01-24T03:30:20.987058+0000","last_unstale":"2023-01-24T03:30:24.992554+0000","last_undegraded":"2023-01-24T03:30:24.992554+0000","last_fullsized":"2023-01-24T03:30:24.992554+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:58:30.391733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.989538+0000","last_change":"2023-01-24T03:30:20.986393+0000","last_active":"2023-01-24T03:30:24.989538+0000","last_peered":"2023-01-24T03:30:24.989538+0000","last_clean":"2023-01-24T03:30:24.989538+0000","last_became_active":"2023-01-24T03:30:20.986089+0000","last_became_peered":"2023-01-24T03:30:20.986089+0000","last_unstale":"2023-01-24T03:30:24.989538+0000","last_undegraded":"2023-01-24T03:30:24.989538+0000","last_fullsized":"2023-01-24T03:30:24.989538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:44:21.992291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990902+0000","last_change":"2023-01-24T03:30:20.991468+0000","last_active":"2023-01-24T03:30:24.990902+0000","last_peered":"2023-01-24T03:30:24.990902+0000","last_clean":"2023-01-24T03:30:24.990902+0000","last_became_active":"2023-01-24T03:30:20.991227+0000","last_became_peered":"2023-01-24T03:30:20.991227+0000","last_unstale":"2023-01-24T03:30:24.990902+0000","last_undegraded":"2023-01-24T03:30:24.990902+0000","last_fullsized":"2023-01-24T03:30:24.990902+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:27:25.806340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992602+0000","last_change":"2023-01-24T03:30:20.989824+0000","last_active":"2023-01-24T03:30:24.992602+0000","last_peered":"2023-01-24T03:30:24.992602+0000","last_clean":"2023-01-24T03:30:24.992602+0000","last_became_active":"2023-01-24T03:30:20.989577+0000","last_became_peered":"2023-01-24T03:30:20.989577+0000","last_unstale":"2023-01-24T03:30:24.992602+0000","last_undegraded":"2023-01-24T03:30:24.992602+0000","last_fullsized":"2023-01-24T03:30:24.992602+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:48:17.948459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992452+0000","last_change":"2023-01-24T03:30:20.996266+0000","last_active":"2023-01-24T03:30:24.992452+0000","last_peered":"2023-01-24T03:30:24.992452+0000","last_clean":"2023-01-24T03:30:24.992452+0000","last_became_active":"2023-01-24T03:30:20.995610+0000","last_became_peered":"2023-01-24T03:30:20.995610+0000","last_unstale":"2023-01-24T03:30:24.992452+0000","last_undegraded":"2023-01-24T03:30:24.992452+0000","last_fullsized":"2023-01-24T03:30:24.992452+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:24:43.022530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.001943+0000","last_change":"2023-01-24T03:30:20.997081+0000","last_active":"2023-01-24T03:30:25.001943+0000","last_peered":"2023-01-24T03:30:25.001943+0000","last_clean":"2023-01-24T03:30:25.001943+0000","last_became_active":"2023-01-24T03:30:20.996837+0000","last_became_peered":"2023-01-24T03:30:20.996837+0000","last_unstale":"2023-01-24T03:30:25.001943+0000","last_undegraded":"2023-01-24T03:30:25.001943+0000","last_fullsized":"2023-01-24T03:30:25.001943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:42:06.134817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2758,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40,"ondisk_log_size":40,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64574,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9180,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":8832,"kb_avail":94362660,"statfs":{"total":96636764160,"available":96627363840,"internally_reserved":0,"allocated":348160,"data_stored":696554,"data_compressed":32287,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":9043968},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8576,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94363264,"statfs":{"total":96636764160,"available":96627982336,"internally_reserved":0,"allocated":253952,"data_stored":104434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9152,"kb_used_data":312,"kb_used_omap":0,"kb_used_meta":8832,"kb_avail":94362688,"statfs":{"total":96636764160,"available":96627392512,"internally_reserved":0,"allocated":319488,"data_stored":694802,"data_compressed":32287,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":9043968},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28000000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8640,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94363200,"statfs":{"total":96636764160,"available":96627916800,"internally_reserved":0,"allocated":253952,"data_stored":104382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8512,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94363328,"statfs":{"total":96636764160,"available":96628047872,"internally_reserved":0,"allocated":253952,"data_stored":104434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8580,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94363260,"statfs":{"total":96636764160,"available":96627978240,"internally_reserved":0,"allocated":258048,"data_stored":104524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.311}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8660,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94363180,"statfs":{"total":96636764160,"available":96627896320,"internally_reserved":0,"allocated":274432,"data_stored":106026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8580,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94363260,"statfs":{"total":96636764160,"available":96627978240,"internally_reserved":0,"allocated":258048,"data_stored":103705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32287,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32287,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-01-24T03:30:38.995 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-01-24T03:30:38.996 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-01-24T03:30:38.996 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-01-24T03:30:38.997 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-01-24T03:30:38.997 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-01-24T03:30:39.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.137+0000 7f7cc0833700 1 Processor -- start 2023-01-24T03:30:39.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.137+0000 7f7cc0833700 1 -- start start 2023-01-24T03:30:39.142 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.137+0000 7f7cc0833700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cbc0c5680 0x7f7cbc156730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:39.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.137+0000 7f7cc0833700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7cbc155e20 0x7f7cbc156c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:39.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.137+0000 7f7cc0833700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cbc0c3d10 con 0x7f7cbc0c5a50 2023-01-24T03:30:39.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.137+0000 7f7cc0833700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cbc0c4eb0 con 0x7f7cbc155e20 2023-01-24T03:30:39.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.137+0000 7f7cc0833700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cbc0c2e20 con 0x7f7cbc0c5680 2023-01-24T03:30:39.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.138+0000 7f7cbaffd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7cbc155e20 0x7f7cbc156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:39.144 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.138+0000 7f7cbb7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cbc0c5680 0x7f7cbc156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:39.144 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.138+0000 7f7cbaffd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7cbc155e20 0x7f7cbc156c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.112:42910/0 (socket says 172.21.15.112:42910) 2023-01-24T03:30:39.144 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.138+0000 7f7cbaffd700 1 -- 172.21.15.112:0/1088842470 learned_addr learned my addr 172.21.15.112:0/1088842470 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:39.144 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.138+0000 7f7cbbfff700 1 --1- 172.21.15.112:0/1088842470 >> v1:172.21.15.142:6789/0 conn(0x7f7cbc0c5a50 0x7f7cbc0cb870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.112:45192/0 (socket says 172.21.15.112:45192) 2023-01-24T03:30:39.144 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.138+0000 7f7cbb7fe700 1 -- 172.21.15.112:0/1088842470 >> v2:172.21.15.142:3300/0 conn(0x7f7cbc155e20 msgr2=0x7f7cbc156c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:39.145 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.138+0000 7f7cbb7fe700 1 --2- 172.21.15.112:0/1088842470 >> v2:172.21.15.142:3300/0 conn(0x7f7cbc155e20 0x7f7cbc156c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:39.145 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.138+0000 7f7cbb7fe700 1 -- 172.21.15.112:0/1088842470 >> v1:172.21.15.142:6789/0 conn(0x7f7cbc0c5a50 legacy=0x7f7cbc0cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:39.145 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.138+0000 7f7cbb7fe700 1 -- 172.21.15.112:0/1088842470 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cbc0c3100 con 0x7f7cbc0c5680 2023-01-24T03:30:39.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.138+0000 7f7cbb7fe700 1 --2- 172.21.15.112:0/1088842470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cbc0c5680 0x7f7cbc156730 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0009790 tx=0x7f7cb0009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe4cfa3ae6505d0a server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:39.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.138+0000 7f7cba7fc700 1 -- 172.21.15.112:0/1088842470 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb0011630 con 0x7f7cbc0c5680 2023-01-24T03:30:39.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.139+0000 7f7cba7fc700 1 -- 172.21.15.112:0/1088842470 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cb0011790 con 0x7f7cbc0c5680 2023-01-24T03:30:39.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.139+0000 7f7cba7fc700 1 -- 172.21.15.112:0/1088842470 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb0011b50 con 0x7f7cbc0c5680 2023-01-24T03:30:39.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.139+0000 7f7cc0833700 1 -- 172.21.15.112:0/1088842470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cbc0c5680 msgr2=0x7f7cbc156730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:39.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.139+0000 7f7cc0833700 1 --2- 172.21.15.112:0/1088842470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cbc0c5680 0x7f7cbc156730 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0009790 tx=0x7f7cb0009060 comp rx=0 tx=0).stop 2023-01-24T03:30:39.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.139+0000 7f7cc0833700 1 -- 172.21.15.112:0/1088842470 shutdown_connections 2023-01-24T03:30:39.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.139+0000 7f7cc0833700 1 --2- 172.21.15.112:0/1088842470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cbc0c5680 0x7f7cbc156730 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:39.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.139+0000 7f7cc0833700 1 --2- 172.21.15.112:0/1088842470 >> v2:172.21.15.142:3300/0 conn(0x7f7cbc155e20 0x7f7cbc156c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:39.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.139+0000 7f7cc0833700 1 -- 172.21.15.112:0/1088842470 >> 172.21.15.112:0/1088842470 conn(0x7f7cbc0685b0 msgr2=0x7f7cbc0689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:39.148 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.139+0000 7f7cc0833700 1 -- 172.21.15.112:0/1088842470 shutdown_connections 2023-01-24T03:30:39.148 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.139+0000 7f7cc0833700 1 -- 172.21.15.112:0/1088842470 wait complete. 2023-01-24T03:30:39.150 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.140+0000 7f7cc0833700 1 Processor -- start 2023-01-24T03:30:39.150 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.140+0000 7f7cc0833700 1 -- start start 2023-01-24T03:30:39.151 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.141+0000 7f7cc0833700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cbc0c5680 0x7f7cbc1662f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:39.151 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.141+0000 7f7cc0833700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7cbc0c5a50 0x7f7cbc0c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:39.151 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.141+0000 7f7cc0833700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7cbc155e20 0x7f7cbc0c1430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:39.151 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.141+0000 7f7cc0833700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cbc0c00f0 con 0x7f7cbc0c5a50 2023-01-24T03:30:39.152 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.141+0000 7f7cc0833700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cbc0c1970 con 0x7f7cbc155e20 2023-01-24T03:30:39.152 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.141+0000 7f7cc0833700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cbc0c28b0 con 0x7f7cbc0c5680 2023-01-24T03:30:39.152 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.141+0000 7f7cbaffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7cbc0c5a50 0x7f7cbc0c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:39.152 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.141+0000 7f7cbaffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7cbc0c5a50 0x7f7cbc0c8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50418/0 (socket says 172.21.15.112:50418) 2023-01-24T03:30:39.153 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.141+0000 7f7cbaffd700 1 -- 172.21.15.112:0/1258337793 learned_addr learned my addr 172.21.15.112:0/1258337793 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:39.153 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.141+0000 7f7cbbfff700 1 --2- 172.21.15.112:0/1258337793 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7cbc155e20 0x7f7cbc0c1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:39.153 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.141+0000 7f7cbb7fe700 1 --2- 172.21.15.112:0/1258337793 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cbc0c5680 0x7f7cbc1662f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:39.153 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.141+0000 7f7cbaffd700 1 -- 172.21.15.112:0/1258337793 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cbc0c5680 msgr2=0x7f7cbc1662f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:39.153 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.141+0000 7f7cbaffd700 1 --2- 172.21.15.112:0/1258337793 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cbc0c5680 0x7f7cbc1662f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:39.154 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.141+0000 7f7cbaffd700 1 -- 172.21.15.112:0/1258337793 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7cbc155e20 msgr2=0x7f7cbc0c1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:39.154 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.141+0000 7f7cbaffd700 1 --2- 172.21.15.112:0/1258337793 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7cbc155e20 0x7f7cbc0c1430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:39.154 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.141+0000 7f7cbaffd700 1 -- 172.21.15.112:0/1258337793 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb00092a0 con 0x7f7cbc0c5a50 2023-01-24T03:30:39.154 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.141+0000 7f7cbaffd700 1 --2- 172.21.15.112:0/1258337793 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7cbc0c5a50 0x7f7cbc0c8c10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7cac00e930 tx=0x7f7cac0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:39.154 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.142+0000 7f7ca37fe700 1 -- 172.21.15.112:0/1258337793 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cac013680 con 0x7f7cbc0c5a50 2023-01-24T03:30:39.155 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.142+0000 7f7ca37fe700 1 -- 172.21.15.112:0/1258337793 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cac0137e0 con 0x7f7cbc0c5a50 2023-01-24T03:30:39.155 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.142+0000 7f7cc0833700 1 -- 172.21.15.112:0/1258337793 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cbc0c05e0 con 0x7f7cbc0c5a50 2023-01-24T03:30:39.155 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.142+0000 7f7ca37fe700 1 -- 172.21.15.112:0/1258337793 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cac013b80 con 0x7f7cbc0c5a50 2023-01-24T03:30:39.155 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.142+0000 7f7cc0833700 1 -- 172.21.15.112:0/1258337793 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cbc0c0300 con 0x7f7cbc0c5a50 2023-01-24T03:30:39.156 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.143+0000 7f7cc0833700 1 -- 172.21.15.112:0/1258337793 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cbc05ea20 con 0x7f7cbc0c5a50 2023-01-24T03:30:39.156 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.143+0000 7f7ca37fe700 1 -- 172.21.15.112:0/1258337793 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f7cac019030 con 0x7f7cbc0c5a50 2023-01-24T03:30:39.156 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.143+0000 7f7ca37fe700 1 --2- 172.21.15.112:0/1258337793 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7c9c0611c0 0x7f7c9c063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:39.156 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.143+0000 7f7cbb7fe700 1 --2- 172.21.15.112:0/1258337793 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7c9c0611c0 0x7f7c9c063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:39.157 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.143+0000 7f7ca37fe700 1 -- 172.21.15.112:0/1258337793 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f7cac07d580 con 0x7f7cbc0c5a50 2023-01-24T03:30:39.157 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.144+0000 7f7cbb7fe700 1 --2- 172.21.15.112:0/1258337793 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7c9c0611c0 0x7f7c9c063670 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0011fd0 tx=0x7f7cb0017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:39.157 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.148+0000 7f7ca37fe700 1 -- 172.21.15.112:0/1258337793 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f7cac04d3c0 con 0x7f7cbc0c5a50 2023-01-24T03:30:39.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.374+0000 7f7cc0833700 1 -- 172.21.15.112:0/1258337793 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f7cbc0c87c0 con 0x7f7cbc0c5a50 2023-01-24T03:30:39.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.377+0000 7f7ca37fe700 1 -- 172.21.15.112:0/1258337793 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+154441 (secure 0 0 0) 0x7f7cac0509e0 con 0x7f7cbc0c5a50 2023-01-24T03:30:39.381 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:30:39.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.379+0000 7f7cc0833700 1 -- 172.21.15.112:0/1258337793 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7c9c0611c0 msgr2=0x7f7c9c063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:39.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.379+0000 7f7cc0833700 1 --2- 172.21.15.112:0/1258337793 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7c9c0611c0 0x7f7c9c063670 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0011fd0 tx=0x7f7cb0017000 comp rx=0 tx=0).stop 2023-01-24T03:30:39.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.379+0000 7f7cc0833700 1 -- 172.21.15.112:0/1258337793 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7cbc0c5a50 msgr2=0x7f7cbc0c8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:39.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.379+0000 7f7cc0833700 1 --2- 172.21.15.112:0/1258337793 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7cbc0c5a50 0x7f7cbc0c8c10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7cac00e930 tx=0x7f7cac0090f0 comp rx=0 tx=0).stop 2023-01-24T03:30:39.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.379+0000 7f7cc0833700 1 -- 172.21.15.112:0/1258337793 shutdown_connections 2023-01-24T03:30:39.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.379+0000 7f7cc0833700 1 --2- 172.21.15.112:0/1258337793 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7c9c0611c0 0x7f7c9c063670 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:39.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.379+0000 7f7cc0833700 1 --2- 172.21.15.112:0/1258337793 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7cbc155e20 0x7f7cbc0c1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:39.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.379+0000 7f7cc0833700 1 --2- 172.21.15.112:0/1258337793 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7cbc0c5680 0x7f7cbc1662f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:39.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.379+0000 7f7cc0833700 1 --2- 172.21.15.112:0/1258337793 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7cbc0c5a50 0x7f7cbc0c8c10 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:39.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.379+0000 7f7cc0833700 1 -- 172.21.15.112:0/1258337793 >> 172.21.15.112:0/1258337793 conn(0x7f7cbc0685b0 msgr2=0x7f7cbc1550e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:39.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.379+0000 7f7cc0833700 1 -- 172.21.15.112:0/1258337793 shutdown_connections 2023-01-24T03:30:39.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.379+0000 7f7cc0833700 1 -- 172.21.15.112:0/1258337793 wait complete. 2023-01-24T03:30:39.394 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":5,"active_gid":4104,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":84276},{"type":"v1","addr":"172.21.15.112:6833","nonce":84276}]},"active_addr":"172.21.15.112:6833/84276","active_change":"2023-01-24T03:30:08.989414+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":4103,"name":"x","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":false,"error_string":"No module named 'packaging'","module_options":{}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":false,"error_string":"No module named 'packaging'","module_options":{}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.112:0","nonce":4248181945}]},{"addrvec":[{"type":"v2","addr":"172.21.15.112:0","nonce":1971001554}]},{"addrvec":[{"type":"v2","addr":"172.21.15.112:0","nonce":4008562818}]},{"addrvec":[{"type":"v2","addr":"172.21.15.112:0","nonce":3158883903}]}]}} 2023-01-24T03:30:39.399 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-01-24T03:30:39.399 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-01-24T03:30:39.399 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T03:30:39.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.508+0000 7f9826b95700 1 Processor -- start 2023-01-24T03:30:39.512 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.508+0000 7f9826b95700 1 -- start start 2023-01-24T03:30:39.519 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.509+0000 7f9826b95700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f9820069c90 0x7f98200cc080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:39.519 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.509+0000 7f9826b95700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f9820155e30 0x7f98200c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:39.520 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.509+0000 7f9826b95700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98200b09e0 con 0x7f98200698c0 2023-01-24T03:30:39.520 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.509+0000 7f9826b95700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98200cc5c0 con 0x7f9820155e30 2023-01-24T03:30:39.520 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.509+0000 7f9826b95700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98200b0c80 con 0x7f9820069c90 2023-01-24T03:30:39.520 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.509+0000 7f9826394700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f9820155e30 0x7f98200c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:39.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.509+0000 7f9825b93700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f98200698c0 0x7f98200ae890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:59646/0 (socket says 172.21.15.112:59646) 2023-01-24T03:30:39.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.509+0000 7f9825b93700 1 -- 172.21.15.112:0/1044586086 learned_addr learned my addr 172.21.15.112:0/1044586086 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:39.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.509+0000 7f9825392700 1 --2- 172.21.15.112:0/1044586086 >> v2:172.21.15.142:3300/0 conn(0x7f9820069c90 0x7f98200cc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:39.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.509+0000 7f9826394700 1 -- 172.21.15.112:0/1044586086 >> v2:172.21.15.142:3300/0 conn(0x7f9820069c90 msgr2=0x7f98200cc080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:39.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.509+0000 7f9826394700 1 --2- 172.21.15.112:0/1044586086 >> v2:172.21.15.142:3300/0 conn(0x7f9820069c90 0x7f98200cc080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:39.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.509+0000 7f9826394700 1 -- 172.21.15.112:0/1044586086 >> v1:172.21.15.112:6789/0 conn(0x7f98200698c0 legacy=0x7f98200ae890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:39.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.509+0000 7f9824b91700 1 -- 172.21.15.112:0/1044586086 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 601905145 0 0) 0x7f98200b09e0 con 0x7f98200698c0 2023-01-24T03:30:39.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.509+0000 7f9826394700 1 -- 172.21.15.112:0/1044586086 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98200b0f40 con 0x7f9820155e30 2023-01-24T03:30:39.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.509+0000 7f9825392700 1 --2- 172.21.15.112:0/1044586086 >> v2:172.21.15.142:3300/0 conn(0x7f9820069c90 0x7f98200cc080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:30:39.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.509+0000 7f9826394700 1 --2- 172.21.15.112:0/1044586086 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9820155e30 0x7f98200c69a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f98200ae610 tx=0x7f9818009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af307a9472c02d51 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:39.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.510+0000 7f9824b91700 1 -- 172.21.15.112:0/1044586086 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9818011630 con 0x7f9820155e30 2023-01-24T03:30:39.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.510+0000 7f9824b91700 1 -- 172.21.15.112:0/1044586086 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9818011790 con 0x7f9820155e30 2023-01-24T03:30:39.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.510+0000 7f9824b91700 1 -- 172.21.15.112:0/1044586086 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9818011b50 con 0x7f9820155e30 2023-01-24T03:30:39.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.510+0000 7f9826b95700 1 -- 172.21.15.112:0/1044586086 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9820155e30 msgr2=0x7f98200c69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:39.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.510+0000 7f9826b95700 1 --2- 172.21.15.112:0/1044586086 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9820155e30 0x7f98200c69a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f98200ae610 tx=0x7f9818009060 comp rx=0 tx=0).stop 2023-01-24T03:30:39.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.510+0000 7f9826b95700 1 -- 172.21.15.112:0/1044586086 shutdown_connections 2023-01-24T03:30:39.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.510+0000 7f9826b95700 1 --2- 172.21.15.112:0/1044586086 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9820155e30 0x7f98200c69a0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:39.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.510+0000 7f9826b95700 1 --2- 172.21.15.112:0/1044586086 >> v2:172.21.15.142:3300/0 conn(0x7f9820069c90 0x7f98200cc080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:39.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.510+0000 7f9826b95700 1 -- 172.21.15.112:0/1044586086 >> 172.21.15.112:0/1044586086 conn(0x7f982005e2c0 msgr2=0x7f98200cb8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:39.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.510+0000 7f9826b95700 1 -- 172.21.15.112:0/1044586086 shutdown_connections 2023-01-24T03:30:39.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.510+0000 7f9826b95700 1 -- 172.21.15.112:0/1044586086 wait complete. 2023-01-24T03:30:39.525 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.511+0000 7f9826b95700 1 Processor -- start 2023-01-24T03:30:39.525 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.511+0000 7f9826b95700 1 -- start start 2023-01-24T03:30:39.525 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.512+0000 7f9826b95700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f98200698c0 0x7f98200c5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:39.525 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.512+0000 7f9826b95700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9820069c90 0x7f98201690a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:39.525 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.512+0000 7f9826b95700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9820155e30 0x7f98200bf810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:39.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.512+0000 7f9826b95700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98200c5a80 con 0x7f9820155e30 2023-01-24T03:30:39.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.512+0000 7f9826b95700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98200cb660 con 0x7f98200698c0 2023-01-24T03:30:39.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.512+0000 7f9826b95700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9820152370 con 0x7f9820069c90 2023-01-24T03:30:39.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.512+0000 7f9826394700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9820155e30 0x7f98200bf810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:39.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.512+0000 7f9826394700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9820155e30 0x7f98200bf810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50436/0 (socket says 172.21.15.112:50436) 2023-01-24T03:30:39.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.512+0000 7f9826394700 1 -- 172.21.15.112:0/1681998170 learned_addr learned my addr 172.21.15.112:0/1681998170 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:39.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.512+0000 7f9825b93700 1 --2- 172.21.15.112:0/1681998170 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f98200698c0 0x7f98200c5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:39.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.512+0000 7f9825392700 1 --2- 172.21.15.112:0/1681998170 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9820069c90 0x7f98201690a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:39.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.512+0000 7f9826394700 1 -- 172.21.15.112:0/1681998170 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9820069c90 msgr2=0x7f98201690a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:39.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.512+0000 7f9826394700 1 --2- 172.21.15.112:0/1681998170 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9820069c90 0x7f98201690a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:39.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.512+0000 7f9826394700 1 -- 172.21.15.112:0/1681998170 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f98200698c0 msgr2=0x7f98200c5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:39.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.512+0000 7f9826394700 1 --2- 172.21.15.112:0/1681998170 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f98200698c0 0x7f98200c5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:39.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.512+0000 7f9826394700 1 -- 172.21.15.112:0/1681998170 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98180092a0 con 0x7f9820155e30 2023-01-24T03:30:39.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.512+0000 7f9826394700 1 --2- 172.21.15.112:0/1681998170 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9820155e30 0x7f98200bf810 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9818009e80 tx=0x7f9818017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:39.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.513+0000 7f981dffb700 1 -- 172.21.15.112:0/1681998170 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9818011b10 con 0x7f9820155e30 2023-01-24T03:30:39.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.513+0000 7f981dffb700 1 -- 172.21.15.112:0/1681998170 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f981801a450 con 0x7f9820155e30 2023-01-24T03:30:39.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.513+0000 7f9826b95700 1 -- 172.21.15.112:0/1681998170 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98201528f0 con 0x7f9820155e30 2023-01-24T03:30:39.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.513+0000 7f981dffb700 1 -- 172.21.15.112:0/1681998170 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f981801a7f0 con 0x7f9820155e30 2023-01-24T03:30:39.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.513+0000 7f9826b95700 1 -- 172.21.15.112:0/1681998170 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9820151fa0 con 0x7f9820155e30 2023-01-24T03:30:39.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.514+0000 7f9826b95700 1 -- 172.21.15.112:0/1681998170 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98201521c0 con 0x7f9820155e30 2023-01-24T03:30:39.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.514+0000 7f981dffb700 1 -- 172.21.15.112:0/1681998170 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f981802b020 con 0x7f9820155e30 2023-01-24T03:30:39.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.514+0000 7f981dffb700 1 --2- 172.21.15.112:0/1681998170 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f98000611c0 0x7f9800063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:39.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.515+0000 7f981dffb700 1 -- 172.21.15.112:0/1681998170 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f981807f330 con 0x7f9820155e30 2023-01-24T03:30:39.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.515+0000 7f9825b93700 1 --2- 172.21.15.112:0/1681998170 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f98000611c0 0x7f9800063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:39.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.515+0000 7f9825b93700 1 --2- 172.21.15.112:0/1681998170 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f98000611c0 0x7f9800063670 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9810001a90 tx=0x7f98100014d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:39.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.519+0000 7f981dffb700 1 -- 172.21.15.112:0/1681998170 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f981804f1f0 con 0x7f9820155e30 2023-01-24T03:30:39.702 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.698+0000 7f9826b95700 1 -- 172.21.15.112:0/1681998170 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f982005ea20 con 0x7f9820155e30 2023-01-24T03:30:39.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.699+0000 7f981dffb700 1 -- 172.21.15.112:0/1681998170 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18755 (secure 0 0 0) 0x7f9818052810 con 0x7f9820155e30 2023-01-24T03:30:39.738 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:30:39.739 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":24,"fsid":"054b9ac9-b7b0-40e5-b932-6c8a955c5a72","created":"2023-01-24T03:30:03.798533+0000","modified":"2023-01-24T03:30:25.373340+0000","last_up_change":"2023-01-24T03:30:13.945316+0000","last_in_change":"2023-01-24T03:30:06.198433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-01-24T03:30:14.931166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-01-24T03:30:15.023730+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-01-24T03:30:19.060597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-01-24T03:30:19.943686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-01-24T03:30:22.999036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"fe6ad231-a39a-42eb-9a3a-49d008dd35e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6809","nonce":84512},{"type":"v1","addr":"172.21.15.112:6811","nonce":84512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":84512},{"type":"v1","addr":"172.21.15.112:6817","nonce":84512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6825","nonce":84512},{"type":"v1","addr":"172.21.15.112:6828","nonce":84512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6819","nonce":84512},{"type":"v1","addr":"172.21.15.112:6822","nonce":84512}]},"public_addr":"172.21.15.112:6811/84512","cluster_addr":"172.21.15.112:6817/84512","heartbeat_back_addr":"172.21.15.112:6828/84512","heartbeat_front_addr":"172.21.15.112:6822/84512","state":["exists","up"]},{"osd":1,"uuid":"8526a1d2-d540-4021-9766-c4bbb30317ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":84513},{"type":"v1","addr":"172.21.15.112:6812","nonce":84513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6815","nonce":84513},{"type":"v1","addr":"172.21.15.112:6818","nonce":84513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6827","nonce":84513},{"type":"v1","addr":"172.21.15.112:6830","nonce":84513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6821","nonce":84513},{"type":"v1","addr":"172.21.15.112:6824","nonce":84513}]},"public_addr":"172.21.15.112:6812/84513","cluster_addr":"172.21.15.112:6818/84513","heartbeat_back_addr":"172.21.15.112:6830/84513","heartbeat_front_addr":"172.21.15.112:6824/84513","state":["exists","up"]},{"osd":2,"uuid":"041ef8e4-4858-432d-9476-cbdb3a0dc47c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":84511},{"type":"v1","addr":"172.21.15.112:6813","nonce":84511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":84511},{"type":"v1","addr":"172.21.15.112:6820","nonce":84511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6829","nonce":84511},{"type":"v1","addr":"172.21.15.112:6831","nonce":84511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6823","nonce":84511},{"type":"v1","addr":"172.21.15.112:6826","nonce":84511}]},"public_addr":"172.21.15.112:6813/84511","cluster_addr":"172.21.15.112:6820/84511","heartbeat_back_addr":"172.21.15.112:6831/84511","heartbeat_front_addr":"172.21.15.112:6826/84511","state":["exists","up"]},{"osd":3,"uuid":"987587cf-1357-4a70-ad85-67b189c67729","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":84510},{"type":"v1","addr":"172.21.15.112:6801","nonce":84510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":84510},{"type":"v1","addr":"172.21.15.112:6803","nonce":84510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":84510},{"type":"v1","addr":"172.21.15.112:6807","nonce":84510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":84510},{"type":"v1","addr":"172.21.15.112:6805","nonce":84510}]},"public_addr":"172.21.15.112:6801/84510","cluster_addr":"172.21.15.112:6803/84510","heartbeat_back_addr":"172.21.15.112:6807/84510","heartbeat_front_addr":"172.21.15.112:6805/84510","state":["exists","up"]},{"osd":4,"uuid":"8503512b-43db-40b8-8b2b-c86cf2e54e17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":84614},{"type":"v1","addr":"172.21.15.142:6801","nonce":84614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":84614},{"type":"v1","addr":"172.21.15.142:6803","nonce":84614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":84614},{"type":"v1","addr":"172.21.15.142:6807","nonce":84614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":84614},{"type":"v1","addr":"172.21.15.142:6805","nonce":84614}]},"public_addr":"172.21.15.142:6801/84614","cluster_addr":"172.21.15.142:6803/84614","heartbeat_back_addr":"172.21.15.142:6807/84614","heartbeat_front_addr":"172.21.15.142:6805/84614","state":["exists","up"]},{"osd":5,"uuid":"d3bc0030-8b08-4645-9ec2-1f8a4e29ef36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":84638},{"type":"v1","addr":"172.21.15.142:6817","nonce":84638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":84638},{"type":"v1","addr":"172.21.15.142:6819","nonce":84638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":84638},{"type":"v1","addr":"172.21.15.142:6823","nonce":84638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":84638},{"type":"v1","addr":"172.21.15.142:6821","nonce":84638}]},"public_addr":"172.21.15.142:6817/84638","cluster_addr":"172.21.15.142:6819/84638","heartbeat_back_addr":"172.21.15.142:6823/84638","heartbeat_front_addr":"172.21.15.142:6821/84638","state":["exists","up"]},{"osd":6,"uuid":"c7649e3b-832d-4e6f-8866-b15e5d814476","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":84645},{"type":"v1","addr":"172.21.15.142:6825","nonce":84645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":84645},{"type":"v1","addr":"172.21.15.142:6827","nonce":84645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":84645},{"type":"v1","addr":"172.21.15.142:6831","nonce":84645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":84645},{"type":"v1","addr":"172.21.15.142:6829","nonce":84645}]},"public_addr":"172.21.15.142:6825/84645","cluster_addr":"172.21.15.142:6827/84645","heartbeat_back_addr":"172.21.15.142:6831/84645","heartbeat_front_addr":"172.21.15.142:6829/84645","state":["exists","up"]},{"osd":7,"uuid":"2ea5b3bf-884e-4b20-b572-1443efdc7477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":84615},{"type":"v1","addr":"172.21.15.142:6809","nonce":84615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":84615},{"type":"v1","addr":"172.21.15.142:6811","nonce":84615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":84615},{"type":"v1","addr":"172.21.15.142:6815","nonce":84615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":84615},{"type":"v1","addr":"172.21.15.142:6813","nonce":84615}]},"public_addr":"172.21.15.142:6809/84615","cluster_addr":"172.21.15.142:6811/84615","heartbeat_back_addr":"172.21.15.142:6815/84615","heartbeat_front_addr":"172.21.15.142:6813/84615","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.370027+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.335683+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.350667+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.350794+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.391119+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.422933+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.400439+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.354018+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T03:30:39.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.701+0000 7f9826b95700 1 -- 172.21.15.112:0/1681998170 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f98000611c0 msgr2=0x7f9800063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:39.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.701+0000 7f9826b95700 1 --2- 172.21.15.112:0/1681998170 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f98000611c0 0x7f9800063670 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9810001a90 tx=0x7f98100014d0 comp rx=0 tx=0).stop 2023-01-24T03:30:39.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.701+0000 7f9826b95700 1 -- 172.21.15.112:0/1681998170 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9820155e30 msgr2=0x7f98200bf810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:39.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.701+0000 7f9826b95700 1 --2- 172.21.15.112:0/1681998170 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9820155e30 0x7f98200bf810 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9818009e80 tx=0x7f9818017000 comp rx=0 tx=0).stop 2023-01-24T03:30:39.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.701+0000 7f9826b95700 1 -- 172.21.15.112:0/1681998170 shutdown_connections 2023-01-24T03:30:39.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.701+0000 7f9826b95700 1 --2- 172.21.15.112:0/1681998170 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f98000611c0 0x7f9800063670 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:39.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.701+0000 7f9826b95700 1 --2- 172.21.15.112:0/1681998170 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9820155e30 0x7f98200bf810 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:39.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.701+0000 7f9826b95700 1 --2- 172.21.15.112:0/1681998170 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f98200698c0 0x7f98200c5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:39.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.701+0000 7f9826b95700 1 --2- 172.21.15.112:0/1681998170 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9820069c90 0x7f98201690a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:39.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.701+0000 7f9826b95700 1 -- 172.21.15.112:0/1681998170 >> 172.21.15.112:0/1681998170 conn(0x7f982005e2c0 msgr2=0x7f9820068970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:39.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.701+0000 7f9826b95700 1 -- 172.21.15.112:0/1681998170 shutdown_connections 2023-01-24T03:30:39.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.701+0000 7f9826b95700 1 -- 172.21.15.112:0/1681998170 wait complete. 2023-01-24T03:30:39.743 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-01-24T03:30:39.743 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T03:30:39.855 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.853+0000 7fcf0d4f4700 1 Processor -- start 2023-01-24T03:30:39.856 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.853+0000 7fcf0d4f4700 1 -- start start 2023-01-24T03:30:39.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.853+0000 7fcf0d4f4700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fcf080b0aa0 0x7fcf08155da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:39.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.853+0000 7fcf0d4f4700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fcf080ca3e0 0x7fcf080ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:39.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.853+0000 7fcf0d4f4700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf080b0f40 con 0x7fcf080c7e40 2023-01-24T03:30:39.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.853+0000 7fcf0d4f4700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf080b1170 con 0x7fcf080b0aa0 2023-01-24T03:30:39.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.853+0000 7fcf0d4f4700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf080b1450 con 0x7fcf080ca3e0 2023-01-24T03:30:39.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.853+0000 7fcefe7fc700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fcf080b0aa0 0x7fcf08155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:39.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.853+0000 7fcf06ffd700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fcf080c7e40 0x7fcf080ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:59660/0 (socket says 172.21.15.112:59660) 2023-01-24T03:30:39.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.853+0000 7fcf06ffd700 1 -- 172.21.15.112:0/3786271800 learned_addr learned my addr 172.21.15.112:0/3786271800 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:39.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.854+0000 7fcefe7fc700 1 -- 172.21.15.112:0/3786271800 >> v2:172.21.15.142:3300/0 conn(0x7fcf080ca3e0 msgr2=0x7fcf080ca7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:39.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.854+0000 7fcefe7fc700 1 --2- 172.21.15.112:0/3786271800 >> v2:172.21.15.142:3300/0 conn(0x7fcf080ca3e0 0x7fcf080ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:39.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.854+0000 7fcefe7fc700 1 -- 172.21.15.112:0/3786271800 >> v1:172.21.15.112:6789/0 conn(0x7fcf080c7e40 legacy=0x7fcf080ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:39.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.854+0000 7fcefe7fc700 1 -- 172.21.15.112:0/3786271800 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf080b1730 con 0x7fcf080b0aa0 2023-01-24T03:30:39.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.854+0000 7fcf067fc700 1 -- 172.21.15.112:0/3786271800 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1492690104 0 0) 0x7fcf080b0f40 con 0x7fcf080c7e40 2023-01-24T03:30:39.862 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.854+0000 7fcefe7fc700 1 --2- 172.21.15.112:0/3786271800 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf080b0aa0 0x7fcf08155da0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fcf0805e810 tx=0x7fcef0009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1c1bd3405d5108e server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:39.862 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.854+0000 7fcf067fc700 1 -- 172.21.15.112:0/3786271800 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef0011630 con 0x7fcf080b0aa0 2023-01-24T03:30:39.862 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.854+0000 7fcf067fc700 1 -- 172.21.15.112:0/3786271800 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcef0011790 con 0x7fcf080b0aa0 2023-01-24T03:30:39.862 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.854+0000 7fcf067fc700 1 -- 172.21.15.112:0/3786271800 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef0011b50 con 0x7fcf080b0aa0 2023-01-24T03:30:39.862 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.855+0000 7fcf0d4f4700 1 -- 172.21.15.112:0/3786271800 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf080b0aa0 msgr2=0x7fcf08155da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:39.863 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.855+0000 7fcf0d4f4700 1 --2- 172.21.15.112:0/3786271800 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf080b0aa0 0x7fcf08155da0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fcf0805e810 tx=0x7fcef0009060 comp rx=0 tx=0).stop 2023-01-24T03:30:39.863 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.855+0000 7fcf0d4f4700 1 -- 172.21.15.112:0/3786271800 shutdown_connections 2023-01-24T03:30:39.863 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.855+0000 7fcf0d4f4700 1 --2- 172.21.15.112:0/3786271800 >> v2:172.21.15.142:3300/0 conn(0x7fcf080ca3e0 0x7fcf080ca7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:39.863 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.855+0000 7fcf0d4f4700 1 --2- 172.21.15.112:0/3786271800 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf080b0aa0 0x7fcf08155da0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:39.863 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.855+0000 7fcf0d4f4700 1 -- 172.21.15.112:0/3786271800 >> 172.21.15.112:0/3786271800 conn(0x7fcf0805e2c0 msgr2=0x7fcf080cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:39.864 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.855+0000 7fcf0d4f4700 1 -- 172.21.15.112:0/3786271800 shutdown_connections 2023-01-24T03:30:39.864 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.855+0000 7fcf0d4f4700 1 -- 172.21.15.112:0/3786271800 wait complete. 2023-01-24T03:30:39.864 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.856+0000 7fcf0d4f4700 1 Processor -- start 2023-01-24T03:30:39.864 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.856+0000 7fcf0d4f4700 1 -- start start 2023-01-24T03:30:39.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.856+0000 7fcf0d4f4700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf080b0aa0 0x7fcf080cb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:39.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.856+0000 7fcf0d4f4700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf080c7e40 0x7fcf08149590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:39.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.856+0000 7fcf0d4f4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf080ca3e0 0x7fcf08149ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:39.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.856+0000 7fcf0d4f4700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0814f120 con 0x7fcf080c7e40 2023-01-24T03:30:39.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.856+0000 7fcf0d4f4700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf08165cf0 con 0x7fcf080b0aa0 2023-01-24T03:30:39.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.856+0000 7fcf0d4f4700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf080c1f20 con 0x7fcf080ca3e0 2023-01-24T03:30:39.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.857+0000 7fcefe7fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf080c7e40 0x7fcf08149590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:39.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.857+0000 7fcefe7fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf080c7e40 0x7fcf08149590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50448/0 (socket says 172.21.15.112:50448) 2023-01-24T03:30:39.869 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.857+0000 7fcefe7fc700 1 -- 172.21.15.112:0/97302206 learned_addr learned my addr 172.21.15.112:0/97302206 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:39.869 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.857+0000 7fcf06ffd700 1 --2- 172.21.15.112:0/97302206 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf080b0aa0 0x7fcf080cb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:39.869 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.857+0000 7fcf077fe700 1 --2- 172.21.15.112:0/97302206 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf080ca3e0 0x7fcf08149ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:39.869 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.857+0000 7fcefe7fc700 1 -- 172.21.15.112:0/97302206 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf080ca3e0 msgr2=0x7fcf08149ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:39.869 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.857+0000 7fcefe7fc700 1 --2- 172.21.15.112:0/97302206 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf080ca3e0 0x7fcf08149ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:39.870 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.857+0000 7fcefe7fc700 1 -- 172.21.15.112:0/97302206 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf080b0aa0 msgr2=0x7fcf080cb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:39.870 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.857+0000 7fcefe7fc700 1 --2- 172.21.15.112:0/97302206 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf080b0aa0 0x7fcf080cb9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:39.870 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.857+0000 7fcefe7fc700 1 -- 172.21.15.112:0/97302206 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcef00092a0 con 0x7fcf080c7e40 2023-01-24T03:30:39.870 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.857+0000 7fcf06ffd700 1 --2- 172.21.15.112:0/97302206 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf080b0aa0 0x7fcf080cb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-01-24T03:30:39.871 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.857+0000 7fcf077fe700 1 --2- 172.21.15.112:0/97302206 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf080ca3e0 0x7fcf08149ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-01-24T03:30:39.871 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.857+0000 7fcefe7fc700 1 --2- 172.21.15.112:0/97302206 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf080c7e40 0x7fcf08149590 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcef0009e80 tx=0x7fcef0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:39.871 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.857+0000 7fceff7fe700 1 -- 172.21.15.112:0/97302206 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef0000bc0 con 0x7fcf080c7e40 2023-01-24T03:30:39.871 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.857+0000 7fceff7fe700 1 -- 172.21.15.112:0/97302206 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcef0011900 con 0x7fcf080c7e40 2023-01-24T03:30:39.871 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.857+0000 7fcf0d4f4700 1 -- 172.21.15.112:0/97302206 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf08165f70 con 0x7fcf080c7e40 2023-01-24T03:30:39.872 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.857+0000 7fceff7fe700 1 -- 172.21.15.112:0/97302206 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef0000d20 con 0x7fcf080c7e40 2023-01-24T03:30:39.872 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.857+0000 7fcf0d4f4700 1 -- 172.21.15.112:0/97302206 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf08166530 con 0x7fcf080c7e40 2023-01-24T03:30:39.872 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.858+0000 7fcf0d4f4700 1 -- 172.21.15.112:0/97302206 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf080c1c80 con 0x7fcf080c7e40 2023-01-24T03:30:39.872 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.858+0000 7fceff7fe700 1 -- 172.21.15.112:0/97302206 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fcef002c030 con 0x7fcf080c7e40 2023-01-24T03:30:39.872 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.859+0000 7fceff7fe700 1 --2- 172.21.15.112:0/97302206 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fceec0611c0 0x7fceec063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:39.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.859+0000 7fceff7fe700 1 -- 172.21.15.112:0/97302206 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fcef007edc0 con 0x7fcf080c7e40 2023-01-24T03:30:39.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.859+0000 7fcf06ffd700 1 --2- 172.21.15.112:0/97302206 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fceec0611c0 0x7fceec063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:39.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.859+0000 7fcf06ffd700 1 --2- 172.21.15.112:0/97302206 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fceec0611c0 0x7fceec063670 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcef80016f0 tx=0x7fcef8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:39.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:39.863+0000 7fceff7fe700 1 -- 172.21.15.112:0/97302206 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fcef004ec00 con 0x7fcf080c7e40 2023-01-24T03:30:40.044 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.041+0000 7fcf0d4f4700 1 -- 172.21.15.112:0/97302206 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf080c5df0 con 0x7fcf080c7e40 2023-01-24T03:30:40.045 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.042+0000 7fceff7fe700 1 -- 172.21.15.112:0/97302206 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18755 (secure 0 0 0) 0x7fcef002a090 con 0x7fcf080c7e40 2023-01-24T03:30:40.045 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:30:40.046 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":24,"fsid":"054b9ac9-b7b0-40e5-b932-6c8a955c5a72","created":"2023-01-24T03:30:03.798533+0000","modified":"2023-01-24T03:30:25.373340+0000","last_up_change":"2023-01-24T03:30:13.945316+0000","last_in_change":"2023-01-24T03:30:06.198433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-01-24T03:30:14.931166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-01-24T03:30:15.023730+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-01-24T03:30:19.060597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-01-24T03:30:19.943686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-01-24T03:30:22.999036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"fe6ad231-a39a-42eb-9a3a-49d008dd35e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6809","nonce":84512},{"type":"v1","addr":"172.21.15.112:6811","nonce":84512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":84512},{"type":"v1","addr":"172.21.15.112:6817","nonce":84512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6825","nonce":84512},{"type":"v1","addr":"172.21.15.112:6828","nonce":84512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6819","nonce":84512},{"type":"v1","addr":"172.21.15.112:6822","nonce":84512}]},"public_addr":"172.21.15.112:6811/84512","cluster_addr":"172.21.15.112:6817/84512","heartbeat_back_addr":"172.21.15.112:6828/84512","heartbeat_front_addr":"172.21.15.112:6822/84512","state":["exists","up"]},{"osd":1,"uuid":"8526a1d2-d540-4021-9766-c4bbb30317ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":84513},{"type":"v1","addr":"172.21.15.112:6812","nonce":84513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6815","nonce":84513},{"type":"v1","addr":"172.21.15.112:6818","nonce":84513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6827","nonce":84513},{"type":"v1","addr":"172.21.15.112:6830","nonce":84513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6821","nonce":84513},{"type":"v1","addr":"172.21.15.112:6824","nonce":84513}]},"public_addr":"172.21.15.112:6812/84513","cluster_addr":"172.21.15.112:6818/84513","heartbeat_back_addr":"172.21.15.112:6830/84513","heartbeat_front_addr":"172.21.15.112:6824/84513","state":["exists","up"]},{"osd":2,"uuid":"041ef8e4-4858-432d-9476-cbdb3a0dc47c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":84511},{"type":"v1","addr":"172.21.15.112:6813","nonce":84511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":84511},{"type":"v1","addr":"172.21.15.112:6820","nonce":84511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6829","nonce":84511},{"type":"v1","addr":"172.21.15.112:6831","nonce":84511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6823","nonce":84511},{"type":"v1","addr":"172.21.15.112:6826","nonce":84511}]},"public_addr":"172.21.15.112:6813/84511","cluster_addr":"172.21.15.112:6820/84511","heartbeat_back_addr":"172.21.15.112:6831/84511","heartbeat_front_addr":"172.21.15.112:6826/84511","state":["exists","up"]},{"osd":3,"uuid":"987587cf-1357-4a70-ad85-67b189c67729","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":84510},{"type":"v1","addr":"172.21.15.112:6801","nonce":84510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":84510},{"type":"v1","addr":"172.21.15.112:6803","nonce":84510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":84510},{"type":"v1","addr":"172.21.15.112:6807","nonce":84510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":84510},{"type":"v1","addr":"172.21.15.112:6805","nonce":84510}]},"public_addr":"172.21.15.112:6801/84510","cluster_addr":"172.21.15.112:6803/84510","heartbeat_back_addr":"172.21.15.112:6807/84510","heartbeat_front_addr":"172.21.15.112:6805/84510","state":["exists","up"]},{"osd":4,"uuid":"8503512b-43db-40b8-8b2b-c86cf2e54e17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":84614},{"type":"v1","addr":"172.21.15.142:6801","nonce":84614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":84614},{"type":"v1","addr":"172.21.15.142:6803","nonce":84614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":84614},{"type":"v1","addr":"172.21.15.142:6807","nonce":84614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":84614},{"type":"v1","addr":"172.21.15.142:6805","nonce":84614}]},"public_addr":"172.21.15.142:6801/84614","cluster_addr":"172.21.15.142:6803/84614","heartbeat_back_addr":"172.21.15.142:6807/84614","heartbeat_front_addr":"172.21.15.142:6805/84614","state":["exists","up"]},{"osd":5,"uuid":"d3bc0030-8b08-4645-9ec2-1f8a4e29ef36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":84638},{"type":"v1","addr":"172.21.15.142:6817","nonce":84638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":84638},{"type":"v1","addr":"172.21.15.142:6819","nonce":84638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":84638},{"type":"v1","addr":"172.21.15.142:6823","nonce":84638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":84638},{"type":"v1","addr":"172.21.15.142:6821","nonce":84638}]},"public_addr":"172.21.15.142:6817/84638","cluster_addr":"172.21.15.142:6819/84638","heartbeat_back_addr":"172.21.15.142:6823/84638","heartbeat_front_addr":"172.21.15.142:6821/84638","state":["exists","up"]},{"osd":6,"uuid":"c7649e3b-832d-4e6f-8866-b15e5d814476","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":84645},{"type":"v1","addr":"172.21.15.142:6825","nonce":84645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":84645},{"type":"v1","addr":"172.21.15.142:6827","nonce":84645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":84645},{"type":"v1","addr":"172.21.15.142:6831","nonce":84645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":84645},{"type":"v1","addr":"172.21.15.142:6829","nonce":84645}]},"public_addr":"172.21.15.142:6825/84645","cluster_addr":"172.21.15.142:6827/84645","heartbeat_back_addr":"172.21.15.142:6831/84645","heartbeat_front_addr":"172.21.15.142:6829/84645","state":["exists","up"]},{"osd":7,"uuid":"2ea5b3bf-884e-4b20-b572-1443efdc7477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":84615},{"type":"v1","addr":"172.21.15.142:6809","nonce":84615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":84615},{"type":"v1","addr":"172.21.15.142:6811","nonce":84615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":84615},{"type":"v1","addr":"172.21.15.142:6815","nonce":84615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":84615},{"type":"v1","addr":"172.21.15.142:6813","nonce":84615}]},"public_addr":"172.21.15.142:6809/84615","cluster_addr":"172.21.15.142:6811/84615","heartbeat_back_addr":"172.21.15.142:6815/84615","heartbeat_front_addr":"172.21.15.142:6813/84615","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.370027+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.335683+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.350667+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.350794+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.391119+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.422933+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.400439+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.354018+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T03:30:40.048 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.044+0000 7fcf0d4f4700 1 -- 172.21.15.112:0/97302206 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fceec0611c0 msgr2=0x7fceec063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.048 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.044+0000 7fcf0d4f4700 1 --2- 172.21.15.112:0/97302206 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fceec0611c0 0x7fceec063670 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcef80016f0 tx=0x7fcef8009040 comp rx=0 tx=0).stop 2023-01-24T03:30:40.049 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.044+0000 7fcf0d4f4700 1 -- 172.21.15.112:0/97302206 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf080c7e40 msgr2=0x7fcf08149590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.049 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.044+0000 7fcf0d4f4700 1 --2- 172.21.15.112:0/97302206 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf080c7e40 0x7fcf08149590 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcef0009e80 tx=0x7fcef0017000 comp rx=0 tx=0).stop 2023-01-24T03:30:40.049 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.044+0000 7fcf0d4f4700 1 -- 172.21.15.112:0/97302206 shutdown_connections 2023-01-24T03:30:40.049 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.044+0000 7fcf0d4f4700 1 --2- 172.21.15.112:0/97302206 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fceec0611c0 0x7fceec063670 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.049 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.044+0000 7fcf0d4f4700 1 --2- 172.21.15.112:0/97302206 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcf080ca3e0 0x7fcf08149ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.044+0000 7fcf0d4f4700 1 --2- 172.21.15.112:0/97302206 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcf080b0aa0 0x7fcf080cb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.044+0000 7fcf0d4f4700 1 --2- 172.21.15.112:0/97302206 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcf080c7e40 0x7fcf08149590 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.044+0000 7fcf0d4f4700 1 -- 172.21.15.112:0/97302206 >> 172.21.15.112:0/97302206 conn(0x7fcf0805e2c0 msgr2=0x7fcf080c60d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:40.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.044+0000 7fcf0d4f4700 1 -- 172.21.15.112:0/97302206 shutdown_connections 2023-01-24T03:30:40.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.044+0000 7fcf0d4f4700 1 -- 172.21.15.112:0/97302206 wait complete. 2023-01-24T03:30:40.060 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-01-24T03:30:40.061 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-01-24T03:30:40.061 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-01-24T03:30:40.062 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-01-24T03:30:40.062 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-01-24T03:30:40.063 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-01-24T03:30:40.063 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-01-24T03:30:40.064 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-01-24T03:30:40.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.249+0000 7f682f945700 1 Processor -- start 2023-01-24T03:30:40.255 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.251+0000 7f682f945700 1 -- start start 2023-01-24T03:30:40.255 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.252+0000 7f682f945700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f682806b7a0 0x7f68280c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.255 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.252+0000 7f682f945700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68280bf190 0x7f68280c2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.256 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.252+0000 7f682f945700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68280bf610 con 0x7f68280c40e0 2023-01-24T03:30:40.256 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.252+0000 7f682f945700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f682806efc0 con 0x7f682806b7a0 2023-01-24T03:30:40.256 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.252+0000 7f682f945700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68280c2960 con 0x7f68280bf190 2023-01-24T03:30:40.256 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.252+0000 7f1f0b4e0700 1 Processor -- start 2023-01-24T03:30:40.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.252+0000 7f682cee0700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f682806b7a0 0x7f68280c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.252+0000 7f682d6e1700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f68280c40e0 0x7f682815d140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.112:45232/0 (socket says 172.21.15.112:45232) 2023-01-24T03:30:40.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.252+0000 7f682d6e1700 1 -- 172.21.15.112:0/59456768 learned_addr learned my addr 172.21.15.112:0/59456768 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:40.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.252+0000 7f682dee2700 1 --2- 172.21.15.112:0/59456768 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68280bf190 0x7f68280c2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.253+0000 7f1f0b4e0700 1 -- start start 2023-01-24T03:30:40.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.254+0000 7f1f0b4e0700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f1f040683b0 0x7f1f040c3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.254+0000 7f1f0b4e0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1f040c1960 0x7f1f040c1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.254+0000 7f1f0b4e0700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f040c2300 con 0x7f1f04067fe0 2023-01-24T03:30:40.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.254+0000 7f1f0b4e0700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f04153f00 con 0x7f1f040683b0 2023-01-24T03:30:40.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.254+0000 7f1f0b4e0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f04159700 con 0x7f1f040c1960 2023-01-24T03:30:40.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.254+0000 7f1f09a7d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1f040c1960 0x7f1f040c1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.254+0000 7f1f09a7d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1f040c1960 0x7f1f040c1dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.112:37764/0 (socket says 172.21.15.112:37764) 2023-01-24T03:30:40.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.254+0000 7f1f09a7d700 1 -- 172.21.15.112:0/3968206292 learned_addr learned my addr 172.21.15.112:0/3968206292 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:40.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.255+0000 7f1f09a7d700 1 -- 172.21.15.112:0/3968206292 >> v2:172.21.15.112:3300/0 conn(0x7f1f040683b0 msgr2=0x7f1f040c3150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-01-24T03:30:40.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.255+0000 7f1f09a7d700 1 --2- 172.21.15.112:0/3968206292 >> v2:172.21.15.112:3300/0 conn(0x7f1f040683b0 0x7f1f040c3150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.263 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.255+0000 7f1f09a7d700 1 -- 172.21.15.112:0/3968206292 >> v1:172.21.15.112:6789/0 conn(0x7f1f04067fe0 legacy=0x7f1f040ad730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:40.264 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.255+0000 7f1f09a7d700 1 -- 172.21.15.112:0/3968206292 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f04159910 con 0x7f1f040c1960 2023-01-24T03:30:40.264 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.255+0000 7f1f09a7d700 1 --2- 172.21.15.112:0/3968206292 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1f040c1960 0x7f1f040c1dc0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1f00006fd0 tx=0x7f1f000090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4250e1e80efc4faf server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.264 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.255+0000 7f1ef3fff700 1 -- 172.21.15.112:0/3968206292 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f00011630 con 0x7f1f040c1960 2023-01-24T03:30:40.264 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.255+0000 7f1ef3fff700 1 -- 172.21.15.112:0/3968206292 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f00011790 con 0x7f1f040c1960 2023-01-24T03:30:40.265 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.255+0000 7f1ef3fff700 1 -- 172.21.15.112:0/3968206292 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f00011b50 con 0x7f1f040c1960 2023-01-24T03:30:40.265 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.256+0000 7f1f0b4e0700 1 -- 172.21.15.112:0/3968206292 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1f040c1960 msgr2=0x7f1f040c1dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.265 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.256+0000 7f1f0b4e0700 1 --2- 172.21.15.112:0/3968206292 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1f040c1960 0x7f1f040c1dc0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1f00006fd0 tx=0x7f1f000090f0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.265 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.256+0000 7f1f0b4e0700 1 -- 172.21.15.112:0/3968206292 shutdown_connections 2023-01-24T03:30:40.265 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.256+0000 7f1f0b4e0700 1 --2- 172.21.15.112:0/3968206292 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1f040c1960 0x7f1f040c1dc0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.266 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.256+0000 7f1f0b4e0700 1 --2- 172.21.15.112:0/3968206292 >> v2:172.21.15.112:3300/0 conn(0x7f1f040683b0 0x7f1f040c3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.266 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.256+0000 7f1f0b4e0700 1 -- 172.21.15.112:0/3968206292 >> 172.21.15.112:0/3968206292 conn(0x7f1f0406b550 msgr2=0x7f1f0406b950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:40.267 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.254+0000 7f682cee0700 1 -- 172.21.15.112:0/59456768 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68280bf190 msgr2=0x7f68280c2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:40.267 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.254+0000 7f682cee0700 1 --2- 172.21.15.112:0/59456768 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68280bf190 0x7f68280c2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.267 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.254+0000 7f682cee0700 1 -- 172.21.15.112:0/59456768 >> v1:172.21.15.142:6789/0 conn(0x7f68280c40e0 legacy=0x7f682815d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.267 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.254+0000 7f682cee0700 1 -- 172.21.15.112:0/59456768 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68280bea90 con 0x7f682806b7a0 2023-01-24T03:30:40.267 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.254+0000 7f682cee0700 1 --2- 172.21.15.112:0/59456768 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f682806b7a0 0x7f68280c44b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6824009760 tx=0x7f6824009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=742f112d2b66bd6d server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.268 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.256+0000 7f6817fff700 1 -- 172.21.15.112:0/59456768 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6824011630 con 0x7f682806b7a0 2023-01-24T03:30:40.268 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.256+0000 7f6817fff700 1 -- 172.21.15.112:0/59456768 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6824011790 con 0x7f682806b7a0 2023-01-24T03:30:40.268 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.256+0000 7f6817fff700 1 -- 172.21.15.112:0/59456768 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6824011b50 con 0x7f682806b7a0 2023-01-24T03:30:40.269 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.256+0000 7f1f0b4e0700 1 -- 172.21.15.112:0/3968206292 shutdown_connections 2023-01-24T03:30:40.269 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.256+0000 7f1f0b4e0700 1 -- 172.21.15.112:0/3968206292 wait complete. 2023-01-24T03:30:40.269 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.256+0000 7f2ede439700 1 Processor -- start 2023-01-24T03:30:40.270 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.257+0000 7f682f945700 1 -- 172.21.15.112:0/59456768 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f682806b7a0 msgr2=0x7f68280c44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.270 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.257+0000 7f682f945700 1 --2- 172.21.15.112:0/59456768 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f682806b7a0 0x7f68280c44b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6824009760 tx=0x7f6824009060 comp rx=0 tx=0).stop 2023-01-24T03:30:40.271 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.257+0000 7f682f945700 1 -- 172.21.15.112:0/59456768 shutdown_connections 2023-01-24T03:30:40.271 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.257+0000 7f682f945700 1 --2- 172.21.15.112:0/59456768 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68280bf190 0x7f68280c2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.271 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.257+0000 7f682f945700 1 --2- 172.21.15.112:0/59456768 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f682806b7a0 0x7f68280c44b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.271 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.257+0000 7f682f945700 1 -- 172.21.15.112:0/59456768 >> 172.21.15.112:0/59456768 conn(0x7f6828066e10 msgr2=0x7f6828067210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:40.271 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.257+0000 7f682f945700 1 -- 172.21.15.112:0/59456768 shutdown_connections 2023-01-24T03:30:40.272 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.257+0000 7f682f945700 1 -- 172.21.15.112:0/59456768 wait complete. 2023-01-24T03:30:40.273 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.257+0000 7f2ede439700 1 -- start start 2023-01-24T03:30:40.273 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.258+0000 7f2ede439700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f2ed8067fe0 0x7f2ed81559a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.273 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.258+0000 7f2ede439700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2ed8069360 0x7f2ed8069b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.273 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.258+0000 7f2ede439700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed80c3c80 con 0x7f2ed806f050 2023-01-24T03:30:40.273 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.258+0000 7f2ede439700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ed80bf6e0 con 0x7f2ed8067fe0 2023-01-24T03:30:40.274 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.258+0000 7f2ede439700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed80c2d90 con 0x7f2ed8069360 2023-01-24T03:30:40.274 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.258+0000 7f1f0b4e0700 1 Processor -- start 2023-01-24T03:30:40.274 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.258+0000 7f1f0b4e0700 1 -- start start 2023-01-24T03:30:40.275 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.258+0000 7f2ed7fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f2ed8067fe0 0x7f2ed81559a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.275 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.258+0000 7f2ed7fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f2ed8067fe0 0x7f2ed81559a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.112:42958/0 (socket says 172.21.15.112:42958) 2023-01-24T03:30:40.275 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.258+0000 7f2ed7fff700 1 -- 172.21.15.112:0/2038408404 learned_addr learned my addr 172.21.15.112:0/2038408404 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:40.276 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.258+0000 7f9d4a939700 1 Processor -- start 2023-01-24T03:30:40.276 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.259+0000 7f15f457d700 1 Processor -- start 2023-01-24T03:30:40.277 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.259+0000 7f682f945700 1 Processor -- start 2023-01-24T03:30:40.277 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.259+0000 7f1f0b4e0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1f04067fe0 0x7f1f040bf7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.277 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.259+0000 7f1f0b4e0700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1f040c1960 0x7f1f040bfd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.278 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.259+0000 7f1f0b4e0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1f04158a50 0x7f1f04140200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.278 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.259+0000 7f1f0b4e0700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0413f8e0 con 0x7f1f04158a50 2023-01-24T03:30:40.278 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.259+0000 7f1f0b4e0700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f04141f60 con 0x7f1f040c1960 2023-01-24T03:30:40.278 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.259+0000 7f1f0b4e0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0413fa90 con 0x7f1f04067fe0 2023-01-24T03:30:40.278 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.259+0000 7f1f08a7b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1f040c1960 0x7f1f040bfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.279 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.259+0000 7f1f08a7b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1f040c1960 0x7f1f040bfd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.112:42966/0 (socket says 172.21.15.112:42966) 2023-01-24T03:30:40.279 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.259+0000 7f1f08a7b700 1 -- 172.21.15.112:0/1531870843 learned_addr learned my addr 172.21.15.112:0/1531870843 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:40.279 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.259+0000 7f1f09a7d700 1 --2- 172.21.15.112:0/1531870843 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1f04158a50 0x7f1f04140200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.279 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.259+0000 7f9d4a939700 1 -- start start 2023-01-24T03:30:40.280 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.259+0000 7f2ed7fff700 1 -- 172.21.15.112:0/2038408404 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2ed8069360 msgr2=0x7f2ed8069b00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-01-24T03:30:40.281 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.259+0000 7f2ed7fff700 1 --2- 172.21.15.112:0/2038408404 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2ed8069360 0x7f2ed8069b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.281 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.259+0000 7f2ed7fff700 1 -- 172.21.15.112:0/2038408404 >> v1:172.21.15.142:6789/0 conn(0x7f2ed806f050 legacy=0x7f2ed806dab0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-01-24T03:30:40.281 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.259+0000 7f2ed7fff700 1 -- 172.21.15.112:0/2038408404 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed80c3070 con 0x7f2ed8067fe0 2023-01-24T03:30:40.281 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.260+0000 7f2ed7fff700 1 --2- 172.21.15.112:0/2038408404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2ed8067fe0 0x7f2ed81559a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2ec800bac0 tx=0x7f2ec800b370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=569792d897f96db server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.282 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.259+0000 7f1f0927c700 1 --2- 172.21.15.112:0/1531870843 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1f04067fe0 0x7f1f040bf7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.282 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.260+0000 7f1f09a7d700 1 -- 172.21.15.112:0/1531870843 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1f04067fe0 msgr2=0x7f1f040bf7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.282 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.260+0000 7f1f09a7d700 1 --2- 172.21.15.112:0/1531870843 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1f04067fe0 0x7f1f040bf7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.282 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.260+0000 7f1f09a7d700 1 -- 172.21.15.112:0/1531870843 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1f040c1960 msgr2=0x7f1f040bfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.283 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.260+0000 7f1f09a7d700 1 --2- 172.21.15.112:0/1531870843 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1f040c1960 0x7f1f040bfd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.283 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.260+0000 7f1f09a7d700 1 -- 172.21.15.112:0/1531870843 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f00009370 con 0x7f1f04158a50 2023-01-24T03:30:40.283 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.260+0000 7f2ed6ffd700 1 -- 172.21.15.112:0/2038408404 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec800da80 con 0x7f2ed8067fe0 2023-01-24T03:30:40.284 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.260+0000 7f9d4a939700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d440c4090 0x7f9d440c2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.284 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.260+0000 7f9d4a939700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f9d4406b7a0 0x7f9d440bf190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.284 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.260+0000 7f9d4a939700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d440c2930 con 0x7f9d440c4460 2023-01-24T03:30:40.285 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.260+0000 7f9d4a939700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d4406ec90 con 0x7f9d4406b7a0 2023-01-24T03:30:40.285 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.260+0000 7f9d4a939700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4406ef30 con 0x7f9d440c4090 2023-01-24T03:30:40.286 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.259+0000 7f682f945700 1 -- start start 2023-01-24T03:30:40.286 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.260+0000 7f682f945700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f682806b7a0 0x7f682814d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.286 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.260+0000 7f682f945700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f68280bf190 0x7f68280b7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.286 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.260+0000 7f682f945700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68280c40e0 0x7f682815c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.286 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.260+0000 7f682f945700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68280c7ee0 con 0x7f682806b7a0 2023-01-24T03:30:40.287 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.260+0000 7f682f945700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68281571f0 con 0x7f68280bf190 2023-01-24T03:30:40.287 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.260+0000 7f682f945700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68280ba4c0 con 0x7f68280c40e0 2023-01-24T03:30:40.287 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.260+0000 7f9d437fe700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f9d440c4460 0x7f9d4415d1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:59696/0 (socket says 172.21.15.112:59696) 2023-01-24T03:30:40.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.260+0000 7f9d437fe700 1 -- 172.21.15.112:0/3425360857 learned_addr learned my addr 172.21.15.112:0/3425360857 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:40.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.260+0000 7f9d48ed6700 1 --2- 172.21.15.112:0/3425360857 >> v2:172.21.15.142:3300/0 conn(0x7f9d4406b7a0 0x7f9d440bf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.260+0000 7f2ed6ffd700 1 -- 172.21.15.112:0/2038408404 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ec800dbe0 con 0x7f2ed8067fe0 2023-01-24T03:30:40.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.260+0000 7f2ed6ffd700 1 -- 172.21.15.112:0/2038408404 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec8017410 con 0x7f2ed8067fe0 2023-01-24T03:30:40.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.259+0000 7f15f457d700 1 -- start start 2023-01-24T03:30:40.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.260+0000 7f15f457d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15ec1523b0 0x7f15ec06e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.260+0000 7f15f457d700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15ec06ebd0 con 0x7f15ec06b210 2023-01-24T03:30:40.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.260+0000 7f15f457d700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15ec154da0 con 0x7f15ec06d210 2023-01-24T03:30:40.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.260+0000 7f15f457d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15ec155040 con 0x7f15ec1523b0 2023-01-24T03:30:40.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.260+0000 7f682cee0700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f68280bf190 0x7f68280b7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.260+0000 7f682cee0700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f68280bf190 0x7f68280b7cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.112:42974/0 (socket says 172.21.15.112:42974) 2023-01-24T03:30:40.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.260+0000 7f682cee0700 1 -- 172.21.15.112:0/1161503996 learned_addr learned my addr 172.21.15.112:0/1161503996 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:40.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.260+0000 7f15f1b18700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f15ec06d210 0x7f15ec06ab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.112:45262/0 (socket says 172.21.15.112:45262) 2023-01-24T03:30:40.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.260+0000 7f15f1b18700 1 -- 172.21.15.112:0/2980596863 learned_addr learned my addr 172.21.15.112:0/2980596863 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:40.292 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.260+0000 7f682dee2700 1 --2- 172.21.15.112:0/1161503996 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68280c40e0 0x7f682815c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.292 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.261+0000 7f1f09a7d700 1 --2- 172.21.15.112:0/1531870843 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1f04158a50 0x7f1f04140200 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1f00009340 tx=0x7f1f00017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.261+0000 7f9d48ed6700 1 -- 172.21.15.112:0/3425360857 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d440c4090 msgr2=0x7f9d440c2dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-01-24T03:30:40.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.261+0000 7f9d48ed6700 1 --2- 172.21.15.112:0/3425360857 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d440c4090 0x7f9d440c2dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.261+0000 7f9d48ed6700 1 -- 172.21.15.112:0/3425360857 >> v1:172.21.15.112:6789/0 conn(0x7f9d440c4460 legacy=0x7f9d4415d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.261+0000 7f9d48ed6700 1 -- 172.21.15.112:0/3425360857 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d440c4850 con 0x7f9d4406b7a0 2023-01-24T03:30:40.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.261+0000 7f682d6e1700 1 --2- 172.21.15.112:0/1161503996 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f682806b7a0 0x7f682814d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.261+0000 7f15f2319700 1 --2- 172.21.15.112:0/2980596863 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15ec1523b0 0x7f15ec06e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.295 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.261+0000 7f9d48ed6700 1 --2- 172.21.15.112:0/3425360857 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d4406b7a0 0x7f9d440bf190 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c009e60 tx=0x7f9d3c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f444ff71eac3cc6 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.295 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.261+0000 7f15f1317700 1 -- 172.21.15.112:0/2980596863 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2174724672 0 0) 0x7f15ec154da0 con 0x7f15ec06d210 2023-01-24T03:30:40.295 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.261+0000 7f15f1317700 1 -- 172.21.15.112:0/2980596863 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15d4005600 con 0x7f15ec06d210 2023-01-24T03:30:40.296 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.261+0000 7f1ef17fa700 1 -- 172.21.15.112:0/1531870843 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f00027450 con 0x7f1f04158a50 2023-01-24T03:30:40.296 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.261+0000 7f1ef17fa700 1 -- 172.21.15.112:0/1531870843 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f000275b0 con 0x7f1f04158a50 2023-01-24T03:30:40.296 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.261+0000 7f1ef17fa700 1 -- 172.21.15.112:0/1531870843 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f00027970 con 0x7f1f04158a50 2023-01-24T03:30:40.296 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.261+0000 7f682dee2700 1 -- 172.21.15.112:0/1161503996 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f68280bf190 msgr2=0x7f68280b7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.297 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.261+0000 7f682dee2700 1 --2- 172.21.15.112:0/1161503996 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f68280bf190 0x7f68280b7cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.297 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.261+0000 7f682dee2700 1 -- 172.21.15.112:0/1161503996 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f682806b7a0 msgr2=0x7f682814d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.297 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.261+0000 7f682dee2700 1 --2- 172.21.15.112:0/1161503996 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f682806b7a0 0x7f682814d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.297 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.261+0000 7f682dee2700 1 -- 172.21.15.112:0/1161503996 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68240092a0 con 0x7f68280c40e0 2023-01-24T03:30:40.298 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.261+0000 7f682d6e1700 1 --2- 172.21.15.112:0/1161503996 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f682806b7a0 0x7f682814d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:30:40.298 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.261+0000 7f9d42ffd700 1 -- 172.21.15.112:0/3425360857 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c00e070 con 0x7f9d4406b7a0 2023-01-24T03:30:40.298 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.261+0000 7f9d42ffd700 1 -- 172.21.15.112:0/3425360857 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d3c00bc30 con 0x7f9d4406b7a0 2023-01-24T03:30:40.298 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.261+0000 7f9d42ffd700 1 -- 172.21.15.112:0/3425360857 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c012430 con 0x7f9d4406b7a0 2023-01-24T03:30:40.299 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.261+0000 7f15f2319700 1 -- 172.21.15.112:0/2980596863 >> v1:172.21.15.142:6789/0 conn(0x7f15ec06d210 legacy=0x7f15ec06ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.299 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.261+0000 7f15f2319700 1 -- 172.21.15.112:0/2980596863 >> v1:172.21.15.112:6789/0 conn(0x7f15ec06b210 legacy=0x7f15ec0c3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.299 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.261+0000 7f15f2319700 1 -- 172.21.15.112:0/2980596863 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15ec155320 con 0x7f15ec1523b0 2023-01-24T03:30:40.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.261+0000 7f1f0b4e0700 1 -- 172.21.15.112:0/1531870843 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f041422f0 con 0x7f1f04158a50 2023-01-24T03:30:40.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.261+0000 7f1f0b4e0700 1 -- 172.21.15.112:0/1531870843 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f041428b0 con 0x7f1f04158a50 2023-01-24T03:30:40.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.261+0000 7f682dee2700 1 --2- 172.21.15.112:0/1161503996 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68280c40e0 0x7f682815c910 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6820007f00 tx=0x7f6820009850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.301 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.261+0000 7f15f2319700 1 --2- 172.21.15.112:0/2980596863 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15ec1523b0 0x7f15ec06e690 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f15dc00a830 tx=0x7f15dc0080e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bed60b90b1f0bf6 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.301 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.262+0000 7f9d4a939700 1 -- 172.21.15.112:0/3425360857 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d4406b7a0 msgr2=0x7f9d440bf190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.302 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.262+0000 7f9d4a939700 1 --2- 172.21.15.112:0/3425360857 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d4406b7a0 0x7f9d440bf190 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c009e60 tx=0x7f9d3c009780 comp rx=0 tx=0).stop 2023-01-24T03:30:40.302 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.262+0000 7f15f1317700 1 -- 172.21.15.112:0/2980596863 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15dc00f400 con 0x7f15ec1523b0 2023-01-24T03:30:40.302 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.262+0000 7f1f0b4e0700 1 -- 172.21.15.112:0/1531870843 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1f0405e610 con 0x7f1f04158a50 2023-01-24T03:30:40.303 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.262+0000 7f2ede439700 1 -- 172.21.15.112:0/2038408404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2ed8067fe0 msgr2=0x7f2ed81559a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.303 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.262+0000 7f2ede439700 1 --2- 172.21.15.112:0/2038408404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2ed8067fe0 0x7f2ed81559a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2ec800bac0 tx=0x7f2ec800b370 comp rx=0 tx=0).stop 2023-01-24T03:30:40.303 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.262+0000 7f15f1317700 1 -- 172.21.15.112:0/2980596863 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15dc00f560 con 0x7f15ec1523b0 2023-01-24T03:30:40.304 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.262+0000 7f15f1317700 1 -- 172.21.15.112:0/2980596863 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15dc00f8e0 con 0x7f15ec1523b0 2023-01-24T03:30:40.304 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.262+0000 7f2ede439700 1 -- 172.21.15.112:0/2038408404 shutdown_connections 2023-01-24T03:30:40.305 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.262+0000 7f2ede439700 1 --2- 172.21.15.112:0/2038408404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2ed8067fe0 0x7f2ed81559a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.305 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.262+0000 7f2ede439700 1 --2- 172.21.15.112:0/2038408404 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2ed8069360 0x7f2ed8069b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.305 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.262+0000 7f2ede439700 1 -- 172.21.15.112:0/2038408404 >> 172.21.15.112:0/2038408404 conn(0x7f2ed806b550 msgr2=0x7f2ed806b950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:40.305 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.262+0000 7f9d4a939700 1 -- 172.21.15.112:0/3425360857 shutdown_connections 2023-01-24T03:30:40.306 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.262+0000 7f9d4a939700 1 --2- 172.21.15.112:0/3425360857 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d4406b7a0 0x7f9d440bf190 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.306 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.262+0000 7f9d4a939700 1 --2- 172.21.15.112:0/3425360857 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d440c4090 0x7f9d440c2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.306 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.262+0000 7f9d4a939700 1 -- 172.21.15.112:0/3425360857 >> 172.21.15.112:0/3425360857 conn(0x7f9d44066e10 msgr2=0x7f9d44067210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:40.306 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.262+0000 7f2ede439700 1 -- 172.21.15.112:0/2038408404 shutdown_connections 2023-01-24T03:30:40.307 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.263+0000 7f15f457d700 1 -- 172.21.15.112:0/2980596863 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15ec1523b0 msgr2=0x7f15ec06e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.307 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.263+0000 7f15f457d700 1 --2- 172.21.15.112:0/2980596863 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15ec1523b0 0x7f15ec06e690 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f15dc00a830 tx=0x7f15dc0080e0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.308 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.262+0000 7f68157fa700 1 -- 172.21.15.112:0/1161503996 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682000ec40 con 0x7f68280c40e0 2023-01-24T03:30:40.308 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.262+0000 7f68157fa700 1 -- 172.21.15.112:0/1161503996 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f682000eda0 con 0x7f68280c40e0 2023-01-24T03:30:40.308 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.262+0000 7f68157fa700 1 -- 172.21.15.112:0/1161503996 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6820014670 con 0x7f68280c40e0 2023-01-24T03:30:40.309 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.262+0000 7f682f945700 1 -- 172.21.15.112:0/1161503996 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68280ba780 con 0x7f68280c40e0 2023-01-24T03:30:40.309 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.262+0000 7f682f945700 1 -- 172.21.15.112:0/1161503996 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6828156ba0 con 0x7f68280c40e0 2023-01-24T03:30:40.309 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.263+0000 7f9d4a939700 1 -- 172.21.15.112:0/3425360857 shutdown_connections 2023-01-24T03:30:40.310 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.263+0000 7f9d4a939700 1 -- 172.21.15.112:0/3425360857 wait complete. 2023-01-24T03:30:40.310 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.262+0000 7f2ede439700 1 -- 172.21.15.112:0/2038408404 wait complete. 2023-01-24T03:30:40.310 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.264+0000 7f682f945700 1 -- 172.21.15.112:0/1161503996 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f68280683d0 con 0x7f68280c40e0 2023-01-24T03:30:40.311 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.264+0000 7f68157fa700 1 -- 172.21.15.112:0/1161503996 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f6820012050 con 0x7f68280c40e0 2023-01-24T03:30:40.311 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.264+0000 7f2ede439700 1 Processor -- start 2023-01-24T03:30:40.312 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.264+0000 7fb551f12700 1 Processor -- start 2023-01-24T03:30:40.312 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.264+0000 7f68157fa700 1 --2- 172.21.15.112:0/1161503996 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f6808061290 0x7f6808063740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.312 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.264+0000 7f9d4a939700 1 Processor -- start 2023-01-24T03:30:40.313 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.264+0000 7f68157fa700 1 -- 172.21.15.112:0/1161503996 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f682007dce0 con 0x7f68280c40e0 2023-01-24T03:30:40.313 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.265+0000 7f68157fa700 1 --2- 172.21.15.112:0/1161503996 >> [v2:172.21.15.142:6808/84615,v1:172.21.15.142:6809/84615] conn(0x7f680806b080 0x7f680806d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.314 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.265+0000 7f68157fa700 1 -- 172.21.15.112:0/1161503996 --> [v2:172.21.15.142:6808/84615,v1:172.21.15.142:6809/84615] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f680806db80 con 0x7f680806b080 2023-01-24T03:30:40.314 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.265+0000 7f68157fa700 1 -- 172.21.15.112:0/1161503996 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6820083340 con 0x7f68280c40e0 2023-01-24T03:30:40.314 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.265+0000 7f682cee0700 1 --2- 172.21.15.112:0/1161503996 >> [v2:172.21.15.142:6808/84615,v1:172.21.15.142:6809/84615] conn(0x7f680806b080 0x7f680806d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.314 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.265+0000 7f682cee0700 1 --2- 172.21.15.112:0/1161503996 >> [v2:172.21.15.142:6808/84615,v1:172.21.15.142:6809/84615] conn(0x7f680806b080 0x7f680806d4b0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.264+0000 7f2ede439700 1 -- start start 2023-01-24T03:30:40.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.265+0000 7f2ede439700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ed8067fe0 0x7f2ed80a6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.265+0000 7f2ede439700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2ed8069360 0x7f2ed8169990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.265+0000 7f2ede439700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2ed806f050 0x7f2ed814ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.265+0000 7f2ede439700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed80a6530 con 0x7f2ed8067fe0 2023-01-24T03:30:40.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.265+0000 7f2ede439700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed80a5640 con 0x7f2ed8069360 2023-01-24T03:30:40.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.265+0000 7f2ede439700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed80a5920 con 0x7f2ed806f050 2023-01-24T03:30:40.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.264+0000 7fb551f12700 1 -- start start 2023-01-24T03:30:40.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.265+0000 7fb551f12700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fb54c06b210 0x7fb54c06e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.265+0000 7fb551f12700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb54c06d210 0x7fb54c0c3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.265+0000 7fb551f12700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb54c06ebd0 con 0x7fb54c1523b0 2023-01-24T03:30:40.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.265+0000 7fb551f12700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb54c154da0 con 0x7fb54c06b210 2023-01-24T03:30:40.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.265+0000 7fb551f12700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb54c155040 con 0x7fb54c06d210 2023-01-24T03:30:40.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.266+0000 7fb54b7fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fb54c1523b0 0x7fb54c06ab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.112:45274/0 (socket says 172.21.15.112:45274) 2023-01-24T03:30:40.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.266+0000 7fb54b7fe700 1 -- 172.21.15.112:0/479088894 learned_addr learned my addr 172.21.15.112:0/479088894 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:40.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.266+0000 7fb54bfff700 1 --2- 172.21.15.112:0/479088894 >> v2:172.21.15.142:3300/0 conn(0x7fb54c06d210 0x7fb54c0c3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.265+0000 7f9d4a939700 1 -- start start 2023-01-24T03:30:40.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.265+0000 7f9d4a939700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d4406b7a0 0x7f9d440bd720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.265+0000 7f9d4a939700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d440c4090 0x7f9d44169d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.265+0000 7f9d4a939700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d440c4460 0x7f9d4416a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.266+0000 7f9d4a939700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d440c64c0 con 0x7f9d4406b7a0 2023-01-24T03:30:40.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.266+0000 7f9d4a939700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d44169840 con 0x7f9d440c4460 2023-01-24T03:30:40.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.266+0000 7f9d4a939700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4416cf10 con 0x7f9d440c4090 2023-01-24T03:30:40.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.266+0000 7f9d48ed6700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d440c4460 0x7f9d4416a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.266+0000 7f9d48ed6700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d440c4460 0x7f9d4416a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.112:43002/0 (socket says 172.21.15.112:43002) 2023-01-24T03:30:40.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.266+0000 7f9d48ed6700 1 -- 172.21.15.112:0/1403724823 learned_addr learned my addr 172.21.15.112:0/1403724823 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:40.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.266+0000 7f9d437fe700 1 --2- 172.21.15.112:0/1403724823 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d440c4090 0x7f9d44169d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.266+0000 7f9d43fff700 1 --2- 172.21.15.112:0/1403724823 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d4406b7a0 0x7f9d440bd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.265+0000 7f15f457d700 1 -- 172.21.15.112:0/2980596863 shutdown_connections 2023-01-24T03:30:40.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.265+0000 7f15f457d700 1 --2- 172.21.15.112:0/2980596863 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15ec1523b0 0x7f15ec06e690 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.265+0000 7f15f457d700 1 -- 172.21.15.112:0/2980596863 >> 172.21.15.112:0/2980596863 conn(0x7f15ec066e10 msgr2=0x7f15ec067210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:40.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.266+0000 7f1ef17fa700 1 -- 172.21.15.112:0/1531870843 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f1f00024070 con 0x7f1f04158a50 2023-01-24T03:30:40.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.266+0000 7fb54a7fc700 1 -- 172.21.15.112:0/479088894 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 168369576 0 0) 0x7fb540001110 con 0x7fb54c1523b0 2023-01-24T03:30:40.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.266+0000 7fb54a7fc700 1 -- 172.21.15.112:0/479088894 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5380033f0 con 0x7fb54c1523b0 2023-01-24T03:30:40.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.266+0000 7fb54affd700 1 --2- 172.21.15.112:0/479088894 >> v2:172.21.15.112:3300/0 conn(0x7fb54c06b210 0x7fb54c06e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.266+0000 7f9d48ed6700 1 -- 172.21.15.112:0/1403724823 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d440c4090 msgr2=0x7f9d44169d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.266+0000 7f9d48ed6700 1 --2- 172.21.15.112:0/1403724823 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d440c4090 0x7f9d44169d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.324 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.266+0000 7f9d48ed6700 1 -- 172.21.15.112:0/1403724823 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d4406b7a0 msgr2=0x7f9d440bd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.324 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.266+0000 7f9d48ed6700 1 --2- 172.21.15.112:0/1403724823 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d4406b7a0 0x7f9d440bd720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.324 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.266+0000 7f9d48ed6700 1 -- 172.21.15.112:0/1403724823 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d3c009a20 con 0x7f9d440c4460 2023-01-24T03:30:40.324 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.266+0000 7f1ef17fa700 1 --2- 172.21.15.112:0/1531870843 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f1ee4061290 0x7f1ee4063740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.266+0000 7fb54bfff700 1 -- 172.21.15.112:0/479088894 >> v2:172.21.15.112:3300/0 conn(0x7fb54c06b210 msgr2=0x7fb54c06e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:40.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.266+0000 7fb54bfff700 1 --2- 172.21.15.112:0/479088894 >> v2:172.21.15.112:3300/0 conn(0x7fb54c06b210 0x7fb54c06e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.266+0000 7fb54bfff700 1 -- 172.21.15.112:0/479088894 >> v1:172.21.15.142:6789/0 conn(0x7fb54c1523b0 legacy=0x7fb54c06ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.266+0000 7fb54bfff700 1 -- 172.21.15.112:0/479088894 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb54c155320 con 0x7fb54c06d210 2023-01-24T03:30:40.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.266+0000 7f1ef17fa700 1 -- 172.21.15.112:0/1531870843 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f1f0007ff50 con 0x7f1f04158a50 2023-01-24T03:30:40.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.266+0000 7f15f457d700 1 -- 172.21.15.112:0/2980596863 shutdown_connections 2023-01-24T03:30:40.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.266+0000 7f15f457d700 1 -- 172.21.15.112:0/2980596863 wait complete. 2023-01-24T03:30:40.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.266+0000 7f2ed77fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2ed8069360 0x7f2ed8169990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.266+0000 7f2ed77fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2ed8069360 0x7f2ed8169990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.112:43004/0 (socket says 172.21.15.112:43004) 2023-01-24T03:30:40.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.266+0000 7f2ed77fe700 1 -- 172.21.15.112:0/4123652931 learned_addr learned my addr 172.21.15.112:0/4123652931 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:40.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.266+0000 7f2edc9d6700 1 --2- 172.21.15.112:0/4123652931 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2ed806f050 0x7f2ed814ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.266+0000 7f2ed7fff700 1 --2- 172.21.15.112:0/4123652931 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ed8067fe0 0x7f2ed80a6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.267+0000 7fb54bfff700 1 --2- 172.21.15.112:0/479088894 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb54c06d210 0x7fb54c0c3cf0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb54c06a350 tx=0x7fb544007620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15019244604dac80 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.266+0000 7f9d48ed6700 1 --2- 172.21.15.112:0/1403724823 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d440c4460 0x7f9d4416a280 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c013000 tx=0x7f9d3c019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.266+0000 7f1ef17fa700 1 --2- 172.21.15.112:0/1531870843 >> [v2:172.21.15.142:6800/84614,v1:172.21.15.142:6801/84614] conn(0x7f1ee406b080 0x7f1ee406d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.267+0000 7f1ef17fa700 1 -- 172.21.15.112:0/1531870843 --> [v2:172.21.15.142:6800/84614,v1:172.21.15.142:6801/84614] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1ee406db80 con 0x7f1ee406b080 2023-01-24T03:30:40.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.267+0000 7f1ef17fa700 1 -- 172.21.15.112:0/1531870843 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1f00080280 con 0x7f1f04158a50 2023-01-24T03:30:40.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.267+0000 7f68157fa700 1 -- 172.21.15.112:0/1161503996 <== osd.7 v2:172.21.15.142:6808/84615 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23747 (crc 0 0 0) 0x7f680806db80 con 0x7f680806b080 2023-01-24T03:30:40.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.267+0000 7f2ed77fe700 1 -- 172.21.15.112:0/4123652931 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2ed806f050 msgr2=0x7f2ed814ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.267+0000 7f2ed77fe700 1 --2- 172.21.15.112:0/4123652931 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2ed806f050 0x7f2ed814ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.267+0000 7f2ed77fe700 1 -- 172.21.15.112:0/4123652931 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ed8067fe0 msgr2=0x7f2ed80a6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.267+0000 7f2ed77fe700 1 --2- 172.21.15.112:0/4123652931 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ed8067fe0 0x7f2ed80a6f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.267+0000 7f2ed77fe700 1 -- 172.21.15.112:0/4123652931 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec800b6a0 con 0x7f2ed8069360 2023-01-24T03:30:40.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.267+0000 7f2ed7fff700 1 --2- 172.21.15.112:0/4123652931 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ed8067fe0 0x7f2ed80a6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:30:40.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.267+0000 7f9d2bfff700 1 -- 172.21.15.112:0/1403724823 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c012e10 con 0x7f9d440c4460 2023-01-24T03:30:40.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.267+0000 7f9d4a939700 1 -- 172.21.15.112:0/1403724823 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d4416d1f0 con 0x7f9d440c4460 2023-01-24T03:30:40.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.267+0000 7f9d4a939700 1 -- 172.21.15.112:0/1403724823 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d4416d7b0 con 0x7f9d440c4460 2023-01-24T03:30:40.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.267+0000 7f2ed77fe700 1 --2- 172.21.15.112:0/4123652931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2ed8069360 0x7f2ed8169990 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc00b490 tx=0x7f2ecc009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.267+0000 7f1f0927c700 1 --2- 172.21.15.112:0/1531870843 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f1ee4061290 0x7f1ee4063740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.267+0000 7f1f08a7b700 1 --2- 172.21.15.112:0/1531870843 >> [v2:172.21.15.142:6800/84614,v1:172.21.15.142:6801/84614] conn(0x7f1ee406b080 0x7f1ee406d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.267+0000 7f1f08a7b700 1 --2- 172.21.15.112:0/1531870843 >> [v2:172.21.15.142:6800/84614,v1:172.21.15.142:6801/84614] conn(0x7f1ee406b080 0x7f1ee406d4b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.267+0000 7f9d2bfff700 1 -- 172.21.15.112:0/1403724823 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d3c01d630 con 0x7f9d440c4460 2023-01-24T03:30:40.335 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.267+0000 7f9d2bfff700 1 -- 172.21.15.112:0/1403724823 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c01d9f0 con 0x7f9d440c4460 2023-01-24T03:30:40.335 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.268+0000 7f682d6e1700 1 --2- 172.21.15.112:0/1161503996 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f6808061290 0x7f6808063740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.336 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.268+0000 7f9d4a939700 1 -- 172.21.15.112:0/1403724823 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9d20000fd0 con 0x7f9d440c4460 2023-01-24T03:30:40.336 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.268+0000 7f15f457d700 1 Processor -- start 2023-01-24T03:30:40.336 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.268+0000 7f2ebffff700 1 -- 172.21.15.112:0/4123652931 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc00f510 con 0x7f2ed8069360 2023-01-24T03:30:40.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.268+0000 7f2ebffff700 1 -- 172.21.15.112:0/4123652931 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ecc00f690 con 0x7f2ed8069360 2023-01-24T03:30:40.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.268+0000 7f2ebffff700 1 -- 172.21.15.112:0/4123652931 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc00fa50 con 0x7f2ed8069360 2023-01-24T03:30:40.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.268+0000 7f682d6e1700 1 --2- 172.21.15.112:0/1161503996 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f6808061290 0x7f6808063740 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f68180016c0 tx=0x7f6818009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.268+0000 7f1ef17fa700 1 -- 172.21.15.112:0/1531870843 <== osd.4 v2:172.21.15.142:6800/84614 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23747 (crc 0 0 0) 0x7f1ee406db80 con 0x7f1ee406b080 2023-01-24T03:30:40.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.268+0000 7f9d2bfff700 1 -- 172.21.15.112:0/1403724823 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f9d3c0458e0 con 0x7f9d440c4460 2023-01-24T03:30:40.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.267+0000 7fb54a7fc700 1 -- 172.21.15.112:0/479088894 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54400dba0 con 0x7fb54c06d210 2023-01-24T03:30:40.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.267+0000 7fb54a7fc700 1 -- 172.21.15.112:0/479088894 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb54400dd00 con 0x7fb54c06d210 2023-01-24T03:30:40.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.267+0000 7fb54a7fc700 1 -- 172.21.15.112:0/479088894 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb544007cc0 con 0x7fb54c06d210 2023-01-24T03:30:40.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.269+0000 7f9d2bfff700 1 --2- 172.21.15.112:0/1403724823 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9d240611f0 0x7f9d240636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.269+0000 7f9d2bfff700 1 -- 172.21.15.112:0/1403724823 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f9d3c080630 con 0x7f9d440c4460 2023-01-24T03:30:40.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.269+0000 7f9d2bfff700 1 --2- 172.21.15.112:0/1403724823 >> [v2:172.21.15.112:6810/84511,v1:172.21.15.112:6813/84511] conn(0x7f9d2406afe0 0x7f9d2406d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.269+0000 7f9d2bfff700 1 -- 172.21.15.112:0/1403724823 --> [v2:172.21.15.112:6810/84511,v1:172.21.15.112:6813/84511] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9d2406dae0 con 0x7f9d2406afe0 2023-01-24T03:30:40.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.269+0000 7f9d2bfff700 1 -- 172.21.15.112:0/1403724823 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d3c080a10 con 0x7f9d440c4460 2023-01-24T03:30:40.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.269+0000 7f9d43fff700 1 --2- 172.21.15.112:0/1403724823 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9d240611f0 0x7f9d240636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.268+0000 7f15f457d700 1 -- start start 2023-01-24T03:30:40.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.269+0000 7f15f457d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15ec06b210 0x7f15ec15da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.269+0000 7f15f457d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15ec06d210 0x7f15ec16d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.269+0000 7f15f457d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f15ec1523b0 0x7f15ec16d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.269+0000 7f15f457d700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ec0b5c40 con 0x7f15ec1523b0 2023-01-24T03:30:40.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.269+0000 7f15f457d700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ec15cd60 con 0x7f15ec06b210 2023-01-24T03:30:40.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.269+0000 7f15f457d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15ec0b3fa0 con 0x7f15ec06d210 2023-01-24T03:30:40.343 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.269+0000 7f15f2b1a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f15ec1523b0 0x7f15ec16d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.343 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.269+0000 7f15f2b1a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f15ec1523b0 0x7f15ec16d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50522/0 (socket says 172.21.15.112:50522) 2023-01-24T03:30:40.343 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.269+0000 7f15f2b1a700 1 -- 172.21.15.112:0/2100666414 learned_addr learned my addr 172.21.15.112:0/2100666414 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:40.343 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.269+0000 7f15f2319700 1 --2- 172.21.15.112:0/2100666414 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15ec06b210 0x7f15ec15da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.343 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.269+0000 7f15f1b18700 1 --2- 172.21.15.112:0/2100666414 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15ec06d210 0x7f15ec16d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.344 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.269+0000 7fb551f12700 1 -- 172.21.15.112:0/479088894 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb54c06d210 msgr2=0x7fb54c0c3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.344 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.269+0000 7fb551f12700 1 --2- 172.21.15.112:0/479088894 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb54c06d210 0x7fb54c0c3cf0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb54c06a350 tx=0x7fb544007620 comp rx=0 tx=0).stop 2023-01-24T03:30:40.344 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.270+0000 7fb551f12700 1 -- 172.21.15.112:0/479088894 shutdown_connections 2023-01-24T03:30:40.345 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.270+0000 7fb551f12700 1 --2- 172.21.15.112:0/479088894 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb54c06d210 0x7fb54c0c3cf0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.345 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.270+0000 7fb551f12700 1 --2- 172.21.15.112:0/479088894 >> v2:172.21.15.112:3300/0 conn(0x7fb54c06b210 0x7fb54c06e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.345 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.270+0000 7fb551f12700 1 -- 172.21.15.112:0/479088894 >> 172.21.15.112:0/479088894 conn(0x7fb54c066e10 msgr2=0x7fb54c067210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:40.345 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.269+0000 7f15f2319700 1 -- 172.21.15.112:0/2100666414 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15ec06d210 msgr2=0x7f15ec16d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.346 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.269+0000 7f15f2319700 1 --2- 172.21.15.112:0/2100666414 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15ec06d210 0x7f15ec16d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.346 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.270+0000 7f15f2319700 1 -- 172.21.15.112:0/2100666414 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f15ec1523b0 msgr2=0x7f15ec16d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.346 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.270+0000 7f15f2319700 1 --2- 172.21.15.112:0/2100666414 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f15ec1523b0 0x7f15ec16d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.346 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.270+0000 7f15f2319700 1 -- 172.21.15.112:0/2100666414 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15dc00a340 con 0x7f15ec06b210 2023-01-24T03:30:40.347 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.270+0000 7f9d43fff700 1 --2- 172.21.15.112:0/1403724823 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9d240611f0 0x7f9d240636a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9d3400b940 tx=0x7f9d34006ee0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.347 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.270+0000 7f2ede439700 1 -- 172.21.15.112:0/4123652931 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ed80a5c00 con 0x7f2ed8069360 2023-01-24T03:30:40.347 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.270+0000 7f2ede439700 1 -- 172.21.15.112:0/4123652931 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ed80a61c0 con 0x7f2ed8069360 2023-01-24T03:30:40.348 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.270+0000 7f15f2319700 1 --2- 172.21.15.112:0/2100666414 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15ec06b210 0x7f15ec15da80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f15ec06bb10 tx=0x7f15dc014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.348 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.271+0000 7f9d437fe700 1 --2- 172.21.15.112:0/1403724823 >> [v2:172.21.15.112:6810/84511,v1:172.21.15.112:6813/84511] conn(0x7f9d2406afe0 0x7f9d2406d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.349 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.271+0000 7f1f0927c700 1 --2- 172.21.15.112:0/1531870843 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f1ee4061290 0x7f1ee4063740 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1efc006fd0 tx=0x7f1efc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.350 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.271+0000 7f15da7fc700 1 -- 172.21.15.112:0/2100666414 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15dc01e070 con 0x7f15ec06b210 2023-01-24T03:30:40.350 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.271+0000 7f15da7fc700 1 -- 172.21.15.112:0/2100666414 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15dc00fc90 con 0x7f15ec06b210 2023-01-24T03:30:40.350 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.271+0000 7f15da7fc700 1 -- 172.21.15.112:0/2100666414 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15dc00f8e0 con 0x7f15ec06b210 2023-01-24T03:30:40.350 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.271+0000 7f15f457d700 1 -- 172.21.15.112:0/2100666414 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15ec0b4210 con 0x7f15ec06b210 2023-01-24T03:30:40.350 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.271+0000 7f15f457d700 1 -- 172.21.15.112:0/2100666414 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15ec0b47d0 con 0x7f15ec06b210 2023-01-24T03:30:40.351 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.271+0000 7f9d437fe700 1 --2- 172.21.15.112:0/1403724823 >> [v2:172.21.15.112:6810/84511,v1:172.21.15.112:6813/84511] conn(0x7f9d2406afe0 0x7f9d2406d410 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.351 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.272+0000 7f15f457d700 1 -- 172.21.15.112:0/2100666414 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f15ec16cc80 con 0x7f15ec06b210 2023-01-24T03:30:40.352 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.273+0000 7f2ede439700 1 -- 172.21.15.112:0/4123652931 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2ed805e610 con 0x7f2ed8069360 2023-01-24T03:30:40.352 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.273+0000 7f15da7fc700 1 -- 172.21.15.112:0/2100666414 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f15dc041890 con 0x7f15ec06b210 2023-01-24T03:30:40.353 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.273+0000 7f9d2bfff700 1 -- 172.21.15.112:0/1403724823 <== osd.2 v2:172.21.15.112:6810/84511 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23747 (crc 0 0 0) 0x7f9d2406dae0 con 0x7f9d2406afe0 2023-01-24T03:30:40.354 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.273+0000 7f15da7fc700 1 --2- 172.21.15.112:0/2100666414 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f15cc061190 0x7f15cc063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.354 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.273+0000 7f15da7fc700 1 -- 172.21.15.112:0/2100666414 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f15dc07ec60 con 0x7f15ec06b210 2023-01-24T03:30:40.354 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.273+0000 7f15da7fc700 1 --2- 172.21.15.112:0/2100666414 >> [v2:172.21.15.112:6800/84510,v1:172.21.15.112:6801/84510] conn(0x7f15cc06af80 0x7f15cc06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.354 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.273+0000 7f15da7fc700 1 -- 172.21.15.112:0/2100666414 --> [v2:172.21.15.112:6800/84510,v1:172.21.15.112:6801/84510] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f15cc06da80 con 0x7f15cc06af80 2023-01-24T03:30:40.354 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.273+0000 7f15f1b18700 1 --2- 172.21.15.112:0/2100666414 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f15cc061190 0x7f15cc063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.355 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.273+0000 7f15da7fc700 1 -- 172.21.15.112:0/2100666414 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f15dc07f040 con 0x7f15ec06b210 2023-01-24T03:30:40.355 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.275+0000 7f15f1b18700 1 --2- 172.21.15.112:0/2100666414 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f15cc061190 0x7f15cc063640 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f15ec0b6f80 tx=0x7f15e8005e60 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.355 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.275+0000 7f365330b700 1 Processor -- start 2023-01-24T03:30:40.356 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.275+0000 7f365330b700 1 -- start start 2023-01-24T03:30:40.356 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.276+0000 7f15f2b1a700 1 --2- 172.21.15.112:0/2100666414 >> [v2:172.21.15.112:6800/84510,v1:172.21.15.112:6801/84510] conn(0x7f15cc06af80 0x7f15cc06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.356 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.276+0000 7fb551f12700 1 -- 172.21.15.112:0/479088894 shutdown_connections 2023-01-24T03:30:40.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.276+0000 7f365330b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f364c06f050 0x7f364c1559a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.276+0000 7f365330b700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f364c0c3c80 con 0x7f364c069360 2023-01-24T03:30:40.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.276+0000 7f365330b700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f364c0c2d90 con 0x7f364c067fe0 2023-01-24T03:30:40.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.276+0000 7f365330b700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f364c0bf6e0 con 0x7f364c06f050 2023-01-24T03:30:40.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.276+0000 7f15f2b1a700 1 --2- 172.21.15.112:0/2100666414 >> [v2:172.21.15.112:6800/84510,v1:172.21.15.112:6801/84510] conn(0x7f15cc06af80 0x7f15cc06d3b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.276+0000 7f36510a7700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f364c069360 0x7f364c05d830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:59754/0 (socket says 172.21.15.112:59754) 2023-01-24T03:30:40.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.276+0000 7f36510a7700 1 -- 172.21.15.112:0/2836795818 learned_addr learned my addr 172.21.15.112:0/2836795818 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:40.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.277+0000 7f363bfff700 1 -- 172.21.15.112:0/2836795818 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1868942311 0 0) 0x7f364c0c2d90 con 0x7f364c067fe0 2023-01-24T03:30:40.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.277+0000 7f363bfff700 1 -- 172.21.15.112:0/2836795818 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36340033f0 con 0x7f364c067fe0 2023-01-24T03:30:40.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.277+0000 7f363bfff700 1 -- 172.21.15.112:0/2836795818 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1433762850 0 0) 0x7f364c0c3c80 con 0x7f364c069360 2023-01-24T03:30:40.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.277+0000 7f363bfff700 1 -- 172.21.15.112:0/2836795818 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f364c0c2d90 con 0x7f364c069360 2023-01-24T03:30:40.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.277+0000 7f363bfff700 1 -- 172.21.15.112:0/2836795818 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1529647097 0 0) 0x7f36340033f0 con 0x7f364c067fe0 2023-01-24T03:30:40.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.277+0000 7f363bfff700 1 -- 172.21.15.112:0/2836795818 >> v2:172.21.15.142:3300/0 conn(0x7f364c06f050 msgr2=0x7f364c1559a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-01-24T03:30:40.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.277+0000 7f363bfff700 1 --2- 172.21.15.112:0/2836795818 >> v2:172.21.15.142:3300/0 conn(0x7f364c06f050 0x7f364c1559a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.277+0000 7f363bfff700 1 -- 172.21.15.112:0/2836795818 >> v1:172.21.15.112:6789/0 conn(0x7f364c069360 legacy=0x7f364c05d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.277+0000 7f363bfff700 1 -- 172.21.15.112:0/2836795818 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f364c0c3070 con 0x7f364c067fe0 2023-01-24T03:30:40.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.277+0000 7f363bfff700 1 -- 172.21.15.112:0/2836795818 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7f36440017f0 con 0x7f364c067fe0 2023-01-24T03:30:40.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.277+0000 7f363bfff700 1 -- 172.21.15.112:0/2836795818 >> v1:172.21.15.142:6789/0 conn(0x7f364c067fe0 legacy=0x7f364c069b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.277+0000 7f363bfff700 1 --2- 172.21.15.112:0/2836795818 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3634004ad0 0x7f3634004ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.277+0000 7f363bfff700 1 --2- 172.21.15.112:0/2836795818 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3634003630 0x7f3634003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.277+0000 7f363bfff700 1 --2- 172.21.15.112:0/2836795818 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3634004130 0x7f363400b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.277+0000 7f363bfff700 1 -- 172.21.15.112:0/2836795818 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364c0bf6e0 con 0x7f3634004ad0 2023-01-24T03:30:40.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.277+0000 7f363bfff700 1 -- 172.21.15.112:0/2836795818 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f363400c0b0 con 0x7f3634003630 2023-01-24T03:30:40.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.277+0000 7f363bfff700 1 -- 172.21.15.112:0/2836795818 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f363400c270 con 0x7f3634004130 2023-01-24T03:30:40.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.277+0000 7f15da7fc700 1 -- 172.21.15.112:0/2100666414 <== osd.3 v2:172.21.15.112:6800/84510 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23747 (crc 0 0 0) 0x7f15cc06da80 con 0x7f15cc06af80 2023-01-24T03:30:40.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.277+0000 7f36510a7700 1 --2- 172.21.15.112:0/2836795818 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3634004ad0 0x7f3634004ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.278+0000 7f36518a8700 1 --2- 172.21.15.112:0/2836795818 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3634004130 0x7f363400b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.278+0000 7f36510a7700 1 -- 172.21.15.112:0/2836795818 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3634004130 msgr2=0x7f363400b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:40.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.278+0000 7f36510a7700 1 --2- 172.21.15.112:0/2836795818 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3634004130 0x7f363400b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.278+0000 7f36510a7700 1 -- 172.21.15.112:0/2836795818 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3634003630 msgr2=0x7f3634003aa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-01-24T03:30:40.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.278+0000 7f36510a7700 1 --2- 172.21.15.112:0/2836795818 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3634003630 0x7f3634003aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.278+0000 7f36510a7700 1 -- 172.21.15.112:0/2836795818 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f364c0c3070 con 0x7f3634004ad0 2023-01-24T03:30:40.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.278+0000 7f36518a8700 1 -- 172.21.15.112:0/2836795818 reap_dead start 2023-01-24T03:30:40.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.278+0000 7f36510a7700 1 --2- 172.21.15.112:0/2836795818 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3634004ad0 0x7f3634004ec0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f36480019a0 tx=0x7f3648007c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=926ac266745906dc server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.278+0000 7f363bfff700 1 -- 172.21.15.112:0/2836795818 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36480070d0 con 0x7f3634004ad0 2023-01-24T03:30:40.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.278+0000 7f363bfff700 1 -- 172.21.15.112:0/2836795818 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3648007230 con 0x7f3634004ad0 2023-01-24T03:30:40.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.278+0000 7f363bfff700 1 -- 172.21.15.112:0/2836795818 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36480075f0 con 0x7f3634004ad0 2023-01-24T03:30:40.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.279+0000 7fb551f12700 1 -- 172.21.15.112:0/479088894 wait complete. 2023-01-24T03:30:40.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.280+0000 7f7bfc9d2700 1 Processor -- start 2023-01-24T03:30:40.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.280+0000 7fb551f12700 1 Processor -- start 2023-01-24T03:30:40.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.280+0000 7fb551f12700 1 -- start start 2023-01-24T03:30:40.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.280+0000 7f7bfc9d2700 1 -- start start 2023-01-24T03:30:40.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.281+0000 7f7bfc9d2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7bf8067fe0 0x7f7bf80c3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.281+0000 7f7bfc9d2700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f7bf80683b0 0x7f7bf80ad730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.281+0000 7f7bfc9d2700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bf8159910 con 0x7f7bf80c1960 2023-01-24T03:30:40.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.281+0000 7f7bfc9d2700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bf8153f00 con 0x7f7bf80683b0 2023-01-24T03:30:40.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.281+0000 7f7bfc9d2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf8159700 con 0x7f7bf8067fe0 2023-01-24T03:30:40.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.281+0000 7f7bf5d9b700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f7bf80683b0 0x7f7bf80ad730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.281+0000 7f7bf5d9b700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f7bf80683b0 0x7f7bf80ad730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50546/0 (socket says 172.21.15.112:50546) 2023-01-24T03:30:40.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.281+0000 7f7bf5d9b700 1 -- 172.21.15.112:0/4265991601 learned_addr learned my addr 172.21.15.112:0/4265991601 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:40.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.281+0000 7fb551f12700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb54c06b210 0x7fb54c0c7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.281+0000 7fb551f12700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb54c06d210 0x7fb54c0c57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.281+0000 7fb551f12700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb54c1523b0 0x7fb54c0c5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.281+0000 7fb551f12700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54c0c65b0 con 0x7fb54c06b210 2023-01-24T03:30:40.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.281+0000 7fb551f12700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54c14f710 con 0x7fb54c1523b0 2023-01-24T03:30:40.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.281+0000 7fb551f12700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb54c14e840 con 0x7fb54c06d210 2023-01-24T03:30:40.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.282+0000 7fb54b7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb54c06b210 0x7fb54c0c7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.282+0000 7fb54b7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb54c06b210 0x7fb54c0c7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50552/0 (socket says 172.21.15.112:50552) 2023-01-24T03:30:40.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.282+0000 7fb54b7fe700 1 -- 172.21.15.112:0/1287892074 learned_addr learned my addr 172.21.15.112:0/1287892074 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:40.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.282+0000 7fb54bfff700 1 --2- 172.21.15.112:0/1287892074 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb54c1523b0 0x7fb54c0c5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.282+0000 7fb54b7fe700 1 -- 172.21.15.112:0/1287892074 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb54c06d210 msgr2=0x7fb54c0c57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.282+0000 7fb54b7fe700 1 --2- 172.21.15.112:0/1287892074 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb54c06d210 0x7fb54c0c57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.282+0000 7f7bf5d9b700 1 -- 172.21.15.112:0/4265991601 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7bf8067fe0 msgr2=0x7f7bf80c3150 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-01-24T03:30:40.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.282+0000 7f7bf5d9b700 1 --2- 172.21.15.112:0/4265991601 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7bf8067fe0 0x7f7bf80c3150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.282+0000 7f7bf5d9b700 1 -- 172.21.15.112:0/4265991601 >> v1:172.21.15.142:6789/0 conn(0x7f7bf80c1960 legacy=0x7f7bf80c1da0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-01-24T03:30:40.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.282+0000 7f7bf5d9b700 1 -- 172.21.15.112:0/4265991601 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf8159bf0 con 0x7f7bf80683b0 2023-01-24T03:30:40.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.282+0000 7fb54b7fe700 1 -- 172.21.15.112:0/1287892074 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb54c1523b0 msgr2=0x7fb54c0c5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.282+0000 7fb54b7fe700 1 --2- 172.21.15.112:0/1287892074 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb54c1523b0 0x7fb54c0c5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.282+0000 7fb54b7fe700 1 -- 172.21.15.112:0/1287892074 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5440078c0 con 0x7fb54c06b210 2023-01-24T03:30:40.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.282+0000 7fb54b7fe700 1 --2- 172.21.15.112:0/1287892074 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb54c06b210 0x7fb54c0c7be0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb54000c4d0 tx=0x7fb540007af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.282+0000 7f7bf5d9b700 1 --2- 172.21.15.112:0/4265991601 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7bf80683b0 0x7f7bf80ad730 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7bec006fd0 tx=0x7f7bec0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5fb5b56fa695821 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.282+0000 7fb5337fe700 1 -- 172.21.15.112:0/1287892074 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5400072e0 con 0x7fb54c06b210 2023-01-24T03:30:40.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.282+0000 7fb551f12700 1 -- 172.21.15.112:0/1287892074 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb54c14eb00 con 0x7fb54c06b210 2023-01-24T03:30:40.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.282+0000 7fb551f12700 1 -- 172.21.15.112:0/1287892074 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb54c14f0c0 con 0x7fb54c06b210 2023-01-24T03:30:40.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.283+0000 7fb5337fe700 1 -- 172.21.15.112:0/1287892074 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb540007480 con 0x7fb54c06b210 2023-01-24T03:30:40.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.283+0000 7fb5337fe700 1 -- 172.21.15.112:0/1287892074 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5400059d0 con 0x7fb54c06b210 2023-01-24T03:30:40.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.283+0000 7fb551f12700 1 -- 172.21.15.112:0/1287892074 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb528000fd0 con 0x7fb54c06b210 2023-01-24T03:30:40.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.285+0000 7fb5337fe700 1 -- 172.21.15.112:0/1287892074 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fb540019030 con 0x7fb54c06b210 2023-01-24T03:30:40.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.285+0000 7fb5337fe700 1 --2- 172.21.15.112:0/1287892074 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb52c0611c0 0x7fb52c063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.285+0000 7fb5337fe700 1 -- 172.21.15.112:0/1287892074 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fb54007c880 con 0x7fb54c06b210 2023-01-24T03:30:40.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.285+0000 7fb5337fe700 1 --2- 172.21.15.112:0/1287892074 >> [v2:172.21.15.112:6808/84513,v1:172.21.15.112:6812/84513] conn(0x7fb52c06af30 0x7fb52c06d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.285+0000 7fb5337fe700 1 -- 172.21.15.112:0/1287892074 --> [v2:172.21.15.112:6808/84513,v1:172.21.15.112:6812/84513] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb52c06da30 con 0x7fb52c06af30 2023-01-24T03:30:40.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.285+0000 7fb5337fe700 1 -- 172.21.15.112:0/1287892074 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb540081220 con 0x7fb54c06b210 2023-01-24T03:30:40.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.285+0000 7fb54bfff700 1 --2- 172.21.15.112:0/1287892074 >> [v2:172.21.15.112:6808/84513,v1:172.21.15.112:6812/84513] conn(0x7fb52c06af30 0x7fb52c06d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.285+0000 7f365330b700 1 -- 172.21.15.112:0/2836795818 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3634004ad0 msgr2=0x7f3634004ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.285+0000 7f365330b700 1 --2- 172.21.15.112:0/2836795818 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3634004ad0 0x7f3634004ec0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f36480019a0 tx=0x7f3648007c40 comp rx=0 tx=0).stop 2023-01-24T03:30:40.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.286+0000 7fb54bfff700 1 --2- 172.21.15.112:0/1287892074 >> [v2:172.21.15.112:6808/84513,v1:172.21.15.112:6812/84513] conn(0x7fb52c06af30 0x7fb52c06d360 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.286+0000 7fb5337fe700 1 -- 172.21.15.112:0/1287892074 <== osd.1 v2:172.21.15.112:6808/84513 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23747 (crc 0 0 0) 0x7fb52c06da30 con 0x7fb52c06af30 2023-01-24T03:30:40.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.287+0000 7f365330b700 1 -- 172.21.15.112:0/2836795818 shutdown_connections 2023-01-24T03:30:40.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.287+0000 7f365330b700 1 --2- 172.21.15.112:0/2836795818 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3634004ad0 0x7f3634004ec0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.287+0000 7f365330b700 1 -- 172.21.15.112:0/2836795818 >> 172.21.15.112:0/2836795818 conn(0x7f364c06b550 msgr2=0x7f364c06b950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:40.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.289+0000 7fb54affd700 1 --2- 172.21.15.112:0/1287892074 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb52c0611c0 0x7fb52c063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.289+0000 7fb54affd700 1 --2- 172.21.15.112:0/1287892074 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb52c0611c0 0x7fb52c063670 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb53c006fd0 tx=0x7fb53c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.291+0000 7f2ebffff700 1 -- 172.21.15.112:0/4123652931 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f2ecc00f070 con 0x7f2ed8069360 2023-01-24T03:30:40.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.289+0000 7f7bf559a700 1 -- 172.21.15.112:0/4265991601 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bec00b690 con 0x7f7bf80683b0 2023-01-24T03:30:40.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.289+0000 7f7bf559a700 1 -- 172.21.15.112:0/4265991601 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bec00b7f0 con 0x7f7bf80683b0 2023-01-24T03:30:40.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.289+0000 7f7bf559a700 1 -- 172.21.15.112:0/4265991601 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bec00bbb0 con 0x7f7bf80683b0 2023-01-24T03:30:40.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.290+0000 7f7bfc9d2700 1 -- 172.21.15.112:0/4265991601 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7bf80683b0 msgr2=0x7f7bf80ad730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.290+0000 7f7bfc9d2700 1 --2- 172.21.15.112:0/4265991601 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7bf80683b0 0x7f7bf80ad730 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7bec006fd0 tx=0x7f7bec0090f0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.290+0000 7f7bfc9d2700 1 -- 172.21.15.112:0/4265991601 shutdown_connections 2023-01-24T03:30:40.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.290+0000 7f7bfc9d2700 1 --2- 172.21.15.112:0/4265991601 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7bf8067fe0 0x7f7bf80c3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.290+0000 7f7bfc9d2700 1 --2- 172.21.15.112:0/4265991601 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7bf80683b0 0x7f7bf80ad730 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.290+0000 7f7bfc9d2700 1 -- 172.21.15.112:0/4265991601 >> 172.21.15.112:0/4265991601 conn(0x7f7bf806b550 msgr2=0x7f7bf806b950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:40.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.291+0000 7f2ebffff700 1 --2- 172.21.15.112:0/4123652931 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2eb8061340 0x7f2eb80637f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.291+0000 7f2ebffff700 1 -- 172.21.15.112:0/4123652931 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f2ecc07d120 con 0x7f2ed8069360 2023-01-24T03:30:40.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.291+0000 7f2ed7fff700 1 --2- 172.21.15.112:0/4123652931 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2eb8061340 0x7f2eb80637f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.291+0000 7f2ebffff700 1 --2- 172.21.15.112:0/4123652931 >> [v2:172.21.15.112:6809/84512,v1:172.21.15.112:6811/84512] conn(0x7f2eb806b130 0x7f2eb806d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.291+0000 7f2ebffff700 1 -- 172.21.15.112:0/4123652931 --> [v2:172.21.15.112:6809/84512,v1:172.21.15.112:6811/84512] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2eb806dc30 con 0x7f2eb806b130 2023-01-24T03:30:40.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.291+0000 7f2ebffff700 1 -- 172.21.15.112:0/4123652931 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ecc07f660 con 0x7f2ed8069360 2023-01-24T03:30:40.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.292+0000 7f2ed7fff700 1 --2- 172.21.15.112:0/4123652931 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2eb8061340 0x7f2eb80637f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2ec8003080 tx=0x7f2ec8019000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.293+0000 7f9d4a939700 1 -- 172.21.15.112:0/1403724823 --> [v2:172.21.15.112:6810/84511,v1:172.21.15.112:6813/84511] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9d20002bb0 con 0x7f9d2406afe0 2023-01-24T03:30:40.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.293+0000 7f9d2bfff700 1 -- 172.21.15.112:0/1403724823 <== osd.2 v2:172.21.15.112:6810/84511 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9d20002bb0 con 0x7f9d2406afe0 2023-01-24T03:30:40.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.294+0000 7f9d4a939700 1 -- 172.21.15.112:0/1403724823 >> [v2:172.21.15.112:6810/84511,v1:172.21.15.112:6813/84511] conn(0x7f9d2406afe0 msgr2=0x7f9d2406d410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.294+0000 7f9d4a939700 1 --2- 172.21.15.112:0/1403724823 >> [v2:172.21.15.112:6810/84511,v1:172.21.15.112:6813/84511] conn(0x7f9d2406afe0 0x7f9d2406d410 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.294+0000 7f9d4a939700 1 -- 172.21.15.112:0/1403724823 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9d240611f0 msgr2=0x7f9d240636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.294+0000 7f9d4a939700 1 --2- 172.21.15.112:0/1403724823 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9d240611f0 0x7f9d240636a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9d3400b940 tx=0x7f9d34006ee0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.294+0000 7f9d4a939700 1 -- 172.21.15.112:0/1403724823 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d440c4460 msgr2=0x7f9d4416a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.294+0000 7f9d4a939700 1 --2- 172.21.15.112:0/1403724823 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d440c4460 0x7f9d4416a280 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c013000 tx=0x7f9d3c019000 comp rx=0 tx=0).stop 2023-01-24T03:30:40.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.295+0000 7f9d48ed6700 1 -- 172.21.15.112:0/1403724823 reap_dead start 2023-01-24T03:30:40.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.295+0000 7f9d4a939700 1 -- 172.21.15.112:0/1403724823 shutdown_connections 2023-01-24T03:30:40.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.295+0000 7f9d4a939700 1 -- 172.21.15.112:0/1403724823 >> 172.21.15.112:0/1403724823 conn(0x7f9d44066e10 msgr2=0x7f9d4415af70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:40.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.295+0000 7f9d4a939700 1 -- 172.21.15.112:0/1403724823 shutdown_connections 2023-01-24T03:30:40.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.295+0000 7f9d4a939700 1 -- 172.21.15.112:0/1403724823 wait complete. 2023-01-24T03:30:40.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.296+0000 7f15f457d700 1 -- 172.21.15.112:0/2100666414 --> [v2:172.21.15.112:6800/84510,v1:172.21.15.112:6801/84510] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f15ec0683d0 con 0x7f15cc06af80 2023-01-24T03:30:40.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.296+0000 7f15da7fc700 1 -- 172.21.15.112:0/2100666414 <== osd.3 v2:172.21.15.112:6800/84510 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f15ec0683d0 con 0x7f15cc06af80 2023-01-24T03:30:40.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.297+0000 7f15f457d700 1 -- 172.21.15.112:0/2100666414 >> [v2:172.21.15.112:6800/84510,v1:172.21.15.112:6801/84510] conn(0x7f15cc06af80 msgr2=0x7f15cc06d3b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.393 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.297+0000 7f15f457d700 1 --2- 172.21.15.112:0/2100666414 >> [v2:172.21.15.112:6800/84510,v1:172.21.15.112:6801/84510] conn(0x7f15cc06af80 0x7f15cc06d3b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.393 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.297+0000 7f365330b700 1 -- 172.21.15.112:0/2836795818 shutdown_connections 2023-01-24T03:30:40.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.297+0000 7f2edc9d6700 1 --2- 172.21.15.112:0/4123652931 >> [v2:172.21.15.112:6809/84512,v1:172.21.15.112:6811/84512] conn(0x7f2eb806b130 0x7f2eb806d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.297+0000 7f15f457d700 1 -- 172.21.15.112:0/2100666414 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f15cc061190 msgr2=0x7f15cc063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.297+0000 7f15f457d700 1 --2- 172.21.15.112:0/2100666414 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f15cc061190 0x7f15cc063640 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f15ec0b6f80 tx=0x7f15e8005e60 comp rx=0 tx=0).stop 2023-01-24T03:30:40.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.297+0000 7f15f457d700 1 -- 172.21.15.112:0/2100666414 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15ec06b210 msgr2=0x7f15ec15da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.297+0000 7f15f457d700 1 --2- 172.21.15.112:0/2100666414 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15ec06b210 0x7f15ec15da80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f15ec06bb10 tx=0x7f15dc014000 comp rx=0 tx=0).stop 2023-01-24T03:30:40.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.297+0000 7f15f2b1a700 1 -- 172.21.15.112:0/2100666414 reap_dead start 2023-01-24T03:30:40.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.297+0000 7f7bfc9d2700 1 -- 172.21.15.112:0/4265991601 shutdown_connections 2023-01-24T03:30:40.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.297+0000 7f2edc9d6700 1 --2- 172.21.15.112:0/4123652931 >> [v2:172.21.15.112:6809/84512,v1:172.21.15.112:6811/84512] conn(0x7f2eb806b130 0x7f2eb806d560 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.298+0000 7f2ebffff700 1 -- 172.21.15.112:0/4123652931 <== osd.0 v2:172.21.15.112:6809/84512 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23747 (crc 0 0 0) 0x7f2eb806dc30 con 0x7f2eb806b130 2023-01-24T03:30:40.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.297+0000 7f7bfc9d2700 1 -- 172.21.15.112:0/4265991601 wait complete. 2023-01-24T03:30:40.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.297+0000 7f365330b700 1 -- 172.21.15.112:0/2836795818 wait complete. 2023-01-24T03:30:40.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.298+0000 7f7bfc9d2700 1 Processor -- start 2023-01-24T03:30:40.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.298+0000 7f365330b700 1 Processor -- start 2023-01-24T03:30:40.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.298+0000 7f365330b700 1 -- start start 2023-01-24T03:30:40.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.300+0000 7f365330b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3634004ad0 0x7f364c0a54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.300+0000 7f365330b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f364c0a7680 0x7f364c0be060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.300+0000 7f365330b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f364c1543e0 0x7f364c151410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.300+0000 7f365330b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364c1727b0 con 0x7f364c1543e0 2023-01-24T03:30:40.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.300+0000 7f365330b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364c0bee40 con 0x7f364c0a7680 2023-01-24T03:30:40.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.300+0000 7f365330b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f364c0bf120 con 0x7f3634004ad0 2023-01-24T03:30:40.401 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.299+0000 7f7bfc9d2700 1 -- start start 2023-01-24T03:30:40.402 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.300+0000 7f7bfc9d2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7bf8067fe0 0x7f7bf81523d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.402 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.300+0000 7f7bfc9d2700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7bf80683b0 0x7f7bf8152910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.402 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.300+0000 7f7bfc9d2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7bf80c1960 0x7f7bf8163d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.402 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.300+0000 7f7bfc9d2700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf81516e0 con 0x7f7bf80683b0 2023-01-24T03:30:40.402 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.300+0000 7f7bfc9d2700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf8151f40 con 0x7f7bf80c1960 2023-01-24T03:30:40.403 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.300+0000 7f7bfc9d2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf80c0af0 con 0x7f7bf8067fe0 2023-01-24T03:30:40.403 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.300+0000 7f7bf5d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7bf80683b0 0x7f7bf8152910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.403 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.300+0000 7f7bf5d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7bf80683b0 0x7f7bf8152910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50568/0 (socket says 172.21.15.112:50568) 2023-01-24T03:30:40.403 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.300+0000 7f7bf5d9b700 1 -- 172.21.15.112:0/1810870069 learned_addr learned my addr 172.21.15.112:0/1810870069 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:40.403 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.300+0000 7f7bf6d9d700 1 --2- 172.21.15.112:0/1810870069 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7bf80c1960 0x7f7bf8163d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.404 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.300+0000 7f7bf6d9d700 1 -- 172.21.15.112:0/1810870069 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7bf8067fe0 msgr2=0x7f7bf81523d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-01-24T03:30:40.404 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.300+0000 7f7bf6d9d700 1 --2- 172.21.15.112:0/1810870069 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7bf8067fe0 0x7f7bf81523d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.404 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.300+0000 7f7bf6d9d700 1 -- 172.21.15.112:0/1810870069 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7bf80683b0 msgr2=0x7f7bf8152910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.405 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.300+0000 7f7bf6d9d700 1 --2- 172.21.15.112:0/1810870069 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7bf80683b0 0x7f7bf8152910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.405 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.300+0000 7f7bf6d9d700 1 -- 172.21.15.112:0/1810870069 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bec009370 con 0x7f7bf80c1960 2023-01-24T03:30:40.405 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.300+0000 7f7bf5d9b700 1 --2- 172.21.15.112:0/1810870069 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7bf80683b0 0x7f7bf8152910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:30:40.405 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.301+0000 7f7bf6d9d700 1 --2- 172.21.15.112:0/1810870069 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7bf80c1960 0x7f7bf8163d60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7bf000b810 tx=0x7f7bf00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.405 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.301+0000 7f7be67fc700 1 -- 172.21.15.112:0/1810870069 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf00105b0 con 0x7f7bf80c1960 2023-01-24T03:30:40.406 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.301+0000 7f7be67fc700 1 -- 172.21.15.112:0/1810870069 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bf0010750 con 0x7f7bf80c1960 2023-01-24T03:30:40.406 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.301+0000 7f7bfc9d2700 1 -- 172.21.15.112:0/1810870069 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bf80c0d70 con 0x7f7bf80c1960 2023-01-24T03:30:40.406 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.301+0000 7f7bfc9d2700 1 -- 172.21.15.112:0/1810870069 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bf8151bd0 con 0x7f7bf80c1960 2023-01-24T03:30:40.407 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.301+0000 7f36510a7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3634004ad0 0x7f364c0a54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.407 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.301+0000 7f36508a6700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f364c0a7680 0x7f364c0be060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.407 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.301+0000 7f36510a7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3634004ad0 0x7f364c0a54a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.112:37872/0 (socket says 172.21.15.112:37872) 2023-01-24T03:30:40.408 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.301+0000 7f36510a7700 1 -- 172.21.15.112:0/3709419392 learned_addr learned my addr 172.21.15.112:0/3709419392 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:40.408 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.301+0000 7f36508a6700 1 --2- 172.21.15.112:0/3709419392 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f364c0a7680 0x7f364c0be060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.112:43068/0 (socket says 172.21.15.112:43068) 2023-01-24T03:30:40.408 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.302+0000 7f36510a7700 1 -- 172.21.15.112:0/3709419392 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f364c0a7680 msgr2=0x7f364c0be060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.408 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.302+0000 7f36510a7700 1 --2- 172.21.15.112:0/3709419392 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f364c0a7680 0x7f364c0be060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.408 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.302+0000 7f36510a7700 1 -- 172.21.15.112:0/3709419392 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f364c1543e0 msgr2=0x7f364c151410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.409 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.302+0000 7f36510a7700 1 --2- 172.21.15.112:0/3709419392 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f364c1543e0 0x7f364c151410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.409 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.302+0000 7f36510a7700 1 -- 172.21.15.112:0/3709419392 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f364c172160 con 0x7f3634004ad0 2023-01-24T03:30:40.409 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.301+0000 7f7be67fc700 1 -- 172.21.15.112:0/1810870069 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf0010b10 con 0x7f7bf80c1960 2023-01-24T03:30:40.410 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.302+0000 7f7bfc9d2700 1 -- 172.21.15.112:0/1810870069 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7bd8000fd0 con 0x7f7bf80c1960 2023-01-24T03:30:40.410 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.302+0000 7f36510a7700 1 --2- 172.21.15.112:0/3709419392 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3634004ad0 0x7f364c0a54a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f363400c080 tx=0x7f3648015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.410 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.302+0000 7f36397fa700 1 -- 172.21.15.112:0/3709419392 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3648007c40 con 0x7f3634004ad0 2023-01-24T03:30:40.411 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.302+0000 7f36397fa700 1 -- 172.21.15.112:0/3709419392 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f364800c7a0 con 0x7f3634004ad0 2023-01-24T03:30:40.411 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.302+0000 7f365330b700 1 -- 172.21.15.112:0/3709419392 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f364c172440 con 0x7f3634004ad0 2023-01-24T03:30:40.411 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.302+0000 7f365330b700 1 -- 172.21.15.112:0/3709419392 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f364c172c40 con 0x7f3634004ad0 2023-01-24T03:30:40.411 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.302+0000 7f36397fa700 1 -- 172.21.15.112:0/3709419392 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364800cb40 con 0x7f3634004ad0 2023-01-24T03:30:40.412 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.303+0000 7f682f945700 1 -- 172.21.15.112:0/1161503996 --> [v2:172.21.15.142:6808/84615,v1:172.21.15.142:6809/84615] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f68280b5850 con 0x7f680806b080 2023-01-24T03:30:40.412 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.303+0000 7f365330b700 1 -- 172.21.15.112:0/3709419392 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3628000fd0 con 0x7f3634004ad0 2023-01-24T03:30:40.412 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.303+0000 7f68157fa700 1 -- 172.21.15.112:0/1161503996 <== osd.7 v2:172.21.15.142:6808/84615 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f68280b5850 con 0x7f680806b080 2023-01-24T03:30:40.413 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.301+0000 7f1f0b4e0700 1 -- 172.21.15.112:0/1531870843 --> [v2:172.21.15.142:6800/84614,v1:172.21.15.142:6801/84614] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1f04159bf0 con 0x7f1ee406b080 2023-01-24T03:30:40.413 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.302+0000 7f1ef17fa700 1 -- 172.21.15.112:0/1531870843 <== osd.4 v2:172.21.15.142:6800/84614 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1f04159bf0 con 0x7f1ee406b080 2023-01-24T03:30:40.413 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.304+0000 7f36397fa700 1 -- 172.21.15.112:0/3709419392 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f364801c040 con 0x7f3634004ad0 2023-01-24T03:30:40.414 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.303+0000 7f7be67fc700 1 -- 172.21.15.112:0/1810870069 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f7bf000f070 con 0x7f7bf80c1960 2023-01-24T03:30:40.414 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.303+0000 7f7be67fc700 1 --2- 172.21.15.112:0/1810870069 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7bd4061240 0x7f7bd40636f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.415 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.304+0000 7f7be67fc700 1 -- 172.21.15.112:0/1810870069 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f7bf007d720 con 0x7f7bf80c1960 2023-01-24T03:30:40.415 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.304+0000 7f7bf659c700 1 --2- 172.21.15.112:0/1810870069 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7bd4061240 0x7f7bd40636f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.415 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.304+0000 7f7be67fc700 1 --2- 172.21.15.112:0/1810870069 >> [v2:172.21.15.142:6816/84638,v1:172.21.15.142:6817/84638] conn(0x7f7bd406afb0 0x7f7bd406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.416 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.304+0000 7f7be67fc700 1 -- 172.21.15.112:0/1810870069 --> [v2:172.21.15.142:6816/84638,v1:172.21.15.142:6817/84638] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7bd406dab0 con 0x7f7bd406afb0 2023-01-24T03:30:40.416 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.304+0000 7f7be67fc700 1 -- 172.21.15.112:0/1810870069 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7bf004d410 con 0x7f7bf80c1960 2023-01-24T03:30:40.416 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.304+0000 7f7bf5d9b700 1 --2- 172.21.15.112:0/1810870069 >> [v2:172.21.15.142:6816/84638,v1:172.21.15.142:6817/84638] conn(0x7f7bd406afb0 0x7f7bd406d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.417 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.304+0000 7f36397fa700 1 --2- 172.21.15.112:0/3709419392 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f362c0611f0 0x7f362c0636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.417 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.304+0000 7f36397fa700 1 -- 172.21.15.112:0/3709419392 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f364807eeb0 con 0x7f3634004ad0 2023-01-24T03:30:40.417 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.304+0000 7f36508a6700 1 --2- 172.21.15.112:0/3709419392 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f362c0611f0 0x7f362c0636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.418 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.304+0000 7f36397fa700 1 --2- 172.21.15.112:0/3709419392 >> [v2:172.21.15.142:6824/84645,v1:172.21.15.142:6825/84645] conn(0x7f362c06afe0 0x7f362c06d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.418 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.304+0000 7f36397fa700 1 -- 172.21.15.112:0/3709419392 --> [v2:172.21.15.142:6824/84645,v1:172.21.15.142:6825/84645] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f362c06dae0 con 0x7f362c06afe0 2023-01-24T03:30:40.418 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.304+0000 7f36397fa700 1 -- 172.21.15.112:0/3709419392 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f364807f1e0 con 0x7f3634004ad0 2023-01-24T03:30:40.419 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.304+0000 7f7bf5d9b700 1 --2- 172.21.15.112:0/1810870069 >> [v2:172.21.15.142:6816/84638,v1:172.21.15.142:6817/84638] conn(0x7f7bd406afb0 0x7f7bd406d3e0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.419 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.304+0000 7f682f945700 1 -- 172.21.15.112:0/1161503996 >> [v2:172.21.15.142:6808/84615,v1:172.21.15.142:6809/84615] conn(0x7f680806b080 msgr2=0x7f680806d4b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.419 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.304+0000 7f682f945700 1 --2- 172.21.15.112:0/1161503996 >> [v2:172.21.15.142:6808/84615,v1:172.21.15.142:6809/84615] conn(0x7f680806b080 0x7f680806d4b0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.420 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.304+0000 7f36518a8700 1 --2- 172.21.15.112:0/3709419392 >> [v2:172.21.15.142:6824/84645,v1:172.21.15.142:6825/84645] conn(0x7f362c06afe0 0x7f362c06d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.420 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.305+0000 7f682f945700 1 -- 172.21.15.112:0/1161503996 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f6808061290 msgr2=0x7f6808063740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.420 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.305+0000 7f682f945700 1 --2- 172.21.15.112:0/1161503996 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f6808061290 0x7f6808063740 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f68180016c0 tx=0x7f6818009040 comp rx=0 tx=0).stop 2023-01-24T03:30:40.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.305+0000 7f682f945700 1 -- 172.21.15.112:0/1161503996 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68280c40e0 msgr2=0x7f682815c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.305+0000 7f682f945700 1 --2- 172.21.15.112:0/1161503996 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f68280c40e0 0x7f682815c910 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6820007f00 tx=0x7f6820009850 comp rx=0 tx=0).stop 2023-01-24T03:30:40.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.305+0000 7f15f457d700 1 -- 172.21.15.112:0/2100666414 shutdown_connections 2023-01-24T03:30:40.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.305+0000 7f15f457d700 1 -- 172.21.15.112:0/2100666414 >> 172.21.15.112:0/2100666414 conn(0x7f15ec066e10 msgr2=0x7f15ec0bfd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:40.422 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.305+0000 7f682dee2700 1 -- 172.21.15.112:0/1161503996 reap_dead start 2023-01-24T03:30:40.422 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.305+0000 7f7bf659c700 1 --2- 172.21.15.112:0/1810870069 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7bd4061240 0x7f7bd40636f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7be80095d0 tx=0x7f7be8006ae0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.423 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.305+0000 7f36508a6700 1 --2- 172.21.15.112:0/3709419392 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f362c0611f0 0x7f362c0636a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f364c0c74b0 tx=0x7f363c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.423 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.305+0000 7f36518a8700 1 --2- 172.21.15.112:0/3709419392 >> [v2:172.21.15.142:6824/84645,v1:172.21.15.142:6825/84645] conn(0x7f362c06afe0 0x7f362c06d410 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.423 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.305+0000 7f7be67fc700 1 -- 172.21.15.112:0/1810870069 <== osd.5 v2:172.21.15.142:6816/84638 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23747 (crc 0 0 0) 0x7f7bd406dab0 con 0x7f7bd406afb0 2023-01-24T03:30:40.424 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.306+0000 7f1f0b4e0700 1 -- 172.21.15.112:0/1531870843 >> [v2:172.21.15.142:6800/84614,v1:172.21.15.142:6801/84614] conn(0x7f1ee406b080 msgr2=0x7f1ee406d4b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.424 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.306+0000 7f1f0b4e0700 1 --2- 172.21.15.112:0/1531870843 >> [v2:172.21.15.142:6800/84614,v1:172.21.15.142:6801/84614] conn(0x7f1ee406b080 0x7f1ee406d4b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.425 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.306+0000 7f1f0b4e0700 1 -- 172.21.15.112:0/1531870843 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f1ee4061290 msgr2=0x7f1ee4063740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.425 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.306+0000 7f1f0b4e0700 1 --2- 172.21.15.112:0/1531870843 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f1ee4061290 0x7f1ee4063740 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1efc006fd0 tx=0x7f1efc008040 comp rx=0 tx=0).stop 2023-01-24T03:30:40.425 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.306+0000 7f1f0b4e0700 1 -- 172.21.15.112:0/1531870843 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1f04158a50 msgr2=0x7f1f04140200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.426 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.306+0000 7f1f0b4e0700 1 --2- 172.21.15.112:0/1531870843 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1f04158a50 0x7f1f04140200 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1f00009340 tx=0x7f1f00017000 comp rx=0 tx=0).stop 2023-01-24T03:30:40.426 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.306+0000 7f36397fa700 1 -- 172.21.15.112:0/3709419392 <== osd.6 v2:172.21.15.142:6824/84645 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23747 (crc 0 0 0) 0x7f362c06dae0 con 0x7f362c06afe0 2023-01-24T03:30:40.426 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.308+0000 7f15f457d700 1 -- 172.21.15.112:0/2100666414 shutdown_connections 2023-01-24T03:30:40.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.309+0000 7f1f09a7d700 1 -- 172.21.15.112:0/1531870843 reap_dead start 2023-01-24T03:30:40.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.309+0000 7f1f0b4e0700 1 -- 172.21.15.112:0/1531870843 shutdown_connections 2023-01-24T03:30:40.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.309+0000 7f1f0b4e0700 1 -- 172.21.15.112:0/1531870843 >> 172.21.15.112:0/1531870843 conn(0x7f1f0406b550 msgr2=0x7f1f0413e240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:40.428 INFO:teuthology.orchestra.run.smithi112.stdout:55834574856 2023-01-24T03:30:40.429 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-01-24T03:30:40.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.310+0000 7fb551f12700 1 -- 172.21.15.112:0/1287892074 --> [v2:172.21.15.112:6808/84513,v1:172.21.15.112:6812/84513] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb528002bb0 con 0x7fb52c06af30 2023-01-24T03:30:40.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.310+0000 7f15f457d700 1 -- 172.21.15.112:0/2100666414 wait complete. 2023-01-24T03:30:40.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.310+0000 7f1f0b4e0700 1 -- 172.21.15.112:0/1531870843 shutdown_connections 2023-01-24T03:30:40.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.310+0000 7f682f945700 1 -- 172.21.15.112:0/1161503996 shutdown_connections 2023-01-24T03:30:40.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.310+0000 7f682f945700 1 -- 172.21.15.112:0/1161503996 >> 172.21.15.112:0/1161503996 conn(0x7f6828066e10 msgr2=0x7f68280bfbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:40.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.310+0000 7f682f945700 1 -- 172.21.15.112:0/1161503996 shutdown_connections 2023-01-24T03:30:40.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.310+0000 7f682f945700 1 -- 172.21.15.112:0/1161503996 wait complete. 2023-01-24T03:30:40.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.310+0000 7f1f0b4e0700 1 -- 172.21.15.112:0/1531870843 wait complete. 2023-01-24T03:30:40.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.312+0000 7fb5337fe700 1 -- 172.21.15.112:0/1287892074 <== osd.1 v2:172.21.15.112:6808/84513 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb528002bb0 con 0x7fb52c06af30 2023-01-24T03:30:40.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.314+0000 7fb551f12700 1 -- 172.21.15.112:0/1287892074 >> [v2:172.21.15.112:6808/84513,v1:172.21.15.112:6812/84513] conn(0x7fb52c06af30 msgr2=0x7fb52c06d360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.314+0000 7fb551f12700 1 --2- 172.21.15.112:0/1287892074 >> [v2:172.21.15.112:6808/84513,v1:172.21.15.112:6812/84513] conn(0x7fb52c06af30 0x7fb52c06d360 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.318+0000 7fb551f12700 1 -- 172.21.15.112:0/1287892074 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb52c0611c0 msgr2=0x7fb52c063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.318+0000 7fb551f12700 1 --2- 172.21.15.112:0/1287892074 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb52c0611c0 0x7fb52c063670 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb53c006fd0 tx=0x7fb53c008040 comp rx=0 tx=0).stop 2023-01-24T03:30:40.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.318+0000 7fb551f12700 1 -- 172.21.15.112:0/1287892074 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb54c06b210 msgr2=0x7fb54c0c7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.318+0000 7fb551f12700 1 --2- 172.21.15.112:0/1287892074 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb54c06b210 0x7fb54c0c7be0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb54000c4d0 tx=0x7fb540007af0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.318+0000 7fb54bfff700 1 -- 172.21.15.112:0/1287892074 reap_dead start 2023-01-24T03:30:40.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.320+0000 7f2ede439700 1 -- 172.21.15.112:0/4123652931 --> [v2:172.21.15.112:6809/84512,v1:172.21.15.112:6811/84512] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2ed806c510 con 0x7f2eb806b130 2023-01-24T03:30:40.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.320+0000 7f2ebffff700 1 -- 172.21.15.112:0/4123652931 <== osd.0 v2:172.21.15.112:6809/84512 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2ed806c510 con 0x7f2eb806b130 2023-01-24T03:30:40.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.322+0000 7fb551f12700 1 -- 172.21.15.112:0/1287892074 shutdown_connections 2023-01-24T03:30:40.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.322+0000 7fb551f12700 1 -- 172.21.15.112:0/1287892074 >> 172.21.15.112:0/1287892074 conn(0x7fb54c066e10 msgr2=0x7fb54c0c2f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:40.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.322+0000 7fb551f12700 1 -- 172.21.15.112:0/1287892074 shutdown_connections 2023-01-24T03:30:40.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.322+0000 7fb551f12700 1 -- 172.21.15.112:0/1287892074 wait complete. 2023-01-24T03:30:40.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.323+0000 7f365330b700 1 -- 172.21.15.112:0/3709419392 --> [v2:172.21.15.142:6824/84645,v1:172.21.15.142:6825/84645] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3628002bb0 con 0x7f362c06afe0 2023-01-24T03:30:40.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.324+0000 7f36397fa700 1 -- 172.21.15.112:0/3709419392 <== osd.6 v2:172.21.15.142:6824/84645 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3628002bb0 con 0x7f362c06afe0 2023-01-24T03:30:40.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.324+0000 7f2ede439700 1 -- 172.21.15.112:0/4123652931 >> [v2:172.21.15.112:6809/84512,v1:172.21.15.112:6811/84512] conn(0x7f2eb806b130 msgr2=0x7f2eb806d560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.324+0000 7f2ede439700 1 --2- 172.21.15.112:0/4123652931 >> [v2:172.21.15.112:6809/84512,v1:172.21.15.112:6811/84512] conn(0x7f2eb806b130 0x7f2eb806d560 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.324+0000 7f2ede439700 1 -- 172.21.15.112:0/4123652931 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2eb8061340 msgr2=0x7f2eb80637f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.324+0000 7f2ede439700 1 --2- 172.21.15.112:0/4123652931 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2eb8061340 0x7f2eb80637f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2ec8003080 tx=0x7f2ec8019000 comp rx=0 tx=0).stop 2023-01-24T03:30:40.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.324+0000 7f2ede439700 1 -- 172.21.15.112:0/4123652931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2ed8069360 msgr2=0x7f2ed8169990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.324+0000 7f2ede439700 1 --2- 172.21.15.112:0/4123652931 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2ed8069360 0x7f2ed8169990 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc00b490 tx=0x7f2ecc009060 comp rx=0 tx=0).stop 2023-01-24T03:30:40.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.324+0000 7f2edc9d6700 1 -- 172.21.15.112:0/4123652931 reap_dead start 2023-01-24T03:30:40.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.324+0000 7f365330b700 1 -- 172.21.15.112:0/3709419392 >> [v2:172.21.15.142:6824/84645,v1:172.21.15.142:6825/84645] conn(0x7f362c06afe0 msgr2=0x7f362c06d410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.324+0000 7f365330b700 1 --2- 172.21.15.112:0/3709419392 >> [v2:172.21.15.142:6824/84645,v1:172.21.15.142:6825/84645] conn(0x7f362c06afe0 0x7f362c06d410 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.324+0000 7f365330b700 1 -- 172.21.15.112:0/3709419392 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f362c0611f0 msgr2=0x7f362c0636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.324+0000 7f365330b700 1 --2- 172.21.15.112:0/3709419392 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f362c0611f0 0x7f362c0636a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f364c0c74b0 tx=0x7f363c008040 comp rx=0 tx=0).stop 2023-01-24T03:30:40.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.324+0000 7f365330b700 1 -- 172.21.15.112:0/3709419392 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3634004ad0 msgr2=0x7f364c0a54a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.324+0000 7f365330b700 1 --2- 172.21.15.112:0/3709419392 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3634004ad0 0x7f364c0a54a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f363400c080 tx=0x7f3648015000 comp rx=0 tx=0).stop 2023-01-24T03:30:40.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.324+0000 7f36518a8700 1 -- 172.21.15.112:0/3709419392 reap_dead start 2023-01-24T03:30:40.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.324+0000 7f365330b700 1 -- 172.21.15.112:0/3709419392 shutdown_connections 2023-01-24T03:30:40.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.324+0000 7f365330b700 1 -- 172.21.15.112:0/3709419392 >> 172.21.15.112:0/3709419392 conn(0x7f364c06b550 msgr2=0x7f364c06d9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:40.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.324+0000 7f365330b700 1 -- 172.21.15.112:0/3709419392 shutdown_connections 2023-01-24T03:30:40.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.324+0000 7f365330b700 1 -- 172.21.15.112:0/3709419392 wait complete. 2023-01-24T03:30:40.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.325+0000 7f2ede439700 1 -- 172.21.15.112:0/4123652931 shutdown_connections 2023-01-24T03:30:40.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.325+0000 7f2ede439700 1 -- 172.21.15.112:0/4123652931 >> 172.21.15.112:0/4123652931 conn(0x7f2ed806b550 msgr2=0x7f2ed8151150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:40.445 INFO:teuthology.orchestra.run.smithi112.stdout:55834574856 2023-01-24T03:30:40.445 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-01-24T03:30:40.446 INFO:teuthology.orchestra.run.smithi112.stdout:55834574856 2023-01-24T03:30:40.447 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-01-24T03:30:40.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.330+0000 7f7bfc9d2700 1 -- 172.21.15.112:0/1810870069 --> [v2:172.21.15.142:6816/84638,v1:172.21.15.142:6817/84638] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7bd8002bb0 con 0x7f7bd406afb0 2023-01-24T03:30:40.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.331+0000 7f2ede439700 1 -- 172.21.15.112:0/4123652931 shutdown_connections 2023-01-24T03:30:40.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.331+0000 7f2ede439700 1 -- 172.21.15.112:0/4123652931 wait complete. 2023-01-24T03:30:40.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.331+0000 7f7be67fc700 1 -- 172.21.15.112:0/1810870069 <== osd.5 v2:172.21.15.142:6816/84638 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7bd8002bb0 con 0x7f7bd406afb0 2023-01-24T03:30:40.450 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.331+0000 7f7bfc9d2700 1 -- 172.21.15.112:0/1810870069 >> [v2:172.21.15.142:6816/84638,v1:172.21.15.142:6817/84638] conn(0x7f7bd406afb0 msgr2=0x7f7bd406d3e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.450 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.331+0000 7f7bfc9d2700 1 --2- 172.21.15.112:0/1810870069 >> [v2:172.21.15.142:6816/84638,v1:172.21.15.142:6817/84638] conn(0x7f7bd406afb0 0x7f7bd406d3e0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.451 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.331+0000 7f7bfc9d2700 1 -- 172.21.15.112:0/1810870069 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7bd4061240 msgr2=0x7f7bd40636f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.451 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.331+0000 7f7bfc9d2700 1 --2- 172.21.15.112:0/1810870069 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7bd4061240 0x7f7bd40636f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7be80095d0 tx=0x7f7be8006ae0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.451 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.331+0000 7f7bfc9d2700 1 -- 172.21.15.112:0/1810870069 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7bf80c1960 msgr2=0x7f7bf8163d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.451 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.331+0000 7f7bfc9d2700 1 --2- 172.21.15.112:0/1810870069 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7bf80c1960 0x7f7bf8163d60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7bf000b810 tx=0x7f7bf00091a0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.451 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.331+0000 7f7bf6d9d700 1 -- 172.21.15.112:0/1810870069 reap_dead start 2023-01-24T03:30:40.452 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.332+0000 7f7bfc9d2700 1 -- 172.21.15.112:0/1810870069 shutdown_connections 2023-01-24T03:30:40.452 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.332+0000 7f7bfc9d2700 1 -- 172.21.15.112:0/1810870069 >> 172.21.15.112:0/1810870069 conn(0x7f7bf806b550 msgr2=0x7f7bf8158020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:40.452 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.332+0000 7f7bfc9d2700 1 -- 172.21.15.112:0/1810870069 shutdown_connections 2023-01-24T03:30:40.452 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.332+0000 7f7bfc9d2700 1 -- 172.21.15.112:0/1810870069 wait complete. 2023-01-24T03:30:40.453 INFO:teuthology.orchestra.run.smithi112.stdout:55834574856 2023-01-24T03:30:40.454 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-01-24T03:30:40.454 INFO:teuthology.orchestra.run.smithi112.stdout:55834574856 2023-01-24T03:30:40.455 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-01-24T03:30:40.456 INFO:teuthology.orchestra.run.smithi112.stdout:55834574856 2023-01-24T03:30:40.456 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-01-24T03:30:40.457 INFO:teuthology.orchestra.run.smithi112.stdout:55834574857 2023-01-24T03:30:40.457 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-01-24T03:30:40.458 INFO:teuthology.orchestra.run.smithi112.stdout:55834574857 2023-01-24T03:30:40.459 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-01-24T03:30:40.679 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.677+0000 7f4018eab700 1 Processor -- start 2023-01-24T03:30:40.681 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.677+0000 7f7eb2e19700 1 Processor -- start 2023-01-24T03:30:40.681 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.677+0000 7f7eb2e19700 1 -- start start 2023-01-24T03:30:40.682 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.678+0000 7f7eb2e19700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7eac0c1d70 0x7f7eac14cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.682 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.678+0000 7f7eb2e19700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f7eac153500 0x7f7eac14d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.682 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.678+0000 7f7eb2e19700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eac0c2b50 con 0x7f7eac0c4850 2023-01-24T03:30:40.683 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.678+0000 7f7eb2e19700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7eac156fc0 con 0x7f7eac153500 2023-01-24T03:30:40.683 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.678+0000 7f7eb2e19700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac0c22b0 con 0x7f7eac0c1d70 2023-01-24T03:30:40.684 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.677+0000 7f4018eab700 1 -- start start 2023-01-24T03:30:40.685 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.678+0000 7f4018eab700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f4014067fe0 0x7f40141559a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.685 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.678+0000 7f4018eab700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f4014069360 0x7f4014069b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.685 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.678+0000 7f4018eab700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f401406f050 0x7f401406a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.685 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.678+0000 7f4018eab700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40140bf6e0 con 0x7f4014069360 2023-01-24T03:30:40.685 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.678+0000 7f4018eab700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40140c3c80 con 0x7f4014067fe0 2023-01-24T03:30:40.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.678+0000 7f4018eab700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40140c2d90 con 0x7f401406f050 2023-01-24T03:30:40.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.678+0000 7f401259c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f4014067fe0 0x7f40141559a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.678+0000 7f401259c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f4014067fe0 0x7f40141559a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.112:43082/0 (socket says 172.21.15.112:43082) 2023-01-24T03:30:40.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.678+0000 7f401259c700 1 -- 172.21.15.112:0/1798983190 learned_addr learned my addr 172.21.15.112:0/1798983190 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:40.687 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.678+0000 7f7eabfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7eac0c1d70 0x7f7eac14cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.687 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.678+0000 7f7eabfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7eac0c1d70 0x7f7eac14cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.112:37888/0 (socket says 172.21.15.112:37888) 2023-01-24T03:30:40.687 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.678+0000 7f7eabfff700 1 -- 172.21.15.112:0/3408272826 learned_addr learned my addr 172.21.15.112:0/3408272826 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:40.688 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.679+0000 7fc14fe85700 1 Processor -- start 2023-01-24T03:30:40.688 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.679+0000 7f401259c700 1 -- 172.21.15.112:0/1798983190 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f401406f050 msgr2=0x7f401406a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:40.689 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.679+0000 7f401259c700 1 --2- 172.21.15.112:0/1798983190 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f401406f050 0x7f401406a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.689 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.679+0000 7f401259c700 1 -- 172.21.15.112:0/1798983190 >> v2:172.21.15.112:3300/0 conn(0x7f4014069360 msgr2=0x7f4014069b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:40.689 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.679+0000 7f401259c700 1 --2- 172.21.15.112:0/1798983190 >> v2:172.21.15.112:3300/0 conn(0x7f4014069360 0x7f4014069b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.689 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.679+0000 7f401259c700 1 -- 172.21.15.112:0/1798983190 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40140c3070 con 0x7f4014067fe0 2023-01-24T03:30:40.690 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.679+0000 7f7eabfff700 1 -- 172.21.15.112:0/3408272826 >> v2:172.21.15.112:3300/0 conn(0x7f7eac153500 msgr2=0x7f7eac14d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:40.690 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.679+0000 7f7eabfff700 1 --2- 172.21.15.112:0/3408272826 >> v2:172.21.15.112:3300/0 conn(0x7f7eac153500 0x7f7eac14d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.690 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.679+0000 7f7eabfff700 1 -- 172.21.15.112:0/3408272826 >> v1:172.21.15.112:6789/0 conn(0x7f7eac0c4850 legacy=0x7f7eac1573e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:40.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.679+0000 7f7eabfff700 1 -- 172.21.15.112:0/3408272826 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eac0c2590 con 0x7f7eac0c1d70 2023-01-24T03:30:40.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.679+0000 7f401259c700 1 --2- 172.21.15.112:0/1798983190 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4014067fe0 0x7f40141559a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f401406d8b0 tx=0x7f400c00ba50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=daa6ecb81f66479b server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.679+0000 7f7eabfff700 1 --2- 172.21.15.112:0/3408272826 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7eac0c1d70 0x7f7eac14cf20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7e98006fd0 tx=0x7f7e980090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58f936acbf05b6df server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.692 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.679+0000 7f401159a700 1 -- 172.21.15.112:0/1798983190 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c010070 con 0x7f4014067fe0 2023-01-24T03:30:40.693 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.679+0000 7f401159a700 1 -- 172.21.15.112:0/1798983190 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f400c00dd10 con 0x7f4014067fe0 2023-01-24T03:30:40.693 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.679+0000 7f401159a700 1 -- 172.21.15.112:0/1798983190 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c016630 con 0x7f4014067fe0 2023-01-24T03:30:40.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.679+0000 7f7eab7fe700 1 -- 172.21.15.112:0/3408272826 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e9800b690 con 0x7f7eac0c1d70 2023-01-24T03:30:40.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.679+0000 7f7eab7fe700 1 -- 172.21.15.112:0/3408272826 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e9800b810 con 0x7f7eac0c1d70 2023-01-24T03:30:40.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.679+0000 7f7eab7fe700 1 -- 172.21.15.112:0/3408272826 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e9800bbd0 con 0x7f7eac0c1d70 2023-01-24T03:30:40.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.680+0000 7f3523267700 1 Processor -- start 2023-01-24T03:30:40.695 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.679+0000 7fc14fe85700 1 -- start start 2023-01-24T03:30:40.695 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.680+0000 7fc14fe85700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fc14806a210 0x7fc1481559a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.696 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.680+0000 7fc14fe85700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fc148069360 0x7fc148151d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.696 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.680+0000 7fc14fe85700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1480c3c80 con 0x7fc148067fe0 2023-01-24T03:30:40.696 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.680+0000 7fc14fe85700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1480bf6e0 con 0x7fc148069360 2023-01-24T03:30:40.696 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.680+0000 7fc14fe85700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1480c2d90 con 0x7fc14806a210 2023-01-24T03:30:40.697 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.680+0000 7fc14d420700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fc14806a210 0x7fc1481559a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.697 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.680+0000 7fc14d420700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fc14806a210 0x7fc1481559a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.112:43092/0 (socket says 172.21.15.112:43092) 2023-01-24T03:30:40.697 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.680+0000 7fc14d420700 1 -- 172.21.15.112:0/2602008149 learned_addr learned my addr 172.21.15.112:0/2602008149 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:40.697 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.680+0000 7f4018eab700 1 -- 172.21.15.112:0/1798983190 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4014067fe0 msgr2=0x7f40141559a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.698 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.680+0000 7f4018eab700 1 --2- 172.21.15.112:0/1798983190 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4014067fe0 0x7f40141559a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f401406d8b0 tx=0x7f400c00ba50 comp rx=0 tx=0).stop 2023-01-24T03:30:40.698 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.680+0000 7fc14d420700 1 -- 172.21.15.112:0/2602008149 >> v2:172.21.15.112:3300/0 conn(0x7fc148069360 msgr2=0x7fc148151d20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-01-24T03:30:40.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.680+0000 7f7eb2e19700 1 -- 172.21.15.112:0/3408272826 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7eac0c1d70 msgr2=0x7f7eac14cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.680+0000 7f7eb2e19700 1 --2- 172.21.15.112:0/3408272826 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7eac0c1d70 0x7f7eac14cf20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7e98006fd0 tx=0x7f7e980090f0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.680+0000 7fc14d420700 1 --2- 172.21.15.112:0/2602008149 >> v2:172.21.15.112:3300/0 conn(0x7fc148069360 0x7fc148151d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.700 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.680+0000 7fc14d420700 1 -- 172.21.15.112:0/2602008149 >> v1:172.21.15.142:6789/0 conn(0x7fc148067fe0 legacy=0x7fc148069b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:40.700 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.680+0000 7fc14d420700 1 -- 172.21.15.112:0/2602008149 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1480c3070 con 0x7fc14806a210 2023-01-24T03:30:40.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.680+0000 7f4018eab700 1 -- 172.21.15.112:0/1798983190 shutdown_connections 2023-01-24T03:30:40.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.680+0000 7f4018eab700 1 --2- 172.21.15.112:0/1798983190 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f401406f050 0x7f401406a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.680+0000 7f4018eab700 1 --2- 172.21.15.112:0/1798983190 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4014067fe0 0x7f40141559a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.680+0000 7f4018eab700 1 --2- 172.21.15.112:0/1798983190 >> v2:172.21.15.112:3300/0 conn(0x7f4014069360 0x7f4014069b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.702 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.680+0000 7f4018eab700 1 -- 172.21.15.112:0/1798983190 >> 172.21.15.112:0/1798983190 conn(0x7f401406b550 msgr2=0x7f401406b950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:40.702 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.680+0000 7f7eb2e19700 1 -- 172.21.15.112:0/3408272826 shutdown_connections 2023-01-24T03:30:40.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.680+0000 7f7eb2e19700 1 --2- 172.21.15.112:0/3408272826 >> v2:172.21.15.112:3300/0 conn(0x7f7eac153500 0x7f7eac14d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.680+0000 7f7eb2e19700 1 --2- 172.21.15.112:0/3408272826 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7eac0c1d70 0x7f7eac14cf20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.680+0000 7f7eb2e19700 1 -- 172.21.15.112:0/3408272826 >> 172.21.15.112:0/3408272826 conn(0x7f7eac0689f0 msgr2=0x7f7eac0c8720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:40.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.681+0000 7f4018eab700 1 -- 172.21.15.112:0/1798983190 shutdown_connections 2023-01-24T03:30:40.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.681+0000 7fc14d420700 1 --2- 172.21.15.112:0/2602008149 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc14806a210 0x7fc1481559a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc1480c7e60 tx=0x7fc1440090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26b01ea5ce983752 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.681+0000 7f7eb2e19700 1 -- 172.21.15.112:0/3408272826 shutdown_connections 2023-01-24T03:30:40.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.681+0000 7fc14cc1f700 1 -- 172.21.15.112:0/2602008149 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14400c740 con 0x7fc14806a210 2023-01-24T03:30:40.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.681+0000 7fc14cc1f700 1 -- 172.21.15.112:0/2602008149 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc14400c8c0 con 0x7fc14806a210 2023-01-24T03:30:40.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.681+0000 7fc14cc1f700 1 -- 172.21.15.112:0/2602008149 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14400cc80 con 0x7fc14806a210 2023-01-24T03:30:40.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.680+0000 7f3523267700 1 -- start start 2023-01-24T03:30:40.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.681+0000 7f3523267700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f351c069360 0x7f351c1559a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.681+0000 7f3523267700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f351c0c3c80 con 0x7f351c067fe0 2023-01-24T03:30:40.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.681+0000 7f3523267700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f351c0c2d90 con 0x7f351c06a210 2023-01-24T03:30:40.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.681+0000 7f3523267700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f351c0bf6e0 con 0x7f351c069360 2023-01-24T03:30:40.707 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.681+0000 7f7eb2e19700 1 -- 172.21.15.112:0/3408272826 wait complete. 2023-01-24T03:30:40.707 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.682+0000 7fc14fe85700 1 -- 172.21.15.112:0/2602008149 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc14806a210 msgr2=0x7fc1481559a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.708 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.682+0000 7fc14fe85700 1 --2- 172.21.15.112:0/2602008149 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc14806a210 0x7fc1481559a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc1480c7e60 tx=0x7fc1440090f0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.708 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.682+0000 7f3521003700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f351c067fe0 0x7f351c069b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:59822/0 (socket says 172.21.15.112:59822) 2023-01-24T03:30:40.708 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.682+0000 7f3521003700 1 -- 172.21.15.112:0/1116059118 learned_addr learned my addr 172.21.15.112:0/1116059118 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:40.709 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.682+0000 7f3573870700 1 Processor -- start 2023-01-24T03:30:40.709 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.682+0000 7f3521804700 1 --2- 172.21.15.112:0/1116059118 >> v2:172.21.15.142:3300/0 conn(0x7f351c069360 0x7f351c1559a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.710 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.682+0000 7f350bfff700 1 -- 172.21.15.112:0/1116059118 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4129336752 0 0) 0x7f351c0c3c80 con 0x7f351c067fe0 2023-01-24T03:30:40.710 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.682+0000 7f350bfff700 1 -- 172.21.15.112:0/1116059118 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3504004500 con 0x7f351c067fe0 2023-01-24T03:30:40.710 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.682+0000 7f350bfff700 1 -- 172.21.15.112:0/1116059118 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3045362159 0 0) 0x7f3504004500 con 0x7f351c067fe0 2023-01-24T03:30:40.711 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.681+0000 7f4018eab700 1 -- 172.21.15.112:0/1798983190 wait complete. 2023-01-24T03:30:40.711 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.682+0000 7f4018eab700 1 Processor -- start 2023-01-24T03:30:40.711 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.682+0000 7f350bfff700 1 -- 172.21.15.112:0/1116059118 >> v2:172.21.15.142:3300/0 conn(0x7f351c069360 msgr2=0x7f351c1559a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:40.712 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.682+0000 7f350bfff700 1 --2- 172.21.15.112:0/1116059118 >> v2:172.21.15.142:3300/0 conn(0x7f351c069360 0x7f351c1559a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.712 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.682+0000 7f350bfff700 1 -- 172.21.15.112:0/1116059118 >> v1:172.21.15.142:6789/0 conn(0x7f351c06a210 legacy=0x7f351c151d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:40.712 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.682+0000 7f350bfff700 1 -- 172.21.15.112:0/1116059118 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f351c0c3070 con 0x7f351c067fe0 2023-01-24T03:30:40.712 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.682+0000 7f7eb2e19700 1 Processor -- start 2023-01-24T03:30:40.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.682+0000 7f3573870700 1 -- start start 2023-01-24T03:30:40.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.683+0000 7f3573870700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f356c068e60 0x7f356c069860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.683+0000 7f3573870700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f356c0c6d00 con 0x7f356c0af100 2023-01-24T03:30:40.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.683+0000 7f3573870700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f356c0c75a0 con 0x7f356c0bf210 2023-01-24T03:30:40.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.683+0000 7f3573870700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f356c0cc4b0 con 0x7f356c068e60 2023-01-24T03:30:40.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7f3570e0b700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f356c0bf210 0x7f356c0ae9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.112:45390/0 (socket says 172.21.15.112:45390) 2023-01-24T03:30:40.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7f3570e0b700 1 -- 172.21.15.112:0/1360220823 learned_addr learned my addr 172.21.15.112:0/1360220823 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:40.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.682+0000 7f350bfff700 1 -- 172.21.15.112:0/1116059118 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7f35140017f0 con 0x7f351c067fe0 2023-01-24T03:30:40.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.682+0000 7f350bfff700 1 -- 172.21.15.112:0/1116059118 >> v1:172.21.15.112:6789/0 conn(0x7f351c067fe0 legacy=0x7f351c069b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.682+0000 7f350bfff700 1 --2- 172.21.15.112:0/1116059118 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35040049f0 0x7f3504004de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.716 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.682+0000 7f350bfff700 1 --2- 172.21.15.112:0/1116059118 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f35040057a0 0x7f3504005c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.716 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.682+0000 7f350bfff700 1 --2- 172.21.15.112:0/1116059118 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f35040061c0 0x7f350400c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.716 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.682+0000 7f350bfff700 1 -- 172.21.15.112:0/1116059118 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351c0bf6e0 con 0x7f35040049f0 2023-01-24T03:30:40.716 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.682+0000 7f350bfff700 1 -- 172.21.15.112:0/1116059118 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350400d0f0 con 0x7f35040057a0 2023-01-24T03:30:40.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.682+0000 7f350bfff700 1 -- 172.21.15.112:0/1116059118 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f350400d2b0 con 0x7f35040061c0 2023-01-24T03:30:40.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.683+0000 7f3521003700 1 --2- 172.21.15.112:0/1116059118 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35040049f0 0x7f3504004de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.683+0000 7f3521003700 1 -- 172.21.15.112:0/1116059118 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f35040061c0 msgr2=0x7f350400c9f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-01-24T03:30:40.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.683+0000 7f3521003700 1 --2- 172.21.15.112:0/1116059118 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f35040061c0 0x7f350400c9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.683+0000 7f3521003700 1 -- 172.21.15.112:0/1116059118 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f35040057a0 msgr2=0x7f3504005c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-01-24T03:30:40.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.683+0000 7f3521003700 1 --2- 172.21.15.112:0/1116059118 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f35040057a0 0x7f3504005c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.683+0000 7f3521003700 1 -- 172.21.15.112:0/1116059118 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3504004500 con 0x7f35040049f0 2023-01-24T03:30:40.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.683+0000 7f3521003700 1 --2- 172.21.15.112:0/1116059118 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35040049f0 0x7f3504004de0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3514002180 tx=0x7f351c0bfc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2662fbb3bf930d3 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7f350bfff700 1 -- 172.21.15.112:0/1116059118 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f351400ce30 con 0x7f35040049f0 2023-01-24T03:30:40.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7f350bfff700 1 -- 172.21.15.112:0/1116059118 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3514005b20 con 0x7f35040049f0 2023-01-24T03:30:40.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7f350bfff700 1 -- 172.21.15.112:0/1116059118 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3514013770 con 0x7f35040049f0 2023-01-24T03:30:40.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7f3523267700 1 -- 172.21.15.112:0/1116059118 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35040049f0 msgr2=0x7f3504004de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7f3523267700 1 --2- 172.21.15.112:0/1116059118 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f35040049f0 0x7f3504004de0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3514002180 tx=0x7f351c0bfc80 comp rx=0 tx=0).stop 2023-01-24T03:30:40.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.682+0000 7f4018eab700 1 -- start start 2023-01-24T03:30:40.720 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7f4018eab700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4014067fe0 0x7f4014158600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.720 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7f4018eab700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4014069360 0x7f40140bb250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.720 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7f4018eab700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f401406f050 0x7f40140bb790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.720 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7f4018eab700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40140b7fd0 con 0x7f401406f050 2023-01-24T03:30:40.720 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7f4018eab700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40140bf300 con 0x7f4014069360 2023-01-24T03:30:40.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7f4018eab700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40140be5f0 con 0x7f4014067fe0 2023-01-24T03:30:40.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7f401259c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4014067fe0 0x7f4014158600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7f401259c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4014067fe0 0x7f4014158600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.112:37916/0 (socket says 172.21.15.112:37916) 2023-01-24T03:30:40.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7f401259c700 1 -- 172.21.15.112:0/2368568765 learned_addr learned my addr 172.21.15.112:0/2368568765 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:40.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7f4012d9d700 1 --2- 172.21.15.112:0/2368568765 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f401406f050 0x7f40140bb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7f4011d9b700 1 --2- 172.21.15.112:0/2368568765 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4014069360 0x7f40140bb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7f4012d9d700 1 -- 172.21.15.112:0/2368568765 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4014067fe0 msgr2=0x7f4014158600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7f4012d9d700 1 --2- 172.21.15.112:0/2368568765 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4014067fe0 0x7f4014158600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7f4012d9d700 1 -- 172.21.15.112:0/2368568765 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4014069360 msgr2=0x7f40140bb250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7f4012d9d700 1 --2- 172.21.15.112:0/2368568765 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4014069360 0x7f40140bb250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7f4012d9d700 1 -- 172.21.15.112:0/2368568765 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f400c00bcf0 con 0x7f401406f050 2023-01-24T03:30:40.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.683+0000 7fc2cad7b700 1 Processor -- start 2023-01-24T03:30:40.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.683+0000 7fc2cad7b700 1 -- start start 2023-01-24T03:30:40.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7fc2cad7b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2c40c40e0 0x7fc2c40c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7fc2cad7b700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fc2c40b7a60 0x7fc2c406b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7fc2cad7b700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2c406e6a0 con 0x7fc2c406d210 2023-01-24T03:30:40.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7fc2cad7b700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2c406ef20 con 0x7fc2c40b7a60 2023-01-24T03:30:40.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7fc2cad7b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c406e960 con 0x7fc2c40c40e0 2023-01-24T03:30:40.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7fc2c3fff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fc2c40b7a60 0x7fc2c406b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7fc2c3fff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fc2c40b7a60 0x7fc2c406b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50628/0 (socket says 172.21.15.112:50628) 2023-01-24T03:30:40.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7fc2c3fff700 1 -- 172.21.15.112:0/4248192309 learned_addr learned my addr 172.21.15.112:0/4248192309 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:40.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.683+0000 7f7eb2e19700 1 -- start start 2023-01-24T03:30:40.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7f7eb2e19700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7eac0c1d70 0x7f7eac14e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7f7eb2e19700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7eac0c4850 0x7f7eac0bfe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7f7eb2e19700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7eac153500 0x7f7eac0c03a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7f7eb2e19700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac1516d0 con 0x7f7eac153500 2023-01-24T03:30:40.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7f7eb2e19700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac0c7f50 con 0x7f7eac0c4850 2023-01-24T03:30:40.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7f7eabfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7eac0c4850 0x7f7eac0bfe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7f7eabfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7eac0c4850 0x7f7eac0bfe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.112:43112/0 (socket says 172.21.15.112:43112) 2023-01-24T03:30:40.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7f7eabfff700 1 -- 172.21.15.112:0/4283257724 learned_addr learned my addr 172.21.15.112:0/4283257724 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:40.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7f7eb13b6700 1 --2- 172.21.15.112:0/4283257724 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7eac153500 0x7f7eac0c03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7f7eb0bb5700 1 --2- 172.21.15.112:0/4283257724 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7eac0c1d70 0x7f7eac14e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7f3521804700 1 -- 172.21.15.112:0/1116059118 reap_dead start 2023-01-24T03:30:40.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7f401259c700 1 --2- 172.21.15.112:0/2368568765 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4014067fe0 0x7f4014158600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:30:40.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7f4011d9b700 1 --2- 172.21.15.112:0/2368568765 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4014069360 0x7f40140bb250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:30:40.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7f357160c700 1 --2- 172.21.15.112:0/1360220823 >> v2:172.21.15.142:3300/0 conn(0x7f356c068e60 0x7f356c069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7f3523267700 1 -- 172.21.15.112:0/1116059118 shutdown_connections 2023-01-24T03:30:40.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7f3523267700 1 -- 172.21.15.112:0/1116059118 >> 172.21.15.112:0/1116059118 conn(0x7f351c06b550 msgr2=0x7f351c06b950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:40.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7fc2c37fe700 1 -- 172.21.15.112:0/4248192309 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 488532263 0 0) 0x7fc2c406e6a0 con 0x7fc2c406d210 2023-01-24T03:30:40.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7fc2c37fe700 1 -- 172.21.15.112:0/4248192309 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2a40033f0 con 0x7fc2c406d210 2023-01-24T03:30:40.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7f355bfff700 1 -- 172.21.15.112:0/1360220823 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 731757768 0 0) 0x7f356c0c75a0 con 0x7f356c0bf210 2023-01-24T03:30:40.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7f3523267700 1 -- 172.21.15.112:0/1116059118 shutdown_connections 2023-01-24T03:30:40.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7f4012d9d700 1 --2- 172.21.15.112:0/2368568765 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f401406f050 0x7f40140bb790 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4008009410 tx=0x7f4008006e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7fc14fe85700 1 -- 172.21.15.112:0/2602008149 shutdown_connections 2023-01-24T03:30:40.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7fc14fe85700 1 --2- 172.21.15.112:0/2602008149 >> v2:172.21.15.112:3300/0 conn(0x7fc148069360 0x7fc148151d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7fc14fe85700 1 --2- 172.21.15.112:0/2602008149 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc14806a210 0x7fc1481559a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7f355bfff700 1 -- 172.21.15.112:0/1360220823 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35540033f0 con 0x7f356c0bf210 2023-01-24T03:30:40.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7fc14fe85700 1 -- 172.21.15.112:0/2602008149 >> 172.21.15.112:0/2602008149 conn(0x7fc14806b550 msgr2=0x7fc14806b950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:40.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7f3523267700 1 -- 172.21.15.112:0/1116059118 wait complete. 2023-01-24T03:30:40.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7fc14fe85700 1 -- 172.21.15.112:0/2602008149 shutdown_connections 2023-01-24T03:30:40.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7f357160c700 1 -- 172.21.15.112:0/1360220823 >> v1:172.21.15.142:6789/0 conn(0x7f356c0bf210 legacy=0x7f356c0ae9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7f357160c700 1 -- 172.21.15.112:0/1360220823 >> v1:172.21.15.112:6789/0 conn(0x7f356c0af100 legacy=0x7f356c15c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:40.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7f357160c700 1 -- 172.21.15.112:0/1360220823 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f356c0cc790 con 0x7f356c068e60 2023-01-24T03:30:40.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7f40027fc700 1 -- 172.21.15.112:0/2368568765 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4008010360 con 0x7f401406f050 2023-01-24T03:30:40.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7f40027fc700 1 -- 172.21.15.112:0/2368568765 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40080104c0 con 0x7f401406f050 2023-01-24T03:30:40.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7f40027fc700 1 -- 172.21.15.112:0/2368568765 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4008010880 con 0x7f401406f050 2023-01-24T03:30:40.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7fc14fe85700 1 -- 172.21.15.112:0/2602008149 wait complete. 2023-01-24T03:30:40.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.684+0000 7f7eb2e19700 1 -- 172.21.15.112:0/4283257724 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eac0c76b0 con 0x7f7eac0c1d70 2023-01-24T03:30:40.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7fc2c37fe700 1 -- 172.21.15.112:0/4248192309 <== mon.1 v1:172.21.15.142:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 221789449 0 0) 0x7fc2a40033f0 con 0x7fc2c406d210 2023-01-24T03:30:40.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7fc2c37fe700 1 -- 172.21.15.112:0/4248192309 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2c40c40e0 msgr2=0x7fc2c40c44b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-01-24T03:30:40.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7fc2c37fe700 1 --2- 172.21.15.112:0/4248192309 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2c40c40e0 0x7fc2c40c44b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7fc2c37fe700 1 -- 172.21.15.112:0/4248192309 >> v2:172.21.15.112:3300/0 conn(0x7fc2c40b7a60 msgr2=0x7fc2c406b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:40.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7fc2c37fe700 1 --2- 172.21.15.112:0/4248192309 >> v2:172.21.15.112:3300/0 conn(0x7fc2c40b7a60 0x7fc2c406b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7f7eabfff700 1 -- 172.21.15.112:0/4283257724 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7eac0c1d70 msgr2=0x7f7eac14e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7f7eabfff700 1 --2- 172.21.15.112:0/4283257724 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7eac0c1d70 0x7f7eac14e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7f7eabfff700 1 -- 172.21.15.112:0/4283257724 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7eac153500 msgr2=0x7f7eac0c03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7f7eabfff700 1 --2- 172.21.15.112:0/4283257724 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7eac153500 0x7f7eac0c03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7f7eabfff700 1 -- 172.21.15.112:0/4283257724 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e98009370 con 0x7f7eac0c4850 2023-01-24T03:30:40.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7fc2c37fe700 1 -- 172.21.15.112:0/4248192309 --> v1:172.21.15.142:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c406ec40 con 0x7fc2c406d210 2023-01-24T03:30:40.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7fc2c37fe700 1 -- 172.21.15.112:0/4248192309 <== mon.1 v1:172.21.15.142:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7fc2bc001470 con 0x7fc2c406d210 2023-01-24T03:30:40.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7fc2c37fe700 1 -- 172.21.15.112:0/4248192309 >> v1:172.21.15.142:6789/0 conn(0x7fc2c406d210 legacy=0x7fc2c4165a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7fc2c37fe700 1 --2- 172.21.15.112:0/4248192309 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2a4003960 0x7fc2a4003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7fc2c37fe700 1 --2- 172.21.15.112:0/4248192309 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2a4004690 0x7fc2a4004b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7f7eabfff700 1 --2- 172.21.15.112:0/4283257724 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7eac0c4850 0x7f7eac0bfe60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7eac0c23f0 tx=0x7f7e98017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7fc2c37fe700 1 --2- 172.21.15.112:0/4248192309 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc2a4005040 0x7fc2a400b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7fc2c37fe700 1 -- 172.21.15.112:0/4248192309 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c406ef20 con 0x7fc2a4005040 2023-01-24T03:30:40.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7fc2c37fe700 1 -- 172.21.15.112:0/4248192309 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c406e960 con 0x7fc2a4004690 2023-01-24T03:30:40.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.685+0000 7fc2c37fe700 1 -- 172.21.15.112:0/4248192309 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a400bf80 con 0x7fc2a4003960 2023-01-24T03:30:40.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.686+0000 7f7ea8ff9700 1 -- 172.21.15.112:0/4283257724 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e98000bc0 con 0x7f7eac0c4850 2023-01-24T03:30:40.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.686+0000 7f7ea8ff9700 1 -- 172.21.15.112:0/4283257724 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e9800b980 con 0x7f7eac0c4850 2023-01-24T03:30:40.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.686+0000 7f7ea8ff9700 1 -- 172.21.15.112:0/4283257724 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e98025420 con 0x7f7eac0c4850 2023-01-24T03:30:40.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.686+0000 7f357160c700 1 --2- 172.21.15.112:0/1360220823 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f356c068e60 0x7f356c069860 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f355c009760 tx=0x7f355c009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1bf26cbc21b8424f server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.686+0000 7fc2c9318700 1 --2- 172.21.15.112:0/4248192309 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc2a4005040 0x7fc2a400b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.686+0000 7fc2c9318700 1 -- 172.21.15.112:0/4248192309 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2a4003960 msgr2=0x7fc2a4003d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-01-24T03:30:40.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.686+0000 7fc2c9318700 1 --2- 172.21.15.112:0/4248192309 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2a4003960 0x7fc2a4003d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.686+0000 7fc2c9318700 1 -- 172.21.15.112:0/4248192309 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2a4004690 msgr2=0x7fc2a4004b00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-01-24T03:30:40.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.686+0000 7fc2c9318700 1 --2- 172.21.15.112:0/4248192309 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2a4004690 0x7fc2a4004b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.686+0000 7fc2c9318700 1 -- 172.21.15.112:0/4248192309 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c406ec40 con 0x7fc2a4005040 2023-01-24T03:30:40.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.686+0000 7f4018eab700 1 -- 172.21.15.112:0/2368568765 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40140bc3a0 con 0x7f401406f050 2023-01-24T03:30:40.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.686+0000 7f4018eab700 1 -- 172.21.15.112:0/2368568765 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40140be820 con 0x7f401406f050 2023-01-24T03:30:40.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.686+0000 7fc2c9318700 1 --2- 172.21.15.112:0/4248192309 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc2a4005040 0x7fc2a400b8e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc00c190 tx=0x7fc2bc007740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aef73da418057e11 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.687+0000 7f355bfff700 1 -- 172.21.15.112:0/1360220823 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355c011630 con 0x7f356c068e60 2023-01-24T03:30:40.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.687+0000 7f355bfff700 1 -- 172.21.15.112:0/1360220823 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f355c011790 con 0x7f356c068e60 2023-01-24T03:30:40.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.687+0000 7f355bfff700 1 -- 172.21.15.112:0/1360220823 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355c011b30 con 0x7f356c068e60 2023-01-24T03:30:40.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.687+0000 7f3523267700 1 Processor -- start 2023-01-24T03:30:40.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.687+0000 7fc14fe85700 1 Processor -- start 2023-01-24T03:30:40.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.687+0000 7f7eb2e19700 1 -- 172.21.15.112:0/4283257724 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eac151410 con 0x7f7eac0c4850 2023-01-24T03:30:40.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.687+0000 7f7eb2e19700 1 -- 172.21.15.112:0/4283257724 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eac0c1910 con 0x7f7eac0c4850 2023-01-24T03:30:40.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.687+0000 7fc14fe85700 1 -- start start 2023-01-24T03:30:40.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.688+0000 7f3573870700 1 -- 172.21.15.112:0/1360220823 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f356c068e60 msgr2=0x7f356c069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.688+0000 7f3573870700 1 --2- 172.21.15.112:0/1360220823 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f356c068e60 0x7f356c069860 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f355c009760 tx=0x7f355c009060 comp rx=0 tx=0).stop 2023-01-24T03:30:40.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.688+0000 7f3573870700 1 -- 172.21.15.112:0/1360220823 shutdown_connections 2023-01-24T03:30:40.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.688+0000 7f3573870700 1 --2- 172.21.15.112:0/1360220823 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f356c068e60 0x7f356c069860 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.688+0000 7f3573870700 1 -- 172.21.15.112:0/1360220823 >> 172.21.15.112:0/1360220823 conn(0x7f356c05e2b0 msgr2=0x7f356c0cba40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:40.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.688+0000 7f3573870700 1 -- 172.21.15.112:0/1360220823 shutdown_connections 2023-01-24T03:30:40.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.688+0000 7f3573870700 1 -- 172.21.15.112:0/1360220823 wait complete. 2023-01-24T03:30:40.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.687+0000 7f3523267700 1 -- start start 2023-01-24T03:30:40.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.688+0000 7f3523267700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f351c150590 0x7f351c169d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.689+0000 7f3523267700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f351c068420 0x7f351c0baae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.689+0000 7f3523267700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f351c0bb020 0x7f351c0bd730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.689+0000 7f3523267700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351c0ba210 con 0x7f351c068420 2023-01-24T03:30:40.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.689+0000 7f3523267700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351c0bcd20 con 0x7f351c0bb020 2023-01-24T03:30:40.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.689+0000 7f3523267700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f351c0bec00 con 0x7f351c150590 2023-01-24T03:30:40.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.689+0000 7f3521003700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f351c150590 0x7f351c169d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.689+0000 7f3521003700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f351c150590 0x7f351c169d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.112:37946/0 (socket says 172.21.15.112:37946) 2023-01-24T03:30:40.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.689+0000 7f3521003700 1 -- 172.21.15.112:0/3628150623 learned_addr learned my addr 172.21.15.112:0/3628150623 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:40.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.689+0000 7f3521804700 1 --2- 172.21.15.112:0/3628150623 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f351c0bb020 0x7f351c0bd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.688+0000 7f4018eab700 1 -- 172.21.15.112:0/2368568765 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f401405e610 con 0x7f401406f050 2023-01-24T03:30:40.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.689+0000 7f40027fc700 1 -- 172.21.15.112:0/2368568765 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f4008019030 con 0x7f401406f050 2023-01-24T03:30:40.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.688+0000 7fc2c37fe700 1 -- 172.21.15.112:0/4248192309 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc007970 con 0x7fc2a4005040 2023-01-24T03:30:40.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.688+0000 7fc2c37fe700 1 -- 172.21.15.112:0/4248192309 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2bc00ca40 con 0x7fc2a4005040 2023-01-24T03:30:40.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.688+0000 7fc2c37fe700 1 -- 172.21.15.112:0/4248192309 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc00cdc0 con 0x7fc2a4005040 2023-01-24T03:30:40.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.689+0000 7f7eb2e19700 1 -- 172.21.15.112:0/4283257724 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7eac154290 con 0x7f7eac0c4850 2023-01-24T03:30:40.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.688+0000 7fc14fe85700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc148067fe0 0x7fc1480baf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.688+0000 7fc14fe85700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc148069360 0x7fc1480bdbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.688+0000 7fc14fe85700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc14806a210 0x7fc1480be100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.688+0000 7fc14fe85700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc148169fe0 con 0x7fc14806a210 2023-01-24T03:30:40.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.688+0000 7fc14fe85700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1480ba3f0 con 0x7fc148069360 2023-01-24T03:30:40.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.688+0000 7fc14fe85700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc14816a4a0 con 0x7fc148067fe0 2023-01-24T03:30:40.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.688+0000 7fc14d420700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc148069360 0x7fc1480bdbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.688+0000 7fc14d420700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc148069360 0x7fc1480bdbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.112:43132/0 (socket says 172.21.15.112:43132) 2023-01-24T03:30:40.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.688+0000 7fc14d420700 1 -- 172.21.15.112:0/431155693 learned_addr learned my addr 172.21.15.112:0/431155693 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:40.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.689+0000 7f3521003700 1 -- 172.21.15.112:0/3628150623 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f351c0bb020 msgr2=0x7f351c0bd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.759 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.689+0000 7f3521003700 1 --2- 172.21.15.112:0/3628150623 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f351c0bb020 0x7f351c0bd730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.759 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.689+0000 7f3521003700 1 -- 172.21.15.112:0/3628150623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f351c068420 msgr2=0x7f351c0baae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.759 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.689+0000 7f3521003700 1 --2- 172.21.15.112:0/3628150623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f351c068420 0x7f351c0baae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.759 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.689+0000 7f3521003700 1 -- 172.21.15.112:0/3628150623 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f351c0bd2a0 con 0x7f351c150590 2023-01-24T03:30:40.760 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.689+0000 7fc2cad7b700 1 -- 172.21.15.112:0/4248192309 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc2a4005040 msgr2=0x7fc2a400b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.760 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.689+0000 7fc2cad7b700 1 --2- 172.21.15.112:0/4248192309 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc2a4005040 0x7fc2a400b8e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc00c190 tx=0x7fc2bc007740 comp rx=0 tx=0).stop 2023-01-24T03:30:40.760 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.689+0000 7fc2c9318700 1 -- 172.21.15.112:0/4248192309 reap_dead start 2023-01-24T03:30:40.761 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.689+0000 7fc14d420700 1 -- 172.21.15.112:0/431155693 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc148067fe0 msgr2=0x7fc1480baf00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-01-24T03:30:40.761 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.689+0000 7fc14d420700 1 --2- 172.21.15.112:0/431155693 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc148067fe0 0x7fc1480baf00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.761 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.689+0000 7fc14d420700 1 -- 172.21.15.112:0/431155693 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc14806a210 msgr2=0x7fc1480be100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.761 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.689+0000 7fc14d420700 1 --2- 172.21.15.112:0/431155693 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc14806a210 0x7fc1480be100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.762 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.689+0000 7fc14d420700 1 -- 172.21.15.112:0/431155693 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc144009370 con 0x7fc148069360 2023-01-24T03:30:40.762 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.689+0000 7fc2cad7b700 1 -- 172.21.15.112:0/4248192309 shutdown_connections 2023-01-24T03:30:40.762 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.689+0000 7fc2cad7b700 1 -- 172.21.15.112:0/4248192309 >> 172.21.15.112:0/4248192309 conn(0x7fc2c4066e10 msgr2=0x7fc2c4067210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:40.763 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.689+0000 7f40027fc700 1 --2- 172.21.15.112:0/2368568765 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3ff00612e0 0x7f3ff0063790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.763 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.689+0000 7f40027fc700 1 -- 172.21.15.112:0/2368568765 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f400807cb00 con 0x7f401406f050 2023-01-24T03:30:40.763 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.689+0000 7f3521003700 1 --2- 172.21.15.112:0/3628150623 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f351c150590 0x7f351c169d60 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f351400caf0 tx=0x7f3514019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.764 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.689+0000 7fc2cad7b700 1 -- 172.21.15.112:0/4248192309 shutdown_connections 2023-01-24T03:30:40.764 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.689+0000 7fc2cad7b700 1 -- 172.21.15.112:0/4248192309 wait complete. 2023-01-24T03:30:40.764 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.689+0000 7f35097fa700 1 -- 172.21.15.112:0/3628150623 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3514013dc0 con 0x7f351c150590 2023-01-24T03:30:40.765 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.690+0000 7f35097fa700 1 -- 172.21.15.112:0/3628150623 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35140070d0 con 0x7f351c150590 2023-01-24T03:30:40.765 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.690+0000 7f3523267700 1 -- 172.21.15.112:0/3628150623 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f351c0bee50 con 0x7f351c150590 2023-01-24T03:30:40.765 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.690+0000 7f3523267700 1 -- 172.21.15.112:0/3628150623 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f351c0bcf30 con 0x7f351c150590 2023-01-24T03:30:40.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.690+0000 7f35097fa700 1 -- 172.21.15.112:0/3628150623 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3514007450 con 0x7f351c150590 2023-01-24T03:30:40.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.690+0000 7f401259c700 1 --2- 172.21.15.112:0/2368568765 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3ff00612e0 0x7f3ff0063790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.767 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.690+0000 7f7ea8ff9700 1 -- 172.21.15.112:0/4283257724 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f7e9802c030 con 0x7f7eac0c4850 2023-01-24T03:30:40.767 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.690+0000 7f7ea8ff9700 1 --2- 172.21.15.112:0/4283257724 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7e94061310 0x7f7e940637c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.767 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.690+0000 7f3573870700 1 Processor -- start 2023-01-24T03:30:40.768 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.690+0000 7f7ea8ff9700 1 -- 172.21.15.112:0/4283257724 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f7e9807ef50 con 0x7f7eac0c4850 2023-01-24T03:30:40.768 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.690+0000 7f401259c700 1 --2- 172.21.15.112:0/2368568765 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3ff00612e0 0x7f3ff0063790 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f400c00ba90 tx=0x7f400c01c000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.769 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.691+0000 7f7eb0bb5700 1 --2- 172.21.15.112:0/4283257724 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7e94061310 0x7f7e940637c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.769 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.690+0000 7fc14d420700 1 --2- 172.21.15.112:0/431155693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc148069360 0x7fc1480bdbc0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc144014000 tx=0x7fc144015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.770 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.690+0000 7fc135ffb700 1 -- 172.21.15.112:0/431155693 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc144023420 con 0x7fc148069360 2023-01-24T03:30:40.770 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.690+0000 7fc135ffb700 1 -- 172.21.15.112:0/431155693 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc14400c050 con 0x7fc148069360 2023-01-24T03:30:40.770 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.690+0000 7fc135ffb700 1 -- 172.21.15.112:0/431155693 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc144023790 con 0x7fc148069360 2023-01-24T03:30:40.770 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.690+0000 7fc14fe85700 1 -- 172.21.15.112:0/431155693 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc14806eb00 con 0x7fc148069360 2023-01-24T03:30:40.770 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.690+0000 7fc14fe85700 1 -- 172.21.15.112:0/431155693 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1480bce00 con 0x7fc148069360 2023-01-24T03:30:40.771 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.691+0000 7fc2cad7b700 1 Processor -- start 2023-01-24T03:30:40.771 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.691+0000 7f3523267700 1 -- 172.21.15.112:0/3628150623 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34f8005130 con 0x7f351c150590 2023-01-24T03:30:40.772 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.691+0000 7fc2cad7b700 1 -- start start 2023-01-24T03:30:40.772 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.691+0000 7f7eb0bb5700 1 --2- 172.21.15.112:0/4283257724 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7e94061310 0x7f7e940637c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7ea000b890 tx=0x7f7ea0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.772 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.691+0000 7f4ca5983700 1 Processor -- start 2023-01-24T03:30:40.773 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.691+0000 7fc14fe85700 1 -- 172.21.15.112:0/431155693 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc14805e610 con 0x7fc148069360 2023-01-24T03:30:40.773 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.692+0000 7f4ca5983700 1 -- start start 2023-01-24T03:30:40.774 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.692+0000 7f35097fa700 1 -- 172.21.15.112:0/3628150623 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f35140077c0 con 0x7f351c150590 2023-01-24T03:30:40.774 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.691+0000 7fb1fadd9700 1 Processor -- start 2023-01-24T03:30:40.775 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.692+0000 7fb1fadd9700 1 -- start start 2023-01-24T03:30:40.775 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.692+0000 7f35097fa700 1 --2- 172.21.15.112:0/3628150623 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f34fc061240 0x7f34fc0636f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.776 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.691+0000 7f3573870700 1 -- start start 2023-01-24T03:30:40.776 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.692+0000 7f3573870700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f356c068e60 0x7f356c0c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.776 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.692+0000 7f3573870700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f356c0af100 0x7f356c15e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.776 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.692+0000 7f3573870700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f356c0bf210 0x7f356c15e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.777 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.692+0000 7f3573870700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f356c169a70 con 0x7f356c0af100 2023-01-24T03:30:40.777 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.692+0000 7f3573870700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f356c15fd10 con 0x7f356c0bf210 2023-01-24T03:30:40.777 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.692+0000 7f3573870700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f356c169d30 con 0x7f356c068e60 2023-01-24T03:30:40.777 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.692+0000 7f3570e0b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f356c0af100 0x7f356c15e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.777 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.692+0000 7f3570e0b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f356c0af100 0x7f356c15e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50656/0 (socket says 172.21.15.112:50656) 2023-01-24T03:30:40.778 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.692+0000 7f3570e0b700 1 -- 172.21.15.112:0/1901013592 learned_addr learned my addr 172.21.15.112:0/1901013592 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:40.778 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.692+0000 7f3571e0d700 1 --2- 172.21.15.112:0/1901013592 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f356c0bf210 0x7f356c15e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.778 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.692+0000 7f35097fa700 1 -- 172.21.15.112:0/3628150623 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f351407f500 con 0x7f351c150590 2023-01-24T03:30:40.778 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.693+0000 7f357160c700 1 --2- 172.21.15.112:0/1901013592 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f356c068e60 0x7f356c0c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.779 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.693+0000 7f4ca5983700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ca005e420 0x7f4ca0069890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.779 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.692+0000 7fc2cad7b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc2a4005040 0x7fc2c41647f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.779 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.692+0000 7fc2cad7b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2c40c6ed0 0x7fc2c40c61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.780 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.692+0000 7fc2cad7b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2c40b9520 0x7fc2c40c6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.780 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.692+0000 7fc2cad7b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c40bcda0 con 0x7fc2a4005040 2023-01-24T03:30:40.780 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.693+0000 7f4ca5983700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ca0155ab0 con 0x7f4ca00694c0 2023-01-24T03:30:40.781 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.693+0000 7f4ca5983700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ca00c70e0 con 0x7f4ca0068d20 2023-01-24T03:30:40.781 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.693+0000 7f4ca5983700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca0155550 con 0x7f4ca005e420 2023-01-24T03:30:40.781 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.692+0000 7fc2cad7b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c4172230 con 0x7fc2c40c6ed0 2023-01-24T03:30:40.781 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.692+0000 7fc2cad7b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c40b9be0 con 0x7fc2c40b9520 2023-01-24T03:30:40.782 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.693+0000 7f3570e0b700 1 -- 172.21.15.112:0/1901013592 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f356c068e60 msgr2=0x7f356c0c9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.782 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.693+0000 7f3570e0b700 1 --2- 172.21.15.112:0/1901013592 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f356c068e60 0x7f356c0c9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.783 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.693+0000 7f3570e0b700 1 -- 172.21.15.112:0/1901013592 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f356c0bf210 msgr2=0x7f356c15e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.783 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.693+0000 7f3570e0b700 1 --2- 172.21.15.112:0/1901013592 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f356c0bf210 0x7f356c15e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.783 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.693+0000 7f3570e0b700 1 -- 172.21.15.112:0/1901013592 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f355c0092a0 con 0x7f356c0af100 2023-01-24T03:30:40.784 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.693+0000 7f357160c700 1 --2- 172.21.15.112:0/1901013592 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f356c068e60 0x7f356c0c9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-01-24T03:30:40.784 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.693+0000 7f3520802700 1 --2- 172.21.15.112:0/3628150623 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f34fc061240 0x7f34fc0636f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.785 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.693+0000 7fc2c3fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2c40c6ed0 0x7fc2c40c61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.785 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.693+0000 7fc2c3fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2c40c6ed0 0x7fc2c40c61f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.112:43160/0 (socket says 172.21.15.112:43160) 2023-01-24T03:30:40.785 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.693+0000 7fc2c3fff700 1 -- 172.21.15.112:0/2312300046 learned_addr learned my addr 172.21.15.112:0/2312300046 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:40.785 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.693+0000 7fc2c8b17700 1 --2- 172.21.15.112:0/2312300046 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc2a4005040 0x7fc2c41647f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.786 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.693+0000 7f3570e0b700 1 --2- 172.21.15.112:0/1901013592 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f356c0af100 0x7f356c15e0b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f356400c7c0 tx=0x7f35640015b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.786 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.694+0000 7f3520802700 1 --2- 172.21.15.112:0/3628150623 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f34fc061240 0x7f34fc0636f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f351c163c30 tx=0x7f35180079a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.787 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.693+0000 7f4c967fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ca005e420 0x7f4ca0069890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.787 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.693+0000 7f4c9f7fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f4ca0068d20 0x7f4ca00ae9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.112:45450/0 (socket says 172.21.15.112:45450) 2023-01-24T03:30:40.787 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.693+0000 7f4c9f7fe700 1 -- 172.21.15.112:0/1438642758 learned_addr learned my addr 172.21.15.112:0/1438642758 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:40.787 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.693+0000 7fc2c3fff700 1 -- 172.21.15.112:0/2312300046 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2c40b9520 msgr2=0x7fc2c40c6730 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-01-24T03:30:40.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.693+0000 7fc2c3fff700 1 --2- 172.21.15.112:0/2312300046 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2c40b9520 0x7fc2c40c6730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.693+0000 7fc2c3fff700 1 -- 172.21.15.112:0/2312300046 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc2a4005040 msgr2=0x7fc2c41647f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.693+0000 7fc2c3fff700 1 --2- 172.21.15.112:0/2312300046 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc2a4005040 0x7fc2c41647f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.693+0000 7fc2c3fff700 1 -- 172.21.15.112:0/2312300046 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c4165110 con 0x7fc2c40c6ed0 2023-01-24T03:30:40.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.693+0000 7fc2c8b17700 1 --2- 172.21.15.112:0/2312300046 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc2a4005040 0x7fc2c41647f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:30:40.789 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.693+0000 7fc2c3fff700 1 --2- 172.21.15.112:0/2312300046 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2c40c6ed0 0x7fc2c40c61f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc2b4009f20 tx=0x7fc2b4008850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.789 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.693+0000 7fb1fadd9700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb1f40bf210 0x7fb1f40ae9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.789 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.693+0000 7fb1fadd9700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fb1f4069230 0x7fb1f40aef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.790 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.693+0000 7fb1fadd9700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1f40c6d00 con 0x7fb1f4068e60 2023-01-24T03:30:40.790 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.693+0000 7fb1fadd9700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1f40c75a0 con 0x7fb1f4069230 2023-01-24T03:30:40.790 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.693+0000 7fb1fadd9700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1f40cc4b0 con 0x7fb1f40bf210 2023-01-24T03:30:40.790 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.693+0000 7fb1f8b75700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fb1f4068e60 0x7fb1f4069860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:59888/0 (socket says 172.21.15.112:59888) 2023-01-24T03:30:40.791 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.693+0000 7fb1f8b75700 1 -- 172.21.15.112:0/4036210562 learned_addr learned my addr 172.21.15.112:0/4036210562 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:40.791 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.693+0000 7fb1f3fff700 1 --2- 172.21.15.112:0/4036210562 >> v2:172.21.15.142:3300/0 conn(0x7fb1f40bf210 0x7fb1f40ae9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.791 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.693+0000 7fb1f9376700 1 --2- 172.21.15.112:0/4036210562 >> v2:172.21.15.112:3300/0 conn(0x7fb1f4069230 0x7fb1f40aef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.791 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.694+0000 7fb1f3fff700 1 -- 172.21.15.112:0/4036210562 >> v2:172.21.15.112:3300/0 conn(0x7fb1f4069230 msgr2=0x7fb1f40aef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:40.792 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.694+0000 7fb1f3fff700 1 --2- 172.21.15.112:0/4036210562 >> v2:172.21.15.112:3300/0 conn(0x7fb1f4069230 0x7fb1f40aef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.792 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.694+0000 7fb1f3fff700 1 -- 172.21.15.112:0/4036210562 >> v1:172.21.15.112:6789/0 conn(0x7fb1f4068e60 legacy=0x7fb1f4069860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.792 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.694+0000 7fb1f3fff700 1 -- 172.21.15.112:0/4036210562 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f40cc790 con 0x7fb1f40bf210 2023-01-24T03:30:40.792 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.694+0000 7f4c967fc700 1 -- 172.21.15.112:0/1438642758 >> v1:172.21.15.142:6789/0 conn(0x7f4ca0068d20 legacy=0x7f4ca00ae9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.793 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.694+0000 7f4c967fc700 1 -- 172.21.15.112:0/1438642758 >> v1:172.21.15.112:6789/0 conn(0x7f4ca00694c0 legacy=0x7f4ca00c68d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-01-24T03:30:40.793 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.694+0000 7f4c967fc700 1 -- 172.21.15.112:0/1438642758 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca00c73c0 con 0x7f4ca005e420 2023-01-24T03:30:40.793 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.694+0000 7fb1f3fff700 1 --2- 172.21.15.112:0/4036210562 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1f40bf210 0x7fb1f40ae9f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb1f40c81d0 tx=0x7fb1e8009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1174425789b541f server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.794 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.694+0000 7f4c967fc700 1 --2- 172.21.15.112:0/1438642758 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ca005e420 0x7f4ca0069890 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4c88009790 tx=0x7f4c88009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a77cd43b4abe0ee server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.794 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.695+0000 7fc2c0ff9700 1 -- 172.21.15.112:0/2312300046 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b400edb0 con 0x7fc2c40c6ed0 2023-01-24T03:30:40.794 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.695+0000 7f7ea8ff9700 1 -- 172.21.15.112:0/4283257724 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f7e9804ed20 con 0x7f7eac0c4850 2023-01-24T03:30:40.795 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.695+0000 7fb1f37fe700 1 -- 172.21.15.112:0/4036210562 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e8011630 con 0x7fb1f40bf210 2023-01-24T03:30:40.795 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.695+0000 7fc2c0ff9700 1 -- 172.21.15.112:0/2312300046 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b4008b70 con 0x7fc2c40c6ed0 2023-01-24T03:30:40.796 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.695+0000 7fc2cad7b700 1 -- 172.21.15.112:0/2312300046 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2c41724b0 con 0x7fc2c40c6ed0 2023-01-24T03:30:40.796 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.695+0000 7fb1f37fe700 1 -- 172.21.15.112:0/4036210562 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1e8011790 con 0x7fb1f40bf210 2023-01-24T03:30:40.796 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.695+0000 7fc2c0ff9700 1 -- 172.21.15.112:0/2312300046 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b4009030 con 0x7fc2c40c6ed0 2023-01-24T03:30:40.797 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.695+0000 7fb1f37fe700 1 -- 172.21.15.112:0/4036210562 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e8011b50 con 0x7fb1f40bf210 2023-01-24T03:30:40.797 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.695+0000 7fc2cad7b700 1 -- 172.21.15.112:0/2312300046 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2c4172a70 con 0x7fc2c40c6ed0 2023-01-24T03:30:40.798 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.696+0000 7fb1fadd9700 1 -- 172.21.15.112:0/4036210562 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1f40bf210 msgr2=0x7fb1f40ae9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.798 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.696+0000 7fb1fadd9700 1 --2- 172.21.15.112:0/4036210562 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1f40bf210 0x7fb1f40ae9f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb1f40c81d0 tx=0x7fb1e8009060 comp rx=0 tx=0).stop 2023-01-24T03:30:40.798 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.696+0000 7fb1fadd9700 1 -- 172.21.15.112:0/4036210562 shutdown_connections 2023-01-24T03:30:40.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.696+0000 7fb1fadd9700 1 --2- 172.21.15.112:0/4036210562 >> v2:172.21.15.112:3300/0 conn(0x7fb1f4069230 0x7fb1f40aef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.696+0000 7fb1fadd9700 1 --2- 172.21.15.112:0/4036210562 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1f40bf210 0x7fb1f40ae9f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.696+0000 7fb1fadd9700 1 -- 172.21.15.112:0/4036210562 >> 172.21.15.112:0/4036210562 conn(0x7fb1f405e2b0 msgr2=0x7fb1f40cba40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:40.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.696+0000 7fb1fadd9700 1 -- 172.21.15.112:0/4036210562 shutdown_connections 2023-01-24T03:30:40.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.696+0000 7fb1fadd9700 1 -- 172.21.15.112:0/4036210562 wait complete. 2023-01-24T03:30:40.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.697+0000 7fc135ffb700 1 -- 172.21.15.112:0/431155693 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fc14400e040 con 0x7fc148069360 2023-01-24T03:30:40.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.697+0000 7fc2cad7b700 1 -- 172.21.15.112:0/2312300046 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2c40683d0 con 0x7fc2c40c6ed0 2023-01-24T03:30:40.801 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.697+0000 7f35597fa700 1 -- 172.21.15.112:0/1901013592 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3564005d30 con 0x7f356c0af100 2023-01-24T03:30:40.801 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.697+0000 7f35597fa700 1 -- 172.21.15.112:0/1901013592 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35640070f0 con 0x7f356c0af100 2023-01-24T03:30:40.801 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.697+0000 7f4c9e7fc700 1 -- 172.21.15.112:0/1438642758 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c88011630 con 0x7f4ca005e420 2023-01-24T03:30:40.802 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.697+0000 7f3573870700 1 -- 172.21.15.112:0/1901013592 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f356c16a010 con 0x7f356c0af100 2023-01-24T03:30:40.802 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.697+0000 7f3573870700 1 -- 172.21.15.112:0/1901013592 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f356c16a5d0 con 0x7f356c0af100 2023-01-24T03:30:40.802 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.697+0000 7f40027fc700 1 -- 172.21.15.112:0/2368568765 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f400804c870 con 0x7f401406f050 2023-01-24T03:30:40.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.697+0000 7f4c9e7fc700 1 -- 172.21.15.112:0/1438642758 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c88011790 con 0x7f4ca005e420 2023-01-24T03:30:40.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.697+0000 7f4c9e7fc700 1 -- 172.21.15.112:0/1438642758 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c88011b50 con 0x7f4ca005e420 2023-01-24T03:30:40.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.698+0000 7f35097fa700 1 -- 172.21.15.112:0/3628150623 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f351404f340 con 0x7f351c150590 2023-01-24T03:30:40.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.698+0000 7fb1fadd9700 1 Processor -- start 2023-01-24T03:30:40.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.697+0000 7fc135ffb700 1 --2- 172.21.15.112:0/431155693 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc128061340 0x7fc1280637f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.805 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.697+0000 7fc135ffb700 1 -- 172.21.15.112:0/431155693 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fc14401f070 con 0x7fc148069360 2023-01-24T03:30:40.805 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.698+0000 7fc14dc21700 1 --2- 172.21.15.112:0/431155693 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc128061340 0x7fc1280637f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.806 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.698+0000 7f4ca5983700 1 -- 172.21.15.112:0/1438642758 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ca005e420 msgr2=0x7f4ca0069890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.806 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.698+0000 7f4ca5983700 1 --2- 172.21.15.112:0/1438642758 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ca005e420 0x7f4ca0069890 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4c88009790 tx=0x7f4c88009060 comp rx=0 tx=0).stop 2023-01-24T03:30:40.806 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.698+0000 7f4ca5983700 1 -- 172.21.15.112:0/1438642758 shutdown_connections 2023-01-24T03:30:40.806 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.698+0000 7f4ca5983700 1 --2- 172.21.15.112:0/1438642758 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ca005e420 0x7f4ca0069890 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.698+0000 7f4ca5983700 1 -- 172.21.15.112:0/1438642758 >> 172.21.15.112:0/1438642758 conn(0x7f4ca005d830 msgr2=0x7f4ca005dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:40.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.697+0000 7f35597fa700 1 -- 172.21.15.112:0/1901013592 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35640074b0 con 0x7f356c0af100 2023-01-24T03:30:40.808 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.698+0000 7f3573870700 1 -- 172.21.15.112:0/1901013592 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3548005130 con 0x7f356c0af100 2023-01-24T03:30:40.808 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.698+0000 7fb1fadd9700 1 -- start start 2023-01-24T03:30:40.808 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.699+0000 7fb1fadd9700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1f4068e60 0x7fb1f40c14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.699+0000 7fb1fadd9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb1f4069230 0x7fb1f40c3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.699+0000 7fb1fadd9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1f40bf210 0x7fb1f40c43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.699+0000 7fb1fadd9700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f40c52f0 con 0x7fb1f40bf210 2023-01-24T03:30:40.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.699+0000 7fb1fadd9700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f40c0290 con 0x7fb1f4068e60 2023-01-24T03:30:40.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.699+0000 7fb1fadd9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f40c39e0 con 0x7fb1f4069230 2023-01-24T03:30:40.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.699+0000 7f35597fa700 1 -- 172.21.15.112:0/1901013592 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f3564010070 con 0x7f356c0af100 2023-01-24T03:30:40.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.699+0000 7f4ca5983700 1 -- 172.21.15.112:0/1438642758 shutdown_connections 2023-01-24T03:30:40.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.699+0000 7f4ca5983700 1 -- 172.21.15.112:0/1438642758 wait complete. 2023-01-24T03:30:40.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.699+0000 7fb1f8b75700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1f4068e60 0x7fb1f40c14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.699+0000 7fb1f8b75700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1f4068e60 0x7fb1f40c14c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.112:43172/0 (socket says 172.21.15.112:43172) 2023-01-24T03:30:40.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.699+0000 7fb1f3fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb1f4069230 0x7fb1f40c3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.699+0000 7fb1f8b75700 1 -- 172.21.15.112:0/4037677652 learned_addr learned my addr 172.21.15.112:0/4037677652 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:40.812 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.700+0000 7f35597fa700 1 --2- 172.21.15.112:0/1901013592 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f354c0611c0 0x7f354c063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.812 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.700+0000 7f35597fa700 1 -- 172.21.15.112:0/1901013592 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f356407d2f0 con 0x7f356c0af100 2023-01-24T03:30:40.813 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.700+0000 7fb1f8b75700 1 -- 172.21.15.112:0/4037677652 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb1f4069230 msgr2=0x7fb1f40c3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.813 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.700+0000 7fb1f8b75700 1 --2- 172.21.15.112:0/4037677652 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb1f4069230 0x7fb1f40c3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.813 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.700+0000 7fb1f8b75700 1 -- 172.21.15.112:0/4037677652 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1f40bf210 msgr2=0x7fb1f40c43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.813 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.700+0000 7fb1f8b75700 1 --2- 172.21.15.112:0/4037677652 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1f40bf210 0x7fb1f40c43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.813 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.700+0000 7fb1f8b75700 1 -- 172.21.15.112:0/4037677652 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e80092a0 con 0x7fb1f4068e60 2023-01-24T03:30:40.814 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.700+0000 7fb1f3fff700 1 --2- 172.21.15.112:0/4037677652 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb1f4069230 0x7fb1f40c3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:30:40.814 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.700+0000 7fb1f8b75700 1 --2- 172.21.15.112:0/4037677652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1f4068e60 0x7fb1f40c14c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb1e000c4f0 tx=0x7fb1e0007db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.814 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.701+0000 7fc14dc21700 1 --2- 172.21.15.112:0/431155693 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc128061340 0x7fc1280637f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc140006fd0 tx=0x7fc140006e50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.815 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.701+0000 7f4ca5983700 1 Processor -- start 2023-01-24T03:30:40.815 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.702+0000 7fc135ffb700 1 -- 172.21.15.112:0/431155693 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fc14404ed40 con 0x7fc148069360 2023-01-24T03:30:40.816 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.701+0000 7f357160c700 1 --2- 172.21.15.112:0/1901013592 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f354c0611c0 0x7f354c063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.816 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.701+0000 7f357160c700 1 --2- 172.21.15.112:0/1901013592 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f354c0611c0 0x7f354c063670 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f355c011fd0 tx=0x7f355c016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.817 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.702+0000 7f4ca5983700 1 -- start start 2023-01-24T03:30:40.817 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.703+0000 7f4ca5983700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ca005e420 0x7f4ca0152b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.817 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.703+0000 7f4ca5983700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ca0068d20 0x7f4ca00cae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.818 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.703+0000 7f4ca5983700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ca00694c0 0x7f4ca0151090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.818 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.703+0000 7f4ca5983700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca0176120 con 0x7f4ca005e420 2023-01-24T03:30:40.818 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.703+0000 7f4ca5983700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca01532c0 con 0x7f4ca0068d20 2023-01-24T03:30:40.818 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.703+0000 7f4ca5983700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca01524d0 con 0x7f4ca00694c0 2023-01-24T03:30:40.819 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.703+0000 7f4c9effd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ca005e420 0x7f4ca0152b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.819 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.703+0000 7f4c9effd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ca005e420 0x7f4ca0152b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50688/0 (socket says 172.21.15.112:50688) 2023-01-24T03:30:40.819 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.703+0000 7f4c9effd700 1 -- 172.21.15.112:0/747456138 learned_addr learned my addr 172.21.15.112:0/747456138 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:40.819 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.703+0000 7f4c967fc700 1 --2- 172.21.15.112:0/747456138 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ca0068d20 0x7f4ca00cae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.819 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.703+0000 7f4c9f7fe700 1 --2- 172.21.15.112:0/747456138 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ca00694c0 0x7f4ca0151090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.819 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.703+0000 7f4c967fc700 1 -- 172.21.15.112:0/747456138 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ca00694c0 msgr2=0x7f4ca0151090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.820 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.703+0000 7f4c967fc700 1 --2- 172.21.15.112:0/747456138 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ca00694c0 0x7f4ca0151090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.820 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.703+0000 7f4c967fc700 1 -- 172.21.15.112:0/747456138 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ca005e420 msgr2=0x7f4ca0152b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:40.820 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.703+0000 7f4c967fc700 1 --2- 172.21.15.112:0/747456138 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ca005e420 0x7f4ca0152b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:40.820 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.703+0000 7f4c967fc700 1 -- 172.21.15.112:0/747456138 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c880092a0 con 0x7f4ca0068d20 2023-01-24T03:30:40.820 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.703+0000 7f4c9effd700 1 --2- 172.21.15.112:0/747456138 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ca005e420 0x7f4ca0152b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:30:40.821 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.704+0000 7f4c967fc700 1 --2- 172.21.15.112:0/747456138 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ca0068d20 0x7f4ca00cae30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4c88015000 tx=0x7f4c88016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.821 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.704+0000 7f4c977fe700 1 -- 172.21.15.112:0/747456138 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c8801ad60 con 0x7f4ca0068d20 2023-01-24T03:30:40.821 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.704+0000 7f4ca5983700 1 -- 172.21.15.112:0/747456138 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ca0152770 con 0x7f4ca0068d20 2023-01-24T03:30:40.822 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.704+0000 7fb1f0ff9700 1 -- 172.21.15.112:0/4037677652 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e00072e0 con 0x7fb1f4068e60 2023-01-24T03:30:40.822 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.704+0000 7fb1f0ff9700 1 -- 172.21.15.112:0/4037677652 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1e0007440 con 0x7fb1f4068e60 2023-01-24T03:30:40.822 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.704+0000 7fb1f0ff9700 1 -- 172.21.15.112:0/4037677652 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e0005780 con 0x7fb1f4068e60 2023-01-24T03:30:40.822 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.705+0000 7f4ca5983700 1 -- 172.21.15.112:0/747456138 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ca0175ad0 con 0x7f4ca0068d20 2023-01-24T03:30:40.823 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.705+0000 7f4c977fe700 1 -- 172.21.15.112:0/747456138 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c8801f070 con 0x7f4ca0068d20 2023-01-24T03:30:40.823 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.705+0000 7f4c977fe700 1 -- 172.21.15.112:0/747456138 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c8801b690 con 0x7f4ca0068d20 2023-01-24T03:30:40.823 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.706+0000 7f35597fa700 1 -- 172.21.15.112:0/1901013592 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f356404d1b0 con 0x7f356c0af100 2023-01-24T03:30:40.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.706+0000 7f4ca5983700 1 -- 172.21.15.112:0/747456138 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ca0176290 con 0x7f4ca0068d20 2023-01-24T03:30:40.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.706+0000 7f4c977fe700 1 -- 172.21.15.112:0/747456138 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f4c88017020 con 0x7f4ca0068d20 2023-01-24T03:30:40.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.706+0000 7fb1fadd9700 1 -- 172.21.15.112:0/4037677652 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1f4169970 con 0x7fb1f4068e60 2023-01-24T03:30:40.825 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.706+0000 7fb1fadd9700 1 -- 172.21.15.112:0/4037677652 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1f4169f10 con 0x7fb1f4068e60 2023-01-24T03:30:40.825 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.707+0000 7f4c977fe700 1 --2- 172.21.15.112:0/747456138 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f4c840611c0 0x7f4c84063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.825 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.707+0000 7f4c977fe700 1 -- 172.21.15.112:0/747456138 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f4c8807ede0 con 0x7f4ca0068d20 2023-01-24T03:30:40.826 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.707+0000 7f4c9effd700 1 --2- 172.21.15.112:0/747456138 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f4c840611c0 0x7f4c84063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.826 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.701+0000 7fc2c0ff9700 1 -- 172.21.15.112:0/2312300046 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fc2b40093a0 con 0x7fc2c40c6ed0 2023-01-24T03:30:40.827 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.702+0000 7fc2c0ff9700 1 --2- 172.21.15.112:0/2312300046 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc2a80611f0 0x7fc2a80636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.827 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.702+0000 7fc2c0ff9700 1 -- 172.21.15.112:0/2312300046 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fc2b4014030 con 0x7fc2c40c6ed0 2023-01-24T03:30:40.827 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.702+0000 7fc2c8b17700 1 --2- 172.21.15.112:0/2312300046 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc2a80611f0 0x7fc2a80636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.827 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.703+0000 7fc2c8b17700 1 --2- 172.21.15.112:0/2312300046 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc2a80611f0 0x7fc2a80636a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc2c40c8610 tx=0x7fc2ac008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.827 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.706+0000 7fc2c0ff9700 1 -- 172.21.15.112:0/2312300046 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fc2b404eaa0 con 0x7fc2c40c6ed0 2023-01-24T03:30:40.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.708+0000 7f4c9effd700 1 --2- 172.21.15.112:0/747456138 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f4c840611c0 0x7f4c84063670 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4c90009d10 tx=0x7f4c90006f10 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.711+0000 7f4c977fe700 1 -- 172.21.15.112:0/747456138 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f4c8804ebc0 con 0x7f4ca0068d20 2023-01-24T03:30:40.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.712+0000 7fb1f0ff9700 1 -- 172.21.15.112:0/4037677652 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fb1e0019030 con 0x7fb1f4068e60 2023-01-24T03:30:40.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.713+0000 7fb1f0ff9700 1 --2- 172.21.15.112:0/4037677652 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb1dc061240 0x7fb1dc0636f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:40.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.713+0000 7fb1f0ff9700 1 -- 172.21.15.112:0/4037677652 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fb1e007c950 con 0x7fb1f4068e60 2023-01-24T03:30:40.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.719+0000 7fb1f3fff700 1 --2- 172.21.15.112:0/4037677652 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb1dc061240 0x7fb1dc0636f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:40.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.720+0000 7fb1f3fff700 1 --2- 172.21.15.112:0/4037677652 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb1dc061240 0x7fb1dc0636f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb1e8009fd0 tx=0x7fb1e8017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:40.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.729+0000 7fb1fadd9700 1 -- 172.21.15.112:0/4037677652 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1f40c3cc0 con 0x7fb1f4068e60 2023-01-24T03:30:40.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:40.734+0000 7fb1f0ff9700 1 -- 172.21.15.112:0/4037677652 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fb1e004a1d0 con 0x7fb1f4068e60 2023-01-24T03:30:41.078 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.075+0000 7f3523267700 1 -- 172.21.15.112:0/3628150623 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f34f80053e0 con 0x7f351c150590 2023-01-24T03:30:41.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.075+0000 7fc2cad7b700 1 -- 172.21.15.112:0/2312300046 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fc2c406a830 con 0x7fc2c40c6ed0 2023-01-24T03:30:41.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.076+0000 7f7eb2e19700 1 -- 172.21.15.112:0/4283257724 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f7eac05e820 con 0x7f7eac0c4850 2023-01-24T03:30:41.082 INFO:teuthology.orchestra.run.smithi112.stdout:55834574854 2023-01-24T03:30:41.082 INFO:teuthology.orchestra.run.smithi112.stdout:55834574854 2023-01-24T03:30:41.083 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.077+0000 7fc2c0ff9700 1 -- 172.21.15.112:0/2312300046 <== mon.1 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc2b40520c0 con 0x7fc2c40c6ed0 2023-01-24T03:30:41.083 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.079+0000 7f7ea8ff9700 1 -- 172.21.15.112:0/4283257724 <== mon.1 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7e98015020 con 0x7f7eac0c4850 2023-01-24T03:30:41.084 INFO:teuthology.orchestra.run.smithi112.stdout:55834574854 2023-01-24T03:30:41.085 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.076+0000 7f35097fa700 1 -- 172.21.15.112:0/3628150623 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3514052960 con 0x7f351c150590 2023-01-24T03:30:41.086 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.079+0000 7f3523267700 1 -- 172.21.15.112:0/3628150623 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f34fc061240 msgr2=0x7f34fc0636f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:41.086 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.079+0000 7f3523267700 1 --2- 172.21.15.112:0/3628150623 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f34fc061240 0x7f34fc0636f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f351c163c30 tx=0x7f35180079a0 comp rx=0 tx=0).stop 2023-01-24T03:30:41.086 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.079+0000 7f3523267700 1 -- 172.21.15.112:0/3628150623 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f351c150590 msgr2=0x7f351c169d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:41.086 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.079+0000 7f3523267700 1 --2- 172.21.15.112:0/3628150623 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f351c150590 0x7f351c169d60 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f351400caf0 tx=0x7f3514019000 comp rx=0 tx=0).stop 2023-01-24T03:30:41.087 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.083+0000 7f3523267700 1 -- 172.21.15.112:0/3628150623 shutdown_connections 2023-01-24T03:30:41.087 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.083+0000 7f3523267700 1 --2- 172.21.15.112:0/3628150623 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f34fc061240 0x7f34fc0636f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:41.087 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.083+0000 7f3523267700 1 --2- 172.21.15.112:0/3628150623 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f351c0bb020 0x7f351c0bd730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:41.087 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.083+0000 7f3523267700 1 --2- 172.21.15.112:0/3628150623 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f351c150590 0x7f351c169d60 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:41.087 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.083+0000 7f3523267700 1 --2- 172.21.15.112:0/3628150623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f351c068420 0x7f351c0baae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:41.088 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.083+0000 7f3523267700 1 -- 172.21.15.112:0/3628150623 >> 172.21.15.112:0/3628150623 conn(0x7f351c06b550 msgr2=0x7f351c153d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:41.088 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.083+0000 7f4018eab700 1 -- 172.21.15.112:0/2368568765 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f401404b340 con 0x7f401406f050 2023-01-24T03:30:41.088 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.083+0000 7f7eb2e19700 1 -- 172.21.15.112:0/4283257724 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7e94061310 msgr2=0x7f7e940637c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:41.089 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.083+0000 7f3523267700 1 -- 172.21.15.112:0/3628150623 shutdown_connections 2023-01-24T03:30:41.089 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.083+0000 7f7eb2e19700 1 --2- 172.21.15.112:0/4283257724 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7e94061310 0x7f7e940637c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7ea000b890 tx=0x7f7ea0008040 comp rx=0 tx=0).stop 2023-01-24T03:30:41.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.083+0000 7f7eb2e19700 1 -- 172.21.15.112:0/4283257724 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7eac0c4850 msgr2=0x7f7eac0bfe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:41.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.083+0000 7f7eb2e19700 1 --2- 172.21.15.112:0/4283257724 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7eac0c4850 0x7f7eac0bfe60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7eac0c23f0 tx=0x7f7e98017000 comp rx=0 tx=0).stop 2023-01-24T03:30:41.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.083+0000 7fc2cad7b700 1 -- 172.21.15.112:0/2312300046 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc2a80611f0 msgr2=0x7fc2a80636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:41.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.083+0000 7fc2cad7b700 1 --2- 172.21.15.112:0/2312300046 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc2a80611f0 0x7fc2a80636a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc2c40c8610 tx=0x7fc2ac008040 comp rx=0 tx=0).stop 2023-01-24T03:30:41.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.083+0000 7fc2cad7b700 1 -- 172.21.15.112:0/2312300046 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2c40c6ed0 msgr2=0x7fc2c40c61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:41.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.083+0000 7fc2cad7b700 1 --2- 172.21.15.112:0/2312300046 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2c40c6ed0 0x7fc2c40c61f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc2b4009f20 tx=0x7fc2b4008850 comp rx=0 tx=0).stop 2023-01-24T03:30:41.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.083+0000 7fc2cad7b700 1 -- 172.21.15.112:0/2312300046 shutdown_connections 2023-01-24T03:30:41.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.083+0000 7fc2cad7b700 1 --2- 172.21.15.112:0/2312300046 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc2a80611f0 0x7fc2a80636a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:41.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.083+0000 7fc2cad7b700 1 --2- 172.21.15.112:0/2312300046 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc2a4005040 0x7fc2c41647f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:41.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.083+0000 7fc2cad7b700 1 --2- 172.21.15.112:0/2312300046 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc2c40b9520 0x7fc2c40c6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:41.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.083+0000 7fc2cad7b700 1 --2- 172.21.15.112:0/2312300046 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc2c40c6ed0 0x7fc2c40c61f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:41.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.083+0000 7fc2cad7b700 1 -- 172.21.15.112:0/2312300046 >> 172.21.15.112:0/2312300046 conn(0x7fc2c4066e10 msgr2=0x7fc2c40bf240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:41.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.083+0000 7fc2cad7b700 1 -- 172.21.15.112:0/2312300046 shutdown_connections 2023-01-24T03:30:41.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.083+0000 7fc2cad7b700 1 -- 172.21.15.112:0/2312300046 wait complete. 2023-01-24T03:30:41.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.083+0000 7f40027fc700 1 -- 172.21.15.112:0/2368568765 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4008017090 con 0x7f401406f050 2023-01-24T03:30:41.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.083+0000 7f7eb2e19700 1 -- 172.21.15.112:0/4283257724 shutdown_connections 2023-01-24T03:30:41.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.083+0000 7f7eb2e19700 1 --2- 172.21.15.112:0/4283257724 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7e94061310 0x7f7e940637c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:41.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.083+0000 7f7eb2e19700 1 --2- 172.21.15.112:0/4283257724 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7eac153500 0x7f7eac0c03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:41.095 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.083+0000 7f7eb2e19700 1 --2- 172.21.15.112:0/4283257724 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7eac0c1d70 0x7f7eac14e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:41.095 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.083+0000 7f7eb2e19700 1 --2- 172.21.15.112:0/4283257724 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7eac0c4850 0x7f7eac0bfe60 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:41.095 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.083+0000 7f7eb2e19700 1 -- 172.21.15.112:0/4283257724 >> 172.21.15.112:0/4283257724 conn(0x7f7eac0689f0 msgr2=0x7f7eac06ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:41.095 INFO:teuthology.orchestra.run.smithi112.stdout:55834574854 2023-01-24T03:30:41.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.084+0000 7f7eb2e19700 1 -- 172.21.15.112:0/4283257724 shutdown_connections 2023-01-24T03:30:41.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.084+0000 7f7eb2e19700 1 -- 172.21.15.112:0/4283257724 wait complete. 2023-01-24T03:30:41.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.086+0000 7f3523267700 1 -- 172.21.15.112:0/3628150623 wait complete. 2023-01-24T03:30:41.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.087+0000 7f4018eab700 1 -- 172.21.15.112:0/2368568765 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3ff00612e0 msgr2=0x7f3ff0063790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:41.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.087+0000 7f4018eab700 1 --2- 172.21.15.112:0/2368568765 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3ff00612e0 0x7f3ff0063790 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f400c00ba90 tx=0x7f400c01c000 comp rx=0 tx=0).stop 2023-01-24T03:30:41.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.087+0000 7f4018eab700 1 -- 172.21.15.112:0/2368568765 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f401406f050 msgr2=0x7f40140bb790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:41.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.087+0000 7f4018eab700 1 --2- 172.21.15.112:0/2368568765 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f401406f050 0x7f40140bb790 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4008009410 tx=0x7f4008006e10 comp rx=0 tx=0).stop 2023-01-24T03:30:41.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.087+0000 7f4018eab700 1 -- 172.21.15.112:0/2368568765 shutdown_connections 2023-01-24T03:30:41.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.087+0000 7f4018eab700 1 --2- 172.21.15.112:0/2368568765 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3ff00612e0 0x7f3ff0063790 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:41.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.087+0000 7f4018eab700 1 --2- 172.21.15.112:0/2368568765 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f401406f050 0x7f40140bb790 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:41.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.087+0000 7f4018eab700 1 --2- 172.21.15.112:0/2368568765 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4014067fe0 0x7f4014158600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:41.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.087+0000 7f4018eab700 1 --2- 172.21.15.112:0/2368568765 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4014069360 0x7f40140bb250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:41.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.087+0000 7f4018eab700 1 -- 172.21.15.112:0/2368568765 >> 172.21.15.112:0/2368568765 conn(0x7f401406b550 msgr2=0x7f4014152bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:41.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.087+0000 7f4018eab700 1 -- 172.21.15.112:0/2368568765 shutdown_connections 2023-01-24T03:30:41.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.087+0000 7f4018eab700 1 -- 172.21.15.112:0/2368568765 wait complete. 2023-01-24T03:30:41.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.087+0000 7f4ca5983700 1 -- 172.21.15.112:0/747456138 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f4ca006f3a0 con 0x7f4ca0068d20 2023-01-24T03:30:41.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.088+0000 7f4c977fe700 1 -- 172.21.15.112:0/747456138 <== mon.1 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4c880521e0 con 0x7f4ca0068d20 2023-01-24T03:30:41.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.089+0000 7f3573870700 1 -- 172.21.15.112:0/1901013592 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f35480053e0 con 0x7f356c0af100 2023-01-24T03:30:41.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.089+0000 7f35597fa700 1 -- 172.21.15.112:0/1901013592 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f35640507d0 con 0x7f356c0af100 2023-01-24T03:30:41.101 INFO:teuthology.orchestra.run.smithi112.stdout:55834574854 2023-01-24T03:30:41.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.093+0000 7f3573870700 1 -- 172.21.15.112:0/1901013592 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f354c0611c0 msgr2=0x7f354c063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:41.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.093+0000 7f3573870700 1 --2- 172.21.15.112:0/1901013592 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f354c0611c0 0x7f354c063670 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f355c011fd0 tx=0x7f355c016000 comp rx=0 tx=0).stop 2023-01-24T03:30:41.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.093+0000 7f3573870700 1 -- 172.21.15.112:0/1901013592 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f356c0af100 msgr2=0x7f356c15e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:41.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.093+0000 7f3573870700 1 --2- 172.21.15.112:0/1901013592 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f356c0af100 0x7f356c15e0b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f356400c7c0 tx=0x7f35640015b0 comp rx=0 tx=0).stop 2023-01-24T03:30:41.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.094+0000 7f3573870700 1 -- 172.21.15.112:0/1901013592 shutdown_connections 2023-01-24T03:30:41.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.094+0000 7f3573870700 1 --2- 172.21.15.112:0/1901013592 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f354c0611c0 0x7f354c063670 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:41.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.094+0000 7f3573870700 1 --2- 172.21.15.112:0/1901013592 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f356c0bf210 0x7f356c15e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:41.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.094+0000 7f3573870700 1 --2- 172.21.15.112:0/1901013592 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f356c068e60 0x7f356c0c9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:41.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.094+0000 7f3573870700 1 --2- 172.21.15.112:0/1901013592 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f356c0af100 0x7f356c15e0b0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:41.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.094+0000 7f3573870700 1 -- 172.21.15.112:0/1901013592 >> 172.21.15.112:0/1901013592 conn(0x7f356c05e2b0 msgr2=0x7f356c069230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:41.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.094+0000 7f3573870700 1 -- 172.21.15.112:0/1901013592 shutdown_connections 2023-01-24T03:30:41.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.094+0000 7f3573870700 1 -- 172.21.15.112:0/1901013592 wait complete. 2023-01-24T03:30:41.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.090+0000 7fc14fe85700 1 -- 172.21.15.112:0/431155693 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fc14804b340 con 0x7fc148069360 2023-01-24T03:30:41.105 INFO:teuthology.orchestra.run.smithi112.stdout:55834574854 2023-01-24T03:30:41.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.097+0000 7fc135ffb700 1 -- 172.21.15.112:0/431155693 <== mon.1 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc144052360 con 0x7fc148069360 2023-01-24T03:30:41.106 INFO:teuthology.orchestra.run.smithi112.stdout:55834574855 2023-01-24T03:30:41.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.102+0000 7f4ca5983700 1 -- 172.21.15.112:0/747456138 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f4c840611c0 msgr2=0x7f4c84063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:41.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.102+0000 7f4ca5983700 1 --2- 172.21.15.112:0/747456138 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f4c840611c0 0x7f4c84063670 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4c90009d10 tx=0x7f4c90006f10 comp rx=0 tx=0).stop 2023-01-24T03:30:41.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.102+0000 7f4ca5983700 1 -- 172.21.15.112:0/747456138 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ca0068d20 msgr2=0x7f4ca00cae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:41.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.102+0000 7f4ca5983700 1 --2- 172.21.15.112:0/747456138 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ca0068d20 0x7f4ca00cae30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4c88015000 tx=0x7f4c88016000 comp rx=0 tx=0).stop 2023-01-24T03:30:41.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.102+0000 7fc14fe85700 1 -- 172.21.15.112:0/431155693 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc128061340 msgr2=0x7fc1280637f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:41.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.102+0000 7fc14fe85700 1 --2- 172.21.15.112:0/431155693 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc128061340 0x7fc1280637f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc140006fd0 tx=0x7fc140006e50 comp rx=0 tx=0).stop 2023-01-24T03:30:41.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.102+0000 7fc14fe85700 1 -- 172.21.15.112:0/431155693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc148069360 msgr2=0x7fc1480bdbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:41.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.102+0000 7fc14fe85700 1 --2- 172.21.15.112:0/431155693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc148069360 0x7fc1480bdbc0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc144014000 tx=0x7fc144015000 comp rx=0 tx=0).stop 2023-01-24T03:30:41.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.103+0000 7fc14fe85700 1 -- 172.21.15.112:0/431155693 shutdown_connections 2023-01-24T03:30:41.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.103+0000 7fc14fe85700 1 --2- 172.21.15.112:0/431155693 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc128061340 0x7fc1280637f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:41.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.103+0000 7fc14fe85700 1 --2- 172.21.15.112:0/431155693 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc14806a210 0x7fc1480be100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:41.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.103+0000 7fc14fe85700 1 --2- 172.21.15.112:0/431155693 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc148067fe0 0x7fc1480baf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:41.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.103+0000 7fc14fe85700 1 --2- 172.21.15.112:0/431155693 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc148069360 0x7fc1480bdbc0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:41.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.103+0000 7fc14fe85700 1 -- 172.21.15.112:0/431155693 >> 172.21.15.112:0/431155693 conn(0x7fc14806b550 msgr2=0x7fc14806e280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:41.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.103+0000 7fc14fe85700 1 -- 172.21.15.112:0/431155693 shutdown_connections 2023-01-24T03:30:41.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.103+0000 7fc14fe85700 1 -- 172.21.15.112:0/431155693 wait complete. 2023-01-24T03:30:41.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.104+0000 7f4ca5983700 1 -- 172.21.15.112:0/747456138 shutdown_connections 2023-01-24T03:30:41.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.104+0000 7f4ca5983700 1 --2- 172.21.15.112:0/747456138 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f4c840611c0 0x7f4c84063670 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:41.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.104+0000 7f4ca5983700 1 --2- 172.21.15.112:0/747456138 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4ca005e420 0x7f4ca0152b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:41.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.104+0000 7f4ca5983700 1 --2- 172.21.15.112:0/747456138 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4ca00694c0 0x7f4ca0151090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:41.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.104+0000 7f4ca5983700 1 --2- 172.21.15.112:0/747456138 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ca0068d20 0x7f4ca00cae30 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:41.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.104+0000 7f4ca5983700 1 -- 172.21.15.112:0/747456138 >> 172.21.15.112:0/747456138 conn(0x7f4ca005d830 msgr2=0x7f4ca00b0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:41.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.104+0000 7fb1fadd9700 1 -- 172.21.15.112:0/4037677652 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fb1f406f3a0 con 0x7fb1f4068e60 2023-01-24T03:30:41.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.104+0000 7f4ca5983700 1 -- 172.21.15.112:0/747456138 shutdown_connections 2023-01-24T03:30:41.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.104+0000 7f4ca5983700 1 -- 172.21.15.112:0/747456138 wait complete. 2023-01-24T03:30:41.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.104+0000 7fb1f0ff9700 1 -- 172.21.15.112:0/4037677652 <== mon.1 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb1e0017030 con 0x7fb1f4068e60 2023-01-24T03:30:41.114 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574854 for osd.6 2023-01-24T03:30:41.115 INFO:teuthology.orchestra.run.smithi112.stdout:55834574855 2023-01-24T03:30:41.116 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574854 for osd.1 2023-01-24T03:30:41.117 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574854 for osd.7 2023-01-24T03:30:41.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.110+0000 7fb1fadd9700 1 -- 172.21.15.112:0/4037677652 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb1dc061240 msgr2=0x7fb1dc0636f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:41.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.110+0000 7fb1fadd9700 1 --2- 172.21.15.112:0/4037677652 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb1dc061240 0x7fb1dc0636f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb1e8009fd0 tx=0x7fb1e8017000 comp rx=0 tx=0).stop 2023-01-24T03:30:41.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.110+0000 7fb1fadd9700 1 -- 172.21.15.112:0/4037677652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1f4068e60 msgr2=0x7fb1f40c14c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:41.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.110+0000 7fb1fadd9700 1 --2- 172.21.15.112:0/4037677652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1f4068e60 0x7fb1f40c14c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb1e000c4f0 tx=0x7fb1e0007db0 comp rx=0 tx=0).stop 2023-01-24T03:30:41.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.111+0000 7fb1fadd9700 1 -- 172.21.15.112:0/4037677652 shutdown_connections 2023-01-24T03:30:41.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.111+0000 7fb1fadd9700 1 --2- 172.21.15.112:0/4037677652 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb1dc061240 0x7fb1dc0636f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:41.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.111+0000 7fb1fadd9700 1 --2- 172.21.15.112:0/4037677652 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1f40bf210 0x7fb1f40c43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:41.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.111+0000 7fb1fadd9700 1 --2- 172.21.15.112:0/4037677652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb1f4068e60 0x7fb1f40c14c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:41.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.111+0000 7fb1fadd9700 1 --2- 172.21.15.112:0/4037677652 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb1f4069230 0x7fb1f40c3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:41.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.111+0000 7fb1fadd9700 1 -- 172.21.15.112:0/4037677652 >> 172.21.15.112:0/4037677652 conn(0x7fb1f405e2b0 msgr2=0x7fb1f40c0470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:41.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.111+0000 7fb1fadd9700 1 -- 172.21.15.112:0/4037677652 shutdown_connections 2023-01-24T03:30:41.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:41.111+0000 7fb1fadd9700 1 -- 172.21.15.112:0/4037677652 wait complete. 2023-01-24T03:30:41.120 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574854 for osd.2 2023-01-24T03:30:41.126 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574854 for osd.3 2023-01-24T03:30:41.129 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574855 for osd.5 2023-01-24T03:30:41.130 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574854 for osd.4 2023-01-24T03:30:41.131 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574855 for osd.0 2023-01-24T03:30:42.115 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-01-24T03:30:42.116 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-01-24T03:30:42.117 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-01-24T03:30:42.122 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-01-24T03:30:42.128 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-01-24T03:30:42.129 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-01-24T03:30:42.130 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-01-24T03:30:42.133 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-01-24T03:30:42.310 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.304+0000 7efd220b1700 1 Processor -- start 2023-01-24T03:30:42.310 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.304+0000 7efd220b1700 1 -- start start 2023-01-24T03:30:42.311 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.305+0000 7efd220b1700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7efd1c0b7a60 0x7efd1c0c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.311 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.305+0000 7efd220b1700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd1c06e6a0 con 0x7efd1c0c40e0 2023-01-24T03:30:42.311 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.305+0000 7efd220b1700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd1c06ef20 con 0x7efd1c06b740 2023-01-24T03:30:42.311 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.305+0000 7efd220b1700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd1c06e960 con 0x7efd1c0b7a60 2023-01-24T03:30:42.312 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.306+0000 7efd1b7fe700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7efd1c0c40e0 0x7efd1c06b030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:59918/0 (socket says 172.21.15.112:59918) 2023-01-24T03:30:42.312 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.306+0000 7efd1b7fe700 1 -- 172.21.15.112:0/919084158 learned_addr learned my addr 172.21.15.112:0/919084158 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:42.312 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.306+0000 7efd1affd700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7efd1c0b7a60 0x7efd1c0c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.312 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.307+0000 7efd1affd700 1 -- 172.21.15.112:0/919084158 >> v1:172.21.15.142:6789/0 conn(0x7efd1c06b740 legacy=0x7efd1c0c1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.312 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.307+0000 7efd1affd700 1 -- 172.21.15.112:0/919084158 >> v1:172.21.15.112:6789/0 conn(0x7efd1c0c40e0 legacy=0x7efd1c06b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.313 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.307+0000 7efd1affd700 1 -- 172.21.15.112:0/919084158 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd1c06ec40 con 0x7efd1c0b7a60 2023-01-24T03:30:42.313 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.307+0000 7efd1a7fc700 1 -- 172.21.15.112:0/919084158 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2440832163 0 0) 0x7efd1c06ef20 con 0x7efd1c06b740 2023-01-24T03:30:42.313 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.308+0000 7fb7b89d5700 1 Processor -- start 2023-01-24T03:30:42.314 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.308+0000 7efd1affd700 1 --2- 172.21.15.112:0/919084158 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efd1c0b7a60 0x7efd1c0c44b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7efd10009bb0 tx=0x7efd10009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6bae4e8b52cd31a server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:42.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.308+0000 7fb7b89d5700 1 -- start start 2023-01-24T03:30:42.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.309+0000 7fb7b89d5700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb7b415cf30 0x7fb7b415d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.309+0000 7fb7b89d5700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fb7b40c4140 0x7fb7b415d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.309+0000 7fb7b89d5700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7b40680c0 con 0x7fb7b406d850 2023-01-24T03:30:42.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.309+0000 7fb7b89d5700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7b40c3d50 con 0x7fb7b40c4140 2023-01-24T03:30:42.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.309+0000 7fb7b89d5700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7b40683a0 con 0x7fb7b415cf30 2023-01-24T03:30:42.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.310+0000 7fb7b259c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb7b415cf30 0x7fb7b415d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.310+0000 7fb7b259c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb7b415cf30 0x7fb7b415d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.112:43198/0 (socket says 172.21.15.112:43198) 2023-01-24T03:30:42.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.310+0000 7fb7b259c700 1 -- 172.21.15.112:0/193982346 learned_addr learned my addr 172.21.15.112:0/193982346 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:42.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.310+0000 7fb7b1d9b700 1 --2- 172.21.15.112:0/193982346 >> v2:172.21.15.112:3300/0 conn(0x7fb7b40c4140 0x7fb7b415d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.310+0000 7fb7b259c700 1 -- 172.21.15.112:0/193982346 >> v2:172.21.15.112:3300/0 conn(0x7fb7b40c4140 msgr2=0x7fb7b415d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:42.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.310+0000 7fb7b259c700 1 --2- 172.21.15.112:0/193982346 >> v2:172.21.15.112:3300/0 conn(0x7fb7b40c4140 0x7fb7b415d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.310+0000 7fb7b259c700 1 -- 172.21.15.112:0/193982346 >> v1:172.21.15.142:6789/0 conn(0x7fb7b406d850 legacy=0x7fb7b41536e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:42.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.310+0000 7fb7b259c700 1 -- 172.21.15.112:0/193982346 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7b4068680 con 0x7fb7b415cf30 2023-01-24T03:30:42.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.310+0000 7fb7b259c700 1 --2- 172.21.15.112:0/193982346 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7b415cf30 0x7fb7b415d300 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb7b406a710 tx=0x7fb7a80090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1aaa40c4d45d2d4 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:42.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.311+0000 7fb7b159a700 1 -- 172.21.15.112:0/193982346 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7a8011630 con 0x7fb7b415cf30 2023-01-24T03:30:42.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.311+0000 7fb7b159a700 1 -- 172.21.15.112:0/193982346 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7a8011790 con 0x7fb7b415cf30 2023-01-24T03:30:42.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.311+0000 7fb7b159a700 1 -- 172.21.15.112:0/193982346 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7a8011b50 con 0x7fb7b415cf30 2023-01-24T03:30:42.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.312+0000 7fb7b89d5700 1 -- 172.21.15.112:0/193982346 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7b415cf30 msgr2=0x7fb7b415d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.312+0000 7fb7b89d5700 1 --2- 172.21.15.112:0/193982346 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7b415cf30 0x7fb7b415d300 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb7b406a710 tx=0x7fb7a80090f0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.311+0000 7efd1a7fc700 1 -- 172.21.15.112:0/919084158 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd1000b8e0 con 0x7efd1c0b7a60 2023-01-24T03:30:42.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.311+0000 7efd1a7fc700 1 -- 172.21.15.112:0/919084158 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd1000ba40 con 0x7efd1c0b7a60 2023-01-24T03:30:42.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.311+0000 7efd1a7fc700 1 -- 172.21.15.112:0/919084158 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd1000bde0 con 0x7efd1c0b7a60 2023-01-24T03:30:42.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.312+0000 7efd220b1700 1 -- 172.21.15.112:0/919084158 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efd1c0b7a60 msgr2=0x7efd1c0c44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.312+0000 7efd220b1700 1 --2- 172.21.15.112:0/919084158 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efd1c0b7a60 0x7efd1c0c44b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7efd10009bb0 tx=0x7efd10009060 comp rx=0 tx=0).stop 2023-01-24T03:30:42.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.312+0000 7efd220b1700 1 -- 172.21.15.112:0/919084158 shutdown_connections 2023-01-24T03:30:42.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.312+0000 7efd220b1700 1 --2- 172.21.15.112:0/919084158 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efd1c0b7a60 0x7efd1c0c44b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.312+0000 7efd220b1700 1 -- 172.21.15.112:0/919084158 >> 172.21.15.112:0/919084158 conn(0x7efd1c066e10 msgr2=0x7efd1c067210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:42.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.312+0000 7efd220b1700 1 -- 172.21.15.112:0/919084158 shutdown_connections 2023-01-24T03:30:42.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.312+0000 7efd220b1700 1 -- 172.21.15.112:0/919084158 wait complete. 2023-01-24T03:30:42.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.314+0000 7efd220b1700 1 Processor -- start 2023-01-24T03:30:42.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.314+0000 7fb7b89d5700 1 -- 172.21.15.112:0/193982346 shutdown_connections 2023-01-24T03:30:42.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.314+0000 7fb7b89d5700 1 --2- 172.21.15.112:0/193982346 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7b415cf30 0x7fb7b415d300 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.314+0000 7fb7b89d5700 1 --2- 172.21.15.112:0/193982346 >> v2:172.21.15.112:3300/0 conn(0x7fb7b40c4140 0x7fb7b415d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.324 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.314+0000 7fb7b89d5700 1 -- 172.21.15.112:0/193982346 >> 172.21.15.112:0/193982346 conn(0x7fb7b406c4f0 msgr2=0x7fb7b405d830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:42.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.314+0000 7efd220b1700 1 -- start start 2023-01-24T03:30:42.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.315+0000 7efd220b1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efd1c06b740 0x7efd1c0c60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.315+0000 7efd220b1700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efd1c0b7a60 0x7efd1c1616a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.315+0000 7efd220b1700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efd1c0c40e0 0x7efd1c15ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.315+0000 7efd220b1700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd1c0c70a0 con 0x7efd1c0c40e0 2023-01-24T03:30:42.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.315+0000 7efd220b1700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd1c1673e0 con 0x7efd1c0b7a60 2023-01-24T03:30:42.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.315+0000 7efd220b1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd1c0bf1b0 con 0x7efd1c06b740 2023-01-24T03:30:42.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.316+0000 7efd1b7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efd1c06b740 0x7efd1c0c60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.316+0000 7efd1b7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efd1c06b740 0x7efd1c0c60e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.112:38012/0 (socket says 172.21.15.112:38012) 2023-01-24T03:30:42.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.316+0000 7efd1b7fe700 1 -- 172.21.15.112:0/4016533414 learned_addr learned my addr 172.21.15.112:0/4016533414 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:42.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.316+0000 7efd1bfff700 1 --2- 172.21.15.112:0/4016533414 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efd1c0c40e0 0x7efd1c15ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.316+0000 7efd1b7fe700 1 -- 172.21.15.112:0/4016533414 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efd1c0b7a60 msgr2=0x7efd1c1616a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-01-24T03:30:42.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.316+0000 7efd1b7fe700 1 --2- 172.21.15.112:0/4016533414 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efd1c0b7a60 0x7efd1c1616a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.316+0000 7efd1b7fe700 1 -- 172.21.15.112:0/4016533414 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efd1c0c40e0 msgr2=0x7efd1c15ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.316+0000 7efd1b7fe700 1 --2- 172.21.15.112:0/4016533414 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efd1c0c40e0 0x7efd1c15ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.316+0000 7efd1b7fe700 1 -- 172.21.15.112:0/4016533414 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd100096f0 con 0x7efd1c06b740 2023-01-24T03:30:42.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.316+0000 7efd1b7fe700 1 --2- 172.21.15.112:0/4016533414 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efd1c06b740 0x7efd1c0c60e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efd1400c890 tx=0x7efd14001670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:42.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.317+0000 7fb7b89d5700 1 -- 172.21.15.112:0/193982346 shutdown_connections 2023-01-24T03:30:42.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.317+0000 7efd037fe700 1 -- 172.21.15.112:0/4016533414 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd14005d30 con 0x7efd1c06b740 2023-01-24T03:30:42.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.317+0000 7efd037fe700 1 -- 172.21.15.112:0/4016533414 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd14005e90 con 0x7efd1c06b740 2023-01-24T03:30:42.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.317+0000 7efd037fe700 1 -- 172.21.15.112:0/4016533414 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd14007300 con 0x7efd1c06b740 2023-01-24T03:30:42.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.317+0000 7efd220b1700 1 -- 172.21.15.112:0/4016533414 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd1c1776c0 con 0x7efd1c06b740 2023-01-24T03:30:42.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.317+0000 7efd220b1700 1 -- 172.21.15.112:0/4016533414 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd1c0c6aa0 con 0x7efd1c06b740 2023-01-24T03:30:42.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.317+0000 7fb7b89d5700 1 -- 172.21.15.112:0/193982346 wait complete. 2023-01-24T03:30:42.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.319+0000 7fb7b89d5700 1 Processor -- start 2023-01-24T03:30:42.335 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.319+0000 7efd220b1700 1 -- 172.21.15.112:0/4016533414 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd1c0683d0 con 0x7efd1c06b740 2023-01-24T03:30:42.336 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.319+0000 7fb7b89d5700 1 -- start start 2023-01-24T03:30:42.336 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.320+0000 7fb7b89d5700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7b406d850 0x7fb7b4159530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.336 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.320+0000 7fb7b89d5700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb7b40c4140 0x7fb7b4159a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.320+0000 7fb7b89d5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb7b415cf30 0x7fb7b415dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.320+0000 7fb7b89d5700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7b4149e20 con 0x7fb7b40c4140 2023-01-24T03:30:42.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.320+0000 7fb7b89d5700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7b4159060 con 0x7fb7b406d850 2023-01-24T03:30:42.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.320+0000 7fb7b89d5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7b415ee60 con 0x7fb7b415cf30 2023-01-24T03:30:42.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.320+0000 7fb7b259c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7b406d850 0x7fb7b4159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.320+0000 7fb7b259c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7b406d850 0x7fb7b4159530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.112:43210/0 (socket says 172.21.15.112:43210) 2023-01-24T03:30:42.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.320+0000 7fb7b259c700 1 -- 172.21.15.112:0/271463905 learned_addr learned my addr 172.21.15.112:0/271463905 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:42.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.321+0000 7fb7b259c700 1 -- 172.21.15.112:0/271463905 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb7b415cf30 msgr2=0x7fb7b415dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.321+0000 7fb7b259c700 1 --2- 172.21.15.112:0/271463905 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb7b415cf30 0x7fb7b415dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.321+0000 7fb7b259c700 1 -- 172.21.15.112:0/271463905 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb7b40c4140 msgr2=0x7fb7b4159a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.321+0000 7fb7b259c700 1 --2- 172.21.15.112:0/271463905 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb7b40c4140 0x7fb7b4159a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.321+0000 7fb7b259c700 1 -- 172.21.15.112:0/271463905 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a8009370 con 0x7fb7b406d850 2023-01-24T03:30:42.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.321+0000 7fb7b259c700 1 --2- 172.21.15.112:0/271463905 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7b406d850 0x7fb7b4159530 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb7a8006fd0 tx=0x7fb7a8017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:42.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.321+0000 7fb7a27fc700 1 -- 172.21.15.112:0/271463905 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7a8000d10 con 0x7fb7b406d850 2023-01-24T03:30:42.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.321+0000 7fb7a27fc700 1 -- 172.21.15.112:0/271463905 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7a8011b10 con 0x7fb7b406d850 2023-01-24T03:30:42.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.321+0000 7fb7a27fc700 1 -- 172.21.15.112:0/271463905 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7a8025620 con 0x7fb7b406d850 2023-01-24T03:30:42.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.320+0000 7efd037fe700 1 -- 172.21.15.112:0/4016533414 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7efd14010070 con 0x7efd1c06b740 2023-01-24T03:30:42.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.321+0000 7efd037fe700 1 --2- 172.21.15.112:0/4016533414 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7efcfc061240 0x7efcfc0636f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.321+0000 7efd037fe700 1 -- 172.21.15.112:0/4016533414 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7efd1407d3f0 con 0x7efd1c06b740 2023-01-24T03:30:42.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.322+0000 7fb7b89d5700 1 -- 172.21.15.112:0/271463905 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7b415f140 con 0x7fb7b406d850 2023-01-24T03:30:42.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.322+0000 7fb7b89d5700 1 -- 172.21.15.112:0/271463905 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7b414a350 con 0x7fb7b406d850 2023-01-24T03:30:42.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.323+0000 7fb7b89d5700 1 -- 172.21.15.112:0/271463905 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7b404b340 con 0x7fb7b406d850 2023-01-24T03:30:42.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.323+0000 7fb7a27fc700 1 -- 172.21.15.112:0/271463905 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fb7a800f050 con 0x7fb7b406d850 2023-01-24T03:30:42.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.324+0000 7fb7a27fc700 1 --2- 172.21.15.112:0/271463905 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb7900611f0 0x7fb7900636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.324+0000 7fb7a27fc700 1 -- 172.21.15.112:0/271463905 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fb7a807f5b0 con 0x7fb7b406d850 2023-01-24T03:30:42.343 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.325+0000 7efd037fe700 1 -- 172.21.15.112:0/4016533414 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7efd1404d1c0 con 0x7efd1c06b740 2023-01-24T03:30:42.343 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.327+0000 7fb7b1d9b700 1 --2- 172.21.15.112:0/271463905 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb7900611f0 0x7fb7900636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.344 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.328+0000 7fb7a27fc700 1 -- 172.21.15.112:0/271463905 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fb7a8052e20 con 0x7fb7b406d850 2023-01-24T03:30:42.344 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.332+0000 7efd1affd700 1 --2- 172.21.15.112:0/4016533414 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7efcfc061240 0x7efcfc0636f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.344 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.332+0000 7efd1affd700 1 --2- 172.21.15.112:0/4016533414 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7efcfc061240 0x7efcfc0636f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efd1c0c8530 tx=0x7efd10016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:42.345 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.335+0000 7fb7b1d9b700 1 --2- 172.21.15.112:0/271463905 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb7900611f0 0x7fb7900636a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb7b40c9120 tx=0x7fb7a4009060 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:42.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.355+0000 7f46ffb6d700 1 Processor -- start 2023-01-24T03:30:42.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.355+0000 7f46ffb6d700 1 -- start start 2023-01-24T03:30:42.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.356+0000 7fda2209e700 1 Processor -- start 2023-01-24T03:30:42.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.356+0000 7f46ffb6d700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f46f8067fe0 0x7f46f8155d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.356+0000 7f46ffb6d700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46f80adeb0 con 0x7f46f80c2780 2023-01-24T03:30:42.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.356+0000 7f46ffb6d700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46f80c2cf0 con 0x7f46f806ebd0 2023-01-24T03:30:42.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.356+0000 7f46ffb6d700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46f80c3590 con 0x7f46f8067fe0 2023-01-24T03:30:42.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.356+0000 7fda2209e700 1 -- start start 2023-01-24T03:30:42.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.356+0000 7fb928ddf700 1 Processor -- start 2023-01-24T03:30:42.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.357+0000 7fb928ddf700 1 -- start start 2023-01-24T03:30:42.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.357+0000 7f46fd909700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f46f8067fe0 0x7f46f8155d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.357+0000 7f46fd108700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f46f80c2780 0x7f46f806e4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:59954/0 (socket says 172.21.15.112:59954) 2023-01-24T03:30:42.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.357+0000 7f46fd108700 1 -- 172.21.15.112:0/1628787736 learned_addr learned my addr 172.21.15.112:0/1628787736 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:42.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.357+0000 7fda2209e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fda1c0c8e20 0x7fda1c069960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.357+0000 7fda2209e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fda1c1569e0 0x7fda1c0ae9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.357+0000 7fda2209e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda1c1591f0 con 0x7fda1c05e340 2023-01-24T03:30:42.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.357+0000 7fda2209e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda1c159a10 con 0x7fda1c1569e0 2023-01-24T03:30:42.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.357+0000 7fda2209e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda1c156e80 con 0x7fda1c0c8e20 2023-01-24T03:30:42.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.357+0000 7fda1affd700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fda1c05e340 0x7fda1c158ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.112:45518/0 (socket says 172.21.15.112:45518) 2023-01-24T03:30:42.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.357+0000 7fda1affd700 1 -- 172.21.15.112:0/2717929013 learned_addr learned my addr 172.21.15.112:0/2717929013 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:42.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.357+0000 7f46fd909700 1 -- 172.21.15.112:0/1628787736 >> v1:172.21.15.142:6789/0 conn(0x7f46f806ebd0 legacy=0x7f46f814efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.357+0000 7f46fd909700 1 -- 172.21.15.112:0/1628787736 >> v1:172.21.15.112:6789/0 conn(0x7f46f80c2780 legacy=0x7f46f806e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.357+0000 7f46fd909700 1 -- 172.21.15.112:0/1628787736 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f80c2fd0 con 0x7f46f8067fe0 2023-01-24T03:30:42.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.357+0000 7fda1b7fe700 1 --2- 172.21.15.112:0/2717929013 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fda1c0c8e20 0x7fda1c069960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.358+0000 7fda1a7fc700 1 -- 172.21.15.112:0/2717929013 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2198276433 0 0) 0x7fda1c1591f0 con 0x7fda1c05e340 2023-01-24T03:30:42.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.358+0000 7fda1a7fc700 1 -- 172.21.15.112:0/2717929013 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda080033f0 con 0x7fda1c05e340 2023-01-24T03:30:42.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.358+0000 7f46fd909700 1 --2- 172.21.15.112:0/1628787736 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46f8067fe0 0x7f46f8155d90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f46f4009760 tx=0x7f46f4009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d44414ecb266bdbb server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:42.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.358+0000 7fda1b7fe700 1 -- 172.21.15.112:0/2717929013 >> v2:172.21.15.142:3300/0 conn(0x7fda1c1569e0 msgr2=0x7fda1c0ae9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-01-24T03:30:42.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.358+0000 7fda1b7fe700 1 --2- 172.21.15.112:0/2717929013 >> v2:172.21.15.142:3300/0 conn(0x7fda1c1569e0 0x7fda1c0ae9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.358+0000 7fda1b7fe700 1 -- 172.21.15.112:0/2717929013 >> v1:172.21.15.142:6789/0 conn(0x7fda1c05e340 legacy=0x7fda1c158ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.358+0000 7fda1b7fe700 1 -- 172.21.15.112:0/2717929013 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda1c159450 con 0x7fda1c0c8e20 2023-01-24T03:30:42.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.358+0000 7fda1b7fe700 1 --2- 172.21.15.112:0/2717929013 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fda1c0c8e20 0x7fda1c069960 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fda0c009760 tx=0x7fda0c009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=285e474a69bc67be server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:42.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.358+0000 7f46fc907700 1 -- 172.21.15.112:0/1628787736 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f4011630 con 0x7f46f8067fe0 2023-01-24T03:30:42.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.358+0000 7f46fc907700 1 -- 172.21.15.112:0/1628787736 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46f4011790 con 0x7f46f8067fe0 2023-01-24T03:30:42.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.358+0000 7f46fc907700 1 -- 172.21.15.112:0/1628787736 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f4011b10 con 0x7f46f8067fe0 2023-01-24T03:30:42.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.358+0000 7fb928ddf700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb9240c4d40 0x7fb9240c7af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.358+0000 7fb928ddf700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fb9240c3ab0 0x7fb924151cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.358+0000 7fb928ddf700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9240c5470 con 0x7fb9240c8640 2023-01-24T03:30:42.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.358+0000 7fb928ddf700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9240bb8a0 con 0x7fb9240c3ab0 2023-01-24T03:30:42.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.358+0000 7fb928ddf700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9240c2f10 con 0x7fb9240c4d40 2023-01-24T03:30:42.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.358+0000 7fb921d9b700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fb9240c8640 0x7fb9240c33a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:59964/0 (socket says 172.21.15.112:59964) 2023-01-24T03:30:42.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.358+0000 7fb921d9b700 1 -- 172.21.15.112:0/1953362543 learned_addr learned my addr 172.21.15.112:0/1953362543 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:42.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.358+0000 7fb92259c700 1 --2- 172.21.15.112:0/1953362543 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb9240c4d40 0x7fb9240c7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.359+0000 7fb922d9d700 1 --2- 172.21.15.112:0/1953362543 >> v2:172.21.15.112:3300/0 conn(0x7fb9240c3ab0 0x7fb924151cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.359+0000 7fb92259c700 1 -- 172.21.15.112:0/1953362543 >> v2:172.21.15.112:3300/0 conn(0x7fb9240c3ab0 msgr2=0x7fb924151cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:42.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.359+0000 7fb92259c700 1 --2- 172.21.15.112:0/1953362543 >> v2:172.21.15.112:3300/0 conn(0x7fb9240c3ab0 0x7fb924151cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.359+0000 7fb92259c700 1 -- 172.21.15.112:0/1953362543 >> v1:172.21.15.112:6789/0 conn(0x7fb9240c8640 legacy=0x7fb9240c33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:42.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.359+0000 7fb92259c700 1 -- 172.21.15.112:0/1953362543 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9240c51b0 con 0x7fb9240c4d40 2023-01-24T03:30:42.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.359+0000 7fda1a7fc700 1 -- 172.21.15.112:0/2717929013 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda0c011630 con 0x7fda1c0c8e20 2023-01-24T03:30:42.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.359+0000 7fda1a7fc700 1 -- 172.21.15.112:0/2717929013 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda0c011790 con 0x7fda1c0c8e20 2023-01-24T03:30:42.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.359+0000 7fda1a7fc700 1 -- 172.21.15.112:0/2717929013 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda0c011b30 con 0x7fda1c0c8e20 2023-01-24T03:30:42.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.359+0000 7fb92259c700 1 --2- 172.21.15.112:0/1953362543 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb9240c4d40 0x7fb9240c7af0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb91c009e60 tx=0x7fb91c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=118e1f194cff5d88 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:42.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.350+0000 7fea7183f700 1 Processor -- start 2023-01-24T03:30:42.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.351+0000 7fea7183f700 1 -- start start 2023-01-24T03:30:42.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.352+0000 7fea7183f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea6c0c40e0 0x7fea6c0c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.352+0000 7fea7183f700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fea6c0b7a60 0x7fea6c06b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.352+0000 7fea7183f700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea6c06e6a0 con 0x7fea6c06d210 2023-01-24T03:30:42.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.352+0000 7fea7183f700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea6c06ef20 con 0x7fea6c0b7a60 2023-01-24T03:30:42.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.352+0000 7fea7183f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea6c06e960 con 0x7fea6c0c40e0 2023-01-24T03:30:42.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.352+0000 7fea6affd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea6c0c40e0 0x7fea6c0c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.352+0000 7fea6affd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea6c0c40e0 0x7fea6c0c44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.112:38030/0 (socket says 172.21.15.112:38030) 2023-01-24T03:30:42.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.352+0000 7fea6affd700 1 -- 172.21.15.112:0/694690548 learned_addr learned my addr 172.21.15.112:0/694690548 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:42.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.359+0000 7fb92159a700 1 -- 172.21.15.112:0/1953362543 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91c00e070 con 0x7fb9240c4d40 2023-01-24T03:30:42.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.359+0000 7f46ffb6d700 1 -- 172.21.15.112:0/1628787736 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46f8067fe0 msgr2=0x7f46f8155d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.359+0000 7f46ffb6d700 1 --2- 172.21.15.112:0/1628787736 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46f8067fe0 0x7f46f8155d90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f46f4009760 tx=0x7f46f4009060 comp rx=0 tx=0).stop 2023-01-24T03:30:42.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.359+0000 7fb92159a700 1 -- 172.21.15.112:0/1953362543 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb91c00bbf0 con 0x7fb9240c4d40 2023-01-24T03:30:42.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.359+0000 7fb92159a700 1 -- 172.21.15.112:0/1953362543 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91c012430 con 0x7fb9240c4d40 2023-01-24T03:30:42.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.360+0000 7fb928ddf700 1 -- 172.21.15.112:0/1953362543 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb9240c4d40 msgr2=0x7fb9240c7af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.360+0000 7fb928ddf700 1 --2- 172.21.15.112:0/1953362543 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb9240c4d40 0x7fb9240c7af0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb91c009e60 tx=0x7fb91c009780 comp rx=0 tx=0).stop 2023-01-24T03:30:42.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.354+0000 7fea6a7fc700 1 --2- 172.21.15.112:0/694690548 >> v2:172.21.15.142:3300/0 conn(0x7fea6c0b7a60 0x7fea6c06b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.359+0000 7fea69ffb700 1 -- 172.21.15.112:0/694690548 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 387867565 0 0) 0x7fea6c06e6a0 con 0x7fea6c06d210 2023-01-24T03:30:42.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.359+0000 7fea69ffb700 1 -- 172.21.15.112:0/694690548 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea540033f0 con 0x7fea6c06d210 2023-01-24T03:30:42.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.359+0000 7fea6affd700 1 -- 172.21.15.112:0/694690548 >> v2:172.21.15.142:3300/0 conn(0x7fea6c0b7a60 msgr2=0x7fea6c06b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:42.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.359+0000 7fea6affd700 1 --2- 172.21.15.112:0/694690548 >> v2:172.21.15.142:3300/0 conn(0x7fea6c0b7a60 0x7fea6c06b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.359+0000 7fea6affd700 1 -- 172.21.15.112:0/694690548 >> v1:172.21.15.112:6789/0 conn(0x7fea6c06d210 legacy=0x7fea6c165a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.359+0000 7fea6affd700 1 -- 172.21.15.112:0/694690548 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea6c06ec40 con 0x7fea6c0c40e0 2023-01-24T03:30:42.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.360+0000 7fea6affd700 1 --2- 172.21.15.112:0/694690548 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea6c0c40e0 0x7fea6c0c44b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fea5c006fd0 tx=0x7fea5c009dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c09e7669c4698d10 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:42.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.359+0000 7f46ffb6d700 1 -- 172.21.15.112:0/1628787736 shutdown_connections 2023-01-24T03:30:42.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.359+0000 7f46ffb6d700 1 --2- 172.21.15.112:0/1628787736 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46f8067fe0 0x7f46f8155d90 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.359+0000 7f46ffb6d700 1 -- 172.21.15.112:0/1628787736 >> 172.21.15.112:0/1628787736 conn(0x7f46f806b550 msgr2=0x7f46f806b950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:42.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.360+0000 7f46ffb6d700 1 -- 172.21.15.112:0/1628787736 shutdown_connections 2023-01-24T03:30:42.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.360+0000 7f46ffb6d700 1 -- 172.21.15.112:0/1628787736 wait complete. 2023-01-24T03:30:42.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.360+0000 7fda2209e700 1 -- 172.21.15.112:0/2717929013 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fda1c0c8e20 msgr2=0x7fda1c069960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.360+0000 7fda2209e700 1 --2- 172.21.15.112:0/2717929013 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fda1c0c8e20 0x7fda1c069960 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fda0c009760 tx=0x7fda0c009060 comp rx=0 tx=0).stop 2023-01-24T03:30:42.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.360+0000 7fda2209e700 1 -- 172.21.15.112:0/2717929013 shutdown_connections 2023-01-24T03:30:42.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.360+0000 7fda2209e700 1 --2- 172.21.15.112:0/2717929013 >> v2:172.21.15.142:3300/0 conn(0x7fda1c1569e0 0x7fda1c0ae9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.360+0000 7fda2209e700 1 --2- 172.21.15.112:0/2717929013 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fda1c0c8e20 0x7fda1c069960 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fda0c009760 tx=0x7fda0c009060 comp rx=0 tx=0).stop 2023-01-24T03:30:42.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.360+0000 7fda2209e700 1 -- 172.21.15.112:0/2717929013 >> 172.21.15.112:0/2717929013 conn(0x7fda1c05d830 msgr2=0x7fda1c05dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:42.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.360+0000 7fea69ffb700 1 -- 172.21.15.112:0/694690548 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5c00c7f0 con 0x7fea6c0c40e0 2023-01-24T03:30:42.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.360+0000 7fea69ffb700 1 -- 172.21.15.112:0/694690548 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea5c00c950 con 0x7fea6c0c40e0 2023-01-24T03:30:42.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.360+0000 7fea69ffb700 1 -- 172.21.15.112:0/694690548 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5c00ccf0 con 0x7fea6c0c40e0 2023-01-24T03:30:42.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.361+0000 7fb928ddf700 1 -- 172.21.15.112:0/1953362543 shutdown_connections 2023-01-24T03:30:42.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.361+0000 7fb928ddf700 1 --2- 172.21.15.112:0/1953362543 >> v2:172.21.15.112:3300/0 conn(0x7fb9240c3ab0 0x7fb924151cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.361+0000 7fb928ddf700 1 --2- 172.21.15.112:0/1953362543 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb9240c4d40 0x7fb9240c7af0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.361+0000 7fb928ddf700 1 -- 172.21.15.112:0/1953362543 >> 172.21.15.112:0/1953362543 conn(0x7fb924066e10 msgr2=0x7fb924067210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:42.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.361+0000 7fea7183f700 1 -- 172.21.15.112:0/694690548 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea6c0c40e0 msgr2=0x7fea6c0c44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.361+0000 7fea7183f700 1 --2- 172.21.15.112:0/694690548 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea6c0c40e0 0x7fea6c0c44b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fea5c006fd0 tx=0x7fea5c009dd0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.361+0000 7fb928ddf700 1 -- 172.21.15.112:0/1953362543 shutdown_connections 2023-01-24T03:30:42.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.361+0000 7fb928ddf700 1 -- 172.21.15.112:0/1953362543 wait complete. 2023-01-24T03:30:42.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.361+0000 7fda2209e700 1 -- 172.21.15.112:0/2717929013 shutdown_connections 2023-01-24T03:30:42.393 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.361+0000 7fea7183f700 1 -- 172.21.15.112:0/694690548 shutdown_connections 2023-01-24T03:30:42.393 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.361+0000 7fea7183f700 1 --2- 172.21.15.112:0/694690548 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea6c0c40e0 0x7fea6c0c44b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.393 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.361+0000 7fea7183f700 1 --2- 172.21.15.112:0/694690548 >> v2:172.21.15.142:3300/0 conn(0x7fea6c0b7a60 0x7fea6c06b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.361+0000 7fea7183f700 1 -- 172.21.15.112:0/694690548 >> 172.21.15.112:0/694690548 conn(0x7fea6c066e10 msgr2=0x7fea6c067210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:42.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.361+0000 7f46ffb6d700 1 Processor -- start 2023-01-24T03:30:42.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.361+0000 7f46ffb6d700 1 -- start start 2023-01-24T03:30:42.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.362+0000 7f46ffb6d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f46f8067fe0 0x7f46f8166e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.362+0000 7f46ffb6d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46f806ebd0 0x7f46f815c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.362+0000 7f46ffb6d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f46f80c2780 0x7f46f8168fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.362+0000 7f46ffb6d700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46f8150340 con 0x7f46f806ebd0 2023-01-24T03:30:42.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.362+0000 7f46ffb6d700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46f8150060 con 0x7f46f80c2780 2023-01-24T03:30:42.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.362+0000 7f46ffb6d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46f815b660 con 0x7f46f8067fe0 2023-01-24T03:30:42.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.363+0000 7f46fd909700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f46f8067fe0 0x7f46f8166e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.363+0000 7f46fd108700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46f806ebd0 0x7f46f815c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.363+0000 7f46fd108700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46f806ebd0 0x7f46f815c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50752/0 (socket says 172.21.15.112:50752) 2023-01-24T03:30:42.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.363+0000 7f46fd108700 1 -- 172.21.15.112:0/627051049 learned_addr learned my addr 172.21.15.112:0/627051049 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:42.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.361+0000 7fda2209e700 1 -- 172.21.15.112:0/2717929013 wait complete. 2023-01-24T03:30:42.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.363+0000 7fb928ddf700 1 Processor -- start 2023-01-24T03:30:42.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.363+0000 7fea7183f700 1 -- 172.21.15.112:0/694690548 shutdown_connections 2023-01-24T03:30:42.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.363+0000 7fda2209e700 1 Processor -- start 2023-01-24T03:30:42.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.363+0000 7fda2209e700 1 -- start start 2023-01-24T03:30:42.400 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.363+0000 7f46fd909700 1 -- 172.21.15.112:0/627051049 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f46f80c2780 msgr2=0x7f46f8168fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-01-24T03:30:42.400 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.363+0000 7f46fd909700 1 --2- 172.21.15.112:0/627051049 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f46f80c2780 0x7f46f8168fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.400 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.363+0000 7f46fd909700 1 -- 172.21.15.112:0/627051049 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46f806ebd0 msgr2=0x7f46f815c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.400 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.363+0000 7f46fd909700 1 --2- 172.21.15.112:0/627051049 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46f806ebd0 0x7f46f815c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.401 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.363+0000 7f46fd909700 1 -- 172.21.15.112:0/627051049 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f40092a0 con 0x7f46f8067fe0 2023-01-24T03:30:42.401 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.363+0000 7f46fd909700 1 --2- 172.21.15.112:0/627051049 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f46f8067fe0 0x7f46f8166e30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f46f4015000 tx=0x7f46f4016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:42.401 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.363+0000 7fea7183f700 1 -- 172.21.15.112:0/694690548 wait complete. 2023-01-24T03:30:42.402 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.364+0000 7fda2209e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fda1c05e340 0x7fda1c166590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.402 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.364+0000 7fda2209e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fda1c1569e0 0x7fda1c158160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.402 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.364+0000 7fda2209e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fda1c167060 0x7fda1c173f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.402 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.364+0000 7fda2209e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda1c0c5690 con 0x7fda1c05e340 2023-01-24T03:30:42.402 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.364+0000 7fda2209e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda1c0be3c0 con 0x7fda1c167060 2023-01-24T03:30:42.403 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.364+0000 7fda2209e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda1c0c9870 con 0x7fda1c1569e0 2023-01-24T03:30:42.403 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.364+0000 7fb928ddf700 1 -- start start 2023-01-24T03:30:42.403 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.365+0000 7fb928ddf700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb9240c3ab0 0x7fb924153e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.404 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.365+0000 7fb928ddf700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb9240c4d40 0x7fb9241613a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.404 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.365+0000 7fb928ddf700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb9240c8640 0x7fb9241618e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.404 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.365+0000 7fb928ddf700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9241643e0 con 0x7fb9240c8640 2023-01-24T03:30:42.404 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.365+0000 7fb928ddf700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb924160f10 con 0x7fb9240c4d40 2023-01-24T03:30:42.405 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.365+0000 7fb928ddf700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb924164630 con 0x7fb9240c3ab0 2023-01-24T03:30:42.405 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.365+0000 7f46e5ffb700 1 -- 172.21.15.112:0/627051049 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f4000d10 con 0x7f46f8067fe0 2023-01-24T03:30:42.405 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.365+0000 7f46e5ffb700 1 -- 172.21.15.112:0/627051049 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46f4020070 con 0x7f46f8067fe0 2023-01-24T03:30:42.406 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.365+0000 7f46e5ffb700 1 -- 172.21.15.112:0/627051049 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f40276b0 con 0x7f46f8067fe0 2023-01-24T03:30:42.406 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.365+0000 7fda1b7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fda1c05e340 0x7fda1c166590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.406 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.365+0000 7fda1b7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fda1c05e340 0x7fda1c166590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50756/0 (socket says 172.21.15.112:50756) 2023-01-24T03:30:42.406 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.365+0000 7fda1b7fe700 1 -- 172.21.15.112:0/3229449885 learned_addr learned my addr 172.21.15.112:0/3229449885 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:42.407 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.365+0000 7fea7183f700 1 Processor -- start 2023-01-24T03:30:42.407 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.365+0000 7fb922d9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb9240c8640 0x7fb9241618e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.408 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.365+0000 7fb922d9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb9240c8640 0x7fb9241618e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50766/0 (socket says 172.21.15.112:50766) 2023-01-24T03:30:42.408 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.365+0000 7fb922d9d700 1 -- 172.21.15.112:0/2362699950 learned_addr learned my addr 172.21.15.112:0/2362699950 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:42.408 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.365+0000 7f46ffb6d700 1 -- 172.21.15.112:0/627051049 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46f815b940 con 0x7f46f8067fe0 2023-01-24T03:30:42.409 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.365+0000 7f46ffb6d700 1 -- 172.21.15.112:0/627051049 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46f815bf00 con 0x7f46f8067fe0 2023-01-24T03:30:42.409 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.366+0000 7fb922d9d700 1 -- 172.21.15.112:0/2362699950 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb9240c3ab0 msgr2=0x7fb924153e20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-01-24T03:30:42.410 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.366+0000 7fb922d9d700 1 --2- 172.21.15.112:0/2362699950 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb9240c3ab0 0x7fb924153e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.410 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.366+0000 7fb922d9d700 1 -- 172.21.15.112:0/2362699950 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb9240c4d40 msgr2=0x7fb9241613a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-01-24T03:30:42.410 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.366+0000 7fb922d9d700 1 --2- 172.21.15.112:0/2362699950 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb9240c4d40 0x7fb9241613a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.410 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.366+0000 7fb922d9d700 1 -- 172.21.15.112:0/2362699950 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb91c009a20 con 0x7fb9240c8640 2023-01-24T03:30:42.411 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.365+0000 7fea7183f700 1 -- start start 2023-01-24T03:30:42.411 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.366+0000 7fda1b7fe700 1 -- 172.21.15.112:0/3229449885 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fda1c1569e0 msgr2=0x7fda1c158160 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-01-24T03:30:42.411 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.366+0000 7fda1b7fe700 1 --2- 172.21.15.112:0/3229449885 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fda1c1569e0 0x7fda1c158160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.412 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.366+0000 7fda1b7fe700 1 -- 172.21.15.112:0/3229449885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fda1c167060 msgr2=0x7fda1c173f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.412 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.366+0000 7fda1b7fe700 1 --2- 172.21.15.112:0/3229449885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fda1c167060 0x7fda1c173f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.412 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.366+0000 7fda1b7fe700 1 -- 172.21.15.112:0/3229449885 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda0c0092a0 con 0x7fda1c05e340 2023-01-24T03:30:42.413 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.366+0000 7fb922d9d700 1 --2- 172.21.15.112:0/2362699950 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb9240c8640 0x7fb9241618e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb91800bfd0 tx=0x7fb918008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:42.413 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.366+0000 7fb9127fc700 1 -- 172.21.15.112:0/2362699950 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9180125f0 con 0x7fb9240c8640 2023-01-24T03:30:42.413 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.366+0000 7fea7183f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea6c06d210 0x7fea6c15ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.414 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.366+0000 7fea7183f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea6c0b7a60 0x7fea6c15eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.414 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.366+0000 7fea7183f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea6c0c40e0 0x7fea6c15dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.414 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.366+0000 7fea7183f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6c17bd10 con 0x7fea6c0b7a60 2023-01-24T03:30:42.414 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.366+0000 7fea7183f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6c0c6b60 con 0x7fea6c0c40e0 2023-01-24T03:30:42.415 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.366+0000 7fea7183f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea6c161a30 con 0x7fea6c06d210 2023-01-24T03:30:42.415 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.367+0000 7fea6affd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea6c06d210 0x7fea6c15ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.416 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.366+0000 7fb9127fc700 1 -- 172.21.15.112:0/2362699950 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb918012750 con 0x7fb9240c8640 2023-01-24T03:30:42.416 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.366+0000 7fb9127fc700 1 -- 172.21.15.112:0/2362699950 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb918012b10 con 0x7fb9240c8640 2023-01-24T03:30:42.416 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.366+0000 7fb928ddf700 1 -- 172.21.15.112:0/2362699950 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb924164910 con 0x7fb9240c8640 2023-01-24T03:30:42.417 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.366+0000 7fb928ddf700 1 -- 172.21.15.112:0/2362699950 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb924164ed0 con 0x7fb9240c8640 2023-01-24T03:30:42.418 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.367+0000 7fea6a7fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea6c0b7a60 0x7fea6c15eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.418 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.367+0000 7fea6a7fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea6c0b7a60 0x7fea6c15eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50768/0 (socket says 172.21.15.112:50768) 2023-01-24T03:30:42.418 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.367+0000 7fea6a7fc700 1 -- 172.21.15.112:0/3895003326 learned_addr learned my addr 172.21.15.112:0/3895003326 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:42.419 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.367+0000 7fea6b7fe700 1 --2- 172.21.15.112:0/3895003326 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea6c0c40e0 0x7fea6c15dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.419 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.367+0000 7fea6a7fc700 1 -- 172.21.15.112:0/3895003326 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea6c06d210 msgr2=0x7fea6c15ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.419 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.367+0000 7fea6a7fc700 1 --2- 172.21.15.112:0/3895003326 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea6c06d210 0x7fea6c15ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.419 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.367+0000 7fea6a7fc700 1 -- 172.21.15.112:0/3895003326 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea6c0c40e0 msgr2=0x7fea6c15dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.419 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.367+0000 7fea6a7fc700 1 --2- 172.21.15.112:0/3895003326 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea6c0c40e0 0x7fea6c15dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.420 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.367+0000 7fea6a7fc700 1 -- 172.21.15.112:0/3895003326 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea5c0090c0 con 0x7fea6c0b7a60 2023-01-24T03:30:42.420 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.368+0000 7fea6a7fc700 1 --2- 172.21.15.112:0/3895003326 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea6c0b7a60 0x7fea6c15eef0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fea6000d890 tx=0x7fea60008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:42.420 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.368+0000 7fb928ddf700 1 -- 172.21.15.112:0/2362699950 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb904005130 con 0x7fb9240c8640 2023-01-24T03:30:42.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.367+0000 7f46e5ffb700 1 -- 172.21.15.112:0/627051049 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f46f4017020 con 0x7f46f8067fe0 2023-01-24T03:30:42.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.368+0000 7f46ffb6d700 1 -- 172.21.15.112:0/627051049 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46f805e610 con 0x7f46f8067fe0 2023-01-24T03:30:42.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.368+0000 7f46e5ffb700 1 --2- 172.21.15.112:0/627051049 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f46d8061340 0x7f46d80637f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.422 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.368+0000 7fda1b7fe700 1 --2- 172.21.15.112:0/3229449885 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fda1c05e340 0x7fda1c166590 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fda0c009270 tx=0x7fda0c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:42.422 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.368+0000 7f46e5ffb700 1 -- 172.21.15.112:0/627051049 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f46f4022070 con 0x7f46f8067fe0 2023-01-24T03:30:42.422 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.368+0000 7fea52ffd700 1 -- 172.21.15.112:0/3895003326 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea60012630 con 0x7fea6c0b7a60 2023-01-24T03:30:42.423 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.368+0000 7f46fd108700 1 --2- 172.21.15.112:0/627051049 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f46d8061340 0x7f46d80637f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.424 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.368+0000 7fea52ffd700 1 -- 172.21.15.112:0/3895003326 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea60012790 con 0x7fea6c0b7a60 2023-01-24T03:30:42.424 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.368+0000 7fea52ffd700 1 -- 172.21.15.112:0/3895003326 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea60012b30 con 0x7fea6c0b7a60 2023-01-24T03:30:42.424 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.368+0000 7fea7183f700 1 -- 172.21.15.112:0/3895003326 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea6c0c80e0 con 0x7fea6c0b7a60 2023-01-24T03:30:42.424 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.368+0000 7fea7183f700 1 -- 172.21.15.112:0/3895003326 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea6c0b4660 con 0x7fea6c0b7a60 2023-01-24T03:30:42.425 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.369+0000 7f46fd108700 1 --2- 172.21.15.112:0/627051049 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f46d8061340 0x7f46d80637f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f46f80c1cd0 tx=0x7f46e8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:42.425 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.369+0000 7fb9127fc700 1 -- 172.21.15.112:0/2362699950 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fb91801e460 con 0x7fb9240c8640 2023-01-24T03:30:42.426 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.369+0000 7fb9127fc700 1 --2- 172.21.15.112:0/2362699950 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb900061290 0x7fb900063740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.426 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.369+0000 7fb9127fc700 1 -- 172.21.15.112:0/2362699950 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fb91807dde0 con 0x7fb9240c8640 2023-01-24T03:30:42.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.370+0000 7fea7183f700 1 -- 172.21.15.112:0/3895003326 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea6c161560 con 0x7fea6c0b7a60 2023-01-24T03:30:42.428 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.369+0000 7fda037fe700 1 -- 172.21.15.112:0/3229449885 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda0c011b10 con 0x7fda1c05e340 2023-01-24T03:30:42.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.369+0000 7fda037fe700 1 -- 172.21.15.112:0/3229449885 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda0c01f070 con 0x7fda1c05e340 2023-01-24T03:30:42.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.369+0000 7fda037fe700 1 -- 172.21.15.112:0/3229449885 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda0c01b660 con 0x7fda1c05e340 2023-01-24T03:30:42.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.369+0000 7fda2209e700 1 -- 172.21.15.112:0/3229449885 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda1c167610 con 0x7fda1c05e340 2023-01-24T03:30:42.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.369+0000 7fda2209e700 1 -- 172.21.15.112:0/3229449885 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda1c167b90 con 0x7fda1c05e340 2023-01-24T03:30:42.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.370+0000 7fda2209e700 1 -- 172.21.15.112:0/3229449885 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda1c0b02c0 con 0x7fda1c05e340 2023-01-24T03:30:42.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.370+0000 7fb92259c700 1 --2- 172.21.15.112:0/2362699950 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb900061290 0x7fb900063740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.371+0000 7fb92259c700 1 --2- 172.21.15.112:0/2362699950 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb900061290 0x7fb900063740 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb91c003400 tx=0x7fb91c018000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:42.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.373+0000 7f46e5ffb700 1 -- 172.21.15.112:0/627051049 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f46f404f890 con 0x7f46f8067fe0 2023-01-24T03:30:42.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.374+0000 7fe68a744700 1 Processor -- start 2023-01-24T03:30:42.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.374+0000 7fe68a744700 1 -- start start 2023-01-24T03:30:42.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.374+0000 7fe68a744700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe68406a210 0x7fe6841559a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.374+0000 7fe68a744700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe684069360 0x7fe68405d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.374+0000 7fe68a744700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6840c3c80 con 0x7fe684067fe0 2023-01-24T03:30:42.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.374+0000 7fe68a744700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6840bf6e0 con 0x7fe684069360 2023-01-24T03:30:42.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.374+0000 7fe68a744700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6840c2d90 con 0x7fe68406a210 2023-01-24T03:30:42.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.375+0000 7fb9127fc700 1 -- 172.21.15.112:0/2362699950 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fb91804dbe0 con 0x7fb9240c8640 2023-01-24T03:30:42.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.375+0000 7fdb108c6700 1 Processor -- start 2023-01-24T03:30:42.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.375+0000 7fdb108c6700 1 -- start start 2023-01-24T03:30:42.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.376+0000 7fea52ffd700 1 -- 172.21.15.112:0/3895003326 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fea60027430 con 0x7fea6c0b7a60 2023-01-24T03:30:42.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.376+0000 7fea52ffd700 1 --2- 172.21.15.112:0/3895003326 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fea4c061290 0x7fea4c063740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.376+0000 7fea52ffd700 1 -- 172.21.15.112:0/3895003326 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fea6007e450 con 0x7fea6c0b7a60 2023-01-24T03:30:42.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.376+0000 7fdb108c6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb0c0c4ef0 0x7fdb0c15e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.376+0000 7fdb108c6700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb0c067b60 con 0x7fdb0c0c2440 2023-01-24T03:30:42.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.376+0000 7fdb108c6700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb0c156660 con 0x7fdb0c0c85d0 2023-01-24T03:30:42.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.376+0000 7fdb108c6700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0c156940 con 0x7fdb0c0c4ef0 2023-01-24T03:30:42.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.377+0000 7fdb09d9b700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fdb0c0c2440 0x7fdb0c0c1240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:60010/0 (socket says 172.21.15.112:60010) 2023-01-24T03:30:42.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.377+0000 7fdb09d9b700 1 -- 172.21.15.112:0/2736560276 learned_addr learned my addr 172.21.15.112:0/2736560276 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:42.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.377+0000 7fdb0ad9d700 1 --2- 172.21.15.112:0/2736560276 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb0c0c4ef0 0x7fdb0c15e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.378+0000 7fda037fe700 1 -- 172.21.15.112:0/3229449885 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fda0c00f050 con 0x7fda1c05e340 2023-01-24T03:30:42.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.375+0000 7fe683fff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fe684067fe0 0x7fe684069b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:59998/0 (socket says 172.21.15.112:59998) 2023-01-24T03:30:42.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.375+0000 7fe683fff700 1 -- 172.21.15.112:0/2957082585 learned_addr learned my addr 172.21.15.112:0/2957082585 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:42.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.376+0000 7fe6837fe700 1 --2- 172.21.15.112:0/2957082585 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe68406a210 0x7fe6841559a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.376+0000 7fe688ce1700 1 --2- 172.21.15.112:0/2957082585 >> v2:172.21.15.142:3300/0 conn(0x7fe684069360 0x7fe68405d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.378+0000 7fda037fe700 1 --2- 172.21.15.112:0/3229449885 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd9fc061290 0x7fd9fc063740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.378+0000 7fda037fe700 1 -- 172.21.15.112:0/3229449885 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fda0c07f300 con 0x7fda1c05e340 2023-01-24T03:30:42.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.379+0000 7fea6affd700 1 --2- 172.21.15.112:0/3895003326 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fea4c061290 0x7fea4c063740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.379+0000 7fea6affd700 1 --2- 172.21.15.112:0/3895003326 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fea4c061290 0x7fea4c063740 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fea5c006fd0 tx=0x7fea5c017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:42.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.380+0000 7fdb0ad9d700 1 -- 172.21.15.112:0/2736560276 >> v1:172.21.15.142:6789/0 conn(0x7fdb0c0c85d0 legacy=0x7fdb0c156dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:42.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.380+0000 7fdb0ad9d700 1 -- 172.21.15.112:0/2736560276 >> v1:172.21.15.112:6789/0 conn(0x7fdb0c0c2440 legacy=0x7fdb0c0c1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.380+0000 7fdb0ad9d700 1 -- 172.21.15.112:0/2736560276 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb0c1574e0 con 0x7fdb0c0c4ef0 2023-01-24T03:30:42.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.380+0000 7fdb0ad9d700 1 --2- 172.21.15.112:0/2736560276 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb0c0c4ef0 0x7fdb0c15e9f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdb04009e90 tx=0x7fdb04009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e0c62fbe63ce780 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:42.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.380+0000 7fdb0959a700 1 -- 172.21.15.112:0/2736560276 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0400e070 con 0x7fdb0c0c4ef0 2023-01-24T03:30:42.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.380+0000 7fdb0959a700 1 -- 172.21.15.112:0/2736560276 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb0400bc30 con 0x7fdb0c0c4ef0 2023-01-24T03:30:42.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.380+0000 7fdb0959a700 1 -- 172.21.15.112:0/2736560276 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb04012430 con 0x7fdb0c0c4ef0 2023-01-24T03:30:42.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.381+0000 7fdb108c6700 1 -- 172.21.15.112:0/2736560276 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb0c0c4ef0 msgr2=0x7fdb0c15e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.381+0000 7fdb108c6700 1 --2- 172.21.15.112:0/2736560276 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb0c0c4ef0 0x7fdb0c15e9f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdb04009e90 tx=0x7fdb04009780 comp rx=0 tx=0).stop 2023-01-24T03:30:42.450 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.382+0000 7fda1affd700 1 --2- 172.21.15.112:0/3229449885 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd9fc061290 0x7fd9fc063740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.450 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.382+0000 7fda1affd700 1 --2- 172.21.15.112:0/3229449885 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd9fc061290 0x7fd9fc063740 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fda10005750 tx=0x7fda100010a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:42.451 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.384+0000 7fea52ffd700 1 -- 172.21.15.112:0/3895003326 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fea6004e2d0 con 0x7fea6c0b7a60 2023-01-24T03:30:42.452 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.385+0000 7fe6837fe700 1 -- 172.21.15.112:0/2957082585 >> v2:172.21.15.142:3300/0 conn(0x7fe684069360 msgr2=0x7fe68405d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:42.452 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.385+0000 7fe6837fe700 1 --2- 172.21.15.112:0/2957082585 >> v2:172.21.15.142:3300/0 conn(0x7fe684069360 0x7fe68405d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.452 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.385+0000 7fe6837fe700 1 -- 172.21.15.112:0/2957082585 >> v1:172.21.15.112:6789/0 conn(0x7fe684067fe0 legacy=0x7fe684069b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.452 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.385+0000 7fe6837fe700 1 -- 172.21.15.112:0/2957082585 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6840c3070 con 0x7fe68406a210 2023-01-24T03:30:42.453 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.386+0000 7fdb108c6700 1 -- 172.21.15.112:0/2736560276 shutdown_connections 2023-01-24T03:30:42.453 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.386+0000 7fdb108c6700 1 --2- 172.21.15.112:0/2736560276 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb0c0c4ef0 0x7fdb0c15e9f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.454 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.386+0000 7fdb108c6700 1 -- 172.21.15.112:0/2736560276 >> 172.21.15.112:0/2736560276 conn(0x7fdb0c066e10 msgr2=0x7fdb0c067210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:42.454 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.389+0000 7fdb108c6700 1 -- 172.21.15.112:0/2736560276 shutdown_connections 2023-01-24T03:30:42.455 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.392+0000 7fda037fe700 1 -- 172.21.15.112:0/3229449885 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fda0c052c90 con 0x7fda1c05e340 2023-01-24T03:30:42.456 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.392+0000 7fe6837fe700 1 --2- 172.21.15.112:0/2957082585 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe68406a210 0x7fe6841559a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe674009760 tx=0x7fe674009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ed7ef65e9edcd5e server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:42.456 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.393+0000 7fe682ffd700 1 -- 172.21.15.112:0/2957082585 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe674011630 con 0x7fe68406a210 2023-01-24T03:30:42.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.393+0000 7fe682ffd700 1 -- 172.21.15.112:0/2957082585 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe674011790 con 0x7fe68406a210 2023-01-24T03:30:42.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.393+0000 7fe682ffd700 1 -- 172.21.15.112:0/2957082585 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe674011b50 con 0x7fe68406a210 2023-01-24T03:30:42.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.394+0000 7fe68a744700 1 -- 172.21.15.112:0/2957082585 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe68406a210 msgr2=0x7fe6841559a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.458 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.394+0000 7fe68a744700 1 --2- 172.21.15.112:0/2957082585 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe68406a210 0x7fe6841559a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe674009760 tx=0x7fe674009060 comp rx=0 tx=0).stop 2023-01-24T03:30:42.458 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.395+0000 7fe68a744700 1 -- 172.21.15.112:0/2957082585 shutdown_connections 2023-01-24T03:30:42.458 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.395+0000 7fe68a744700 1 --2- 172.21.15.112:0/2957082585 >> v2:172.21.15.142:3300/0 conn(0x7fe684069360 0x7fe68405d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.459 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.395+0000 7fe68a744700 1 --2- 172.21.15.112:0/2957082585 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe68406a210 0x7fe6841559a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.459 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.395+0000 7fe68a744700 1 -- 172.21.15.112:0/2957082585 >> 172.21.15.112:0/2957082585 conn(0x7fe68406b550 msgr2=0x7fe68406b950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:42.460 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.400+0000 7fdb108c6700 1 -- 172.21.15.112:0/2736560276 wait complete. 2023-01-24T03:30:42.461 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.401+0000 7fdb108c6700 1 Processor -- start 2023-01-24T03:30:42.461 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.401+0000 7fdb108c6700 1 -- start start 2023-01-24T03:30:42.461 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.402+0000 7fdb108c6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb0c0c2440 0x7fdb0c16d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.462 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.402+0000 7fe68a744700 1 -- 172.21.15.112:0/2957082585 shutdown_connections 2023-01-24T03:30:42.463 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.403+0000 7fe68a744700 1 -- 172.21.15.112:0/2957082585 wait complete. 2023-01-24T03:30:42.463 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.404+0000 7fe68a744700 1 Processor -- start 2023-01-24T03:30:42.464 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.403+0000 7fdb108c6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdb0c0c4ef0 0x7fdb0c14ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.464 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.403+0000 7fdb108c6700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb0c153790 0x7fdb0c16a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.464 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.403+0000 7fdb108c6700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0c16a1a0 con 0x7fdb0c0c4ef0 2023-01-24T03:30:42.464 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.403+0000 7fdb108c6700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0c150d70 con 0x7fdb0c153790 2023-01-24T03:30:42.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.403+0000 7fdb108c6700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0c14b600 con 0x7fdb0c0c2440 2023-01-24T03:30:42.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.403+0000 7fdb0ad9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb0c153790 0x7fdb0c16a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.403+0000 7fdb0ad9d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb0c153790 0x7fdb0c16a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.112:43288/0 (socket says 172.21.15.112:43288) 2023-01-24T03:30:42.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.403+0000 7fdb0ad9d700 1 -- 172.21.15.112:0/3311201623 learned_addr learned my addr 172.21.15.112:0/3311201623 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:42.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.403+0000 7fdb0ad9d700 1 -- 172.21.15.112:0/3311201623 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb0c0c2440 msgr2=0x7fdb0c16d480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-01-24T03:30:42.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.403+0000 7fdb0ad9d700 1 --2- 172.21.15.112:0/3311201623 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb0c0c2440 0x7fdb0c16d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.403+0000 7fdb0ad9d700 1 -- 172.21.15.112:0/3311201623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdb0c0c4ef0 msgr2=0x7fdb0c14ad50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-01-24T03:30:42.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.403+0000 7fdb0ad9d700 1 --2- 172.21.15.112:0/3311201623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdb0c0c4ef0 0x7fdb0c14ad50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.403+0000 7fdb0ad9d700 1 -- 172.21.15.112:0/3311201623 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb04009a20 con 0x7fdb0c153790 2023-01-24T03:30:42.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.404+0000 7fdb0ad9d700 1 --2- 172.21.15.112:0/3311201623 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb0c153790 0x7fdb0c16a970 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c06c3f0 tx=0x7fdb04018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:42.467 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.404+0000 7fdafa7fc700 1 -- 172.21.15.112:0/3311201623 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0401d630 con 0x7fdb0c153790 2023-01-24T03:30:42.467 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.404+0000 7fdafa7fc700 1 -- 172.21.15.112:0/3311201623 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb0401d7b0 con 0x7fdb0c153790 2023-01-24T03:30:42.467 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.404+0000 7fdafa7fc700 1 -- 172.21.15.112:0/3311201623 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0401db70 con 0x7fdb0c153790 2023-01-24T03:30:42.467 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.405+0000 7fe68a744700 1 -- start start 2023-01-24T03:30:42.468 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.406+0000 7fe68a744700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe684067fe0 0x7fe684175470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.468 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.406+0000 7fe68a744700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe684069360 0x7fe6841759b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.468 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.406+0000 7fe68a744700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe68406a210 0x7fe6840bde60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.406+0000 7fe68a744700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68414e6e0 con 0x7fe684067fe0 2023-01-24T03:30:42.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.406+0000 7fe68a744700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6840bd9d0 con 0x7fe684069360 2023-01-24T03:30:42.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.406+0000 7fe68a744700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe684175ff0 con 0x7fe68406a210 2023-01-24T03:30:42.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.406+0000 7fdb108c6700 1 -- 172.21.15.112:0/3311201623 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb0c169870 con 0x7fdb0c153790 2023-01-24T03:30:42.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.406+0000 7fdb108c6700 1 -- 172.21.15.112:0/3311201623 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb0c169e30 con 0x7fdb0c153790 2023-01-24T03:30:42.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.406+0000 7fe6837fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe684069360 0x7fe6841759b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.406+0000 7fe688ce1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe68406a210 0x7fe6840bde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.471 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.406+0000 7fe6837fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe684069360 0x7fe6841759b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.112:43292/0 (socket says 172.21.15.112:43292) 2023-01-24T03:30:42.471 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.406+0000 7fe6837fe700 1 -- 172.21.15.112:0/3951785473 learned_addr learned my addr 172.21.15.112:0/3951785473 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:42.471 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.407+0000 7fe6837fe700 1 -- 172.21.15.112:0/3951785473 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe68406a210 msgr2=0x7fe6840bde60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.471 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.407+0000 7fe6837fe700 1 --2- 172.21.15.112:0/3951785473 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe68406a210 0x7fe6840bde60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.407+0000 7fe6837fe700 1 -- 172.21.15.112:0/3951785473 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe684067fe0 msgr2=0x7fe684175470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.407+0000 7fe6837fe700 1 --2- 172.21.15.112:0/3951785473 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe684067fe0 0x7fe684175470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.407+0000 7fe6837fe700 1 -- 172.21.15.112:0/3951785473 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6740092a0 con 0x7fe684069360 2023-01-24T03:30:42.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.407+0000 7fe6837fe700 1 --2- 172.21.15.112:0/3951785473 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe684069360 0x7fe6841759b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe674009e00 tx=0x7fe674017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:42.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.407+0000 7fe66bfff700 1 -- 172.21.15.112:0/3951785473 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe674011900 con 0x7fe684069360 2023-01-24T03:30:42.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.407+0000 7fe66bfff700 1 -- 172.21.15.112:0/3951785473 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe674011a80 con 0x7fe684069360 2023-01-24T03:30:42.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.407+0000 7fe66bfff700 1 -- 172.21.15.112:0/3951785473 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67401a680 con 0x7fe684069360 2023-01-24T03:30:42.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.408+0000 7fdafa7fc700 1 -- 172.21.15.112:0/3311201623 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fdb0402f3f0 con 0x7fdb0c153790 2023-01-24T03:30:42.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.409+0000 7fdafa7fc700 1 --2- 172.21.15.112:0/3311201623 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fdae80612e0 0x7fdae8063790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.474 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.409+0000 7fdafa7fc700 1 -- 172.21.15.112:0/3311201623 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fdb04080250 con 0x7fdb0c153790 2023-01-24T03:30:42.474 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.408+0000 7fe68a744700 1 -- 172.21.15.112:0/3951785473 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe68414e120 con 0x7fe684069360 2023-01-24T03:30:42.474 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.408+0000 7fe68a744700 1 -- 172.21.15.112:0/3951785473 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe684070520 con 0x7fe684069360 2023-01-24T03:30:42.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.410+0000 7fdb108c6700 1 -- 172.21.15.112:0/3311201623 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb0c0681e0 con 0x7fdb0c153790 2023-01-24T03:30:42.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.410+0000 7fe68a744700 1 -- 172.21.15.112:0/3951785473 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe684155e30 con 0x7fe684069360 2023-01-24T03:30:42.476 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.411+0000 7fe66bfff700 1 -- 172.21.15.112:0/3951785473 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fe67400f050 con 0x7fe684069360 2023-01-24T03:30:42.476 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.411+0000 7fdb0a59c700 1 --2- 172.21.15.112:0/3311201623 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fdae80612e0 0x7fdae8063790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.477 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.411+0000 7fe66bfff700 1 --2- 172.21.15.112:0/3951785473 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fe670061450 0x7fe670063900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.477 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.411+0000 7fe66bfff700 1 -- 172.21.15.112:0/3951785473 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fe67407fa20 con 0x7fe684069360 2023-01-24T03:30:42.477 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.412+0000 7fe683fff700 1 --2- 172.21.15.112:0/3951785473 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fe670061450 0x7fe670063900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.478 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.412+0000 7fdb0a59c700 1 --2- 172.21.15.112:0/3311201623 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fdae80612e0 0x7fdae8063790 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c06c3c0 tx=0x7fdafc007980 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:42.478 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.412+0000 7fe683fff700 1 --2- 172.21.15.112:0/3951785473 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fe670061450 0x7fe670063900 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe678001530 tx=0x7fe6780079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:42.479 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.414+0000 7fdafa7fc700 1 -- 172.21.15.112:0/3311201623 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fdb0404ffc0 con 0x7fdb0c153790 2023-01-24T03:30:42.479 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.418+0000 7fe66bfff700 1 -- 172.21.15.112:0/3951785473 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fe67404f870 con 0x7fe684069360 2023-01-24T03:30:42.695 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.692+0000 7fb928ddf700 1 -- 172.21.15.112:0/2362699950 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fb9040053e0 con 0x7fb9240c8640 2023-01-24T03:30:42.698 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.696+0000 7fb9127fc700 1 -- 172.21.15.112:0/2362699950 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb918051200 con 0x7fb9240c8640 2023-01-24T03:30:42.703 INFO:teuthology.orchestra.run.smithi112.stdout:55834574856 2023-01-24T03:30:42.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.704+0000 7fb928ddf700 1 -- 172.21.15.112:0/2362699950 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb900061290 msgr2=0x7fb900063740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.707 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.704+0000 7fb928ddf700 1 --2- 172.21.15.112:0/2362699950 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb900061290 0x7fb900063740 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb91c003400 tx=0x7fb91c018000 comp rx=0 tx=0).stop 2023-01-24T03:30:42.707 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.704+0000 7fb928ddf700 1 -- 172.21.15.112:0/2362699950 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb9240c8640 msgr2=0x7fb9241618e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.707 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.704+0000 7fb928ddf700 1 --2- 172.21.15.112:0/2362699950 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb9240c8640 0x7fb9241618e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb91800bfd0 tx=0x7fb918008040 comp rx=0 tx=0).stop 2023-01-24T03:30:42.708 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.704+0000 7fb928ddf700 1 -- 172.21.15.112:0/2362699950 shutdown_connections 2023-01-24T03:30:42.708 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.704+0000 7fb928ddf700 1 --2- 172.21.15.112:0/2362699950 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb900061290 0x7fb900063740 secure :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb91c003400 tx=0x7fb91c018000 comp rx=0 tx=0).stop 2023-01-24T03:30:42.708 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.704+0000 7fb928ddf700 1 --2- 172.21.15.112:0/2362699950 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb9240c8640 0x7fb9241618e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.708 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.704+0000 7fb928ddf700 1 --2- 172.21.15.112:0/2362699950 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb9240c3ab0 0x7fb924153e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.709 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.704+0000 7fb928ddf700 1 --2- 172.21.15.112:0/2362699950 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb9240c4d40 0x7fb9241613a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.709 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.704+0000 7fb928ddf700 1 -- 172.21.15.112:0/2362699950 >> 172.21.15.112:0/2362699950 conn(0x7fb924066e10 msgr2=0x7fb9241547a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:42.709 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.705+0000 7fb928ddf700 1 -- 172.21.15.112:0/2362699950 shutdown_connections 2023-01-24T03:30:42.709 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.705+0000 7fb928ddf700 1 -- 172.21.15.112:0/2362699950 wait complete. 2023-01-24T03:30:42.710 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.707+0000 7efd220b1700 1 -- 172.21.15.112:0/4016533414 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7efd1c06a8c0 con 0x7efd1c06b740 2023-01-24T03:30:42.711 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.708+0000 7efd037fe700 1 -- 172.21.15.112:0/4016533414 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efd140507e0 con 0x7efd1c06b740 2023-01-24T03:30:42.713 INFO:teuthology.orchestra.run.smithi112.stdout:55834574856 2023-01-24T03:30:42.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.711+0000 7fb7b89d5700 1 -- 172.21.15.112:0/271463905 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fb7b40c71b0 con 0x7fb7b406d850 2023-01-24T03:30:42.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.712+0000 7fb7a27fc700 1 -- 172.21.15.112:0/271463905 <== mon.1 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb7a8015020 con 0x7fb7b406d850 2023-01-24T03:30:42.715 INFO:teuthology.orchestra.run.smithi112.stdout:55834574856 2023-01-24T03:30:42.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.715+0000 7fb7b89d5700 1 -- 172.21.15.112:0/271463905 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb7900611f0 msgr2=0x7fb7900636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.715+0000 7fb7b89d5700 1 --2- 172.21.15.112:0/271463905 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb7900611f0 0x7fb7900636a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb7b40c9120 tx=0x7fb7a4009060 comp rx=0 tx=0).stop 2023-01-24T03:30:42.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.715+0000 7fb7b89d5700 1 -- 172.21.15.112:0/271463905 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7b406d850 msgr2=0x7fb7b4159530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.715+0000 7fb7b89d5700 1 --2- 172.21.15.112:0/271463905 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7b406d850 0x7fb7b4159530 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb7a8006fd0 tx=0x7fb7a8017000 comp rx=0 tx=0).stop 2023-01-24T03:30:42.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.719+0000 7fb7b89d5700 1 -- 172.21.15.112:0/271463905 shutdown_connections 2023-01-24T03:30:42.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.719+0000 7fb7b89d5700 1 --2- 172.21.15.112:0/271463905 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb7900611f0 0x7fb7900636a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.719+0000 7fb7b89d5700 1 --2- 172.21.15.112:0/271463905 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb7b415cf30 0x7fb7b415dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.719+0000 7fb7b89d5700 1 --2- 172.21.15.112:0/271463905 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7b406d850 0x7fb7b4159530 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.719+0000 7fb7b89d5700 1 --2- 172.21.15.112:0/271463905 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb7b40c4140 0x7fb7b4159a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.719+0000 7fb7b89d5700 1 -- 172.21.15.112:0/271463905 >> 172.21.15.112:0/271463905 conn(0x7fb7b406c4f0 msgr2=0x7fb7b40c8460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:42.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.719+0000 7fb7b89d5700 1 -- 172.21.15.112:0/271463905 shutdown_connections 2023-01-24T03:30:42.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.719+0000 7fb7b89d5700 1 -- 172.21.15.112:0/271463905 wait complete. 2023-01-24T03:30:42.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.720+0000 7efd220b1700 1 -- 172.21.15.112:0/4016533414 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7efcfc061240 msgr2=0x7efcfc0636f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.720+0000 7efd220b1700 1 --2- 172.21.15.112:0/4016533414 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7efcfc061240 0x7efcfc0636f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efd1c0c8530 tx=0x7efd10016000 comp rx=0 tx=0).stop 2023-01-24T03:30:42.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.720+0000 7efd220b1700 1 -- 172.21.15.112:0/4016533414 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efd1c06b740 msgr2=0x7efd1c0c60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.720+0000 7efd220b1700 1 --2- 172.21.15.112:0/4016533414 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efd1c06b740 0x7efd1c0c60e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efd1400c890 tx=0x7efd14001670 comp rx=0 tx=0).stop 2023-01-24T03:30:42.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.722+0000 7efd220b1700 1 -- 172.21.15.112:0/4016533414 shutdown_connections 2023-01-24T03:30:42.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.722+0000 7efd220b1700 1 --2- 172.21.15.112:0/4016533414 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7efcfc061240 0x7efcfc0636f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.722+0000 7efd220b1700 1 --2- 172.21.15.112:0/4016533414 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efd1c0c40e0 0x7efd1c15ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.722+0000 7efd220b1700 1 --2- 172.21.15.112:0/4016533414 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7efd1c06b740 0x7efd1c0c60e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.722+0000 7efd220b1700 1 --2- 172.21.15.112:0/4016533414 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efd1c0b7a60 0x7efd1c1616a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.722+0000 7efd220b1700 1 -- 172.21.15.112:0/4016533414 >> 172.21.15.112:0/4016533414 conn(0x7efd1c066e10 msgr2=0x7efd1c164740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:42.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.722+0000 7efd220b1700 1 -- 172.21.15.112:0/4016533414 shutdown_connections 2023-01-24T03:30:42.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.722+0000 7efd220b1700 1 -- 172.21.15.112:0/4016533414 wait complete. 2023-01-24T03:30:42.730 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.7 2023-01-24T03:30:42.730 DEBUG:teuthology.parallel:result is None 2023-01-24T03:30:42.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.730+0000 7f46ffb6d700 1 -- 172.21.15.112:0/627051049 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f46f805e610 con 0x7f46f8067fe0 2023-01-24T03:30:42.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.731+0000 7f46e5ffb700 1 -- 172.21.15.112:0/627051049 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f46f4011b10 con 0x7f46f8067fe0 2023-01-24T03:30:42.734 INFO:teuthology.orchestra.run.smithi112.stdout:55834574856 2023-01-24T03:30:42.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.734+0000 7f46ffb6d700 1 -- 172.21.15.112:0/627051049 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f46d8061340 msgr2=0x7f46d80637f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.734+0000 7f46ffb6d700 1 --2- 172.21.15.112:0/627051049 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f46d8061340 0x7f46d80637f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f46f80c1cd0 tx=0x7f46e8009040 comp rx=0 tx=0).stop 2023-01-24T03:30:42.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.734+0000 7f46ffb6d700 1 -- 172.21.15.112:0/627051049 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f46f8067fe0 msgr2=0x7f46f8166e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.734+0000 7f46ffb6d700 1 --2- 172.21.15.112:0/627051049 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f46f8067fe0 0x7f46f8166e30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f46f4015000 tx=0x7f46f4016000 comp rx=0 tx=0).stop 2023-01-24T03:30:42.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.736+0000 7f46ffb6d700 1 -- 172.21.15.112:0/627051049 shutdown_connections 2023-01-24T03:30:42.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.736+0000 7f46ffb6d700 1 --2- 172.21.15.112:0/627051049 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f46d8061340 0x7f46d80637f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.736+0000 7f46ffb6d700 1 --2- 172.21.15.112:0/627051049 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f46f80c2780 0x7f46f8168fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.736+0000 7f46ffb6d700 1 --2- 172.21.15.112:0/627051049 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f46f8067fe0 0x7f46f8166e30 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.736+0000 7f46ffb6d700 1 --2- 172.21.15.112:0/627051049 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46f806ebd0 0x7f46f815c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.736+0000 7f46ffb6d700 1 -- 172.21.15.112:0/627051049 >> 172.21.15.112:0/627051049 conn(0x7f46f806b550 msgr2=0x7f46f8153b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:42.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.737+0000 7f46ffb6d700 1 -- 172.21.15.112:0/627051049 shutdown_connections 2023-01-24T03:30:42.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.737+0000 7f46ffb6d700 1 -- 172.21.15.112:0/627051049 wait complete. 2023-01-24T03:30:42.743 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.6 2023-01-24T03:30:42.744 DEBUG:teuthology.parallel:result is None 2023-01-24T03:30:42.745 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.1 2023-01-24T03:30:42.745 DEBUG:teuthology.parallel:result is None 2023-01-24T03:30:42.757 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.3 2023-01-24T03:30:42.758 DEBUG:teuthology.parallel:result is None 2023-01-24T03:30:42.763 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.759+0000 7fea7183f700 1 -- 172.21.15.112:0/3895003326 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fea6c161560 con 0x7fea6c0b7a60 2023-01-24T03:30:42.763 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.760+0000 7fea52ffd700 1 -- 172.21.15.112:0/3895003326 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fea6c161560 con 0x7fea6c0b7a60 2023-01-24T03:30:42.764 INFO:teuthology.orchestra.run.smithi112.stdout:55834574856 2023-01-24T03:30:42.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.763+0000 7fea7183f700 1 -- 172.21.15.112:0/3895003326 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fea4c061290 msgr2=0x7fea4c063740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.763+0000 7fea7183f700 1 --2- 172.21.15.112:0/3895003326 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fea4c061290 0x7fea4c063740 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fea5c006fd0 tx=0x7fea5c017000 comp rx=0 tx=0).stop 2023-01-24T03:30:42.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.763+0000 7fea7183f700 1 -- 172.21.15.112:0/3895003326 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea6c0b7a60 msgr2=0x7fea6c15eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.763+0000 7fea7183f700 1 --2- 172.21.15.112:0/3895003326 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea6c0b7a60 0x7fea6c15eef0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fea6000d890 tx=0x7fea60008040 comp rx=0 tx=0).stop 2023-01-24T03:30:42.767 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.763+0000 7fea7183f700 1 -- 172.21.15.112:0/3895003326 shutdown_connections 2023-01-24T03:30:42.767 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.763+0000 7fea7183f700 1 --2- 172.21.15.112:0/3895003326 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fea4c061290 0x7fea4c063740 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.767 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.763+0000 7fea7183f700 1 --2- 172.21.15.112:0/3895003326 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea6c0c40e0 0x7fea6c15dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.767 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.763+0000 7fea7183f700 1 --2- 172.21.15.112:0/3895003326 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea6c06d210 0x7fea6c15ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.768 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.763+0000 7fea7183f700 1 --2- 172.21.15.112:0/3895003326 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea6c0b7a60 0x7fea6c15eef0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.768 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.763+0000 7fea7183f700 1 -- 172.21.15.112:0/3895003326 >> 172.21.15.112:0/3895003326 conn(0x7fea6c066e10 msgr2=0x7fea6c167f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:42.768 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.763+0000 7fea7183f700 1 -- 172.21.15.112:0/3895003326 shutdown_connections 2023-01-24T03:30:42.768 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.763+0000 7fea7183f700 1 -- 172.21.15.112:0/3895003326 wait complete. 2023-01-24T03:30:42.772 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.770+0000 7fda2209e700 1 -- 172.21.15.112:0/3229449885 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fda1c06f3a0 con 0x7fda1c05e340 2023-01-24T03:30:42.773 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.770+0000 7fda037fe700 1 -- 172.21.15.112:0/3229449885 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fda0c052880 con 0x7fda1c05e340 2023-01-24T03:30:42.773 INFO:teuthology.orchestra.run.smithi112.stdout:55834574856 2023-01-24T03:30:42.776 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.773+0000 7fda2209e700 1 -- 172.21.15.112:0/3229449885 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd9fc061290 msgr2=0x7fd9fc063740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.777 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.773+0000 7fda2209e700 1 --2- 172.21.15.112:0/3229449885 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd9fc061290 0x7fd9fc063740 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fda10005750 tx=0x7fda100010a0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.777 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.773+0000 7fda2209e700 1 -- 172.21.15.112:0/3229449885 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fda1c05e340 msgr2=0x7fda1c166590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.777 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.773+0000 7fda2209e700 1 --2- 172.21.15.112:0/3229449885 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fda1c05e340 0x7fda1c166590 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fda0c009270 tx=0x7fda0c017000 comp rx=0 tx=0).stop 2023-01-24T03:30:42.777 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.773+0000 7fda2209e700 1 -- 172.21.15.112:0/3229449885 shutdown_connections 2023-01-24T03:30:42.777 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.773+0000 7fda2209e700 1 --2- 172.21.15.112:0/3229449885 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd9fc061290 0x7fd9fc063740 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.778 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.773+0000 7fda2209e700 1 --2- 172.21.15.112:0/3229449885 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fda1c05e340 0x7fda1c166590 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.778 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.773+0000 7fda2209e700 1 --2- 172.21.15.112:0/3229449885 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fda1c167060 0x7fda1c173f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.778 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.773+0000 7fda2209e700 1 --2- 172.21.15.112:0/3229449885 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fda1c1569e0 0x7fda1c158160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.778 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.773+0000 7fda2209e700 1 -- 172.21.15.112:0/3229449885 >> 172.21.15.112:0/3229449885 conn(0x7fda1c05d830 msgr2=0x7fda1c0c6510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:42.778 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.773+0000 7fda2209e700 1 -- 172.21.15.112:0/3229449885 shutdown_connections 2023-01-24T03:30:42.779 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.773+0000 7fda2209e700 1 -- 172.21.15.112:0/3229449885 wait complete. 2023-01-24T03:30:42.780 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.2 2023-01-24T03:30:42.780 DEBUG:teuthology.parallel:result is None 2023-01-24T03:30:42.792 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.4 2023-01-24T03:30:42.793 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.790+0000 7fdb108c6700 1 -- 172.21.15.112:0/3311201623 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fdb0c04b340 con 0x7fdb0c153790 2023-01-24T03:30:42.793 DEBUG:teuthology.parallel:result is None 2023-01-24T03:30:42.794 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.790+0000 7fdafa7fc700 1 -- 172.21.15.112:0/3311201623 <== mon.1 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdb04056e60 con 0x7fdb0c153790 2023-01-24T03:30:42.794 INFO:teuthology.orchestra.run.smithi112.stdout:55834574857 2023-01-24T03:30:42.795 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.792+0000 7fdb108c6700 1 -- 172.21.15.112:0/3311201623 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fdae80612e0 msgr2=0x7fdae8063790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.796 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.792+0000 7fdb108c6700 1 --2- 172.21.15.112:0/3311201623 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fdae80612e0 0x7fdae8063790 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c06c3c0 tx=0x7fdafc007980 comp rx=0 tx=0).stop 2023-01-24T03:30:42.796 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.792+0000 7fdb108c6700 1 -- 172.21.15.112:0/3311201623 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb0c153790 msgr2=0x7fdb0c16a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.796 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.792+0000 7fdb108c6700 1 --2- 172.21.15.112:0/3311201623 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb0c153790 0x7fdb0c16a970 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c06c3f0 tx=0x7fdb04018000 comp rx=0 tx=0).stop 2023-01-24T03:30:42.796 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.792+0000 7fdb108c6700 1 -- 172.21.15.112:0/3311201623 shutdown_connections 2023-01-24T03:30:42.796 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.792+0000 7fdb108c6700 1 --2- 172.21.15.112:0/3311201623 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fdae80612e0 0x7fdae8063790 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.797 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.792+0000 7fdb108c6700 1 --2- 172.21.15.112:0/3311201623 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb0c153790 0x7fdb0c16a970 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.797 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.792+0000 7fdb108c6700 1 --2- 172.21.15.112:0/3311201623 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb0c0c2440 0x7fdb0c16d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.797 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.792+0000 7fdb108c6700 1 --2- 172.21.15.112:0/3311201623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdb0c0c4ef0 0x7fdb0c14ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.797 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.792+0000 7fdb108c6700 1 -- 172.21.15.112:0/3311201623 >> 172.21.15.112:0/3311201623 conn(0x7fdb0c066e10 msgr2=0x7fdb0c154c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:42.797 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.792+0000 7fdb108c6700 1 -- 172.21.15.112:0/3311201623 shutdown_connections 2023-01-24T03:30:42.798 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.792+0000 7fdb108c6700 1 -- 172.21.15.112:0/3311201623 wait complete. 2023-01-24T03:30:42.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.802+0000 7fe68a744700 1 -- 172.21.15.112:0/3951785473 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fe68406ae30 con 0x7fe684069360 2023-01-24T03:30:42.805 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.802+0000 7fe66bfff700 1 -- 172.21.15.112:0/3951785473 <== mon.1 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe674015020 con 0x7fe684069360 2023-01-24T03:30:42.805 INFO:teuthology.orchestra.run.smithi112.stdout:55834574857 2023-01-24T03:30:42.807 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.5 2023-01-24T03:30:42.807 DEBUG:teuthology.parallel:result is None 2023-01-24T03:30:42.808 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.805+0000 7fe68a744700 1 -- 172.21.15.112:0/3951785473 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fe670061450 msgr2=0x7fe670063900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.805+0000 7fe68a744700 1 --2- 172.21.15.112:0/3951785473 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fe670061450 0x7fe670063900 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe678001530 tx=0x7fe6780079b0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.805+0000 7fe68a744700 1 -- 172.21.15.112:0/3951785473 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe684069360 msgr2=0x7fe6841759b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.805+0000 7fe68a744700 1 --2- 172.21.15.112:0/3951785473 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe684069360 0x7fe6841759b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe674009e00 tx=0x7fe674017000 comp rx=0 tx=0).stop 2023-01-24T03:30:42.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.805+0000 7fe68a744700 1 -- 172.21.15.112:0/3951785473 shutdown_connections 2023-01-24T03:30:42.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.805+0000 7fe68a744700 1 --2- 172.21.15.112:0/3951785473 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fe670061450 0x7fe670063900 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.805+0000 7fe68a744700 1 --2- 172.21.15.112:0/3951785473 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe684067fe0 0x7fe684175470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.805+0000 7fe68a744700 1 --2- 172.21.15.112:0/3951785473 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe68406a210 0x7fe6840bde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.805+0000 7fe68a744700 1 --2- 172.21.15.112:0/3951785473 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe684069360 0x7fe6841759b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.805+0000 7fe68a744700 1 -- 172.21.15.112:0/3951785473 >> 172.21.15.112:0/3951785473 conn(0x7fe68406b550 msgr2=0x7fe68406e540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:42.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.805+0000 7fe68a744700 1 -- 172.21.15.112:0/3951785473 shutdown_connections 2023-01-24T03:30:42.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.805+0000 7fe68a744700 1 -- 172.21.15.112:0/3951785473 wait complete. 2023-01-24T03:30:42.820 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.0 2023-01-24T03:30:42.820 DEBUG:teuthology.parallel:result is None 2023-01-24T03:30:42.820 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-01-24T03:30:42.821 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-24T03:30:42.972 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.969+0000 7f27bdc47700 1 Processor -- start 2023-01-24T03:30:42.972 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.969+0000 7f27bdc47700 1 -- start start 2023-01-24T03:30:42.975 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.970+0000 7f27bdc47700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f27b80b7550 0x7f27b806a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.975 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.970+0000 7f27bdc47700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f27b80b7920 0x7f27b81535e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.975 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.970+0000 7f27bdc47700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27b806d070 con 0x7f27b8153b20 2023-01-24T03:30:42.976 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.970+0000 7f27bdc47700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27b81573b0 con 0x7f27b80b7550 2023-01-24T03:30:42.976 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.970+0000 7f27bdc47700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27b8153150 con 0x7f27b80b7920 2023-01-24T03:30:42.976 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.970+0000 7f27bcc45700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f27b80b7550 0x7f27b806a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.977 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.970+0000 7f27bcc45700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f27b80b7550 0x7f27b806a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50806/0 (socket says 172.21.15.112:50806) 2023-01-24T03:30:42.977 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.970+0000 7f27bcc45700 1 -- 172.21.15.112:0/3532464425 learned_addr learned my addr 172.21.15.112:0/3532464425 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:42.977 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.970+0000 7f27b7fff700 1 --2- 172.21.15.112:0/3532464425 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f27b80b7920 0x7f27b81535e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.977 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.970+0000 7f27bcc45700 1 -- 172.21.15.112:0/3532464425 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f27b80b7920 msgr2=0x7f27b81535e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:42.977 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.970+0000 7f27bcc45700 1 --2- 172.21.15.112:0/3532464425 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f27b80b7920 0x7f27b81535e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.970+0000 7f27bcc45700 1 -- 172.21.15.112:0/3532464425 >> v1:172.21.15.142:6789/0 conn(0x7f27b8153b20 legacy=0x7f27b80c3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:42.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.970+0000 7f27bcc45700 1 -- 172.21.15.112:0/3532464425 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b8156830 con 0x7f27b80b7550 2023-01-24T03:30:42.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.970+0000 7f27b7fff700 1 --2- 172.21.15.112:0/3532464425 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f27b80b7920 0x7f27b81535e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-01-24T03:30:42.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.970+0000 7f27bcc45700 1 --2- 172.21.15.112:0/3532464425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27b80b7550 0x7f27b806a5c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f27a8006fd0 tx=0x7f27a80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eeafec3f04cb47e server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:42.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.971+0000 7f27b77fe700 1 -- 172.21.15.112:0/3532464425 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a800b690 con 0x7f27b80b7550 2023-01-24T03:30:42.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.971+0000 7f27b77fe700 1 -- 172.21.15.112:0/3532464425 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27a800b810 con 0x7f27b80b7550 2023-01-24T03:30:42.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.971+0000 7f27b77fe700 1 -- 172.21.15.112:0/3532464425 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a800bbd0 con 0x7f27b80b7550 2023-01-24T03:30:42.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.971+0000 7f27bdc47700 1 -- 172.21.15.112:0/3532464425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27b80b7550 msgr2=0x7f27b806a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.971+0000 7f27bdc47700 1 --2- 172.21.15.112:0/3532464425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27b80b7550 0x7f27b806a5c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f27a8006fd0 tx=0x7f27a80090f0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.971+0000 7f27bdc47700 1 -- 172.21.15.112:0/3532464425 shutdown_connections 2023-01-24T03:30:42.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.971+0000 7f27bdc47700 1 --2- 172.21.15.112:0/3532464425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27b80b7550 0x7f27b806a5c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.980 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.971+0000 7f27bdc47700 1 --2- 172.21.15.112:0/3532464425 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f27b80b7920 0x7f27b81535e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.980 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.971+0000 7f27bdc47700 1 -- 172.21.15.112:0/3532464425 >> 172.21.15.112:0/3532464425 conn(0x7f27b8066e10 msgr2=0x7f27b8067210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:42.980 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.971+0000 7f27bdc47700 1 -- 172.21.15.112:0/3532464425 shutdown_connections 2023-01-24T03:30:42.980 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.971+0000 7f27bdc47700 1 -- 172.21.15.112:0/3532464425 wait complete. 2023-01-24T03:30:42.980 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.972+0000 7f27bdc47700 1 Processor -- start 2023-01-24T03:30:42.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.972+0000 7f27bdc47700 1 -- start start 2023-01-24T03:30:42.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.973+0000 7f27bdc47700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27b80b7550 0x7f27b80b8e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.973+0000 7f27bdc47700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f27b80b7920 0x7f27b8163090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.985 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.973+0000 7f27bdc47700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f27b8153b20 0x7f27b81635d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.985 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.973+0000 7f27bdc47700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b814fb10 con 0x7f27b80b7550 2023-01-24T03:30:42.985 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.973+0000 7f27bdc47700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b80b62b0 con 0x7f27b8153b20 2023-01-24T03:30:42.985 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.973+0000 7f27bdc47700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27b8152300 con 0x7f27b80b7920 2023-01-24T03:30:42.985 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.973+0000 7f27bcc45700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27b80b7550 0x7f27b80b8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.986 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.973+0000 7f27bcc45700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27b80b7550 0x7f27b80b8e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50812/0 (socket says 172.21.15.112:50812) 2023-01-24T03:30:42.986 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.973+0000 7f27bcc45700 1 -- 172.21.15.112:0/2369914041 learned_addr learned my addr 172.21.15.112:0/2369914041 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:42.986 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.973+0000 7f27b7fff700 1 --2- 172.21.15.112:0/2369914041 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f27b80b7920 0x7f27b8163090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.986 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.973+0000 7f27bd446700 1 --2- 172.21.15.112:0/2369914041 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f27b8153b20 0x7f27b81635d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.986 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.973+0000 7f27bcc45700 1 -- 172.21.15.112:0/2369914041 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f27b80b7920 msgr2=0x7f27b8163090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.987 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.973+0000 7f27bcc45700 1 --2- 172.21.15.112:0/2369914041 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f27b80b7920 0x7f27b8163090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.987 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.973+0000 7f27bcc45700 1 -- 172.21.15.112:0/2369914041 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f27b8153b20 msgr2=0x7f27b81635d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:42.987 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.973+0000 7f27bcc45700 1 --2- 172.21.15.112:0/2369914041 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f27b8153b20 0x7f27b81635d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:42.987 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.973+0000 7f27bcc45700 1 -- 172.21.15.112:0/2369914041 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a8009370 con 0x7f27b80b7550 2023-01-24T03:30:42.987 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.974+0000 7f27bcc45700 1 --2- 172.21.15.112:0/2369914041 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27b80b7550 0x7f27b80b8e40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f27a8013000 tx=0x7f27a8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:42.987 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.974+0000 7f27b4ff9700 1 -- 172.21.15.112:0/2369914041 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a8000c90 con 0x7f27b80b7550 2023-01-24T03:30:42.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.974+0000 7f27b4ff9700 1 -- 172.21.15.112:0/2369914041 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27a800b980 con 0x7f27b80b7550 2023-01-24T03:30:42.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.974+0000 7f27bdc47700 1 -- 172.21.15.112:0/2369914041 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27b81525c0 con 0x7f27b80b7550 2023-01-24T03:30:42.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.974+0000 7f27b4ff9700 1 -- 172.21.15.112:0/2369914041 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a8025420 con 0x7f27b80b7550 2023-01-24T03:30:42.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.974+0000 7f27bdc47700 1 -- 172.21.15.112:0/2369914041 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27b80b5980 con 0x7f27b80b7550 2023-01-24T03:30:42.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.975+0000 7f27bdc47700 1 -- 172.21.15.112:0/2369914041 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27b80714e0 con 0x7f27b80b7550 2023-01-24T03:30:42.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.976+0000 7f27b4ff9700 1 -- 172.21.15.112:0/2369914041 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f27a802c030 con 0x7f27b80b7550 2023-01-24T03:30:42.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.976+0000 7f27b4ff9700 1 --2- 172.21.15.112:0/2369914041 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f279c0611f0 0x7f279c0636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:42.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.976+0000 7f27b4ff9700 1 -- 172.21.15.112:0/2369914041 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f27a807f200 con 0x7f27b80b7550 2023-01-24T03:30:42.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.976+0000 7f27b7fff700 1 --2- 172.21.15.112:0/2369914041 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f279c0611f0 0x7f279c0636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:42.990 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.976+0000 7f27b7fff700 1 --2- 172.21.15.112:0/2369914041 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f279c0611f0 0x7f279c0636a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f27b80be5b0 tx=0x7f27a0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:42.990 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:42.980+0000 7f27b4ff9700 1 -- 172.21.15.112:0/2369914041 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f27a804f0c0 con 0x7f27b80b7550 2023-01-24T03:30:43.156 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.153+0000 7f27bdc47700 1 -- 172.21.15.112:0/2369914041 --> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f27b805ea20 con 0x7f279c0611f0 2023-01-24T03:30:43.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.162+0000 7f27b4ff9700 1 -- 172.21.15.112:0/2369914041 <== mgr.4104 v2:172.21.15.112:6832/84276 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+365109 (secure 0 0 0) 0x7f27b805ea20 con 0x7f279c0611f0 2023-01-24T03:30:43.171 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:30:43.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.166+0000 7f27bdc47700 1 -- 172.21.15.112:0/2369914041 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f279c0611f0 msgr2=0x7f279c0636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:43.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.166+0000 7f27bdc47700 1 --2- 172.21.15.112:0/2369914041 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f279c0611f0 0x7f279c0636a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f27b80be5b0 tx=0x7f27a0008040 comp rx=0 tx=0).stop 2023-01-24T03:30:43.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.166+0000 7f27bdc47700 1 -- 172.21.15.112:0/2369914041 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27b80b7550 msgr2=0x7f27b80b8e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:43.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.166+0000 7f27bdc47700 1 --2- 172.21.15.112:0/2369914041 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27b80b7550 0x7f27b80b8e40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f27a8013000 tx=0x7f27a8017000 comp rx=0 tx=0).stop 2023-01-24T03:30:43.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.166+0000 7f27bdc47700 1 -- 172.21.15.112:0/2369914041 shutdown_connections 2023-01-24T03:30:43.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.166+0000 7f27bdc47700 1 --2- 172.21.15.112:0/2369914041 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f279c0611f0 0x7f279c0636a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:43.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.166+0000 7f27bdc47700 1 --2- 172.21.15.112:0/2369914041 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27b80b7550 0x7f27b80b8e40 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:43.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.166+0000 7f27bdc47700 1 --2- 172.21.15.112:0/2369914041 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f27b8153b20 0x7f27b81635d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:43.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.166+0000 7f27bdc47700 1 --2- 172.21.15.112:0/2369914041 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f27b80b7920 0x7f27b8163090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:43.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.166+0000 7f27bdc47700 1 -- 172.21.15.112:0/2369914041 >> 172.21.15.112:0/2369914041 conn(0x7f27b8066e10 msgr2=0x7f27b806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:43.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.166+0000 7f27bdc47700 1 -- 172.21.15.112:0/2369914041 shutdown_connections 2023-01-24T03:30:43.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.166+0000 7f27bdc47700 1 -- 172.21.15.112:0/2369914041 wait complete. 2023-01-24T03:30:43.176 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2023-01-24T03:30:43.181 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2023-01-24T03:30:43.007270+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594851,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":69880,"kb_used_data":2168,"kb_used_omap":0,"kb_used_meta":67648,"kb_avail":754904840,"statfs":{"total":773094113280,"available":773022556160,"internally_reserved":0,"allocated":2220032,"data_stored":2023215,"data_compressed":64574,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":69271552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2177,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009801"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993289+0000","last_change":"2023-01-24T03:30:20.986308+0000","last_active":"2023-01-24T03:30:24.993289+0000","last_peered":"2023-01-24T03:30:24.993289+0000","last_clean":"2023-01-24T03:30:24.993289+0000","last_became_active":"2023-01-24T03:30:20.986025+0000","last_became_peered":"2023-01-24T03:30:20.986025+0000","last_unstale":"2023-01-24T03:30:24.993289+0000","last_undegraded":"2023-01-24T03:30:24.993289+0000","last_fullsized":"2023-01-24T03:30:24.993289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:47:58.616698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.006202+0000","last_change":"2023-01-24T03:30:20.994478+0000","last_active":"2023-01-24T03:30:25.006202+0000","last_peered":"2023-01-24T03:30:25.006202+0000","last_clean":"2023-01-24T03:30:25.006202+0000","last_became_active":"2023-01-24T03:30:20.994207+0000","last_became_peered":"2023-01-24T03:30:20.994207+0000","last_unstale":"2023-01-24T03:30:25.006202+0000","last_undegraded":"2023-01-24T03:30:25.006202+0000","last_fullsized":"2023-01-24T03:30:25.006202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:02:53.230231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993177+0000","last_change":"2023-01-24T03:30:20.989651+0000","last_active":"2023-01-24T03:30:24.993177+0000","last_peered":"2023-01-24T03:30:24.993177+0000","last_clean":"2023-01-24T03:30:24.993177+0000","last_became_active":"2023-01-24T03:30:20.987585+0000","last_became_peered":"2023-01-24T03:30:20.987585+0000","last_unstale":"2023-01-24T03:30:24.993177+0000","last_undegraded":"2023-01-24T03:30:24.993177+0000","last_fullsized":"2023-01-24T03:30:24.993177+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:36:02.177544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990511+0000","last_change":"2023-01-24T03:30:20.993601+0000","last_active":"2023-01-24T03:30:24.990511+0000","last_peered":"2023-01-24T03:30:24.990511+0000","last_clean":"2023-01-24T03:30:24.990511+0000","last_became_active":"2023-01-24T03:30:20.993418+0000","last_became_peered":"2023-01-24T03:30:20.993418+0000","last_unstale":"2023-01-24T03:30:24.990511+0000","last_undegraded":"2023-01-24T03:30:24.990511+0000","last_fullsized":"2023-01-24T03:30:24.990511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:47:50.260497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996430+0000","last_change":"2023-01-24T03:30:20.991605+0000","last_active":"2023-01-24T03:30:24.996430+0000","last_peered":"2023-01-24T03:30:24.996430+0000","last_clean":"2023-01-24T03:30:24.996430+0000","last_became_active":"2023-01-24T03:30:20.991361+0000","last_became_peered":"2023-01-24T03:30:20.991361+0000","last_unstale":"2023-01-24T03:30:24.996430+0000","last_undegraded":"2023-01-24T03:30:24.996430+0000","last_fullsized":"2023-01-24T03:30:24.996430+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:31:25.418430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993288+0000","last_change":"2023-01-24T03:30:20.985058+0000","last_active":"2023-01-24T03:30:24.993288+0000","last_peered":"2023-01-24T03:30:24.993288+0000","last_clean":"2023-01-24T03:30:24.993288+0000","last_became_active":"2023-01-24T03:30:20.984754+0000","last_became_peered":"2023-01-24T03:30:20.984754+0000","last_unstale":"2023-01-24T03:30:24.993288+0000","last_undegraded":"2023-01-24T03:30:24.993288+0000","last_fullsized":"2023-01-24T03:30:24.993288+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:12:03.225109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996279+0000","last_change":"2023-01-24T03:30:21.000890+0000","last_active":"2023-01-24T03:30:24.996279+0000","last_peered":"2023-01-24T03:30:24.996279+0000","last_clean":"2023-01-24T03:30:24.996279+0000","last_became_active":"2023-01-24T03:30:21.000438+0000","last_became_peered":"2023-01-24T03:30:21.000438+0000","last_unstale":"2023-01-24T03:30:24.996279+0000","last_undegraded":"2023-01-24T03:30:24.996279+0000","last_fullsized":"2023-01-24T03:30:24.996279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:17:12.999664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.004717+0000","last_change":"2023-01-24T03:30:20.986753+0000","last_active":"2023-01-24T03:30:25.004717+0000","last_peered":"2023-01-24T03:30:25.004717+0000","last_clean":"2023-01-24T03:30:25.004717+0000","last_became_active":"2023-01-24T03:30:20.986524+0000","last_became_peered":"2023-01-24T03:30:20.986524+0000","last_unstale":"2023-01-24T03:30:25.004717+0000","last_undegraded":"2023-01-24T03:30:25.004717+0000","last_fullsized":"2023-01-24T03:30:25.004717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:24:55.699379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992350+0000","last_change":"2023-01-24T03:30:20.991136+0000","last_active":"2023-01-24T03:30:24.992350+0000","last_peered":"2023-01-24T03:30:24.992350+0000","last_clean":"2023-01-24T03:30:24.992350+0000","last_became_active":"2023-01-24T03:30:20.990925+0000","last_became_peered":"2023-01-24T03:30:20.990925+0000","last_unstale":"2023-01-24T03:30:24.992350+0000","last_undegraded":"2023-01-24T03:30:24.992350+0000","last_fullsized":"2023-01-24T03:30:24.992350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:45:37.485199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.006401+0000","last_change":"2023-01-24T03:30:20.995730+0000","last_active":"2023-01-24T03:30:25.006401+0000","last_peered":"2023-01-24T03:30:25.006401+0000","last_clean":"2023-01-24T03:30:25.006401+0000","last_became_active":"2023-01-24T03:30:20.994880+0000","last_became_peered":"2023-01-24T03:30:20.994880+0000","last_unstale":"2023-01-24T03:30:25.006401+0000","last_undegraded":"2023-01-24T03:30:25.006401+0000","last_fullsized":"2023-01-24T03:30:25.006401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:07:21.546386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993144+0000","last_change":"2023-01-24T03:30:20.993328+0000","last_active":"2023-01-24T03:30:24.993144+0000","last_peered":"2023-01-24T03:30:24.993144+0000","last_clean":"2023-01-24T03:30:24.993144+0000","last_became_active":"2023-01-24T03:30:20.992730+0000","last_became_peered":"2023-01-24T03:30:20.992730+0000","last_unstale":"2023-01-24T03:30:24.993144+0000","last_undegraded":"2023-01-24T03:30:24.993144+0000","last_fullsized":"2023-01-24T03:30:24.993144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:43:39.048550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.994605+0000","last_change":"2023-01-24T03:30:20.984993+0000","last_active":"2023-01-24T03:30:24.994605+0000","last_peered":"2023-01-24T03:30:24.994605+0000","last_clean":"2023-01-24T03:30:24.994605+0000","last_became_active":"2023-01-24T03:30:20.984540+0000","last_became_peered":"2023-01-24T03:30:20.984540+0000","last_unstale":"2023-01-24T03:30:24.994605+0000","last_undegraded":"2023-01-24T03:30:24.994605+0000","last_fullsized":"2023-01-24T03:30:24.994605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:16:39.993359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.995936+0000","last_change":"2023-01-24T03:30:20.986567+0000","last_active":"2023-01-24T03:30:24.995936+0000","last_peered":"2023-01-24T03:30:24.995936+0000","last_clean":"2023-01-24T03:30:24.995936+0000","last_became_active":"2023-01-24T03:30:20.986224+0000","last_became_peered":"2023-01-24T03:30:20.986224+0000","last_unstale":"2023-01-24T03:30:24.995936+0000","last_undegraded":"2023-01-24T03:30:24.995936+0000","last_fullsized":"2023-01-24T03:30:24.995936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:09:14.692354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005087+0000","last_change":"2023-01-24T03:30:20.988843+0000","last_active":"2023-01-24T03:30:25.005087+0000","last_peered":"2023-01-24T03:30:25.005087+0000","last_clean":"2023-01-24T03:30:25.005087+0000","last_became_active":"2023-01-24T03:30:20.988486+0000","last_became_peered":"2023-01-24T03:30:20.988486+0000","last_unstale":"2023-01-24T03:30:25.005087+0000","last_undegraded":"2023-01-24T03:30:25.005087+0000","last_fullsized":"2023-01-24T03:30:25.005087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:18:20.538844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996110+0000","last_change":"2023-01-24T03:30:21.000738+0000","last_active":"2023-01-24T03:30:24.996110+0000","last_peered":"2023-01-24T03:30:24.996110+0000","last_clean":"2023-01-24T03:30:24.996110+0000","last_became_active":"2023-01-24T03:30:21.000458+0000","last_became_peered":"2023-01-24T03:30:21.000458+0000","last_unstale":"2023-01-24T03:30:24.996110+0000","last_undegraded":"2023-01-24T03:30:24.996110+0000","last_fullsized":"2023-01-24T03:30:24.996110+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:21:42.876426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992383+0000","last_change":"2023-01-24T03:30:20.985062+0000","last_active":"2023-01-24T03:30:24.992383+0000","last_peered":"2023-01-24T03:30:24.992383+0000","last_clean":"2023-01-24T03:30:24.992383+0000","last_became_active":"2023-01-24T03:30:20.984759+0000","last_became_peered":"2023-01-24T03:30:20.984759+0000","last_unstale":"2023-01-24T03:30:24.992383+0000","last_undegraded":"2023-01-24T03:30:24.992383+0000","last_fullsized":"2023-01-24T03:30:24.992383+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:38:34.615748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992439+0000","last_change":"2023-01-24T03:30:20.989080+0000","last_active":"2023-01-24T03:30:24.992439+0000","last_peered":"2023-01-24T03:30:24.992439+0000","last_clean":"2023-01-24T03:30:24.992439+0000","last_became_active":"2023-01-24T03:30:20.988823+0000","last_became_peered":"2023-01-24T03:30:20.988823+0000","last_unstale":"2023-01-24T03:30:24.992439+0000","last_undegraded":"2023-01-24T03:30:24.992439+0000","last_fullsized":"2023-01-24T03:30:24.992439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:22:21.807370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993008+0000","last_change":"2023-01-24T03:30:20.984222+0000","last_active":"2023-01-24T03:30:24.993008+0000","last_peered":"2023-01-24T03:30:24.993008+0000","last_clean":"2023-01-24T03:30:24.993008+0000","last_became_active":"2023-01-24T03:30:20.984002+0000","last_became_peered":"2023-01-24T03:30:20.984002+0000","last_unstale":"2023-01-24T03:30:24.993008+0000","last_undegraded":"2023-01-24T03:30:24.993008+0000","last_fullsized":"2023-01-24T03:30:24.993008+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:54:24.005486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.997598+0000","last_change":"2023-01-24T03:30:20.988197+0000","last_active":"2023-01-24T03:30:24.997598+0000","last_peered":"2023-01-24T03:30:24.997598+0000","last_clean":"2023-01-24T03:30:24.997598+0000","last_became_active":"2023-01-24T03:30:20.987941+0000","last_became_peered":"2023-01-24T03:30:20.987941+0000","last_unstale":"2023-01-24T03:30:24.997598+0000","last_undegraded":"2023-01-24T03:30:24.997598+0000","last_fullsized":"2023-01-24T03:30:24.997598+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T15:08:29.659964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992957+0000","last_change":"2023-01-24T03:30:20.988701+0000","last_active":"2023-01-24T03:30:24.992957+0000","last_peered":"2023-01-24T03:30:24.992957+0000","last_clean":"2023-01-24T03:30:24.992957+0000","last_became_active":"2023-01-24T03:30:20.988476+0000","last_became_peered":"2023-01-24T03:30:20.988476+0000","last_unstale":"2023-01-24T03:30:24.992957+0000","last_undegraded":"2023-01-24T03:30:24.992957+0000","last_fullsized":"2023-01-24T03:30:24.992957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:33:29.463194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996546+0000","last_change":"2023-01-24T03:30:21.001125+0000","last_active":"2023-01-24T03:30:24.996546+0000","last_peered":"2023-01-24T03:30:24.996546+0000","last_clean":"2023-01-24T03:30:24.996546+0000","last_became_active":"2023-01-24T03:30:21.000873+0000","last_became_peered":"2023-01-24T03:30:21.000873+0000","last_unstale":"2023-01-24T03:30:24.996546+0000","last_undegraded":"2023-01-24T03:30:24.996546+0000","last_fullsized":"2023-01-24T03:30:24.996546+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:22:16.136875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993432+0000","last_change":"2023-01-24T03:30:20.992639+0000","last_active":"2023-01-24T03:30:24.993432+0000","last_peered":"2023-01-24T03:30:24.993432+0000","last_clean":"2023-01-24T03:30:24.993432+0000","last_became_active":"2023-01-24T03:30:20.992386+0000","last_became_peered":"2023-01-24T03:30:20.992386+0000","last_unstale":"2023-01-24T03:30:24.993432+0000","last_undegraded":"2023-01-24T03:30:24.993432+0000","last_fullsized":"2023-01-24T03:30:24.993432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:29:17.912198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992608+0000","last_change":"2023-01-24T03:30:20.994066+0000","last_active":"2023-01-24T03:30:24.992608+0000","last_peered":"2023-01-24T03:30:24.992608+0000","last_clean":"2023-01-24T03:30:24.992608+0000","last_became_active":"2023-01-24T03:30:20.993820+0000","last_became_peered":"2023-01-24T03:30:20.993820+0000","last_unstale":"2023-01-24T03:30:24.992608+0000","last_undegraded":"2023-01-24T03:30:24.992608+0000","last_fullsized":"2023-01-24T03:30:24.992608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:31:23.874058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996188+0000","last_change":"2023-01-24T03:30:21.001652+0000","last_active":"2023-01-24T03:30:24.996188+0000","last_peered":"2023-01-24T03:30:24.996188+0000","last_clean":"2023-01-24T03:30:24.996188+0000","last_became_active":"2023-01-24T03:30:21.001400+0000","last_became_peered":"2023-01-24T03:30:21.001400+0000","last_unstale":"2023-01-24T03:30:24.996188+0000","last_undegraded":"2023-01-24T03:30:24.996188+0000","last_fullsized":"2023-01-24T03:30:24.996188+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:55:53.249816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002243+0000","last_change":"2023-01-24T03:30:20.998304+0000","last_active":"2023-01-24T03:30:25.002243+0000","last_peered":"2023-01-24T03:30:25.002243+0000","last_clean":"2023-01-24T03:30:25.002243+0000","last_became_active":"2023-01-24T03:30:20.996288+0000","last_became_peered":"2023-01-24T03:30:20.996288+0000","last_unstale":"2023-01-24T03:30:25.002243+0000","last_undegraded":"2023-01-24T03:30:25.002243+0000","last_fullsized":"2023-01-24T03:30:25.002243+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:11:08.120197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002638+0000","last_change":"2023-01-24T03:30:20.996489+0000","last_active":"2023-01-24T03:30:25.002638+0000","last_peered":"2023-01-24T03:30:25.002638+0000","last_clean":"2023-01-24T03:30:25.002638+0000","last_became_active":"2023-01-24T03:30:20.994371+0000","last_became_peered":"2023-01-24T03:30:20.994371+0000","last_unstale":"2023-01-24T03:30:25.002638+0000","last_undegraded":"2023-01-24T03:30:25.002638+0000","last_fullsized":"2023-01-24T03:30:25.002638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:52:21.369928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005744+0000","last_change":"2023-01-24T03:30:20.997753+0000","last_active":"2023-01-24T03:30:25.005744+0000","last_peered":"2023-01-24T03:30:25.005744+0000","last_clean":"2023-01-24T03:30:25.005744+0000","last_became_active":"2023-01-24T03:30:20.997475+0000","last_became_peered":"2023-01-24T03:30:20.997475+0000","last_unstale":"2023-01-24T03:30:25.005744+0000","last_undegraded":"2023-01-24T03:30:25.005744+0000","last_fullsized":"2023-01-24T03:30:25.005744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:43:16.199843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.013267+0000","last_change":"2023-01-24T03:30:25.004944+0000","last_active":"2023-01-24T03:30:25.013267+0000","last_peered":"2023-01-24T03:30:25.013267+0000","last_clean":"2023-01-24T03:30:25.013267+0000","last_became_active":"2023-01-24T03:30:25.004590+0000","last_became_peered":"2023-01-24T03:30:25.004590+0000","last_unstale":"2023-01-24T03:30:25.013267+0000","last_undegraded":"2023-01-24T03:30:25.013267+0000","last_fullsized":"2023-01-24T03:30:25.013267+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:04:11.054225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2,0],"acting":[4,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"24'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.380450+0000","last_change":"2023-01-24T03:30:20.607211+0000","last_active":"2023-01-24T03:30:25.380450+0000","last_peered":"2023-01-24T03:30:25.380450+0000","last_clean":"2023-01-24T03:30:25.380450+0000","last_became_active":"2023-01-24T03:30:20.606855+0000","last_became_peered":"2023-01-24T03:30:20.606855+0000","last_unstale":"2023-01-24T03:30:25.380450+0000","last_undegraded":"2023-01-24T03:30:25.380450+0000","last_fullsized":"2023-01-24T03:30:25.380450+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:42:29.300089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991276+0000","last_change":"2023-01-24T03:30:20.993128+0000","last_active":"2023-01-24T03:30:24.991276+0000","last_peered":"2023-01-24T03:30:24.991276+0000","last_clean":"2023-01-24T03:30:24.991276+0000","last_became_active":"2023-01-24T03:30:20.992967+0000","last_became_peered":"2023-01-24T03:30:20.992967+0000","last_unstale":"2023-01-24T03:30:24.991276+0000","last_undegraded":"2023-01-24T03:30:24.991276+0000","last_fullsized":"2023-01-24T03:30:24.991276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:09:59.170218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.008814+0000","last_change":"2023-01-24T03:30:25.006569+0000","last_active":"2023-01-24T03:30:25.008814+0000","last_peered":"2023-01-24T03:30:25.008814+0000","last_clean":"2023-01-24T03:30:25.008814+0000","last_became_active":"2023-01-24T03:30:25.006199+0000","last_became_peered":"2023-01-24T03:30:25.006199+0000","last_unstale":"2023-01-24T03:30:25.008814+0000","last_undegraded":"2023-01-24T03:30:25.008814+0000","last_fullsized":"2023-01-24T03:30:25.008814+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:31:46.839485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,7],"acting":[6,4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.001971+0000","last_change":"2023-01-24T03:30:20.612738+0000","last_active":"2023-01-24T03:30:25.001971+0000","last_peered":"2023-01-24T03:30:25.001971+0000","last_clean":"2023-01-24T03:30:25.001971+0000","last_became_active":"2023-01-24T03:30:20.612375+0000","last_became_peered":"2023-01-24T03:30:20.612375+0000","last_unstale":"2023-01-24T03:30:25.001971+0000","last_undegraded":"2023-01-24T03:30:25.001971+0000","last_fullsized":"2023-01-24T03:30:25.001971+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:17:43.251945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993741+0000","last_change":"2023-01-24T03:30:20.996368+0000","last_active":"2023-01-24T03:30:24.993741+0000","last_peered":"2023-01-24T03:30:24.993741+0000","last_clean":"2023-01-24T03:30:24.993741+0000","last_became_active":"2023-01-24T03:30:20.996051+0000","last_became_peered":"2023-01-24T03:30:20.996051+0000","last_unstale":"2023-01-24T03:30:24.993741+0000","last_undegraded":"2023-01-24T03:30:24.993741+0000","last_fullsized":"2023-01-24T03:30:24.993741+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:53:28.193862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.014921+0000","last_change":"2023-01-24T03:30:25.008528+0000","last_active":"2023-01-24T03:30:25.014921+0000","last_peered":"2023-01-24T03:30:25.014921+0000","last_clean":"2023-01-24T03:30:25.014921+0000","last_became_active":"2023-01-24T03:30:25.008146+0000","last_became_peered":"2023-01-24T03:30:25.008146+0000","last_unstale":"2023-01-24T03:30:25.014921+0000","last_undegraded":"2023-01-24T03:30:25.014921+0000","last_fullsized":"2023-01-24T03:30:25.014921+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:17:11.457359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1,2],"acting":[0,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991341+0000","last_change":"2023-01-24T03:30:20.608932+0000","last_active":"2023-01-24T03:30:24.991341+0000","last_peered":"2023-01-24T03:30:24.991341+0000","last_clean":"2023-01-24T03:30:24.991341+0000","last_became_active":"2023-01-24T03:30:20.608135+0000","last_became_peered":"2023-01-24T03:30:20.608135+0000","last_unstale":"2023-01-24T03:30:24.991341+0000","last_undegraded":"2023-01-24T03:30:24.991341+0000","last_fullsized":"2023-01-24T03:30:24.991341+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:38:30.012279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990753+0000","last_change":"2023-01-24T03:30:20.986490+0000","last_active":"2023-01-24T03:30:24.990753+0000","last_peered":"2023-01-24T03:30:24.990753+0000","last_clean":"2023-01-24T03:30:24.990753+0000","last_became_active":"2023-01-24T03:30:20.986245+0000","last_became_peered":"2023-01-24T03:30:20.986245+0000","last_unstale":"2023-01-24T03:30:24.990753+0000","last_undegraded":"2023-01-24T03:30:24.990753+0000","last_fullsized":"2023-01-24T03:30:24.990753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:38:30.801032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.012186+0000","last_change":"2023-01-24T03:30:25.008221+0000","last_active":"2023-01-24T03:30:25.012186+0000","last_peered":"2023-01-24T03:30:25.012186+0000","last_clean":"2023-01-24T03:30:25.012186+0000","last_became_active":"2023-01-24T03:30:25.007855+0000","last_became_peered":"2023-01-24T03:30:25.007855+0000","last_unstale":"2023-01-24T03:30:25.012186+0000","last_undegraded":"2023-01-24T03:30:25.012186+0000","last_fullsized":"2023-01-24T03:30:25.012186+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:45:15.317067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,1],"acting":[6,4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992445+0000","last_change":"2023-01-24T03:30:20.607385+0000","last_active":"2023-01-24T03:30:24.992445+0000","last_peered":"2023-01-24T03:30:24.992445+0000","last_clean":"2023-01-24T03:30:24.992445+0000","last_became_active":"2023-01-24T03:30:20.607123+0000","last_became_peered":"2023-01-24T03:30:20.607123+0000","last_unstale":"2023-01-24T03:30:24.992445+0000","last_undegraded":"2023-01-24T03:30:24.992445+0000","last_fullsized":"2023-01-24T03:30:24.992445+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T15:24:00.474758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991258+0000","last_change":"2023-01-24T03:30:20.994124+0000","last_active":"2023-01-24T03:30:24.991258+0000","last_peered":"2023-01-24T03:30:24.991258+0000","last_clean":"2023-01-24T03:30:24.991258+0000","last_became_active":"2023-01-24T03:30:20.993876+0000","last_became_peered":"2023-01-24T03:30:20.993876+0000","last_unstale":"2023-01-24T03:30:24.991258+0000","last_undegraded":"2023-01-24T03:30:24.991258+0000","last_fullsized":"2023-01-24T03:30:24.991258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:07:26.252741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.011535+0000","last_change":"2023-01-24T03:30:25.006312+0000","last_active":"2023-01-24T03:30:25.011535+0000","last_peered":"2023-01-24T03:30:25.011535+0000","last_clean":"2023-01-24T03:30:25.011535+0000","last_became_active":"2023-01-24T03:30:25.005946+0000","last_became_peered":"2023-01-24T03:30:25.005946+0000","last_unstale":"2023-01-24T03:30:25.011535+0000","last_undegraded":"2023-01-24T03:30:25.011535+0000","last_fullsized":"2023-01-24T03:30:25.011535+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:35:02.760190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6,0],"acting":[7,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.006326+0000","last_change":"2023-01-24T03:30:20.610825+0000","last_active":"2023-01-24T03:30:25.006326+0000","last_peered":"2023-01-24T03:30:25.006326+0000","last_clean":"2023-01-24T03:30:25.006326+0000","last_became_active":"2023-01-24T03:30:20.610540+0000","last_became_peered":"2023-01-24T03:30:20.610540+0000","last_unstale":"2023-01-24T03:30:25.006326+0000","last_undegraded":"2023-01-24T03:30:25.006326+0000","last_fullsized":"2023-01-24T03:30:25.006326+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:23:15.279283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990818+0000","last_change":"2023-01-24T03:30:20.994605+0000","last_active":"2023-01-24T03:30:24.990818+0000","last_peered":"2023-01-24T03:30:24.990818+0000","last_clean":"2023-01-24T03:30:24.990818+0000","last_became_active":"2023-01-24T03:30:20.994356+0000","last_became_peered":"2023-01-24T03:30:20.994356+0000","last_unstale":"2023-01-24T03:30:24.990818+0000","last_undegraded":"2023-01-24T03:30:24.990818+0000","last_fullsized":"2023-01-24T03:30:24.990818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:06:08.797922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.018595+0000","last_change":"2023-01-24T03:30:25.012301+0000","last_active":"2023-01-24T03:30:25.018595+0000","last_peered":"2023-01-24T03:30:25.018595+0000","last_clean":"2023-01-24T03:30:25.018595+0000","last_became_active":"2023-01-24T03:30:25.011891+0000","last_became_peered":"2023-01-24T03:30:25.011891+0000","last_unstale":"2023-01-24T03:30:25.018595+0000","last_undegraded":"2023-01-24T03:30:25.018595+0000","last_fullsized":"2023-01-24T03:30:25.018595+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:15:46.012675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,5],"acting":[3,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"24'3","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.986137+0000","last_change":"2023-01-24T03:30:20.609828+0000","last_active":"2023-01-24T03:30:25.986137+0000","last_peered":"2023-01-24T03:30:25.986137+0000","last_clean":"2023-01-24T03:30:25.986137+0000","last_became_active":"2023-01-24T03:30:20.609492+0000","last_became_peered":"2023-01-24T03:30:20.609492+0000","last_unstale":"2023-01-24T03:30:25.986137+0000","last_undegraded":"2023-01-24T03:30:25.986137+0000","last_fullsized":"2023-01-24T03:30:25.986137+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:22:53.430711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002553+0000","last_change":"2023-01-24T03:30:20.999747+0000","last_active":"2023-01-24T03:30:25.002553+0000","last_peered":"2023-01-24T03:30:25.002553+0000","last_clean":"2023-01-24T03:30:25.002553+0000","last_became_active":"2023-01-24T03:30:20.999003+0000","last_became_peered":"2023-01-24T03:30:20.999003+0000","last_unstale":"2023-01-24T03:30:25.002553+0000","last_undegraded":"2023-01-24T03:30:25.002553+0000","last_fullsized":"2023-01-24T03:30:25.002553+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:19:22.205251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016573+0000","last_change":"2023-01-24T03:30:25.010659+0000","last_active":"2023-01-24T03:30:25.016573+0000","last_peered":"2023-01-24T03:30:25.016573+0000","last_clean":"2023-01-24T03:30:25.016573+0000","last_became_active":"2023-01-24T03:30:25.010237+0000","last_became_peered":"2023-01-24T03:30:25.010237+0000","last_unstale":"2023-01-24T03:30:25.016573+0000","last_undegraded":"2023-01-24T03:30:25.016573+0000","last_fullsized":"2023-01-24T03:30:25.016573+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:01:53.501070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3,7],"acting":[1,5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"24'9","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:34.376852+0000","last_change":"2023-01-24T03:30:20.610682+0000","last_active":"2023-01-24T03:30:34.376852+0000","last_peered":"2023-01-24T03:30:34.376852+0000","last_clean":"2023-01-24T03:30:34.376852+0000","last_became_active":"2023-01-24T03:30:20.610463+0000","last_became_peered":"2023-01-24T03:30:20.610463+0000","last_unstale":"2023-01-24T03:30:34.376852+0000","last_undegraded":"2023-01-24T03:30:34.376852+0000","last_fullsized":"2023-01-24T03:30:34.376852+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:25:00.447596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3713,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992845+0000","last_change":"2023-01-24T03:30:20.987326+0000","last_active":"2023-01-24T03:30:24.992845+0000","last_peered":"2023-01-24T03:30:24.992845+0000","last_clean":"2023-01-24T03:30:24.992845+0000","last_became_active":"2023-01-24T03:30:20.987042+0000","last_became_peered":"2023-01-24T03:30:20.987042+0000","last_unstale":"2023-01-24T03:30:24.992845+0000","last_undegraded":"2023-01-24T03:30:24.992845+0000","last_fullsized":"2023-01-24T03:30:24.992845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:57:43.257624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.020022+0000","last_change":"2023-01-24T03:30:25.012004+0000","last_active":"2023-01-24T03:30:25.020022+0000","last_peered":"2023-01-24T03:30:25.020022+0000","last_clean":"2023-01-24T03:30:25.020022+0000","last_became_active":"2023-01-24T03:30:25.007410+0000","last_became_peered":"2023-01-24T03:30:25.007410+0000","last_unstale":"2023-01-24T03:30:25.020022+0000","last_undegraded":"2023-01-24T03:30:25.020022+0000","last_fullsized":"2023-01-24T03:30:25.020022+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:34:21.681011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3,0],"acting":[5,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"24'2","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985201+0000","last_change":"2023-01-24T03:30:20.613026+0000","last_active":"2023-01-24T03:30:25.985201+0000","last_peered":"2023-01-24T03:30:25.985201+0000","last_clean":"2023-01-24T03:30:25.985201+0000","last_became_active":"2023-01-24T03:30:20.612605+0000","last_became_peered":"2023-01-24T03:30:20.612605+0000","last_unstale":"2023-01-24T03:30:25.985201+0000","last_undegraded":"2023-01-24T03:30:25.985201+0000","last_fullsized":"2023-01-24T03:30:25.985201+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:45:08.016740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991113+0000","last_change":"2023-01-24T03:30:20.991929+0000","last_active":"2023-01-24T03:30:24.991113+0000","last_peered":"2023-01-24T03:30:24.991113+0000","last_clean":"2023-01-24T03:30:24.991113+0000","last_became_active":"2023-01-24T03:30:20.991693+0000","last_became_peered":"2023-01-24T03:30:20.991693+0000","last_unstale":"2023-01-24T03:30:24.991113+0000","last_undegraded":"2023-01-24T03:30:24.991113+0000","last_fullsized":"2023-01-24T03:30:24.991113+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:55:51.637568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.018472+0000","last_change":"2023-01-24T03:30:25.010572+0000","last_active":"2023-01-24T03:30:25.018472+0000","last_peered":"2023-01-24T03:30:25.018472+0000","last_clean":"2023-01-24T03:30:25.018472+0000","last_became_active":"2023-01-24T03:30:25.010193+0000","last_became_peered":"2023-01-24T03:30:25.010193+0000","last_unstale":"2023-01-24T03:30:25.018472+0000","last_undegraded":"2023-01-24T03:30:25.018472+0000","last_fullsized":"2023-01-24T03:30:25.018472+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:25:39.676179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,1],"acting":[3,7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993579+0000","last_change":"2023-01-24T03:30:20.612047+0000","last_active":"2023-01-24T03:30:24.993579+0000","last_peered":"2023-01-24T03:30:24.993579+0000","last_clean":"2023-01-24T03:30:24.993579+0000","last_became_active":"2023-01-24T03:30:20.611798+0000","last_became_peered":"2023-01-24T03:30:20.611798+0000","last_unstale":"2023-01-24T03:30:24.993579+0000","last_undegraded":"2023-01-24T03:30:24.993579+0000","last_fullsized":"2023-01-24T03:30:24.993579+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:01:41.731689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005194+0000","last_change":"2023-01-24T03:30:20.996432+0000","last_active":"2023-01-24T03:30:25.005194+0000","last_peered":"2023-01-24T03:30:25.005194+0000","last_clean":"2023-01-24T03:30:25.005194+0000","last_became_active":"2023-01-24T03:30:20.996173+0000","last_became_peered":"2023-01-24T03:30:20.996173+0000","last_unstale":"2023-01-24T03:30:25.005194+0000","last_undegraded":"2023-01-24T03:30:25.005194+0000","last_fullsized":"2023-01-24T03:30:25.005194+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:13:31.596587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.018507+0000","last_change":"2023-01-24T03:30:25.008479+0000","last_active":"2023-01-24T03:30:25.018507+0000","last_peered":"2023-01-24T03:30:25.018507+0000","last_clean":"2023-01-24T03:30:25.018507+0000","last_became_active":"2023-01-24T03:30:25.007858+0000","last_became_peered":"2023-01-24T03:30:25.007858+0000","last_unstale":"2023-01-24T03:30:25.018507+0000","last_undegraded":"2023-01-24T03:30:25.018507+0000","last_fullsized":"2023-01-24T03:30:25.018507+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:10:38.425874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,0],"acting":[3,1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996134+0000","last_change":"2023-01-24T03:30:20.612848+0000","last_active":"2023-01-24T03:30:24.996134+0000","last_peered":"2023-01-24T03:30:24.996134+0000","last_clean":"2023-01-24T03:30:24.996134+0000","last_became_active":"2023-01-24T03:30:20.612600+0000","last_became_peered":"2023-01-24T03:30:20.612600+0000","last_unstale":"2023-01-24T03:30:24.996134+0000","last_undegraded":"2023-01-24T03:30:24.996134+0000","last_fullsized":"2023-01-24T03:30:24.996134+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:46:45.634472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993883+0000","last_change":"2023-01-24T03:30:20.991133+0000","last_active":"2023-01-24T03:30:24.993883+0000","last_peered":"2023-01-24T03:30:24.993883+0000","last_clean":"2023-01-24T03:30:24.993883+0000","last_became_active":"2023-01-24T03:30:20.990856+0000","last_became_peered":"2023-01-24T03:30:20.990856+0000","last_unstale":"2023-01-24T03:30:24.993883+0000","last_undegraded":"2023-01-24T03:30:24.993883+0000","last_fullsized":"2023-01-24T03:30:24.993883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:27:32.163224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.022452+0000","last_change":"2023-01-24T03:30:25.017038+0000","last_active":"2023-01-24T03:30:25.022452+0000","last_peered":"2023-01-24T03:30:25.022452+0000","last_clean":"2023-01-24T03:30:25.022452+0000","last_became_active":"2023-01-24T03:30:25.016684+0000","last_became_peered":"2023-01-24T03:30:25.016684+0000","last_unstale":"2023-01-24T03:30:25.022452+0000","last_undegraded":"2023-01-24T03:30:25.022452+0000","last_fullsized":"2023-01-24T03:30:25.022452+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:57:30.028834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1,6],"acting":[5,3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"24'8","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.988793+0000","last_change":"2023-01-24T03:30:20.604617+0000","last_active":"2023-01-24T03:30:25.988793+0000","last_peered":"2023-01-24T03:30:25.988793+0000","last_clean":"2023-01-24T03:30:25.988793+0000","last_became_active":"2023-01-24T03:30:20.604445+0000","last_became_peered":"2023-01-24T03:30:20.604445+0000","last_unstale":"2023-01-24T03:30:25.988793+0000","last_undegraded":"2023-01-24T03:30:25.988793+0000","last_fullsized":"2023-01-24T03:30:25.988793+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:06:51.460895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993017+0000","last_change":"2023-01-24T03:30:20.995872+0000","last_active":"2023-01-24T03:30:24.993017+0000","last_peered":"2023-01-24T03:30:24.993017+0000","last_clean":"2023-01-24T03:30:24.993017+0000","last_became_active":"2023-01-24T03:30:20.995599+0000","last_became_peered":"2023-01-24T03:30:20.995599+0000","last_unstale":"2023-01-24T03:30:24.993017+0000","last_undegraded":"2023-01-24T03:30:24.993017+0000","last_fullsized":"2023-01-24T03:30:24.993017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:53:54.356899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.020130+0000","last_change":"2023-01-24T03:30:25.016101+0000","last_active":"2023-01-24T03:30:25.020130+0000","last_peered":"2023-01-24T03:30:25.020130+0000","last_clean":"2023-01-24T03:30:25.020130+0000","last_became_active":"2023-01-24T03:30:25.015705+0000","last_became_peered":"2023-01-24T03:30:25.015705+0000","last_unstale":"2023-01-24T03:30:25.020130+0000","last_undegraded":"2023-01-24T03:30:25.020130+0000","last_fullsized":"2023-01-24T03:30:25.020130+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:54:37.798191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3,4],"acting":[7,6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"24'1","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.446876+0000","last_change":"2023-01-24T03:30:20.610632+0000","last_active":"2023-01-24T03:30:25.446876+0000","last_peered":"2023-01-24T03:30:25.446876+0000","last_clean":"2023-01-24T03:30:25.446876+0000","last_became_active":"2023-01-24T03:30:20.610367+0000","last_became_peered":"2023-01-24T03:30:20.610367+0000","last_unstale":"2023-01-24T03:30:25.446876+0000","last_undegraded":"2023-01-24T03:30:25.446876+0000","last_fullsized":"2023-01-24T03:30:25.446876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:26:05.938178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.004623+0000","last_change":"2023-01-24T03:30:20.996493+0000","last_active":"2023-01-24T03:30:25.004623+0000","last_peered":"2023-01-24T03:30:25.004623+0000","last_clean":"2023-01-24T03:30:25.004623+0000","last_became_active":"2023-01-24T03:30:20.996229+0000","last_became_peered":"2023-01-24T03:30:20.996229+0000","last_unstale":"2023-01-24T03:30:25.004623+0000","last_undegraded":"2023-01-24T03:30:25.004623+0000","last_fullsized":"2023-01-24T03:30:25.004623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:23:53.642522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016195+0000","last_change":"2023-01-24T03:30:25.009215+0000","last_active":"2023-01-24T03:30:25.016195+0000","last_peered":"2023-01-24T03:30:25.016195+0000","last_clean":"2023-01-24T03:30:25.016195+0000","last_became_active":"2023-01-24T03:30:25.008213+0000","last_became_peered":"2023-01-24T03:30:25.008213+0000","last_unstale":"2023-01-24T03:30:25.016195+0000","last_undegraded":"2023-01-24T03:30:25.016195+0000","last_fullsized":"2023-01-24T03:30:25.016195+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:48:37.517172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,5],"acting":[2,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"24'2","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985223+0000","last_change":"2023-01-24T03:30:20.612158+0000","last_active":"2023-01-24T03:30:25.985223+0000","last_peered":"2023-01-24T03:30:25.985223+0000","last_clean":"2023-01-24T03:30:25.985223+0000","last_became_active":"2023-01-24T03:30:20.611931+0000","last_became_peered":"2023-01-24T03:30:20.611931+0000","last_unstale":"2023-01-24T03:30:25.985223+0000","last_undegraded":"2023-01-24T03:30:25.985223+0000","last_fullsized":"2023-01-24T03:30:25.985223+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:30:37.532390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992500+0000","last_change":"2023-01-24T03:30:20.991582+0000","last_active":"2023-01-24T03:30:24.992500+0000","last_peered":"2023-01-24T03:30:24.992500+0000","last_clean":"2023-01-24T03:30:24.992500+0000","last_became_active":"2023-01-24T03:30:20.991335+0000","last_became_peered":"2023-01-24T03:30:20.991335+0000","last_unstale":"2023-01-24T03:30:24.992500+0000","last_undegraded":"2023-01-24T03:30:24.992500+0000","last_fullsized":"2023-01-24T03:30:24.992500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:45:20.786712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016224+0000","last_change":"2023-01-24T03:30:25.008058+0000","last_active":"2023-01-24T03:30:25.016224+0000","last_peered":"2023-01-24T03:30:25.016224+0000","last_clean":"2023-01-24T03:30:25.016224+0000","last_became_active":"2023-01-24T03:30:25.007711+0000","last_became_peered":"2023-01-24T03:30:25.007711+0000","last_unstale":"2023-01-24T03:30:25.016224+0000","last_undegraded":"2023-01-24T03:30:25.016224+0000","last_fullsized":"2023-01-24T03:30:25.016224+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:26:56.270674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5,6],"acting":[2,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"24'1","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.447163+0000","last_change":"2023-01-24T03:30:20.610495+0000","last_active":"2023-01-24T03:30:25.447163+0000","last_peered":"2023-01-24T03:30:25.447163+0000","last_clean":"2023-01-24T03:30:25.447163+0000","last_became_active":"2023-01-24T03:30:20.610035+0000","last_became_peered":"2023-01-24T03:30:20.610035+0000","last_unstale":"2023-01-24T03:30:25.447163+0000","last_undegraded":"2023-01-24T03:30:25.447163+0000","last_fullsized":"2023-01-24T03:30:25.447163+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:25:30.944275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.989631+0000","last_change":"2023-01-24T03:30:20.994362+0000","last_active":"2023-01-24T03:30:24.989631+0000","last_peered":"2023-01-24T03:30:24.989631+0000","last_clean":"2023-01-24T03:30:24.989631+0000","last_became_active":"2023-01-24T03:30:20.994101+0000","last_became_peered":"2023-01-24T03:30:20.994101+0000","last_unstale":"2023-01-24T03:30:24.989631+0000","last_undegraded":"2023-01-24T03:30:24.989631+0000","last_fullsized":"2023-01-24T03:30:24.989631+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:55:12.837646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.015999+0000","last_change":"2023-01-24T03:30:25.006995+0000","last_active":"2023-01-24T03:30:25.015999+0000","last_peered":"2023-01-24T03:30:25.015999+0000","last_clean":"2023-01-24T03:30:25.015999+0000","last_became_active":"2023-01-24T03:30:25.006473+0000","last_became_peered":"2023-01-24T03:30:25.006473+0000","last_unstale":"2023-01-24T03:30:25.015999+0000","last_undegraded":"2023-01-24T03:30:25.015999+0000","last_fullsized":"2023-01-24T03:30:25.015999+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:48:16.320162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3,6],"acting":[2,4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005342+0000","last_change":"2023-01-24T03:30:20.606526+0000","last_active":"2023-01-24T03:30:25.005342+0000","last_peered":"2023-01-24T03:30:25.005342+0000","last_clean":"2023-01-24T03:30:25.005342+0000","last_became_active":"2023-01-24T03:30:20.606301+0000","last_became_peered":"2023-01-24T03:30:20.606301+0000","last_unstale":"2023-01-24T03:30:25.005342+0000","last_undegraded":"2023-01-24T03:30:25.005342+0000","last_fullsized":"2023-01-24T03:30:25.005342+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:43:28.702475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992428+0000","last_change":"2023-01-24T03:30:20.993379+0000","last_active":"2023-01-24T03:30:24.992428+0000","last_peered":"2023-01-24T03:30:24.992428+0000","last_clean":"2023-01-24T03:30:24.992428+0000","last_became_active":"2023-01-24T03:30:20.993151+0000","last_became_peered":"2023-01-24T03:30:20.993151+0000","last_unstale":"2023-01-24T03:30:24.992428+0000","last_undegraded":"2023-01-24T03:30:24.992428+0000","last_fullsized":"2023-01-24T03:30:24.992428+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:00:45.845082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.020912+0000","last_change":"2023-01-24T03:30:25.013701+0000","last_active":"2023-01-24T03:30:25.020912+0000","last_peered":"2023-01-24T03:30:25.020912+0000","last_clean":"2023-01-24T03:30:25.020912+0000","last_became_active":"2023-01-24T03:30:25.013290+0000","last_became_peered":"2023-01-24T03:30:25.013290+0000","last_unstale":"2023-01-24T03:30:25.020912+0000","last_undegraded":"2023-01-24T03:30:25.020912+0000","last_fullsized":"2023-01-24T03:30:25.020912+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:32:17.340583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5,6],"acting":[2,7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"24'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985860+0000","last_change":"2023-01-24T03:30:20.609656+0000","last_active":"2023-01-24T03:30:25.985860+0000","last_peered":"2023-01-24T03:30:25.985860+0000","last_clean":"2023-01-24T03:30:25.985860+0000","last_became_active":"2023-01-24T03:30:20.609374+0000","last_became_peered":"2023-01-24T03:30:20.609374+0000","last_unstale":"2023-01-24T03:30:25.985860+0000","last_undegraded":"2023-01-24T03:30:25.985860+0000","last_fullsized":"2023-01-24T03:30:25.985860+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:58:12.653746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992726+0000","last_change":"2023-01-24T03:30:20.986089+0000","last_active":"2023-01-24T03:30:24.992726+0000","last_peered":"2023-01-24T03:30:24.992726+0000","last_clean":"2023-01-24T03:30:24.992726+0000","last_became_active":"2023-01-24T03:30:20.985857+0000","last_became_peered":"2023-01-24T03:30:20.985857+0000","last_unstale":"2023-01-24T03:30:24.992726+0000","last_undegraded":"2023-01-24T03:30:24.992726+0000","last_fullsized":"2023-01-24T03:30:24.992726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:00:57.338328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016603+0000","last_change":"2023-01-24T03:30:25.009962+0000","last_active":"2023-01-24T03:30:25.016603+0000","last_peered":"2023-01-24T03:30:25.016603+0000","last_clean":"2023-01-24T03:30:25.016603+0000","last_became_active":"2023-01-24T03:30:25.009623+0000","last_became_peered":"2023-01-24T03:30:25.009623+0000","last_unstale":"2023-01-24T03:30:25.016603+0000","last_undegraded":"2023-01-24T03:30:25.016603+0000","last_fullsized":"2023-01-24T03:30:25.016603+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:24:42.958531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0,6],"acting":[1,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.989699+0000","last_change":"2023-01-24T03:30:20.608114+0000","last_active":"2023-01-24T03:30:24.989699+0000","last_peered":"2023-01-24T03:30:24.989699+0000","last_clean":"2023-01-24T03:30:24.989699+0000","last_became_active":"2023-01-24T03:30:20.607816+0000","last_became_peered":"2023-01-24T03:30:20.607816+0000","last_unstale":"2023-01-24T03:30:24.989699+0000","last_undegraded":"2023-01-24T03:30:24.989699+0000","last_fullsized":"2023-01-24T03:30:24.989699+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:40:00.784496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992908+0000","last_change":"2023-01-24T03:30:20.982589+0000","last_active":"2023-01-24T03:30:24.992908+0000","last_peered":"2023-01-24T03:30:24.992908+0000","last_clean":"2023-01-24T03:30:24.992908+0000","last_became_active":"2023-01-24T03:30:20.979779+0000","last_became_peered":"2023-01-24T03:30:20.979779+0000","last_unstale":"2023-01-24T03:30:24.992908+0000","last_undegraded":"2023-01-24T03:30:24.992908+0000","last_fullsized":"2023-01-24T03:30:24.992908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:01:18.491363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.019938+0000","last_change":"2023-01-24T03:30:25.013327+0000","last_active":"2023-01-24T03:30:25.019938+0000","last_peered":"2023-01-24T03:30:25.019938+0000","last_clean":"2023-01-24T03:30:25.019938+0000","last_became_active":"2023-01-24T03:30:25.011303+0000","last_became_peered":"2023-01-24T03:30:25.011303+0000","last_unstale":"2023-01-24T03:30:25.019938+0000","last_undegraded":"2023-01-24T03:30:25.019938+0000","last_fullsized":"2023-01-24T03:30:25.019938+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:14:43.169287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6,2],"acting":[5,4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992390+0000","last_change":"2023-01-24T03:30:20.602369+0000","last_active":"2023-01-24T03:30:24.992390+0000","last_peered":"2023-01-24T03:30:24.992390+0000","last_clean":"2023-01-24T03:30:24.992390+0000","last_became_active":"2023-01-24T03:30:20.601996+0000","last_became_peered":"2023-01-24T03:30:20.601996+0000","last_unstale":"2023-01-24T03:30:24.992390+0000","last_undegraded":"2023-01-24T03:30:24.992390+0000","last_fullsized":"2023-01-24T03:30:24.992390+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:33:55.328080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993109+0000","last_change":"2023-01-24T03:30:20.988738+0000","last_active":"2023-01-24T03:30:24.993109+0000","last_peered":"2023-01-24T03:30:24.993109+0000","last_clean":"2023-01-24T03:30:24.993109+0000","last_became_active":"2023-01-24T03:30:20.987190+0000","last_became_peered":"2023-01-24T03:30:20.987190+0000","last_unstale":"2023-01-24T03:30:24.993109+0000","last_undegraded":"2023-01-24T03:30:24.993109+0000","last_fullsized":"2023-01-24T03:30:24.993109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:32:05.689674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.013303+0000","last_change":"2023-01-24T03:30:25.008401+0000","last_active":"2023-01-24T03:30:25.013303+0000","last_peered":"2023-01-24T03:30:25.013303+0000","last_clean":"2023-01-24T03:30:25.013303+0000","last_became_active":"2023-01-24T03:30:25.007973+0000","last_became_peered":"2023-01-24T03:30:25.007973+0000","last_unstale":"2023-01-24T03:30:25.013303+0000","last_undegraded":"2023-01-24T03:30:25.013303+0000","last_fullsized":"2023-01-24T03:30:25.013303+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:36:22.339295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,2],"acting":[4,5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991180+0000","last_change":"2023-01-24T03:30:20.608145+0000","last_active":"2023-01-24T03:30:24.991180+0000","last_peered":"2023-01-24T03:30:24.991180+0000","last_clean":"2023-01-24T03:30:24.991180+0000","last_became_active":"2023-01-24T03:30:20.607165+0000","last_became_peered":"2023-01-24T03:30:20.607165+0000","last_unstale":"2023-01-24T03:30:24.991180+0000","last_undegraded":"2023-01-24T03:30:24.991180+0000","last_fullsized":"2023-01-24T03:30:24.991180+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:58:56.490482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002416+0000","last_change":"2023-01-24T03:30:20.998777+0000","last_active":"2023-01-24T03:30:25.002416+0000","last_peered":"2023-01-24T03:30:25.002416+0000","last_clean":"2023-01-24T03:30:25.002416+0000","last_became_active":"2023-01-24T03:30:20.998519+0000","last_became_peered":"2023-01-24T03:30:20.998519+0000","last_unstale":"2023-01-24T03:30:25.002416+0000","last_undegraded":"2023-01-24T03:30:25.002416+0000","last_fullsized":"2023-01-24T03:30:25.002416+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:55:55.320556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.012455+0000","last_change":"2023-01-24T03:30:25.008154+0000","last_active":"2023-01-24T03:30:25.012455+0000","last_peered":"2023-01-24T03:30:25.012455+0000","last_clean":"2023-01-24T03:30:25.012455+0000","last_became_active":"2023-01-24T03:30:25.007797+0000","last_became_peered":"2023-01-24T03:30:25.007797+0000","last_unstale":"2023-01-24T03:30:25.012455+0000","last_undegraded":"2023-01-24T03:30:25.012455+0000","last_fullsized":"2023-01-24T03:30:25.012455+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:29:05.153733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7,0],"acting":[4,3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"24'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985835+0000","last_change":"2023-01-24T03:30:20.610690+0000","last_active":"2023-01-24T03:30:25.985835+0000","last_peered":"2023-01-24T03:30:25.985835+0000","last_clean":"2023-01-24T03:30:25.985835+0000","last_became_active":"2023-01-24T03:30:20.610409+0000","last_became_peered":"2023-01-24T03:30:20.610409+0000","last_unstale":"2023-01-24T03:30:25.985835+0000","last_undegraded":"2023-01-24T03:30:25.985835+0000","last_fullsized":"2023-01-24T03:30:25.985835+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:43:37.740181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993033+0000","last_change":"2023-01-24T03:30:16.085885+0000","last_active":"2023-01-24T03:30:24.993033+0000","last_peered":"2023-01-24T03:30:24.993033+0000","last_clean":"2023-01-24T03:30:24.993033+0000","last_became_active":"2023-01-24T03:30:16.085525+0000","last_became_peered":"2023-01-24T03:30:16.085525+0000","last_unstale":"2023-01-24T03:30:24.993033+0000","last_undegraded":"2023-01-24T03:30:24.993033+0000","last_fullsized":"2023-01-24T03:30:24.993033+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:57:36.499644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991099+0000","last_change":"2023-01-24T03:30:20.996370+0000","last_active":"2023-01-24T03:30:24.991099+0000","last_peered":"2023-01-24T03:30:24.991099+0000","last_clean":"2023-01-24T03:30:24.991099+0000","last_became_active":"2023-01-24T03:30:20.996122+0000","last_became_peered":"2023-01-24T03:30:20.996122+0000","last_unstale":"2023-01-24T03:30:24.991099+0000","last_undegraded":"2023-01-24T03:30:24.991099+0000","last_fullsized":"2023-01-24T03:30:24.991099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:13:06.913474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.014170+0000","last_change":"2023-01-24T03:30:25.007262+0000","last_active":"2023-01-24T03:30:25.014170+0000","last_peered":"2023-01-24T03:30:25.014170+0000","last_clean":"2023-01-24T03:30:25.014170+0000","last_became_active":"2023-01-24T03:30:25.006862+0000","last_became_peered":"2023-01-24T03:30:25.006862+0000","last_unstale":"2023-01-24T03:30:25.014170+0000","last_undegraded":"2023-01-24T03:30:25.014170+0000","last_fullsized":"2023-01-24T03:30:25.014170+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:42:05.268862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,0],"acting":[3,1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991165+0000","last_change":"2023-01-24T03:30:16.057240+0000","last_active":"2023-01-24T03:30:24.991165+0000","last_peered":"2023-01-24T03:30:24.991165+0000","last_clean":"2023-01-24T03:30:24.991165+0000","last_became_active":"2023-01-24T03:30:16.056941+0000","last_became_peered":"2023-01-24T03:30:16.056941+0000","last_unstale":"2023-01-24T03:30:24.991165+0000","last_undegraded":"2023-01-24T03:30:24.991165+0000","last_fullsized":"2023-01-24T03:30:24.991165+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:22:26.168527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"24'1","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.447279+0000","last_change":"2023-01-24T03:30:20.606130+0000","last_active":"2023-01-24T03:30:25.447279+0000","last_peered":"2023-01-24T03:30:25.447279+0000","last_clean":"2023-01-24T03:30:25.447279+0000","last_became_active":"2023-01-24T03:30:20.605899+0000","last_became_peered":"2023-01-24T03:30:20.605899+0000","last_unstale":"2023-01-24T03:30:25.447279+0000","last_undegraded":"2023-01-24T03:30:25.447279+0000","last_fullsized":"2023-01-24T03:30:25.447279+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:30:47.956569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992411+0000","last_change":"2023-01-24T03:30:20.988906+0000","last_active":"2023-01-24T03:30:24.992411+0000","last_peered":"2023-01-24T03:30:24.992411+0000","last_clean":"2023-01-24T03:30:24.992411+0000","last_became_active":"2023-01-24T03:30:20.988688+0000","last_became_peered":"2023-01-24T03:30:20.988688+0000","last_unstale":"2023-01-24T03:30:24.992411+0000","last_undegraded":"2023-01-24T03:30:24.992411+0000","last_fullsized":"2023-01-24T03:30:24.992411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:25:43.365167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.019966+0000","last_change":"2023-01-24T03:30:25.013834+0000","last_active":"2023-01-24T03:30:25.019966+0000","last_peered":"2023-01-24T03:30:25.019966+0000","last_clean":"2023-01-24T03:30:25.019966+0000","last_became_active":"2023-01-24T03:30:25.011774+0000","last_became_peered":"2023-01-24T03:30:25.011774+0000","last_unstale":"2023-01-24T03:30:25.019966+0000","last_undegraded":"2023-01-24T03:30:25.019966+0000","last_fullsized":"2023-01-24T03:30:25.019966+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:02:50.158049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0,4],"acting":[5,1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002878+0000","last_change":"2023-01-24T03:30:16.051822+0000","last_active":"2023-01-24T03:30:25.002878+0000","last_peered":"2023-01-24T03:30:25.002878+0000","last_clean":"2023-01-24T03:30:25.002878+0000","last_became_active":"2023-01-24T03:30:16.051525+0000","last_became_peered":"2023-01-24T03:30:16.051525+0000","last_unstale":"2023-01-24T03:30:25.002878+0000","last_undegraded":"2023-01-24T03:30:25.002878+0000","last_fullsized":"2023-01-24T03:30:25.002878+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:37:30.802424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"16'117","reported_seq":166,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996317+0000","last_change":"2023-01-24T03:30:16.054907+0000","last_active":"2023-01-24T03:30:24.996317+0000","last_peered":"2023-01-24T03:30:24.996317+0000","last_clean":"2023-01-24T03:30:24.996317+0000","last_became_active":"2023-01-24T03:30:16.054071+0000","last_became_peered":"2023-01-24T03:30:16.054071+0000","last_unstale":"2023-01-24T03:30:24.996317+0000","last_undegraded":"2023-01-24T03:30:24.996317+0000","last_fullsized":"2023-01-24T03:30:24.996317+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:31:26.790766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992331+0000","last_change":"2023-01-24T03:30:20.607914+0000","last_active":"2023-01-24T03:30:24.992331+0000","last_peered":"2023-01-24T03:30:24.992331+0000","last_clean":"2023-01-24T03:30:24.992331+0000","last_became_active":"2023-01-24T03:30:20.607663+0000","last_became_peered":"2023-01-24T03:30:20.607663+0000","last_unstale":"2023-01-24T03:30:24.992331+0000","last_undegraded":"2023-01-24T03:30:24.992331+0000","last_fullsized":"2023-01-24T03:30:24.992331+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:32:24.271747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993825+0000","last_change":"2023-01-24T03:30:20.998105+0000","last_active":"2023-01-24T03:30:24.993825+0000","last_peered":"2023-01-24T03:30:24.993825+0000","last_clean":"2023-01-24T03:30:24.993825+0000","last_became_active":"2023-01-24T03:30:20.997871+0000","last_became_peered":"2023-01-24T03:30:20.997871+0000","last_unstale":"2023-01-24T03:30:24.993825+0000","last_undegraded":"2023-01-24T03:30:24.993825+0000","last_fullsized":"2023-01-24T03:30:24.993825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:47:42.740428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016190+0000","last_change":"2023-01-24T03:30:25.008079+0000","last_active":"2023-01-24T03:30:25.016190+0000","last_peered":"2023-01-24T03:30:25.016190+0000","last_clean":"2023-01-24T03:30:25.016190+0000","last_became_active":"2023-01-24T03:30:25.007596+0000","last_became_peered":"2023-01-24T03:30:25.007596+0000","last_unstale":"2023-01-24T03:30:25.016190+0000","last_undegraded":"2023-01-24T03:30:25.016190+0000","last_fullsized":"2023-01-24T03:30:25.016190+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:30:57.179885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1,7],"acting":[2,5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996872+0000","last_change":"2023-01-24T03:30:16.272445+0000","last_active":"2023-01-24T03:30:24.996872+0000","last_peered":"2023-01-24T03:30:24.996872+0000","last_clean":"2023-01-24T03:30:24.996872+0000","last_became_active":"2023-01-24T03:30:16.272184+0000","last_became_peered":"2023-01-24T03:30:16.272184+0000","last_unstale":"2023-01-24T03:30:24.996872+0000","last_undegraded":"2023-01-24T03:30:24.996872+0000","last_fullsized":"2023-01-24T03:30:24.996872+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:17:05.555068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002328+0000","last_change":"2023-01-24T03:30:20.612832+0000","last_active":"2023-01-24T03:30:25.002328+0000","last_peered":"2023-01-24T03:30:25.002328+0000","last_clean":"2023-01-24T03:30:25.002328+0000","last_became_active":"2023-01-24T03:30:20.612569+0000","last_became_peered":"2023-01-24T03:30:20.612569+0000","last_unstale":"2023-01-24T03:30:25.002328+0000","last_undegraded":"2023-01-24T03:30:25.002328+0000","last_fullsized":"2023-01-24T03:30:25.002328+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:37:40.534015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002330+0000","last_change":"2023-01-24T03:30:20.998171+0000","last_active":"2023-01-24T03:30:25.002330+0000","last_peered":"2023-01-24T03:30:25.002330+0000","last_clean":"2023-01-24T03:30:25.002330+0000","last_became_active":"2023-01-24T03:30:20.994887+0000","last_became_peered":"2023-01-24T03:30:20.994887+0000","last_unstale":"2023-01-24T03:30:25.002330+0000","last_undegraded":"2023-01-24T03:30:25.002330+0000","last_fullsized":"2023-01-24T03:30:25.002330+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:57:40.909930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.020081+0000","last_change":"2023-01-24T03:30:25.014948+0000","last_active":"2023-01-24T03:30:25.020081+0000","last_peered":"2023-01-24T03:30:25.020081+0000","last_clean":"2023-01-24T03:30:25.020081+0000","last_became_active":"2023-01-24T03:30:25.014572+0000","last_became_peered":"2023-01-24T03:30:25.014572+0000","last_unstale":"2023-01-24T03:30:25.020081+0000","last_undegraded":"2023-01-24T03:30:25.020081+0000","last_fullsized":"2023-01-24T03:30:25.020081+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:59:13.108074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5,3],"acting":[7,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996112+0000","last_change":"2023-01-24T03:30:16.272574+0000","last_active":"2023-01-24T03:30:24.996112+0000","last_peered":"2023-01-24T03:30:24.996112+0000","last_clean":"2023-01-24T03:30:24.996112+0000","last_became_active":"2023-01-24T03:30:16.272252+0000","last_became_peered":"2023-01-24T03:30:16.272252+0000","last_unstale":"2023-01-24T03:30:24.996112+0000","last_undegraded":"2023-01-24T03:30:24.996112+0000","last_fullsized":"2023-01-24T03:30:24.996112+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:12:08.393585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991181+0000","last_change":"2023-01-24T03:30:20.606878+0000","last_active":"2023-01-24T03:30:24.991181+0000","last_peered":"2023-01-24T03:30:24.991181+0000","last_clean":"2023-01-24T03:30:24.991181+0000","last_became_active":"2023-01-24T03:30:20.606442+0000","last_became_peered":"2023-01-24T03:30:20.606442+0000","last_unstale":"2023-01-24T03:30:24.991181+0000","last_undegraded":"2023-01-24T03:30:24.991181+0000","last_fullsized":"2023-01-24T03:30:24.991181+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:26:16.916080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990646+0000","last_change":"2023-01-24T03:30:20.990840+0000","last_active":"2023-01-24T03:30:24.990646+0000","last_peered":"2023-01-24T03:30:24.990646+0000","last_clean":"2023-01-24T03:30:24.990646+0000","last_became_active":"2023-01-24T03:30:20.990615+0000","last_became_peered":"2023-01-24T03:30:20.990615+0000","last_unstale":"2023-01-24T03:30:24.990646+0000","last_undegraded":"2023-01-24T03:30:24.990646+0000","last_fullsized":"2023-01-24T03:30:24.990646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T15:06:37.670127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.013106+0000","last_change":"2023-01-24T03:30:25.006375+0000","last_active":"2023-01-24T03:30:25.013106+0000","last_peered":"2023-01-24T03:30:25.013106+0000","last_clean":"2023-01-24T03:30:25.013106+0000","last_became_active":"2023-01-24T03:30:25.006027+0000","last_became_peered":"2023-01-24T03:30:25.006027+0000","last_unstale":"2023-01-24T03:30:25.013106+0000","last_undegraded":"2023-01-24T03:30:25.013106+0000","last_fullsized":"2023-01-24T03:30:25.013106+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:51:29.481545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,6],"acting":[0,1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005695+0000","last_change":"2023-01-24T03:30:16.272475+0000","last_active":"2023-01-24T03:30:25.005695+0000","last_peered":"2023-01-24T03:30:25.005695+0000","last_clean":"2023-01-24T03:30:25.005695+0000","last_became_active":"2023-01-24T03:30:16.272140+0000","last_became_peered":"2023-01-24T03:30:16.272140+0000","last_unstale":"2023-01-24T03:30:25.005695+0000","last_undegraded":"2023-01-24T03:30:25.005695+0000","last_fullsized":"2023-01-24T03:30:25.005695+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:48:16.300876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992670+0000","last_change":"2023-01-24T03:30:20.609391+0000","last_active":"2023-01-24T03:30:24.992670+0000","last_peered":"2023-01-24T03:30:24.992670+0000","last_clean":"2023-01-24T03:30:24.992670+0000","last_became_active":"2023-01-24T03:30:20.609124+0000","last_became_peered":"2023-01-24T03:30:20.609124+0000","last_unstale":"2023-01-24T03:30:24.992670+0000","last_undegraded":"2023-01-24T03:30:24.992670+0000","last_fullsized":"2023-01-24T03:30:24.992670+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:36:49.616649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.001859+0000","last_change":"2023-01-24T03:30:20.996985+0000","last_active":"2023-01-24T03:30:25.001859+0000","last_peered":"2023-01-24T03:30:25.001859+0000","last_clean":"2023-01-24T03:30:25.001859+0000","last_became_active":"2023-01-24T03:30:20.996612+0000","last_became_peered":"2023-01-24T03:30:20.996612+0000","last_unstale":"2023-01-24T03:30:25.001859+0000","last_undegraded":"2023-01-24T03:30:25.001859+0000","last_fullsized":"2023-01-24T03:30:25.001859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:10:50.378902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.012218+0000","last_change":"2023-01-24T03:30:25.008155+0000","last_active":"2023-01-24T03:30:25.012218+0000","last_peered":"2023-01-24T03:30:25.012218+0000","last_clean":"2023-01-24T03:30:25.012218+0000","last_became_active":"2023-01-24T03:30:25.007674+0000","last_became_peered":"2023-01-24T03:30:25.007674+0000","last_unstale":"2023-01-24T03:30:25.012218+0000","last_undegraded":"2023-01-24T03:30:25.012218+0000","last_fullsized":"2023-01-24T03:30:25.012218+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:55:46.448808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,2],"acting":[6,0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990827+0000","last_change":"2023-01-24T03:30:16.272372+0000","last_active":"2023-01-24T03:30:24.990827+0000","last_peered":"2023-01-24T03:30:24.990827+0000","last_clean":"2023-01-24T03:30:24.990827+0000","last_became_active":"2023-01-24T03:30:16.272104+0000","last_became_peered":"2023-01-24T03:30:16.272104+0000","last_unstale":"2023-01-24T03:30:24.990827+0000","last_undegraded":"2023-01-24T03:30:24.990827+0000","last_fullsized":"2023-01-24T03:30:24.990827+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:15:10.865596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002114+0000","last_change":"2023-01-24T03:30:20.612284+0000","last_active":"2023-01-24T03:30:25.002114+0000","last_peered":"2023-01-24T03:30:25.002114+0000","last_clean":"2023-01-24T03:30:25.002114+0000","last_became_active":"2023-01-24T03:30:20.612029+0000","last_became_peered":"2023-01-24T03:30:20.612029+0000","last_unstale":"2023-01-24T03:30:25.002114+0000","last_undegraded":"2023-01-24T03:30:25.002114+0000","last_fullsized":"2023-01-24T03:30:25.002114+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:18:58.755974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992643+0000","last_change":"2023-01-24T03:30:20.989834+0000","last_active":"2023-01-24T03:30:24.992643+0000","last_peered":"2023-01-24T03:30:24.992643+0000","last_clean":"2023-01-24T03:30:24.992643+0000","last_became_active":"2023-01-24T03:30:20.989645+0000","last_became_peered":"2023-01-24T03:30:20.989645+0000","last_unstale":"2023-01-24T03:30:24.992643+0000","last_undegraded":"2023-01-24T03:30:24.992643+0000","last_fullsized":"2023-01-24T03:30:24.992643+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:42:06.477043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016259+0000","last_change":"2023-01-24T03:30:25.011952+0000","last_active":"2023-01-24T03:30:25.016259+0000","last_peered":"2023-01-24T03:30:25.016259+0000","last_clean":"2023-01-24T03:30:25.016259+0000","last_became_active":"2023-01-24T03:30:25.011612+0000","last_became_peered":"2023-01-24T03:30:25.011612+0000","last_unstale":"2023-01-24T03:30:25.016259+0000","last_undegraded":"2023-01-24T03:30:25.016259+0000","last_fullsized":"2023-01-24T03:30:25.016259+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:51:52.027285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6,5],"acting":[0,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990404+0000","last_change":"2023-01-24T03:30:16.048979+0000","last_active":"2023-01-24T03:30:24.990404+0000","last_peered":"2023-01-24T03:30:24.990404+0000","last_clean":"2023-01-24T03:30:24.990404+0000","last_became_active":"2023-01-24T03:30:16.048641+0000","last_became_peered":"2023-01-24T03:30:16.048641+0000","last_unstale":"2023-01-24T03:30:24.990404+0000","last_undegraded":"2023-01-24T03:30:24.990404+0000","last_fullsized":"2023-01-24T03:30:24.990404+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T15:11:26.211024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005087+0000","last_change":"2023-01-24T03:30:20.606906+0000","last_active":"2023-01-24T03:30:25.005087+0000","last_peered":"2023-01-24T03:30:25.005087+0000","last_clean":"2023-01-24T03:30:25.005087+0000","last_became_active":"2023-01-24T03:30:20.606640+0000","last_became_peered":"2023-01-24T03:30:20.606640+0000","last_unstale":"2023-01-24T03:30:25.005087+0000","last_undegraded":"2023-01-24T03:30:25.005087+0000","last_fullsized":"2023-01-24T03:30:25.005087+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:21:09.008653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002357+0000","last_change":"2023-01-24T03:30:20.999886+0000","last_active":"2023-01-24T03:30:25.002357+0000","last_peered":"2023-01-24T03:30:25.002357+0000","last_clean":"2023-01-24T03:30:25.002357+0000","last_became_active":"2023-01-24T03:30:20.999394+0000","last_became_peered":"2023-01-24T03:30:20.999394+0000","last_unstale":"2023-01-24T03:30:25.002357+0000","last_undegraded":"2023-01-24T03:30:25.002357+0000","last_fullsized":"2023-01-24T03:30:25.002357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:30:22.395611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016356+0000","last_change":"2023-01-24T03:30:25.006190+0000","last_active":"2023-01-24T03:30:25.016356+0000","last_peered":"2023-01-24T03:30:25.016356+0000","last_clean":"2023-01-24T03:30:25.016356+0000","last_became_active":"2023-01-24T03:30:25.005825+0000","last_became_peered":"2023-01-24T03:30:25.005825+0000","last_unstale":"2023-01-24T03:30:25.016356+0000","last_undegraded":"2023-01-24T03:30:25.016356+0000","last_fullsized":"2023-01-24T03:30:25.016356+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:51:50.888938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,0],"acting":[1,3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"24'1","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.446961+0000","last_change":"2023-01-24T03:30:20.611079+0000","last_active":"2023-01-24T03:30:25.446961+0000","last_peered":"2023-01-24T03:30:25.446961+0000","last_clean":"2023-01-24T03:30:25.446961+0000","last_became_active":"2023-01-24T03:30:20.610841+0000","last_became_peered":"2023-01-24T03:30:20.610841+0000","last_unstale":"2023-01-24T03:30:25.446961+0000","last_undegraded":"2023-01-24T03:30:25.446961+0000","last_fullsized":"2023-01-24T03:30:25.446961+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:34:24.922661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.994693+0000","last_change":"2023-01-24T03:30:20.991418+0000","last_active":"2023-01-24T03:30:24.994693+0000","last_peered":"2023-01-24T03:30:24.994693+0000","last_clean":"2023-01-24T03:30:24.994693+0000","last_became_active":"2023-01-24T03:30:20.991167+0000","last_became_peered":"2023-01-24T03:30:20.991167+0000","last_unstale":"2023-01-24T03:30:24.994693+0000","last_undegraded":"2023-01-24T03:30:24.994693+0000","last_fullsized":"2023-01-24T03:30:24.994693+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:51:52.736691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.013519+0000","last_change":"2023-01-24T03:30:25.010394+0000","last_active":"2023-01-24T03:30:25.013519+0000","last_peered":"2023-01-24T03:30:25.013519+0000","last_clean":"2023-01-24T03:30:25.013519+0000","last_became_active":"2023-01-24T03:30:25.010044+0000","last_became_peered":"2023-01-24T03:30:25.010044+0000","last_unstale":"2023-01-24T03:30:25.013519+0000","last_undegraded":"2023-01-24T03:30:25.013519+0000","last_fullsized":"2023-01-24T03:30:25.013519+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:08:55.250184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1,6],"acting":[7,4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"24'2","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.984877+0000","last_change":"2023-01-24T03:30:20.607538+0000","last_active":"2023-01-24T03:30:25.984877+0000","last_peered":"2023-01-24T03:30:25.984877+0000","last_clean":"2023-01-24T03:30:25.984877+0000","last_became_active":"2023-01-24T03:30:20.607285+0000","last_became_peered":"2023-01-24T03:30:20.607285+0000","last_unstale":"2023-01-24T03:30:25.984877+0000","last_undegraded":"2023-01-24T03:30:25.984877+0000","last_fullsized":"2023-01-24T03:30:25.984877+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:42:21.836603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992857+0000","last_change":"2023-01-24T03:30:20.986791+0000","last_active":"2023-01-24T03:30:24.992857+0000","last_peered":"2023-01-24T03:30:24.992857+0000","last_clean":"2023-01-24T03:30:24.992857+0000","last_became_active":"2023-01-24T03:30:20.986516+0000","last_became_peered":"2023-01-24T03:30:20.986516+0000","last_unstale":"2023-01-24T03:30:24.992857+0000","last_undegraded":"2023-01-24T03:30:24.992857+0000","last_fullsized":"2023-01-24T03:30:24.992857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:53:19.191166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.015383+0000","last_change":"2023-01-24T03:30:25.010751+0000","last_active":"2023-01-24T03:30:25.015383+0000","last_peered":"2023-01-24T03:30:25.015383+0000","last_clean":"2023-01-24T03:30:25.015383+0000","last_became_active":"2023-01-24T03:30:25.010332+0000","last_became_peered":"2023-01-24T03:30:25.010332+0000","last_unstale":"2023-01-24T03:30:25.015383+0000","last_undegraded":"2023-01-24T03:30:25.015383+0000","last_fullsized":"2023-01-24T03:30:25.015383+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:48:30.190712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7,4],"acting":[5,0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"24'2","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985166+0000","last_change":"2023-01-24T03:30:20.608509+0000","last_active":"2023-01-24T03:30:25.985166+0000","last_peered":"2023-01-24T03:30:25.985166+0000","last_clean":"2023-01-24T03:30:25.985166+0000","last_became_active":"2023-01-24T03:30:20.608313+0000","last_became_peered":"2023-01-24T03:30:20.608313+0000","last_unstale":"2023-01-24T03:30:25.985166+0000","last_undegraded":"2023-01-24T03:30:25.985166+0000","last_fullsized":"2023-01-24T03:30:25.985166+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:31:52.253588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992448+0000","last_change":"2023-01-24T03:30:20.992334+0000","last_active":"2023-01-24T03:30:24.992448+0000","last_peered":"2023-01-24T03:30:24.992448+0000","last_clean":"2023-01-24T03:30:24.992448+0000","last_became_active":"2023-01-24T03:30:20.992027+0000","last_became_peered":"2023-01-24T03:30:20.992027+0000","last_unstale":"2023-01-24T03:30:24.992448+0000","last_undegraded":"2023-01-24T03:30:24.992448+0000","last_fullsized":"2023-01-24T03:30:24.992448+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:57:56.985507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.012177+0000","last_change":"2023-01-24T03:30:25.004167+0000","last_active":"2023-01-24T03:30:25.012177+0000","last_peered":"2023-01-24T03:30:25.012177+0000","last_clean":"2023-01-24T03:30:25.012177+0000","last_became_active":"2023-01-24T03:30:25.003752+0000","last_became_peered":"2023-01-24T03:30:25.003752+0000","last_unstale":"2023-01-24T03:30:25.012177+0000","last_undegraded":"2023-01-24T03:30:25.012177+0000","last_fullsized":"2023-01-24T03:30:25.012177+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:19:16.285529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"24'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.448123+0000","last_change":"2023-01-24T03:30:20.603542+0000","last_active":"2023-01-24T03:30:25.448123+0000","last_peered":"2023-01-24T03:30:25.448123+0000","last_clean":"2023-01-24T03:30:25.448123+0000","last_became_active":"2023-01-24T03:30:20.603287+0000","last_became_peered":"2023-01-24T03:30:20.603287+0000","last_unstale":"2023-01-24T03:30:25.448123+0000","last_undegraded":"2023-01-24T03:30:25.448123+0000","last_fullsized":"2023-01-24T03:30:25.448123+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:03:56.865763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991098+0000","last_change":"2023-01-24T03:30:20.989519+0000","last_active":"2023-01-24T03:30:24.991098+0000","last_peered":"2023-01-24T03:30:24.991098+0000","last_clean":"2023-01-24T03:30:24.991098+0000","last_became_active":"2023-01-24T03:30:20.989286+0000","last_became_peered":"2023-01-24T03:30:20.989286+0000","last_unstale":"2023-01-24T03:30:24.991098+0000","last_undegraded":"2023-01-24T03:30:24.991098+0000","last_fullsized":"2023-01-24T03:30:24.991098+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:17:30.829327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016619+0000","last_change":"2023-01-24T03:30:25.010262+0000","last_active":"2023-01-24T03:30:25.016619+0000","last_peered":"2023-01-24T03:30:25.016619+0000","last_clean":"2023-01-24T03:30:25.016619+0000","last_became_active":"2023-01-24T03:30:25.009930+0000","last_became_peered":"2023-01-24T03:30:25.009930+0000","last_unstale":"2023-01-24T03:30:25.016619+0000","last_undegraded":"2023-01-24T03:30:25.016619+0000","last_fullsized":"2023-01-24T03:30:25.016619+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:36:17.807046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7,2],"acting":[1,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"24'4","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985911+0000","last_change":"2023-01-24T03:30:20.607334+0000","last_active":"2023-01-24T03:30:25.985911+0000","last_peered":"2023-01-24T03:30:25.985911+0000","last_clean":"2023-01-24T03:30:25.985911+0000","last_became_active":"2023-01-24T03:30:20.607079+0000","last_became_peered":"2023-01-24T03:30:20.607079+0000","last_unstale":"2023-01-24T03:30:25.985911+0000","last_undegraded":"2023-01-24T03:30:25.985911+0000","last_fullsized":"2023-01-24T03:30:25.985911+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:05:18.886532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992554+0000","last_change":"2023-01-24T03:30:20.987567+0000","last_active":"2023-01-24T03:30:24.992554+0000","last_peered":"2023-01-24T03:30:24.992554+0000","last_clean":"2023-01-24T03:30:24.992554+0000","last_became_active":"2023-01-24T03:30:20.987058+0000","last_became_peered":"2023-01-24T03:30:20.987058+0000","last_unstale":"2023-01-24T03:30:24.992554+0000","last_undegraded":"2023-01-24T03:30:24.992554+0000","last_fullsized":"2023-01-24T03:30:24.992554+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:58:30.391733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.989538+0000","last_change":"2023-01-24T03:30:20.986393+0000","last_active":"2023-01-24T03:30:24.989538+0000","last_peered":"2023-01-24T03:30:24.989538+0000","last_clean":"2023-01-24T03:30:24.989538+0000","last_became_active":"2023-01-24T03:30:20.986089+0000","last_became_peered":"2023-01-24T03:30:20.986089+0000","last_unstale":"2023-01-24T03:30:24.989538+0000","last_undegraded":"2023-01-24T03:30:24.989538+0000","last_fullsized":"2023-01-24T03:30:24.989538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:44:21.992291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990902+0000","last_change":"2023-01-24T03:30:20.991468+0000","last_active":"2023-01-24T03:30:24.990902+0000","last_peered":"2023-01-24T03:30:24.990902+0000","last_clean":"2023-01-24T03:30:24.990902+0000","last_became_active":"2023-01-24T03:30:20.991227+0000","last_became_peered":"2023-01-24T03:30:20.991227+0000","last_unstale":"2023-01-24T03:30:24.990902+0000","last_undegraded":"2023-01-24T03:30:24.990902+0000","last_fullsized":"2023-01-24T03:30:24.990902+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:27:25.806340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992602+0000","last_change":"2023-01-24T03:30:20.989824+0000","last_active":"2023-01-24T03:30:24.992602+0000","last_peered":"2023-01-24T03:30:24.992602+0000","last_clean":"2023-01-24T03:30:24.992602+0000","last_became_active":"2023-01-24T03:30:20.989577+0000","last_became_peered":"2023-01-24T03:30:20.989577+0000","last_unstale":"2023-01-24T03:30:24.992602+0000","last_undegraded":"2023-01-24T03:30:24.992602+0000","last_fullsized":"2023-01-24T03:30:24.992602+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:48:17.948459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992452+0000","last_change":"2023-01-24T03:30:20.996266+0000","last_active":"2023-01-24T03:30:24.992452+0000","last_peered":"2023-01-24T03:30:24.992452+0000","last_clean":"2023-01-24T03:30:24.992452+0000","last_became_active":"2023-01-24T03:30:20.995610+0000","last_became_peered":"2023-01-24T03:30:20.995610+0000","last_unstale":"2023-01-24T03:30:24.992452+0000","last_undegraded":"2023-01-24T03:30:24.992452+0000","last_fullsized":"2023-01-24T03:30:24.992452+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:24:43.022530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.001943+0000","last_change":"2023-01-24T03:30:20.997081+0000","last_active":"2023-01-24T03:30:25.001943+0000","last_peered":"2023-01-24T03:30:25.001943+0000","last_clean":"2023-01-24T03:30:25.001943+0000","last_became_active":"2023-01-24T03:30:20.996837+0000","last_became_peered":"2023-01-24T03:30:20.996837+0000","last_unstale":"2023-01-24T03:30:25.001943+0000","last_undegraded":"2023-01-24T03:30:25.001943+0000","last_fullsized":"2023-01-24T03:30:25.001943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:42:06.134817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":4483,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":8966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64574,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9180,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":8832,"kb_avail":94362660,"statfs":{"total":96636764160,"available":96627363840,"internally_reserved":0,"allocated":348160,"data_stored":698731,"data_compressed":32287,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":9043968},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8576,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94363264,"statfs":{"total":96636764160,"available":96627982336,"internally_reserved":0,"allocated":253952,"data_stored":104434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9152,"kb_used_data":312,"kb_used_omap":0,"kb_used_meta":8832,"kb_avail":94362688,"statfs":{"total":96636764160,"available":96627392512,"internally_reserved":0,"allocated":319488,"data_stored":694802,"data_compressed":32287,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":9043968},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]}]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8640,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94363200,"statfs":{"total":96636764160,"available":96627916800,"internally_reserved":0,"allocated":253952,"data_stored":104382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.255}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8512,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94363328,"statfs":{"total":96636764160,"available":96628047872,"internally_reserved":0,"allocated":253952,"data_stored":104434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.254}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8580,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94363260,"statfs":{"total":96636764160,"available":96627978240,"internally_reserved":0,"allocated":258048,"data_stored":104524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8660,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94363180,"statfs":{"total":96636764160,"available":96627896320,"internally_reserved":0,"allocated":274432,"data_stored":108203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]}]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8580,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94363260,"statfs":{"total":96636764160,"available":96627978240,"internally_reserved":0,"allocated":258048,"data_stored":103705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32287,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32287,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":3929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-01-24T03:30:43.188 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-24T03:30:43.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.330+0000 7f8a9d9de700 1 Processor -- start 2023-01-24T03:30:43.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.330+0000 7f8a9d9de700 1 -- start start 2023-01-24T03:30:43.336 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.331+0000 7f8a9d9de700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f8a980c3560 0x7f8a980c1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:43.336 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.331+0000 7f8a9d9de700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f8a9806d320 0x7f8a980c4790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:43.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.331+0000 7f8a9d9de700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a98153270 con 0x7f8a9806cf50 2023-01-24T03:30:43.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.331+0000 7f8a9d9de700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a98153e80 con 0x7f8a980c3560 2023-01-24T03:30:43.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.331+0000 7f8a9d9de700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a980c2870 con 0x7f8a9806d320 2023-01-24T03:30:43.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.331+0000 7f8a967fc700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f8a980c3560 0x7f8a980c1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:43.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.331+0000 7f8a96ffd700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f8a9806cf50 0x7f8a980c2e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:60040/0 (socket says 172.21.15.112:60040) 2023-01-24T03:30:43.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.331+0000 7f8a96ffd700 1 -- 172.21.15.112:0/1328640001 learned_addr learned my addr 172.21.15.112:0/1328640001 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:43.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.331+0000 7f8a967fc700 1 -- 172.21.15.112:0/1328640001 >> v2:172.21.15.142:3300/0 conn(0x7f8a9806d320 msgr2=0x7f8a980c4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:43.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.331+0000 7f8a977fe700 1 --2- 172.21.15.112:0/1328640001 >> v2:172.21.15.142:3300/0 conn(0x7f8a9806d320 0x7f8a980c4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:43.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.331+0000 7f8a967fc700 1 --2- 172.21.15.112:0/1328640001 >> v2:172.21.15.142:3300/0 conn(0x7f8a9806d320 0x7f8a980c4790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:43.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.331+0000 7f8a95ffb700 1 -- 172.21.15.112:0/1328640001 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3294914030 0 0) 0x7f8a98153270 con 0x7f8a9806cf50 2023-01-24T03:30:43.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.331+0000 7f8a967fc700 1 -- 172.21.15.112:0/1328640001 >> v1:172.21.15.112:6789/0 conn(0x7f8a9806cf50 legacy=0x7f8a980c2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:43.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.331+0000 7f8a967fc700 1 -- 172.21.15.112:0/1328640001 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a981534e0 con 0x7f8a980c3560 2023-01-24T03:30:43.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.332+0000 7f8a967fc700 1 --2- 172.21.15.112:0/1328640001 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a980c3560 0x7f8a980c1560 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8a9806eee0 tx=0x7f8a80009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fc36002889d673b server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:43.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.332+0000 7f8a95ffb700 1 -- 172.21.15.112:0/1328640001 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a80011630 con 0x7f8a980c3560 2023-01-24T03:30:43.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.332+0000 7f8a95ffb700 1 -- 172.21.15.112:0/1328640001 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a80011790 con 0x7f8a980c3560 2023-01-24T03:30:43.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.332+0000 7f8a95ffb700 1 -- 172.21.15.112:0/1328640001 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a80011b50 con 0x7f8a980c3560 2023-01-24T03:30:43.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.332+0000 7f8a9d9de700 1 -- 172.21.15.112:0/1328640001 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a980c3560 msgr2=0x7f8a980c1560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:43.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.332+0000 7f8a9d9de700 1 --2- 172.21.15.112:0/1328640001 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a980c3560 0x7f8a980c1560 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8a9806eee0 tx=0x7f8a80009060 comp rx=0 tx=0).stop 2023-01-24T03:30:43.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.332+0000 7f8a9d9de700 1 -- 172.21.15.112:0/1328640001 shutdown_connections 2023-01-24T03:30:43.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.332+0000 7f8a9d9de700 1 --2- 172.21.15.112:0/1328640001 >> v2:172.21.15.142:3300/0 conn(0x7f8a9806d320 0x7f8a980c4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:43.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.332+0000 7f8a9d9de700 1 --2- 172.21.15.112:0/1328640001 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a980c3560 0x7f8a980c1560 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:43.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.332+0000 7f8a9d9de700 1 -- 172.21.15.112:0/1328640001 >> 172.21.15.112:0/1328640001 conn(0x7f8a980c9820 msgr2=0x7f8a980c9c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:43.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.332+0000 7f8a9d9de700 1 -- 172.21.15.112:0/1328640001 shutdown_connections 2023-01-24T03:30:43.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.332+0000 7f8a9d9de700 1 -- 172.21.15.112:0/1328640001 wait complete. 2023-01-24T03:30:43.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.333+0000 7f8a9d9de700 1 Processor -- start 2023-01-24T03:30:43.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.333+0000 7f8a9d9de700 1 -- start start 2023-01-24T03:30:43.344 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.334+0000 7f8a9d9de700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a9806cf50 0x7f8a980c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:43.345 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.334+0000 7f8a9d9de700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a9806d320 0x7f8a980bfb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:43.345 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.334+0000 7f8a9d9de700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a980c3560 0x7f8a9815cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:43.345 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.334+0000 7f8a9d9de700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a980bddd0 con 0x7f8a980c3560 2023-01-24T03:30:43.345 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.334+0000 7f8a9d9de700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a980bf8a0 con 0x7f8a9806d320 2023-01-24T03:30:43.346 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.334+0000 7f8a9d9de700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a980bd4c0 con 0x7f8a9806cf50 2023-01-24T03:30:43.346 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.334+0000 7f8a977fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a980c3560 0x7f8a9815cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:43.346 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.334+0000 7f8a977fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a980c3560 0x7f8a9815cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50828/0 (socket says 172.21.15.112:50828) 2023-01-24T03:30:43.346 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.334+0000 7f8a977fe700 1 -- 172.21.15.112:0/1399017591 learned_addr learned my addr 172.21.15.112:0/1399017591 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:43.346 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.334+0000 7f8a96ffd700 1 --2- 172.21.15.112:0/1399017591 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a9806cf50 0x7f8a980c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:43.347 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.334+0000 7f8a967fc700 1 --2- 172.21.15.112:0/1399017591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a9806d320 0x7f8a980bfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:43.347 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.334+0000 7f8a977fe700 1 -- 172.21.15.112:0/1399017591 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a9806cf50 msgr2=0x7f8a980c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:43.347 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.334+0000 7f8a977fe700 1 --2- 172.21.15.112:0/1399017591 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a9806cf50 0x7f8a980c5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:43.347 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.334+0000 7f8a977fe700 1 -- 172.21.15.112:0/1399017591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a9806d320 msgr2=0x7f8a980bfb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:43.347 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.334+0000 7f8a977fe700 1 --2- 172.21.15.112:0/1399017591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a9806d320 0x7f8a980bfb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:43.347 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.334+0000 7f8a977fe700 1 -- 172.21.15.112:0/1399017591 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a800092a0 con 0x7f8a980c3560 2023-01-24T03:30:43.348 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.335+0000 7f8a977fe700 1 --2- 172.21.15.112:0/1399017591 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a980c3560 0x7f8a9815cf40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8a9000c680 tx=0x7f8a90008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:43.348 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.335+0000 7f8a8effd700 1 -- 172.21.15.112:0/1399017591 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a90011600 con 0x7f8a980c3560 2023-01-24T03:30:43.348 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.335+0000 7f8a8effd700 1 -- 172.21.15.112:0/1399017591 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a90011760 con 0x7f8a980c3560 2023-01-24T03:30:43.348 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.335+0000 7f8a9d9de700 1 -- 172.21.15.112:0/1399017591 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a980c0540 con 0x7f8a980c3560 2023-01-24T03:30:43.349 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.335+0000 7f8a8effd700 1 -- 172.21.15.112:0/1399017591 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a90011b20 con 0x7f8a980c3560 2023-01-24T03:30:43.349 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.335+0000 7f8a9d9de700 1 -- 172.21.15.112:0/1399017591 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a980bda60 con 0x7f8a980c3560 2023-01-24T03:30:43.349 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.336+0000 7f8a9d9de700 1 -- 172.21.15.112:0/1399017591 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a9804bb80 con 0x7f8a980c3560 2023-01-24T03:30:43.349 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.336+0000 7f8a8effd700 1 -- 172.21.15.112:0/1399017591 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f8a90020070 con 0x7f8a980c3560 2023-01-24T03:30:43.349 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.336+0000 7f8a8effd700 1 --2- 172.21.15.112:0/1399017591 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f8a7c0611c0 0x7f8a7c063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:43.350 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.336+0000 7f8a8effd700 1 -- 172.21.15.112:0/1399017591 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f8a9007d280 con 0x7f8a980c3560 2023-01-24T03:30:43.350 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.336+0000 7f8a96ffd700 1 --2- 172.21.15.112:0/1399017591 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f8a7c0611c0 0x7f8a7c063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:43.350 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.337+0000 7f8a96ffd700 1 --2- 172.21.15.112:0/1399017591 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f8a7c0611c0 0x7f8a7c063670 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8a88001560 tx=0x7f8a88009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:43.350 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.340+0000 7f8a8effd700 1 -- 172.21.15.112:0/1399017591 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f8a9004d060 con 0x7f8a980c3560 2023-01-24T03:30:43.519 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.516+0000 7f8a9d9de700 1 -- 172.21.15.112:0/1399017591 --> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8a980bdf10 con 0x7f8a7c0611c0 2023-01-24T03:30:43.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.526+0000 7f8a8effd700 1 -- 172.21.15.112:0/1399017591 <== mgr.4104 v2:172.21.15.112:6832/84276 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+365109 (secure 0 0 0) 0x7f8a980bdf10 con 0x7f8a7c0611c0 2023-01-24T03:30:43.534 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:30:43.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.529+0000 7f8a9d9de700 1 -- 172.21.15.112:0/1399017591 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f8a7c0611c0 msgr2=0x7f8a7c063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:43.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.529+0000 7f8a9d9de700 1 --2- 172.21.15.112:0/1399017591 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f8a7c0611c0 0x7f8a7c063670 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8a88001560 tx=0x7f8a88009040 comp rx=0 tx=0).stop 2023-01-24T03:30:43.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.529+0000 7f8a9d9de700 1 -- 172.21.15.112:0/1399017591 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a980c3560 msgr2=0x7f8a9815cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:43.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.529+0000 7f8a9d9de700 1 --2- 172.21.15.112:0/1399017591 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a980c3560 0x7f8a9815cf40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8a9000c680 tx=0x7f8a90008040 comp rx=0 tx=0).stop 2023-01-24T03:30:43.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.530+0000 7f8a9d9de700 1 -- 172.21.15.112:0/1399017591 shutdown_connections 2023-01-24T03:30:43.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.530+0000 7f8a9d9de700 1 --2- 172.21.15.112:0/1399017591 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f8a7c0611c0 0x7f8a7c063670 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:43.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.530+0000 7f8a9d9de700 1 --2- 172.21.15.112:0/1399017591 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a980c3560 0x7f8a9815cf40 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:43.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.530+0000 7f8a9d9de700 1 --2- 172.21.15.112:0/1399017591 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a9806cf50 0x7f8a980c5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:43.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.530+0000 7f8a9d9de700 1 --2- 172.21.15.112:0/1399017591 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a9806d320 0x7f8a980bfb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:43.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.530+0000 7f8a9d9de700 1 -- 172.21.15.112:0/1399017591 >> 172.21.15.112:0/1399017591 conn(0x7f8a980c9820 msgr2=0x7f8a9814cf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:43.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.530+0000 7f8a9d9de700 1 -- 172.21.15.112:0/1399017591 shutdown_connections 2023-01-24T03:30:43.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.530+0000 7f8a9d9de700 1 -- 172.21.15.112:0/1399017591 wait complete. 2023-01-24T03:30:43.540 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2023-01-24T03:30:43.544 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2023-01-24T03:30:43.007270+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594851,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":69880,"kb_used_data":2168,"kb_used_omap":0,"kb_used_meta":67648,"kb_avail":754904840,"statfs":{"total":773094113280,"available":773022556160,"internally_reserved":0,"allocated":2220032,"data_stored":2023215,"data_compressed":64574,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":69271552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2177,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009801"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993289+0000","last_change":"2023-01-24T03:30:20.986308+0000","last_active":"2023-01-24T03:30:24.993289+0000","last_peered":"2023-01-24T03:30:24.993289+0000","last_clean":"2023-01-24T03:30:24.993289+0000","last_became_active":"2023-01-24T03:30:20.986025+0000","last_became_peered":"2023-01-24T03:30:20.986025+0000","last_unstale":"2023-01-24T03:30:24.993289+0000","last_undegraded":"2023-01-24T03:30:24.993289+0000","last_fullsized":"2023-01-24T03:30:24.993289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:47:58.616698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.006202+0000","last_change":"2023-01-24T03:30:20.994478+0000","last_active":"2023-01-24T03:30:25.006202+0000","last_peered":"2023-01-24T03:30:25.006202+0000","last_clean":"2023-01-24T03:30:25.006202+0000","last_became_active":"2023-01-24T03:30:20.994207+0000","last_became_peered":"2023-01-24T03:30:20.994207+0000","last_unstale":"2023-01-24T03:30:25.006202+0000","last_undegraded":"2023-01-24T03:30:25.006202+0000","last_fullsized":"2023-01-24T03:30:25.006202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:02:53.230231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993177+0000","last_change":"2023-01-24T03:30:20.989651+0000","last_active":"2023-01-24T03:30:24.993177+0000","last_peered":"2023-01-24T03:30:24.993177+0000","last_clean":"2023-01-24T03:30:24.993177+0000","last_became_active":"2023-01-24T03:30:20.987585+0000","last_became_peered":"2023-01-24T03:30:20.987585+0000","last_unstale":"2023-01-24T03:30:24.993177+0000","last_undegraded":"2023-01-24T03:30:24.993177+0000","last_fullsized":"2023-01-24T03:30:24.993177+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:36:02.177544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990511+0000","last_change":"2023-01-24T03:30:20.993601+0000","last_active":"2023-01-24T03:30:24.990511+0000","last_peered":"2023-01-24T03:30:24.990511+0000","last_clean":"2023-01-24T03:30:24.990511+0000","last_became_active":"2023-01-24T03:30:20.993418+0000","last_became_peered":"2023-01-24T03:30:20.993418+0000","last_unstale":"2023-01-24T03:30:24.990511+0000","last_undegraded":"2023-01-24T03:30:24.990511+0000","last_fullsized":"2023-01-24T03:30:24.990511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:47:50.260497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996430+0000","last_change":"2023-01-24T03:30:20.991605+0000","last_active":"2023-01-24T03:30:24.996430+0000","last_peered":"2023-01-24T03:30:24.996430+0000","last_clean":"2023-01-24T03:30:24.996430+0000","last_became_active":"2023-01-24T03:30:20.991361+0000","last_became_peered":"2023-01-24T03:30:20.991361+0000","last_unstale":"2023-01-24T03:30:24.996430+0000","last_undegraded":"2023-01-24T03:30:24.996430+0000","last_fullsized":"2023-01-24T03:30:24.996430+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:31:25.418430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993288+0000","last_change":"2023-01-24T03:30:20.985058+0000","last_active":"2023-01-24T03:30:24.993288+0000","last_peered":"2023-01-24T03:30:24.993288+0000","last_clean":"2023-01-24T03:30:24.993288+0000","last_became_active":"2023-01-24T03:30:20.984754+0000","last_became_peered":"2023-01-24T03:30:20.984754+0000","last_unstale":"2023-01-24T03:30:24.993288+0000","last_undegraded":"2023-01-24T03:30:24.993288+0000","last_fullsized":"2023-01-24T03:30:24.993288+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:12:03.225109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996279+0000","last_change":"2023-01-24T03:30:21.000890+0000","last_active":"2023-01-24T03:30:24.996279+0000","last_peered":"2023-01-24T03:30:24.996279+0000","last_clean":"2023-01-24T03:30:24.996279+0000","last_became_active":"2023-01-24T03:30:21.000438+0000","last_became_peered":"2023-01-24T03:30:21.000438+0000","last_unstale":"2023-01-24T03:30:24.996279+0000","last_undegraded":"2023-01-24T03:30:24.996279+0000","last_fullsized":"2023-01-24T03:30:24.996279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:17:12.999664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.004717+0000","last_change":"2023-01-24T03:30:20.986753+0000","last_active":"2023-01-24T03:30:25.004717+0000","last_peered":"2023-01-24T03:30:25.004717+0000","last_clean":"2023-01-24T03:30:25.004717+0000","last_became_active":"2023-01-24T03:30:20.986524+0000","last_became_peered":"2023-01-24T03:30:20.986524+0000","last_unstale":"2023-01-24T03:30:25.004717+0000","last_undegraded":"2023-01-24T03:30:25.004717+0000","last_fullsized":"2023-01-24T03:30:25.004717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:24:55.699379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992350+0000","last_change":"2023-01-24T03:30:20.991136+0000","last_active":"2023-01-24T03:30:24.992350+0000","last_peered":"2023-01-24T03:30:24.992350+0000","last_clean":"2023-01-24T03:30:24.992350+0000","last_became_active":"2023-01-24T03:30:20.990925+0000","last_became_peered":"2023-01-24T03:30:20.990925+0000","last_unstale":"2023-01-24T03:30:24.992350+0000","last_undegraded":"2023-01-24T03:30:24.992350+0000","last_fullsized":"2023-01-24T03:30:24.992350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:45:37.485199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.006401+0000","last_change":"2023-01-24T03:30:20.995730+0000","last_active":"2023-01-24T03:30:25.006401+0000","last_peered":"2023-01-24T03:30:25.006401+0000","last_clean":"2023-01-24T03:30:25.006401+0000","last_became_active":"2023-01-24T03:30:20.994880+0000","last_became_peered":"2023-01-24T03:30:20.994880+0000","last_unstale":"2023-01-24T03:30:25.006401+0000","last_undegraded":"2023-01-24T03:30:25.006401+0000","last_fullsized":"2023-01-24T03:30:25.006401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:07:21.546386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993144+0000","last_change":"2023-01-24T03:30:20.993328+0000","last_active":"2023-01-24T03:30:24.993144+0000","last_peered":"2023-01-24T03:30:24.993144+0000","last_clean":"2023-01-24T03:30:24.993144+0000","last_became_active":"2023-01-24T03:30:20.992730+0000","last_became_peered":"2023-01-24T03:30:20.992730+0000","last_unstale":"2023-01-24T03:30:24.993144+0000","last_undegraded":"2023-01-24T03:30:24.993144+0000","last_fullsized":"2023-01-24T03:30:24.993144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:43:39.048550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.994605+0000","last_change":"2023-01-24T03:30:20.984993+0000","last_active":"2023-01-24T03:30:24.994605+0000","last_peered":"2023-01-24T03:30:24.994605+0000","last_clean":"2023-01-24T03:30:24.994605+0000","last_became_active":"2023-01-24T03:30:20.984540+0000","last_became_peered":"2023-01-24T03:30:20.984540+0000","last_unstale":"2023-01-24T03:30:24.994605+0000","last_undegraded":"2023-01-24T03:30:24.994605+0000","last_fullsized":"2023-01-24T03:30:24.994605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:16:39.993359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.995936+0000","last_change":"2023-01-24T03:30:20.986567+0000","last_active":"2023-01-24T03:30:24.995936+0000","last_peered":"2023-01-24T03:30:24.995936+0000","last_clean":"2023-01-24T03:30:24.995936+0000","last_became_active":"2023-01-24T03:30:20.986224+0000","last_became_peered":"2023-01-24T03:30:20.986224+0000","last_unstale":"2023-01-24T03:30:24.995936+0000","last_undegraded":"2023-01-24T03:30:24.995936+0000","last_fullsized":"2023-01-24T03:30:24.995936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:09:14.692354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005087+0000","last_change":"2023-01-24T03:30:20.988843+0000","last_active":"2023-01-24T03:30:25.005087+0000","last_peered":"2023-01-24T03:30:25.005087+0000","last_clean":"2023-01-24T03:30:25.005087+0000","last_became_active":"2023-01-24T03:30:20.988486+0000","last_became_peered":"2023-01-24T03:30:20.988486+0000","last_unstale":"2023-01-24T03:30:25.005087+0000","last_undegraded":"2023-01-24T03:30:25.005087+0000","last_fullsized":"2023-01-24T03:30:25.005087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:18:20.538844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996110+0000","last_change":"2023-01-24T03:30:21.000738+0000","last_active":"2023-01-24T03:30:24.996110+0000","last_peered":"2023-01-24T03:30:24.996110+0000","last_clean":"2023-01-24T03:30:24.996110+0000","last_became_active":"2023-01-24T03:30:21.000458+0000","last_became_peered":"2023-01-24T03:30:21.000458+0000","last_unstale":"2023-01-24T03:30:24.996110+0000","last_undegraded":"2023-01-24T03:30:24.996110+0000","last_fullsized":"2023-01-24T03:30:24.996110+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:21:42.876426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992383+0000","last_change":"2023-01-24T03:30:20.985062+0000","last_active":"2023-01-24T03:30:24.992383+0000","last_peered":"2023-01-24T03:30:24.992383+0000","last_clean":"2023-01-24T03:30:24.992383+0000","last_became_active":"2023-01-24T03:30:20.984759+0000","last_became_peered":"2023-01-24T03:30:20.984759+0000","last_unstale":"2023-01-24T03:30:24.992383+0000","last_undegraded":"2023-01-24T03:30:24.992383+0000","last_fullsized":"2023-01-24T03:30:24.992383+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:38:34.615748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992439+0000","last_change":"2023-01-24T03:30:20.989080+0000","last_active":"2023-01-24T03:30:24.992439+0000","last_peered":"2023-01-24T03:30:24.992439+0000","last_clean":"2023-01-24T03:30:24.992439+0000","last_became_active":"2023-01-24T03:30:20.988823+0000","last_became_peered":"2023-01-24T03:30:20.988823+0000","last_unstale":"2023-01-24T03:30:24.992439+0000","last_undegraded":"2023-01-24T03:30:24.992439+0000","last_fullsized":"2023-01-24T03:30:24.992439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:22:21.807370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993008+0000","last_change":"2023-01-24T03:30:20.984222+0000","last_active":"2023-01-24T03:30:24.993008+0000","last_peered":"2023-01-24T03:30:24.993008+0000","last_clean":"2023-01-24T03:30:24.993008+0000","last_became_active":"2023-01-24T03:30:20.984002+0000","last_became_peered":"2023-01-24T03:30:20.984002+0000","last_unstale":"2023-01-24T03:30:24.993008+0000","last_undegraded":"2023-01-24T03:30:24.993008+0000","last_fullsized":"2023-01-24T03:30:24.993008+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:54:24.005486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.997598+0000","last_change":"2023-01-24T03:30:20.988197+0000","last_active":"2023-01-24T03:30:24.997598+0000","last_peered":"2023-01-24T03:30:24.997598+0000","last_clean":"2023-01-24T03:30:24.997598+0000","last_became_active":"2023-01-24T03:30:20.987941+0000","last_became_peered":"2023-01-24T03:30:20.987941+0000","last_unstale":"2023-01-24T03:30:24.997598+0000","last_undegraded":"2023-01-24T03:30:24.997598+0000","last_fullsized":"2023-01-24T03:30:24.997598+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T15:08:29.659964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992957+0000","last_change":"2023-01-24T03:30:20.988701+0000","last_active":"2023-01-24T03:30:24.992957+0000","last_peered":"2023-01-24T03:30:24.992957+0000","last_clean":"2023-01-24T03:30:24.992957+0000","last_became_active":"2023-01-24T03:30:20.988476+0000","last_became_peered":"2023-01-24T03:30:20.988476+0000","last_unstale":"2023-01-24T03:30:24.992957+0000","last_undegraded":"2023-01-24T03:30:24.992957+0000","last_fullsized":"2023-01-24T03:30:24.992957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:33:29.463194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996546+0000","last_change":"2023-01-24T03:30:21.001125+0000","last_active":"2023-01-24T03:30:24.996546+0000","last_peered":"2023-01-24T03:30:24.996546+0000","last_clean":"2023-01-24T03:30:24.996546+0000","last_became_active":"2023-01-24T03:30:21.000873+0000","last_became_peered":"2023-01-24T03:30:21.000873+0000","last_unstale":"2023-01-24T03:30:24.996546+0000","last_undegraded":"2023-01-24T03:30:24.996546+0000","last_fullsized":"2023-01-24T03:30:24.996546+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:22:16.136875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993432+0000","last_change":"2023-01-24T03:30:20.992639+0000","last_active":"2023-01-24T03:30:24.993432+0000","last_peered":"2023-01-24T03:30:24.993432+0000","last_clean":"2023-01-24T03:30:24.993432+0000","last_became_active":"2023-01-24T03:30:20.992386+0000","last_became_peered":"2023-01-24T03:30:20.992386+0000","last_unstale":"2023-01-24T03:30:24.993432+0000","last_undegraded":"2023-01-24T03:30:24.993432+0000","last_fullsized":"2023-01-24T03:30:24.993432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:29:17.912198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992608+0000","last_change":"2023-01-24T03:30:20.994066+0000","last_active":"2023-01-24T03:30:24.992608+0000","last_peered":"2023-01-24T03:30:24.992608+0000","last_clean":"2023-01-24T03:30:24.992608+0000","last_became_active":"2023-01-24T03:30:20.993820+0000","last_became_peered":"2023-01-24T03:30:20.993820+0000","last_unstale":"2023-01-24T03:30:24.992608+0000","last_undegraded":"2023-01-24T03:30:24.992608+0000","last_fullsized":"2023-01-24T03:30:24.992608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:31:23.874058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996188+0000","last_change":"2023-01-24T03:30:21.001652+0000","last_active":"2023-01-24T03:30:24.996188+0000","last_peered":"2023-01-24T03:30:24.996188+0000","last_clean":"2023-01-24T03:30:24.996188+0000","last_became_active":"2023-01-24T03:30:21.001400+0000","last_became_peered":"2023-01-24T03:30:21.001400+0000","last_unstale":"2023-01-24T03:30:24.996188+0000","last_undegraded":"2023-01-24T03:30:24.996188+0000","last_fullsized":"2023-01-24T03:30:24.996188+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:55:53.249816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002243+0000","last_change":"2023-01-24T03:30:20.998304+0000","last_active":"2023-01-24T03:30:25.002243+0000","last_peered":"2023-01-24T03:30:25.002243+0000","last_clean":"2023-01-24T03:30:25.002243+0000","last_became_active":"2023-01-24T03:30:20.996288+0000","last_became_peered":"2023-01-24T03:30:20.996288+0000","last_unstale":"2023-01-24T03:30:25.002243+0000","last_undegraded":"2023-01-24T03:30:25.002243+0000","last_fullsized":"2023-01-24T03:30:25.002243+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:11:08.120197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002638+0000","last_change":"2023-01-24T03:30:20.996489+0000","last_active":"2023-01-24T03:30:25.002638+0000","last_peered":"2023-01-24T03:30:25.002638+0000","last_clean":"2023-01-24T03:30:25.002638+0000","last_became_active":"2023-01-24T03:30:20.994371+0000","last_became_peered":"2023-01-24T03:30:20.994371+0000","last_unstale":"2023-01-24T03:30:25.002638+0000","last_undegraded":"2023-01-24T03:30:25.002638+0000","last_fullsized":"2023-01-24T03:30:25.002638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:52:21.369928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005744+0000","last_change":"2023-01-24T03:30:20.997753+0000","last_active":"2023-01-24T03:30:25.005744+0000","last_peered":"2023-01-24T03:30:25.005744+0000","last_clean":"2023-01-24T03:30:25.005744+0000","last_became_active":"2023-01-24T03:30:20.997475+0000","last_became_peered":"2023-01-24T03:30:20.997475+0000","last_unstale":"2023-01-24T03:30:25.005744+0000","last_undegraded":"2023-01-24T03:30:25.005744+0000","last_fullsized":"2023-01-24T03:30:25.005744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:43:16.199843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.013267+0000","last_change":"2023-01-24T03:30:25.004944+0000","last_active":"2023-01-24T03:30:25.013267+0000","last_peered":"2023-01-24T03:30:25.013267+0000","last_clean":"2023-01-24T03:30:25.013267+0000","last_became_active":"2023-01-24T03:30:25.004590+0000","last_became_peered":"2023-01-24T03:30:25.004590+0000","last_unstale":"2023-01-24T03:30:25.013267+0000","last_undegraded":"2023-01-24T03:30:25.013267+0000","last_fullsized":"2023-01-24T03:30:25.013267+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:04:11.054225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2,0],"acting":[4,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"24'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.380450+0000","last_change":"2023-01-24T03:30:20.607211+0000","last_active":"2023-01-24T03:30:25.380450+0000","last_peered":"2023-01-24T03:30:25.380450+0000","last_clean":"2023-01-24T03:30:25.380450+0000","last_became_active":"2023-01-24T03:30:20.606855+0000","last_became_peered":"2023-01-24T03:30:20.606855+0000","last_unstale":"2023-01-24T03:30:25.380450+0000","last_undegraded":"2023-01-24T03:30:25.380450+0000","last_fullsized":"2023-01-24T03:30:25.380450+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:42:29.300089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991276+0000","last_change":"2023-01-24T03:30:20.993128+0000","last_active":"2023-01-24T03:30:24.991276+0000","last_peered":"2023-01-24T03:30:24.991276+0000","last_clean":"2023-01-24T03:30:24.991276+0000","last_became_active":"2023-01-24T03:30:20.992967+0000","last_became_peered":"2023-01-24T03:30:20.992967+0000","last_unstale":"2023-01-24T03:30:24.991276+0000","last_undegraded":"2023-01-24T03:30:24.991276+0000","last_fullsized":"2023-01-24T03:30:24.991276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:09:59.170218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.008814+0000","last_change":"2023-01-24T03:30:25.006569+0000","last_active":"2023-01-24T03:30:25.008814+0000","last_peered":"2023-01-24T03:30:25.008814+0000","last_clean":"2023-01-24T03:30:25.008814+0000","last_became_active":"2023-01-24T03:30:25.006199+0000","last_became_peered":"2023-01-24T03:30:25.006199+0000","last_unstale":"2023-01-24T03:30:25.008814+0000","last_undegraded":"2023-01-24T03:30:25.008814+0000","last_fullsized":"2023-01-24T03:30:25.008814+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:31:46.839485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,7],"acting":[6,4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.001971+0000","last_change":"2023-01-24T03:30:20.612738+0000","last_active":"2023-01-24T03:30:25.001971+0000","last_peered":"2023-01-24T03:30:25.001971+0000","last_clean":"2023-01-24T03:30:25.001971+0000","last_became_active":"2023-01-24T03:30:20.612375+0000","last_became_peered":"2023-01-24T03:30:20.612375+0000","last_unstale":"2023-01-24T03:30:25.001971+0000","last_undegraded":"2023-01-24T03:30:25.001971+0000","last_fullsized":"2023-01-24T03:30:25.001971+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:17:43.251945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993741+0000","last_change":"2023-01-24T03:30:20.996368+0000","last_active":"2023-01-24T03:30:24.993741+0000","last_peered":"2023-01-24T03:30:24.993741+0000","last_clean":"2023-01-24T03:30:24.993741+0000","last_became_active":"2023-01-24T03:30:20.996051+0000","last_became_peered":"2023-01-24T03:30:20.996051+0000","last_unstale":"2023-01-24T03:30:24.993741+0000","last_undegraded":"2023-01-24T03:30:24.993741+0000","last_fullsized":"2023-01-24T03:30:24.993741+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:53:28.193862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.014921+0000","last_change":"2023-01-24T03:30:25.008528+0000","last_active":"2023-01-24T03:30:25.014921+0000","last_peered":"2023-01-24T03:30:25.014921+0000","last_clean":"2023-01-24T03:30:25.014921+0000","last_became_active":"2023-01-24T03:30:25.008146+0000","last_became_peered":"2023-01-24T03:30:25.008146+0000","last_unstale":"2023-01-24T03:30:25.014921+0000","last_undegraded":"2023-01-24T03:30:25.014921+0000","last_fullsized":"2023-01-24T03:30:25.014921+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:17:11.457359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1,2],"acting":[0,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991341+0000","last_change":"2023-01-24T03:30:20.608932+0000","last_active":"2023-01-24T03:30:24.991341+0000","last_peered":"2023-01-24T03:30:24.991341+0000","last_clean":"2023-01-24T03:30:24.991341+0000","last_became_active":"2023-01-24T03:30:20.608135+0000","last_became_peered":"2023-01-24T03:30:20.608135+0000","last_unstale":"2023-01-24T03:30:24.991341+0000","last_undegraded":"2023-01-24T03:30:24.991341+0000","last_fullsized":"2023-01-24T03:30:24.991341+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:38:30.012279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990753+0000","last_change":"2023-01-24T03:30:20.986490+0000","last_active":"2023-01-24T03:30:24.990753+0000","last_peered":"2023-01-24T03:30:24.990753+0000","last_clean":"2023-01-24T03:30:24.990753+0000","last_became_active":"2023-01-24T03:30:20.986245+0000","last_became_peered":"2023-01-24T03:30:20.986245+0000","last_unstale":"2023-01-24T03:30:24.990753+0000","last_undegraded":"2023-01-24T03:30:24.990753+0000","last_fullsized":"2023-01-24T03:30:24.990753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:38:30.801032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.012186+0000","last_change":"2023-01-24T03:30:25.008221+0000","last_active":"2023-01-24T03:30:25.012186+0000","last_peered":"2023-01-24T03:30:25.012186+0000","last_clean":"2023-01-24T03:30:25.012186+0000","last_became_active":"2023-01-24T03:30:25.007855+0000","last_became_peered":"2023-01-24T03:30:25.007855+0000","last_unstale":"2023-01-24T03:30:25.012186+0000","last_undegraded":"2023-01-24T03:30:25.012186+0000","last_fullsized":"2023-01-24T03:30:25.012186+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:45:15.317067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,1],"acting":[6,4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992445+0000","last_change":"2023-01-24T03:30:20.607385+0000","last_active":"2023-01-24T03:30:24.992445+0000","last_peered":"2023-01-24T03:30:24.992445+0000","last_clean":"2023-01-24T03:30:24.992445+0000","last_became_active":"2023-01-24T03:30:20.607123+0000","last_became_peered":"2023-01-24T03:30:20.607123+0000","last_unstale":"2023-01-24T03:30:24.992445+0000","last_undegraded":"2023-01-24T03:30:24.992445+0000","last_fullsized":"2023-01-24T03:30:24.992445+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T15:24:00.474758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991258+0000","last_change":"2023-01-24T03:30:20.994124+0000","last_active":"2023-01-24T03:30:24.991258+0000","last_peered":"2023-01-24T03:30:24.991258+0000","last_clean":"2023-01-24T03:30:24.991258+0000","last_became_active":"2023-01-24T03:30:20.993876+0000","last_became_peered":"2023-01-24T03:30:20.993876+0000","last_unstale":"2023-01-24T03:30:24.991258+0000","last_undegraded":"2023-01-24T03:30:24.991258+0000","last_fullsized":"2023-01-24T03:30:24.991258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:07:26.252741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.011535+0000","last_change":"2023-01-24T03:30:25.006312+0000","last_active":"2023-01-24T03:30:25.011535+0000","last_peered":"2023-01-24T03:30:25.011535+0000","last_clean":"2023-01-24T03:30:25.011535+0000","last_became_active":"2023-01-24T03:30:25.005946+0000","last_became_peered":"2023-01-24T03:30:25.005946+0000","last_unstale":"2023-01-24T03:30:25.011535+0000","last_undegraded":"2023-01-24T03:30:25.011535+0000","last_fullsized":"2023-01-24T03:30:25.011535+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:35:02.760190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6,0],"acting":[7,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.006326+0000","last_change":"2023-01-24T03:30:20.610825+0000","last_active":"2023-01-24T03:30:25.006326+0000","last_peered":"2023-01-24T03:30:25.006326+0000","last_clean":"2023-01-24T03:30:25.006326+0000","last_became_active":"2023-01-24T03:30:20.610540+0000","last_became_peered":"2023-01-24T03:30:20.610540+0000","last_unstale":"2023-01-24T03:30:25.006326+0000","last_undegraded":"2023-01-24T03:30:25.006326+0000","last_fullsized":"2023-01-24T03:30:25.006326+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:23:15.279283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990818+0000","last_change":"2023-01-24T03:30:20.994605+0000","last_active":"2023-01-24T03:30:24.990818+0000","last_peered":"2023-01-24T03:30:24.990818+0000","last_clean":"2023-01-24T03:30:24.990818+0000","last_became_active":"2023-01-24T03:30:20.994356+0000","last_became_peered":"2023-01-24T03:30:20.994356+0000","last_unstale":"2023-01-24T03:30:24.990818+0000","last_undegraded":"2023-01-24T03:30:24.990818+0000","last_fullsized":"2023-01-24T03:30:24.990818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:06:08.797922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.018595+0000","last_change":"2023-01-24T03:30:25.012301+0000","last_active":"2023-01-24T03:30:25.018595+0000","last_peered":"2023-01-24T03:30:25.018595+0000","last_clean":"2023-01-24T03:30:25.018595+0000","last_became_active":"2023-01-24T03:30:25.011891+0000","last_became_peered":"2023-01-24T03:30:25.011891+0000","last_unstale":"2023-01-24T03:30:25.018595+0000","last_undegraded":"2023-01-24T03:30:25.018595+0000","last_fullsized":"2023-01-24T03:30:25.018595+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:15:46.012675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,5],"acting":[3,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"24'3","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.986137+0000","last_change":"2023-01-24T03:30:20.609828+0000","last_active":"2023-01-24T03:30:25.986137+0000","last_peered":"2023-01-24T03:30:25.986137+0000","last_clean":"2023-01-24T03:30:25.986137+0000","last_became_active":"2023-01-24T03:30:20.609492+0000","last_became_peered":"2023-01-24T03:30:20.609492+0000","last_unstale":"2023-01-24T03:30:25.986137+0000","last_undegraded":"2023-01-24T03:30:25.986137+0000","last_fullsized":"2023-01-24T03:30:25.986137+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:22:53.430711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002553+0000","last_change":"2023-01-24T03:30:20.999747+0000","last_active":"2023-01-24T03:30:25.002553+0000","last_peered":"2023-01-24T03:30:25.002553+0000","last_clean":"2023-01-24T03:30:25.002553+0000","last_became_active":"2023-01-24T03:30:20.999003+0000","last_became_peered":"2023-01-24T03:30:20.999003+0000","last_unstale":"2023-01-24T03:30:25.002553+0000","last_undegraded":"2023-01-24T03:30:25.002553+0000","last_fullsized":"2023-01-24T03:30:25.002553+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:19:22.205251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016573+0000","last_change":"2023-01-24T03:30:25.010659+0000","last_active":"2023-01-24T03:30:25.016573+0000","last_peered":"2023-01-24T03:30:25.016573+0000","last_clean":"2023-01-24T03:30:25.016573+0000","last_became_active":"2023-01-24T03:30:25.010237+0000","last_became_peered":"2023-01-24T03:30:25.010237+0000","last_unstale":"2023-01-24T03:30:25.016573+0000","last_undegraded":"2023-01-24T03:30:25.016573+0000","last_fullsized":"2023-01-24T03:30:25.016573+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:01:53.501070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3,7],"acting":[1,5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"24'9","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:34.376852+0000","last_change":"2023-01-24T03:30:20.610682+0000","last_active":"2023-01-24T03:30:34.376852+0000","last_peered":"2023-01-24T03:30:34.376852+0000","last_clean":"2023-01-24T03:30:34.376852+0000","last_became_active":"2023-01-24T03:30:20.610463+0000","last_became_peered":"2023-01-24T03:30:20.610463+0000","last_unstale":"2023-01-24T03:30:34.376852+0000","last_undegraded":"2023-01-24T03:30:34.376852+0000","last_fullsized":"2023-01-24T03:30:34.376852+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:25:00.447596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3713,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992845+0000","last_change":"2023-01-24T03:30:20.987326+0000","last_active":"2023-01-24T03:30:24.992845+0000","last_peered":"2023-01-24T03:30:24.992845+0000","last_clean":"2023-01-24T03:30:24.992845+0000","last_became_active":"2023-01-24T03:30:20.987042+0000","last_became_peered":"2023-01-24T03:30:20.987042+0000","last_unstale":"2023-01-24T03:30:24.992845+0000","last_undegraded":"2023-01-24T03:30:24.992845+0000","last_fullsized":"2023-01-24T03:30:24.992845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:57:43.257624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.020022+0000","last_change":"2023-01-24T03:30:25.012004+0000","last_active":"2023-01-24T03:30:25.020022+0000","last_peered":"2023-01-24T03:30:25.020022+0000","last_clean":"2023-01-24T03:30:25.020022+0000","last_became_active":"2023-01-24T03:30:25.007410+0000","last_became_peered":"2023-01-24T03:30:25.007410+0000","last_unstale":"2023-01-24T03:30:25.020022+0000","last_undegraded":"2023-01-24T03:30:25.020022+0000","last_fullsized":"2023-01-24T03:30:25.020022+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:34:21.681011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3,0],"acting":[5,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"24'2","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985201+0000","last_change":"2023-01-24T03:30:20.613026+0000","last_active":"2023-01-24T03:30:25.985201+0000","last_peered":"2023-01-24T03:30:25.985201+0000","last_clean":"2023-01-24T03:30:25.985201+0000","last_became_active":"2023-01-24T03:30:20.612605+0000","last_became_peered":"2023-01-24T03:30:20.612605+0000","last_unstale":"2023-01-24T03:30:25.985201+0000","last_undegraded":"2023-01-24T03:30:25.985201+0000","last_fullsized":"2023-01-24T03:30:25.985201+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:45:08.016740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991113+0000","last_change":"2023-01-24T03:30:20.991929+0000","last_active":"2023-01-24T03:30:24.991113+0000","last_peered":"2023-01-24T03:30:24.991113+0000","last_clean":"2023-01-24T03:30:24.991113+0000","last_became_active":"2023-01-24T03:30:20.991693+0000","last_became_peered":"2023-01-24T03:30:20.991693+0000","last_unstale":"2023-01-24T03:30:24.991113+0000","last_undegraded":"2023-01-24T03:30:24.991113+0000","last_fullsized":"2023-01-24T03:30:24.991113+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:55:51.637568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.018472+0000","last_change":"2023-01-24T03:30:25.010572+0000","last_active":"2023-01-24T03:30:25.018472+0000","last_peered":"2023-01-24T03:30:25.018472+0000","last_clean":"2023-01-24T03:30:25.018472+0000","last_became_active":"2023-01-24T03:30:25.010193+0000","last_became_peered":"2023-01-24T03:30:25.010193+0000","last_unstale":"2023-01-24T03:30:25.018472+0000","last_undegraded":"2023-01-24T03:30:25.018472+0000","last_fullsized":"2023-01-24T03:30:25.018472+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:25:39.676179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,1],"acting":[3,7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993579+0000","last_change":"2023-01-24T03:30:20.612047+0000","last_active":"2023-01-24T03:30:24.993579+0000","last_peered":"2023-01-24T03:30:24.993579+0000","last_clean":"2023-01-24T03:30:24.993579+0000","last_became_active":"2023-01-24T03:30:20.611798+0000","last_became_peered":"2023-01-24T03:30:20.611798+0000","last_unstale":"2023-01-24T03:30:24.993579+0000","last_undegraded":"2023-01-24T03:30:24.993579+0000","last_fullsized":"2023-01-24T03:30:24.993579+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:01:41.731689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005194+0000","last_change":"2023-01-24T03:30:20.996432+0000","last_active":"2023-01-24T03:30:25.005194+0000","last_peered":"2023-01-24T03:30:25.005194+0000","last_clean":"2023-01-24T03:30:25.005194+0000","last_became_active":"2023-01-24T03:30:20.996173+0000","last_became_peered":"2023-01-24T03:30:20.996173+0000","last_unstale":"2023-01-24T03:30:25.005194+0000","last_undegraded":"2023-01-24T03:30:25.005194+0000","last_fullsized":"2023-01-24T03:30:25.005194+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:13:31.596587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.018507+0000","last_change":"2023-01-24T03:30:25.008479+0000","last_active":"2023-01-24T03:30:25.018507+0000","last_peered":"2023-01-24T03:30:25.018507+0000","last_clean":"2023-01-24T03:30:25.018507+0000","last_became_active":"2023-01-24T03:30:25.007858+0000","last_became_peered":"2023-01-24T03:30:25.007858+0000","last_unstale":"2023-01-24T03:30:25.018507+0000","last_undegraded":"2023-01-24T03:30:25.018507+0000","last_fullsized":"2023-01-24T03:30:25.018507+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:10:38.425874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,0],"acting":[3,1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996134+0000","last_change":"2023-01-24T03:30:20.612848+0000","last_active":"2023-01-24T03:30:24.996134+0000","last_peered":"2023-01-24T03:30:24.996134+0000","last_clean":"2023-01-24T03:30:24.996134+0000","last_became_active":"2023-01-24T03:30:20.612600+0000","last_became_peered":"2023-01-24T03:30:20.612600+0000","last_unstale":"2023-01-24T03:30:24.996134+0000","last_undegraded":"2023-01-24T03:30:24.996134+0000","last_fullsized":"2023-01-24T03:30:24.996134+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:46:45.634472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993883+0000","last_change":"2023-01-24T03:30:20.991133+0000","last_active":"2023-01-24T03:30:24.993883+0000","last_peered":"2023-01-24T03:30:24.993883+0000","last_clean":"2023-01-24T03:30:24.993883+0000","last_became_active":"2023-01-24T03:30:20.990856+0000","last_became_peered":"2023-01-24T03:30:20.990856+0000","last_unstale":"2023-01-24T03:30:24.993883+0000","last_undegraded":"2023-01-24T03:30:24.993883+0000","last_fullsized":"2023-01-24T03:30:24.993883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:27:32.163224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.022452+0000","last_change":"2023-01-24T03:30:25.017038+0000","last_active":"2023-01-24T03:30:25.022452+0000","last_peered":"2023-01-24T03:30:25.022452+0000","last_clean":"2023-01-24T03:30:25.022452+0000","last_became_active":"2023-01-24T03:30:25.016684+0000","last_became_peered":"2023-01-24T03:30:25.016684+0000","last_unstale":"2023-01-24T03:30:25.022452+0000","last_undegraded":"2023-01-24T03:30:25.022452+0000","last_fullsized":"2023-01-24T03:30:25.022452+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:57:30.028834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1,6],"acting":[5,3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"24'8","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.988793+0000","last_change":"2023-01-24T03:30:20.604617+0000","last_active":"2023-01-24T03:30:25.988793+0000","last_peered":"2023-01-24T03:30:25.988793+0000","last_clean":"2023-01-24T03:30:25.988793+0000","last_became_active":"2023-01-24T03:30:20.604445+0000","last_became_peered":"2023-01-24T03:30:20.604445+0000","last_unstale":"2023-01-24T03:30:25.988793+0000","last_undegraded":"2023-01-24T03:30:25.988793+0000","last_fullsized":"2023-01-24T03:30:25.988793+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:06:51.460895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993017+0000","last_change":"2023-01-24T03:30:20.995872+0000","last_active":"2023-01-24T03:30:24.993017+0000","last_peered":"2023-01-24T03:30:24.993017+0000","last_clean":"2023-01-24T03:30:24.993017+0000","last_became_active":"2023-01-24T03:30:20.995599+0000","last_became_peered":"2023-01-24T03:30:20.995599+0000","last_unstale":"2023-01-24T03:30:24.993017+0000","last_undegraded":"2023-01-24T03:30:24.993017+0000","last_fullsized":"2023-01-24T03:30:24.993017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:53:54.356899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.020130+0000","last_change":"2023-01-24T03:30:25.016101+0000","last_active":"2023-01-24T03:30:25.020130+0000","last_peered":"2023-01-24T03:30:25.020130+0000","last_clean":"2023-01-24T03:30:25.020130+0000","last_became_active":"2023-01-24T03:30:25.015705+0000","last_became_peered":"2023-01-24T03:30:25.015705+0000","last_unstale":"2023-01-24T03:30:25.020130+0000","last_undegraded":"2023-01-24T03:30:25.020130+0000","last_fullsized":"2023-01-24T03:30:25.020130+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:54:37.798191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3,4],"acting":[7,6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"24'1","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.446876+0000","last_change":"2023-01-24T03:30:20.610632+0000","last_active":"2023-01-24T03:30:25.446876+0000","last_peered":"2023-01-24T03:30:25.446876+0000","last_clean":"2023-01-24T03:30:25.446876+0000","last_became_active":"2023-01-24T03:30:20.610367+0000","last_became_peered":"2023-01-24T03:30:20.610367+0000","last_unstale":"2023-01-24T03:30:25.446876+0000","last_undegraded":"2023-01-24T03:30:25.446876+0000","last_fullsized":"2023-01-24T03:30:25.446876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:26:05.938178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.004623+0000","last_change":"2023-01-24T03:30:20.996493+0000","last_active":"2023-01-24T03:30:25.004623+0000","last_peered":"2023-01-24T03:30:25.004623+0000","last_clean":"2023-01-24T03:30:25.004623+0000","last_became_active":"2023-01-24T03:30:20.996229+0000","last_became_peered":"2023-01-24T03:30:20.996229+0000","last_unstale":"2023-01-24T03:30:25.004623+0000","last_undegraded":"2023-01-24T03:30:25.004623+0000","last_fullsized":"2023-01-24T03:30:25.004623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:23:53.642522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016195+0000","last_change":"2023-01-24T03:30:25.009215+0000","last_active":"2023-01-24T03:30:25.016195+0000","last_peered":"2023-01-24T03:30:25.016195+0000","last_clean":"2023-01-24T03:30:25.016195+0000","last_became_active":"2023-01-24T03:30:25.008213+0000","last_became_peered":"2023-01-24T03:30:25.008213+0000","last_unstale":"2023-01-24T03:30:25.016195+0000","last_undegraded":"2023-01-24T03:30:25.016195+0000","last_fullsized":"2023-01-24T03:30:25.016195+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:48:37.517172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,5],"acting":[2,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"24'2","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985223+0000","last_change":"2023-01-24T03:30:20.612158+0000","last_active":"2023-01-24T03:30:25.985223+0000","last_peered":"2023-01-24T03:30:25.985223+0000","last_clean":"2023-01-24T03:30:25.985223+0000","last_became_active":"2023-01-24T03:30:20.611931+0000","last_became_peered":"2023-01-24T03:30:20.611931+0000","last_unstale":"2023-01-24T03:30:25.985223+0000","last_undegraded":"2023-01-24T03:30:25.985223+0000","last_fullsized":"2023-01-24T03:30:25.985223+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:30:37.532390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992500+0000","last_change":"2023-01-24T03:30:20.991582+0000","last_active":"2023-01-24T03:30:24.992500+0000","last_peered":"2023-01-24T03:30:24.992500+0000","last_clean":"2023-01-24T03:30:24.992500+0000","last_became_active":"2023-01-24T03:30:20.991335+0000","last_became_peered":"2023-01-24T03:30:20.991335+0000","last_unstale":"2023-01-24T03:30:24.992500+0000","last_undegraded":"2023-01-24T03:30:24.992500+0000","last_fullsized":"2023-01-24T03:30:24.992500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:45:20.786712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016224+0000","last_change":"2023-01-24T03:30:25.008058+0000","last_active":"2023-01-24T03:30:25.016224+0000","last_peered":"2023-01-24T03:30:25.016224+0000","last_clean":"2023-01-24T03:30:25.016224+0000","last_became_active":"2023-01-24T03:30:25.007711+0000","last_became_peered":"2023-01-24T03:30:25.007711+0000","last_unstale":"2023-01-24T03:30:25.016224+0000","last_undegraded":"2023-01-24T03:30:25.016224+0000","last_fullsized":"2023-01-24T03:30:25.016224+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:26:56.270674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5,6],"acting":[2,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"24'1","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.447163+0000","last_change":"2023-01-24T03:30:20.610495+0000","last_active":"2023-01-24T03:30:25.447163+0000","last_peered":"2023-01-24T03:30:25.447163+0000","last_clean":"2023-01-24T03:30:25.447163+0000","last_became_active":"2023-01-24T03:30:20.610035+0000","last_became_peered":"2023-01-24T03:30:20.610035+0000","last_unstale":"2023-01-24T03:30:25.447163+0000","last_undegraded":"2023-01-24T03:30:25.447163+0000","last_fullsized":"2023-01-24T03:30:25.447163+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:25:30.944275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.989631+0000","last_change":"2023-01-24T03:30:20.994362+0000","last_active":"2023-01-24T03:30:24.989631+0000","last_peered":"2023-01-24T03:30:24.989631+0000","last_clean":"2023-01-24T03:30:24.989631+0000","last_became_active":"2023-01-24T03:30:20.994101+0000","last_became_peered":"2023-01-24T03:30:20.994101+0000","last_unstale":"2023-01-24T03:30:24.989631+0000","last_undegraded":"2023-01-24T03:30:24.989631+0000","last_fullsized":"2023-01-24T03:30:24.989631+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:55:12.837646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.015999+0000","last_change":"2023-01-24T03:30:25.006995+0000","last_active":"2023-01-24T03:30:25.015999+0000","last_peered":"2023-01-24T03:30:25.015999+0000","last_clean":"2023-01-24T03:30:25.015999+0000","last_became_active":"2023-01-24T03:30:25.006473+0000","last_became_peered":"2023-01-24T03:30:25.006473+0000","last_unstale":"2023-01-24T03:30:25.015999+0000","last_undegraded":"2023-01-24T03:30:25.015999+0000","last_fullsized":"2023-01-24T03:30:25.015999+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:48:16.320162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3,6],"acting":[2,4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005342+0000","last_change":"2023-01-24T03:30:20.606526+0000","last_active":"2023-01-24T03:30:25.005342+0000","last_peered":"2023-01-24T03:30:25.005342+0000","last_clean":"2023-01-24T03:30:25.005342+0000","last_became_active":"2023-01-24T03:30:20.606301+0000","last_became_peered":"2023-01-24T03:30:20.606301+0000","last_unstale":"2023-01-24T03:30:25.005342+0000","last_undegraded":"2023-01-24T03:30:25.005342+0000","last_fullsized":"2023-01-24T03:30:25.005342+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:43:28.702475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992428+0000","last_change":"2023-01-24T03:30:20.993379+0000","last_active":"2023-01-24T03:30:24.992428+0000","last_peered":"2023-01-24T03:30:24.992428+0000","last_clean":"2023-01-24T03:30:24.992428+0000","last_became_active":"2023-01-24T03:30:20.993151+0000","last_became_peered":"2023-01-24T03:30:20.993151+0000","last_unstale":"2023-01-24T03:30:24.992428+0000","last_undegraded":"2023-01-24T03:30:24.992428+0000","last_fullsized":"2023-01-24T03:30:24.992428+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:00:45.845082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.020912+0000","last_change":"2023-01-24T03:30:25.013701+0000","last_active":"2023-01-24T03:30:25.020912+0000","last_peered":"2023-01-24T03:30:25.020912+0000","last_clean":"2023-01-24T03:30:25.020912+0000","last_became_active":"2023-01-24T03:30:25.013290+0000","last_became_peered":"2023-01-24T03:30:25.013290+0000","last_unstale":"2023-01-24T03:30:25.020912+0000","last_undegraded":"2023-01-24T03:30:25.020912+0000","last_fullsized":"2023-01-24T03:30:25.020912+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:32:17.340583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5,6],"acting":[2,7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"24'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985860+0000","last_change":"2023-01-24T03:30:20.609656+0000","last_active":"2023-01-24T03:30:25.985860+0000","last_peered":"2023-01-24T03:30:25.985860+0000","last_clean":"2023-01-24T03:30:25.985860+0000","last_became_active":"2023-01-24T03:30:20.609374+0000","last_became_peered":"2023-01-24T03:30:20.609374+0000","last_unstale":"2023-01-24T03:30:25.985860+0000","last_undegraded":"2023-01-24T03:30:25.985860+0000","last_fullsized":"2023-01-24T03:30:25.985860+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:58:12.653746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992726+0000","last_change":"2023-01-24T03:30:20.986089+0000","last_active":"2023-01-24T03:30:24.992726+0000","last_peered":"2023-01-24T03:30:24.992726+0000","last_clean":"2023-01-24T03:30:24.992726+0000","last_became_active":"2023-01-24T03:30:20.985857+0000","last_became_peered":"2023-01-24T03:30:20.985857+0000","last_unstale":"2023-01-24T03:30:24.992726+0000","last_undegraded":"2023-01-24T03:30:24.992726+0000","last_fullsized":"2023-01-24T03:30:24.992726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:00:57.338328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016603+0000","last_change":"2023-01-24T03:30:25.009962+0000","last_active":"2023-01-24T03:30:25.016603+0000","last_peered":"2023-01-24T03:30:25.016603+0000","last_clean":"2023-01-24T03:30:25.016603+0000","last_became_active":"2023-01-24T03:30:25.009623+0000","last_became_peered":"2023-01-24T03:30:25.009623+0000","last_unstale":"2023-01-24T03:30:25.016603+0000","last_undegraded":"2023-01-24T03:30:25.016603+0000","last_fullsized":"2023-01-24T03:30:25.016603+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:24:42.958531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0,6],"acting":[1,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.989699+0000","last_change":"2023-01-24T03:30:20.608114+0000","last_active":"2023-01-24T03:30:24.989699+0000","last_peered":"2023-01-24T03:30:24.989699+0000","last_clean":"2023-01-24T03:30:24.989699+0000","last_became_active":"2023-01-24T03:30:20.607816+0000","last_became_peered":"2023-01-24T03:30:20.607816+0000","last_unstale":"2023-01-24T03:30:24.989699+0000","last_undegraded":"2023-01-24T03:30:24.989699+0000","last_fullsized":"2023-01-24T03:30:24.989699+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:40:00.784496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992908+0000","last_change":"2023-01-24T03:30:20.982589+0000","last_active":"2023-01-24T03:30:24.992908+0000","last_peered":"2023-01-24T03:30:24.992908+0000","last_clean":"2023-01-24T03:30:24.992908+0000","last_became_active":"2023-01-24T03:30:20.979779+0000","last_became_peered":"2023-01-24T03:30:20.979779+0000","last_unstale":"2023-01-24T03:30:24.992908+0000","last_undegraded":"2023-01-24T03:30:24.992908+0000","last_fullsized":"2023-01-24T03:30:24.992908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:01:18.491363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.019938+0000","last_change":"2023-01-24T03:30:25.013327+0000","last_active":"2023-01-24T03:30:25.019938+0000","last_peered":"2023-01-24T03:30:25.019938+0000","last_clean":"2023-01-24T03:30:25.019938+0000","last_became_active":"2023-01-24T03:30:25.011303+0000","last_became_peered":"2023-01-24T03:30:25.011303+0000","last_unstale":"2023-01-24T03:30:25.019938+0000","last_undegraded":"2023-01-24T03:30:25.019938+0000","last_fullsized":"2023-01-24T03:30:25.019938+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:14:43.169287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6,2],"acting":[5,4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992390+0000","last_change":"2023-01-24T03:30:20.602369+0000","last_active":"2023-01-24T03:30:24.992390+0000","last_peered":"2023-01-24T03:30:24.992390+0000","last_clean":"2023-01-24T03:30:24.992390+0000","last_became_active":"2023-01-24T03:30:20.601996+0000","last_became_peered":"2023-01-24T03:30:20.601996+0000","last_unstale":"2023-01-24T03:30:24.992390+0000","last_undegraded":"2023-01-24T03:30:24.992390+0000","last_fullsized":"2023-01-24T03:30:24.992390+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:33:55.328080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993109+0000","last_change":"2023-01-24T03:30:20.988738+0000","last_active":"2023-01-24T03:30:24.993109+0000","last_peered":"2023-01-24T03:30:24.993109+0000","last_clean":"2023-01-24T03:30:24.993109+0000","last_became_active":"2023-01-24T03:30:20.987190+0000","last_became_peered":"2023-01-24T03:30:20.987190+0000","last_unstale":"2023-01-24T03:30:24.993109+0000","last_undegraded":"2023-01-24T03:30:24.993109+0000","last_fullsized":"2023-01-24T03:30:24.993109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:32:05.689674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.013303+0000","last_change":"2023-01-24T03:30:25.008401+0000","last_active":"2023-01-24T03:30:25.013303+0000","last_peered":"2023-01-24T03:30:25.013303+0000","last_clean":"2023-01-24T03:30:25.013303+0000","last_became_active":"2023-01-24T03:30:25.007973+0000","last_became_peered":"2023-01-24T03:30:25.007973+0000","last_unstale":"2023-01-24T03:30:25.013303+0000","last_undegraded":"2023-01-24T03:30:25.013303+0000","last_fullsized":"2023-01-24T03:30:25.013303+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:36:22.339295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,2],"acting":[4,5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991180+0000","last_change":"2023-01-24T03:30:20.608145+0000","last_active":"2023-01-24T03:30:24.991180+0000","last_peered":"2023-01-24T03:30:24.991180+0000","last_clean":"2023-01-24T03:30:24.991180+0000","last_became_active":"2023-01-24T03:30:20.607165+0000","last_became_peered":"2023-01-24T03:30:20.607165+0000","last_unstale":"2023-01-24T03:30:24.991180+0000","last_undegraded":"2023-01-24T03:30:24.991180+0000","last_fullsized":"2023-01-24T03:30:24.991180+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:58:56.490482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002416+0000","last_change":"2023-01-24T03:30:20.998777+0000","last_active":"2023-01-24T03:30:25.002416+0000","last_peered":"2023-01-24T03:30:25.002416+0000","last_clean":"2023-01-24T03:30:25.002416+0000","last_became_active":"2023-01-24T03:30:20.998519+0000","last_became_peered":"2023-01-24T03:30:20.998519+0000","last_unstale":"2023-01-24T03:30:25.002416+0000","last_undegraded":"2023-01-24T03:30:25.002416+0000","last_fullsized":"2023-01-24T03:30:25.002416+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:55:55.320556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.012455+0000","last_change":"2023-01-24T03:30:25.008154+0000","last_active":"2023-01-24T03:30:25.012455+0000","last_peered":"2023-01-24T03:30:25.012455+0000","last_clean":"2023-01-24T03:30:25.012455+0000","last_became_active":"2023-01-24T03:30:25.007797+0000","last_became_peered":"2023-01-24T03:30:25.007797+0000","last_unstale":"2023-01-24T03:30:25.012455+0000","last_undegraded":"2023-01-24T03:30:25.012455+0000","last_fullsized":"2023-01-24T03:30:25.012455+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:29:05.153733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7,0],"acting":[4,3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"24'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985835+0000","last_change":"2023-01-24T03:30:20.610690+0000","last_active":"2023-01-24T03:30:25.985835+0000","last_peered":"2023-01-24T03:30:25.985835+0000","last_clean":"2023-01-24T03:30:25.985835+0000","last_became_active":"2023-01-24T03:30:20.610409+0000","last_became_peered":"2023-01-24T03:30:20.610409+0000","last_unstale":"2023-01-24T03:30:25.985835+0000","last_undegraded":"2023-01-24T03:30:25.985835+0000","last_fullsized":"2023-01-24T03:30:25.985835+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:43:37.740181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993033+0000","last_change":"2023-01-24T03:30:16.085885+0000","last_active":"2023-01-24T03:30:24.993033+0000","last_peered":"2023-01-24T03:30:24.993033+0000","last_clean":"2023-01-24T03:30:24.993033+0000","last_became_active":"2023-01-24T03:30:16.085525+0000","last_became_peered":"2023-01-24T03:30:16.085525+0000","last_unstale":"2023-01-24T03:30:24.993033+0000","last_undegraded":"2023-01-24T03:30:24.993033+0000","last_fullsized":"2023-01-24T03:30:24.993033+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:57:36.499644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991099+0000","last_change":"2023-01-24T03:30:20.996370+0000","last_active":"2023-01-24T03:30:24.991099+0000","last_peered":"2023-01-24T03:30:24.991099+0000","last_clean":"2023-01-24T03:30:24.991099+0000","last_became_active":"2023-01-24T03:30:20.996122+0000","last_became_peered":"2023-01-24T03:30:20.996122+0000","last_unstale":"2023-01-24T03:30:24.991099+0000","last_undegraded":"2023-01-24T03:30:24.991099+0000","last_fullsized":"2023-01-24T03:30:24.991099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:13:06.913474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.014170+0000","last_change":"2023-01-24T03:30:25.007262+0000","last_active":"2023-01-24T03:30:25.014170+0000","last_peered":"2023-01-24T03:30:25.014170+0000","last_clean":"2023-01-24T03:30:25.014170+0000","last_became_active":"2023-01-24T03:30:25.006862+0000","last_became_peered":"2023-01-24T03:30:25.006862+0000","last_unstale":"2023-01-24T03:30:25.014170+0000","last_undegraded":"2023-01-24T03:30:25.014170+0000","last_fullsized":"2023-01-24T03:30:25.014170+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:42:05.268862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,0],"acting":[3,1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991165+0000","last_change":"2023-01-24T03:30:16.057240+0000","last_active":"2023-01-24T03:30:24.991165+0000","last_peered":"2023-01-24T03:30:24.991165+0000","last_clean":"2023-01-24T03:30:24.991165+0000","last_became_active":"2023-01-24T03:30:16.056941+0000","last_became_peered":"2023-01-24T03:30:16.056941+0000","last_unstale":"2023-01-24T03:30:24.991165+0000","last_undegraded":"2023-01-24T03:30:24.991165+0000","last_fullsized":"2023-01-24T03:30:24.991165+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:22:26.168527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"24'1","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.447279+0000","last_change":"2023-01-24T03:30:20.606130+0000","last_active":"2023-01-24T03:30:25.447279+0000","last_peered":"2023-01-24T03:30:25.447279+0000","last_clean":"2023-01-24T03:30:25.447279+0000","last_became_active":"2023-01-24T03:30:20.605899+0000","last_became_peered":"2023-01-24T03:30:20.605899+0000","last_unstale":"2023-01-24T03:30:25.447279+0000","last_undegraded":"2023-01-24T03:30:25.447279+0000","last_fullsized":"2023-01-24T03:30:25.447279+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:30:47.956569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992411+0000","last_change":"2023-01-24T03:30:20.988906+0000","last_active":"2023-01-24T03:30:24.992411+0000","last_peered":"2023-01-24T03:30:24.992411+0000","last_clean":"2023-01-24T03:30:24.992411+0000","last_became_active":"2023-01-24T03:30:20.988688+0000","last_became_peered":"2023-01-24T03:30:20.988688+0000","last_unstale":"2023-01-24T03:30:24.992411+0000","last_undegraded":"2023-01-24T03:30:24.992411+0000","last_fullsized":"2023-01-24T03:30:24.992411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:25:43.365167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.019966+0000","last_change":"2023-01-24T03:30:25.013834+0000","last_active":"2023-01-24T03:30:25.019966+0000","last_peered":"2023-01-24T03:30:25.019966+0000","last_clean":"2023-01-24T03:30:25.019966+0000","last_became_active":"2023-01-24T03:30:25.011774+0000","last_became_peered":"2023-01-24T03:30:25.011774+0000","last_unstale":"2023-01-24T03:30:25.019966+0000","last_undegraded":"2023-01-24T03:30:25.019966+0000","last_fullsized":"2023-01-24T03:30:25.019966+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:02:50.158049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0,4],"acting":[5,1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002878+0000","last_change":"2023-01-24T03:30:16.051822+0000","last_active":"2023-01-24T03:30:25.002878+0000","last_peered":"2023-01-24T03:30:25.002878+0000","last_clean":"2023-01-24T03:30:25.002878+0000","last_became_active":"2023-01-24T03:30:16.051525+0000","last_became_peered":"2023-01-24T03:30:16.051525+0000","last_unstale":"2023-01-24T03:30:25.002878+0000","last_undegraded":"2023-01-24T03:30:25.002878+0000","last_fullsized":"2023-01-24T03:30:25.002878+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:37:30.802424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"16'117","reported_seq":166,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996317+0000","last_change":"2023-01-24T03:30:16.054907+0000","last_active":"2023-01-24T03:30:24.996317+0000","last_peered":"2023-01-24T03:30:24.996317+0000","last_clean":"2023-01-24T03:30:24.996317+0000","last_became_active":"2023-01-24T03:30:16.054071+0000","last_became_peered":"2023-01-24T03:30:16.054071+0000","last_unstale":"2023-01-24T03:30:24.996317+0000","last_undegraded":"2023-01-24T03:30:24.996317+0000","last_fullsized":"2023-01-24T03:30:24.996317+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:31:26.790766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992331+0000","last_change":"2023-01-24T03:30:20.607914+0000","last_active":"2023-01-24T03:30:24.992331+0000","last_peered":"2023-01-24T03:30:24.992331+0000","last_clean":"2023-01-24T03:30:24.992331+0000","last_became_active":"2023-01-24T03:30:20.607663+0000","last_became_peered":"2023-01-24T03:30:20.607663+0000","last_unstale":"2023-01-24T03:30:24.992331+0000","last_undegraded":"2023-01-24T03:30:24.992331+0000","last_fullsized":"2023-01-24T03:30:24.992331+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:32:24.271747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993825+0000","last_change":"2023-01-24T03:30:20.998105+0000","last_active":"2023-01-24T03:30:24.993825+0000","last_peered":"2023-01-24T03:30:24.993825+0000","last_clean":"2023-01-24T03:30:24.993825+0000","last_became_active":"2023-01-24T03:30:20.997871+0000","last_became_peered":"2023-01-24T03:30:20.997871+0000","last_unstale":"2023-01-24T03:30:24.993825+0000","last_undegraded":"2023-01-24T03:30:24.993825+0000","last_fullsized":"2023-01-24T03:30:24.993825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:47:42.740428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016190+0000","last_change":"2023-01-24T03:30:25.008079+0000","last_active":"2023-01-24T03:30:25.016190+0000","last_peered":"2023-01-24T03:30:25.016190+0000","last_clean":"2023-01-24T03:30:25.016190+0000","last_became_active":"2023-01-24T03:30:25.007596+0000","last_became_peered":"2023-01-24T03:30:25.007596+0000","last_unstale":"2023-01-24T03:30:25.016190+0000","last_undegraded":"2023-01-24T03:30:25.016190+0000","last_fullsized":"2023-01-24T03:30:25.016190+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:30:57.179885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1,7],"acting":[2,5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996872+0000","last_change":"2023-01-24T03:30:16.272445+0000","last_active":"2023-01-24T03:30:24.996872+0000","last_peered":"2023-01-24T03:30:24.996872+0000","last_clean":"2023-01-24T03:30:24.996872+0000","last_became_active":"2023-01-24T03:30:16.272184+0000","last_became_peered":"2023-01-24T03:30:16.272184+0000","last_unstale":"2023-01-24T03:30:24.996872+0000","last_undegraded":"2023-01-24T03:30:24.996872+0000","last_fullsized":"2023-01-24T03:30:24.996872+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:17:05.555068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002328+0000","last_change":"2023-01-24T03:30:20.612832+0000","last_active":"2023-01-24T03:30:25.002328+0000","last_peered":"2023-01-24T03:30:25.002328+0000","last_clean":"2023-01-24T03:30:25.002328+0000","last_became_active":"2023-01-24T03:30:20.612569+0000","last_became_peered":"2023-01-24T03:30:20.612569+0000","last_unstale":"2023-01-24T03:30:25.002328+0000","last_undegraded":"2023-01-24T03:30:25.002328+0000","last_fullsized":"2023-01-24T03:30:25.002328+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:37:40.534015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002330+0000","last_change":"2023-01-24T03:30:20.998171+0000","last_active":"2023-01-24T03:30:25.002330+0000","last_peered":"2023-01-24T03:30:25.002330+0000","last_clean":"2023-01-24T03:30:25.002330+0000","last_became_active":"2023-01-24T03:30:20.994887+0000","last_became_peered":"2023-01-24T03:30:20.994887+0000","last_unstale":"2023-01-24T03:30:25.002330+0000","last_undegraded":"2023-01-24T03:30:25.002330+0000","last_fullsized":"2023-01-24T03:30:25.002330+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:57:40.909930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.020081+0000","last_change":"2023-01-24T03:30:25.014948+0000","last_active":"2023-01-24T03:30:25.020081+0000","last_peered":"2023-01-24T03:30:25.020081+0000","last_clean":"2023-01-24T03:30:25.020081+0000","last_became_active":"2023-01-24T03:30:25.014572+0000","last_became_peered":"2023-01-24T03:30:25.014572+0000","last_unstale":"2023-01-24T03:30:25.020081+0000","last_undegraded":"2023-01-24T03:30:25.020081+0000","last_fullsized":"2023-01-24T03:30:25.020081+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:59:13.108074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5,3],"acting":[7,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996112+0000","last_change":"2023-01-24T03:30:16.272574+0000","last_active":"2023-01-24T03:30:24.996112+0000","last_peered":"2023-01-24T03:30:24.996112+0000","last_clean":"2023-01-24T03:30:24.996112+0000","last_became_active":"2023-01-24T03:30:16.272252+0000","last_became_peered":"2023-01-24T03:30:16.272252+0000","last_unstale":"2023-01-24T03:30:24.996112+0000","last_undegraded":"2023-01-24T03:30:24.996112+0000","last_fullsized":"2023-01-24T03:30:24.996112+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:12:08.393585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991181+0000","last_change":"2023-01-24T03:30:20.606878+0000","last_active":"2023-01-24T03:30:24.991181+0000","last_peered":"2023-01-24T03:30:24.991181+0000","last_clean":"2023-01-24T03:30:24.991181+0000","last_became_active":"2023-01-24T03:30:20.606442+0000","last_became_peered":"2023-01-24T03:30:20.606442+0000","last_unstale":"2023-01-24T03:30:24.991181+0000","last_undegraded":"2023-01-24T03:30:24.991181+0000","last_fullsized":"2023-01-24T03:30:24.991181+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:26:16.916080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990646+0000","last_change":"2023-01-24T03:30:20.990840+0000","last_active":"2023-01-24T03:30:24.990646+0000","last_peered":"2023-01-24T03:30:24.990646+0000","last_clean":"2023-01-24T03:30:24.990646+0000","last_became_active":"2023-01-24T03:30:20.990615+0000","last_became_peered":"2023-01-24T03:30:20.990615+0000","last_unstale":"2023-01-24T03:30:24.990646+0000","last_undegraded":"2023-01-24T03:30:24.990646+0000","last_fullsized":"2023-01-24T03:30:24.990646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T15:06:37.670127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.013106+0000","last_change":"2023-01-24T03:30:25.006375+0000","last_active":"2023-01-24T03:30:25.013106+0000","last_peered":"2023-01-24T03:30:25.013106+0000","last_clean":"2023-01-24T03:30:25.013106+0000","last_became_active":"2023-01-24T03:30:25.006027+0000","last_became_peered":"2023-01-24T03:30:25.006027+0000","last_unstale":"2023-01-24T03:30:25.013106+0000","last_undegraded":"2023-01-24T03:30:25.013106+0000","last_fullsized":"2023-01-24T03:30:25.013106+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:51:29.481545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,6],"acting":[0,1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005695+0000","last_change":"2023-01-24T03:30:16.272475+0000","last_active":"2023-01-24T03:30:25.005695+0000","last_peered":"2023-01-24T03:30:25.005695+0000","last_clean":"2023-01-24T03:30:25.005695+0000","last_became_active":"2023-01-24T03:30:16.272140+0000","last_became_peered":"2023-01-24T03:30:16.272140+0000","last_unstale":"2023-01-24T03:30:25.005695+0000","last_undegraded":"2023-01-24T03:30:25.005695+0000","last_fullsized":"2023-01-24T03:30:25.005695+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:48:16.300876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992670+0000","last_change":"2023-01-24T03:30:20.609391+0000","last_active":"2023-01-24T03:30:24.992670+0000","last_peered":"2023-01-24T03:30:24.992670+0000","last_clean":"2023-01-24T03:30:24.992670+0000","last_became_active":"2023-01-24T03:30:20.609124+0000","last_became_peered":"2023-01-24T03:30:20.609124+0000","last_unstale":"2023-01-24T03:30:24.992670+0000","last_undegraded":"2023-01-24T03:30:24.992670+0000","last_fullsized":"2023-01-24T03:30:24.992670+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:36:49.616649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.001859+0000","last_change":"2023-01-24T03:30:20.996985+0000","last_active":"2023-01-24T03:30:25.001859+0000","last_peered":"2023-01-24T03:30:25.001859+0000","last_clean":"2023-01-24T03:30:25.001859+0000","last_became_active":"2023-01-24T03:30:20.996612+0000","last_became_peered":"2023-01-24T03:30:20.996612+0000","last_unstale":"2023-01-24T03:30:25.001859+0000","last_undegraded":"2023-01-24T03:30:25.001859+0000","last_fullsized":"2023-01-24T03:30:25.001859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:10:50.378902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.012218+0000","last_change":"2023-01-24T03:30:25.008155+0000","last_active":"2023-01-24T03:30:25.012218+0000","last_peered":"2023-01-24T03:30:25.012218+0000","last_clean":"2023-01-24T03:30:25.012218+0000","last_became_active":"2023-01-24T03:30:25.007674+0000","last_became_peered":"2023-01-24T03:30:25.007674+0000","last_unstale":"2023-01-24T03:30:25.012218+0000","last_undegraded":"2023-01-24T03:30:25.012218+0000","last_fullsized":"2023-01-24T03:30:25.012218+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:55:46.448808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,2],"acting":[6,0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990827+0000","last_change":"2023-01-24T03:30:16.272372+0000","last_active":"2023-01-24T03:30:24.990827+0000","last_peered":"2023-01-24T03:30:24.990827+0000","last_clean":"2023-01-24T03:30:24.990827+0000","last_became_active":"2023-01-24T03:30:16.272104+0000","last_became_peered":"2023-01-24T03:30:16.272104+0000","last_unstale":"2023-01-24T03:30:24.990827+0000","last_undegraded":"2023-01-24T03:30:24.990827+0000","last_fullsized":"2023-01-24T03:30:24.990827+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:15:10.865596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002114+0000","last_change":"2023-01-24T03:30:20.612284+0000","last_active":"2023-01-24T03:30:25.002114+0000","last_peered":"2023-01-24T03:30:25.002114+0000","last_clean":"2023-01-24T03:30:25.002114+0000","last_became_active":"2023-01-24T03:30:20.612029+0000","last_became_peered":"2023-01-24T03:30:20.612029+0000","last_unstale":"2023-01-24T03:30:25.002114+0000","last_undegraded":"2023-01-24T03:30:25.002114+0000","last_fullsized":"2023-01-24T03:30:25.002114+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:18:58.755974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992643+0000","last_change":"2023-01-24T03:30:20.989834+0000","last_active":"2023-01-24T03:30:24.992643+0000","last_peered":"2023-01-24T03:30:24.992643+0000","last_clean":"2023-01-24T03:30:24.992643+0000","last_became_active":"2023-01-24T03:30:20.989645+0000","last_became_peered":"2023-01-24T03:30:20.989645+0000","last_unstale":"2023-01-24T03:30:24.992643+0000","last_undegraded":"2023-01-24T03:30:24.992643+0000","last_fullsized":"2023-01-24T03:30:24.992643+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:42:06.477043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016259+0000","last_change":"2023-01-24T03:30:25.011952+0000","last_active":"2023-01-24T03:30:25.016259+0000","last_peered":"2023-01-24T03:30:25.016259+0000","last_clean":"2023-01-24T03:30:25.016259+0000","last_became_active":"2023-01-24T03:30:25.011612+0000","last_became_peered":"2023-01-24T03:30:25.011612+0000","last_unstale":"2023-01-24T03:30:25.016259+0000","last_undegraded":"2023-01-24T03:30:25.016259+0000","last_fullsized":"2023-01-24T03:30:25.016259+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:51:52.027285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6,5],"acting":[0,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990404+0000","last_change":"2023-01-24T03:30:16.048979+0000","last_active":"2023-01-24T03:30:24.990404+0000","last_peered":"2023-01-24T03:30:24.990404+0000","last_clean":"2023-01-24T03:30:24.990404+0000","last_became_active":"2023-01-24T03:30:16.048641+0000","last_became_peered":"2023-01-24T03:30:16.048641+0000","last_unstale":"2023-01-24T03:30:24.990404+0000","last_undegraded":"2023-01-24T03:30:24.990404+0000","last_fullsized":"2023-01-24T03:30:24.990404+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T15:11:26.211024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005087+0000","last_change":"2023-01-24T03:30:20.606906+0000","last_active":"2023-01-24T03:30:25.005087+0000","last_peered":"2023-01-24T03:30:25.005087+0000","last_clean":"2023-01-24T03:30:25.005087+0000","last_became_active":"2023-01-24T03:30:20.606640+0000","last_became_peered":"2023-01-24T03:30:20.606640+0000","last_unstale":"2023-01-24T03:30:25.005087+0000","last_undegraded":"2023-01-24T03:30:25.005087+0000","last_fullsized":"2023-01-24T03:30:25.005087+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:21:09.008653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002357+0000","last_change":"2023-01-24T03:30:20.999886+0000","last_active":"2023-01-24T03:30:25.002357+0000","last_peered":"2023-01-24T03:30:25.002357+0000","last_clean":"2023-01-24T03:30:25.002357+0000","last_became_active":"2023-01-24T03:30:20.999394+0000","last_became_peered":"2023-01-24T03:30:20.999394+0000","last_unstale":"2023-01-24T03:30:25.002357+0000","last_undegraded":"2023-01-24T03:30:25.002357+0000","last_fullsized":"2023-01-24T03:30:25.002357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:30:22.395611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016356+0000","last_change":"2023-01-24T03:30:25.006190+0000","last_active":"2023-01-24T03:30:25.016356+0000","last_peered":"2023-01-24T03:30:25.016356+0000","last_clean":"2023-01-24T03:30:25.016356+0000","last_became_active":"2023-01-24T03:30:25.005825+0000","last_became_peered":"2023-01-24T03:30:25.005825+0000","last_unstale":"2023-01-24T03:30:25.016356+0000","last_undegraded":"2023-01-24T03:30:25.016356+0000","last_fullsized":"2023-01-24T03:30:25.016356+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:51:50.888938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,0],"acting":[1,3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"24'1","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.446961+0000","last_change":"2023-01-24T03:30:20.611079+0000","last_active":"2023-01-24T03:30:25.446961+0000","last_peered":"2023-01-24T03:30:25.446961+0000","last_clean":"2023-01-24T03:30:25.446961+0000","last_became_active":"2023-01-24T03:30:20.610841+0000","last_became_peered":"2023-01-24T03:30:20.610841+0000","last_unstale":"2023-01-24T03:30:25.446961+0000","last_undegraded":"2023-01-24T03:30:25.446961+0000","last_fullsized":"2023-01-24T03:30:25.446961+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:34:24.922661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.994693+0000","last_change":"2023-01-24T03:30:20.991418+0000","last_active":"2023-01-24T03:30:24.994693+0000","last_peered":"2023-01-24T03:30:24.994693+0000","last_clean":"2023-01-24T03:30:24.994693+0000","last_became_active":"2023-01-24T03:30:20.991167+0000","last_became_peered":"2023-01-24T03:30:20.991167+0000","last_unstale":"2023-01-24T03:30:24.994693+0000","last_undegraded":"2023-01-24T03:30:24.994693+0000","last_fullsized":"2023-01-24T03:30:24.994693+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:51:52.736691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.013519+0000","last_change":"2023-01-24T03:30:25.010394+0000","last_active":"2023-01-24T03:30:25.013519+0000","last_peered":"2023-01-24T03:30:25.013519+0000","last_clean":"2023-01-24T03:30:25.013519+0000","last_became_active":"2023-01-24T03:30:25.010044+0000","last_became_peered":"2023-01-24T03:30:25.010044+0000","last_unstale":"2023-01-24T03:30:25.013519+0000","last_undegraded":"2023-01-24T03:30:25.013519+0000","last_fullsized":"2023-01-24T03:30:25.013519+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:08:55.250184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1,6],"acting":[7,4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"24'2","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.984877+0000","last_change":"2023-01-24T03:30:20.607538+0000","last_active":"2023-01-24T03:30:25.984877+0000","last_peered":"2023-01-24T03:30:25.984877+0000","last_clean":"2023-01-24T03:30:25.984877+0000","last_became_active":"2023-01-24T03:30:20.607285+0000","last_became_peered":"2023-01-24T03:30:20.607285+0000","last_unstale":"2023-01-24T03:30:25.984877+0000","last_undegraded":"2023-01-24T03:30:25.984877+0000","last_fullsized":"2023-01-24T03:30:25.984877+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:42:21.836603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992857+0000","last_change":"2023-01-24T03:30:20.986791+0000","last_active":"2023-01-24T03:30:24.992857+0000","last_peered":"2023-01-24T03:30:24.992857+0000","last_clean":"2023-01-24T03:30:24.992857+0000","last_became_active":"2023-01-24T03:30:20.986516+0000","last_became_peered":"2023-01-24T03:30:20.986516+0000","last_unstale":"2023-01-24T03:30:24.992857+0000","last_undegraded":"2023-01-24T03:30:24.992857+0000","last_fullsized":"2023-01-24T03:30:24.992857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:53:19.191166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.015383+0000","last_change":"2023-01-24T03:30:25.010751+0000","last_active":"2023-01-24T03:30:25.015383+0000","last_peered":"2023-01-24T03:30:25.015383+0000","last_clean":"2023-01-24T03:30:25.015383+0000","last_became_active":"2023-01-24T03:30:25.010332+0000","last_became_peered":"2023-01-24T03:30:25.010332+0000","last_unstale":"2023-01-24T03:30:25.015383+0000","last_undegraded":"2023-01-24T03:30:25.015383+0000","last_fullsized":"2023-01-24T03:30:25.015383+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:48:30.190712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7,4],"acting":[5,0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"24'2","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985166+0000","last_change":"2023-01-24T03:30:20.608509+0000","last_active":"2023-01-24T03:30:25.985166+0000","last_peered":"2023-01-24T03:30:25.985166+0000","last_clean":"2023-01-24T03:30:25.985166+0000","last_became_active":"2023-01-24T03:30:20.608313+0000","last_became_peered":"2023-01-24T03:30:20.608313+0000","last_unstale":"2023-01-24T03:30:25.985166+0000","last_undegraded":"2023-01-24T03:30:25.985166+0000","last_fullsized":"2023-01-24T03:30:25.985166+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:31:52.253588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992448+0000","last_change":"2023-01-24T03:30:20.992334+0000","last_active":"2023-01-24T03:30:24.992448+0000","last_peered":"2023-01-24T03:30:24.992448+0000","last_clean":"2023-01-24T03:30:24.992448+0000","last_became_active":"2023-01-24T03:30:20.992027+0000","last_became_peered":"2023-01-24T03:30:20.992027+0000","last_unstale":"2023-01-24T03:30:24.992448+0000","last_undegraded":"2023-01-24T03:30:24.992448+0000","last_fullsized":"2023-01-24T03:30:24.992448+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:57:56.985507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.012177+0000","last_change":"2023-01-24T03:30:25.004167+0000","last_active":"2023-01-24T03:30:25.012177+0000","last_peered":"2023-01-24T03:30:25.012177+0000","last_clean":"2023-01-24T03:30:25.012177+0000","last_became_active":"2023-01-24T03:30:25.003752+0000","last_became_peered":"2023-01-24T03:30:25.003752+0000","last_unstale":"2023-01-24T03:30:25.012177+0000","last_undegraded":"2023-01-24T03:30:25.012177+0000","last_fullsized":"2023-01-24T03:30:25.012177+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:19:16.285529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"24'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.448123+0000","last_change":"2023-01-24T03:30:20.603542+0000","last_active":"2023-01-24T03:30:25.448123+0000","last_peered":"2023-01-24T03:30:25.448123+0000","last_clean":"2023-01-24T03:30:25.448123+0000","last_became_active":"2023-01-24T03:30:20.603287+0000","last_became_peered":"2023-01-24T03:30:20.603287+0000","last_unstale":"2023-01-24T03:30:25.448123+0000","last_undegraded":"2023-01-24T03:30:25.448123+0000","last_fullsized":"2023-01-24T03:30:25.448123+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:03:56.865763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991098+0000","last_change":"2023-01-24T03:30:20.989519+0000","last_active":"2023-01-24T03:30:24.991098+0000","last_peered":"2023-01-24T03:30:24.991098+0000","last_clean":"2023-01-24T03:30:24.991098+0000","last_became_active":"2023-01-24T03:30:20.989286+0000","last_became_peered":"2023-01-24T03:30:20.989286+0000","last_unstale":"2023-01-24T03:30:24.991098+0000","last_undegraded":"2023-01-24T03:30:24.991098+0000","last_fullsized":"2023-01-24T03:30:24.991098+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:17:30.829327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016619+0000","last_change":"2023-01-24T03:30:25.010262+0000","last_active":"2023-01-24T03:30:25.016619+0000","last_peered":"2023-01-24T03:30:25.016619+0000","last_clean":"2023-01-24T03:30:25.016619+0000","last_became_active":"2023-01-24T03:30:25.009930+0000","last_became_peered":"2023-01-24T03:30:25.009930+0000","last_unstale":"2023-01-24T03:30:25.016619+0000","last_undegraded":"2023-01-24T03:30:25.016619+0000","last_fullsized":"2023-01-24T03:30:25.016619+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:36:17.807046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7,2],"acting":[1,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"24'4","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985911+0000","last_change":"2023-01-24T03:30:20.607334+0000","last_active":"2023-01-24T03:30:25.985911+0000","last_peered":"2023-01-24T03:30:25.985911+0000","last_clean":"2023-01-24T03:30:25.985911+0000","last_became_active":"2023-01-24T03:30:20.607079+0000","last_became_peered":"2023-01-24T03:30:20.607079+0000","last_unstale":"2023-01-24T03:30:25.985911+0000","last_undegraded":"2023-01-24T03:30:25.985911+0000","last_fullsized":"2023-01-24T03:30:25.985911+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:05:18.886532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992554+0000","last_change":"2023-01-24T03:30:20.987567+0000","last_active":"2023-01-24T03:30:24.992554+0000","last_peered":"2023-01-24T03:30:24.992554+0000","last_clean":"2023-01-24T03:30:24.992554+0000","last_became_active":"2023-01-24T03:30:20.987058+0000","last_became_peered":"2023-01-24T03:30:20.987058+0000","last_unstale":"2023-01-24T03:30:24.992554+0000","last_undegraded":"2023-01-24T03:30:24.992554+0000","last_fullsized":"2023-01-24T03:30:24.992554+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:58:30.391733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.989538+0000","last_change":"2023-01-24T03:30:20.986393+0000","last_active":"2023-01-24T03:30:24.989538+0000","last_peered":"2023-01-24T03:30:24.989538+0000","last_clean":"2023-01-24T03:30:24.989538+0000","last_became_active":"2023-01-24T03:30:20.986089+0000","last_became_peered":"2023-01-24T03:30:20.986089+0000","last_unstale":"2023-01-24T03:30:24.989538+0000","last_undegraded":"2023-01-24T03:30:24.989538+0000","last_fullsized":"2023-01-24T03:30:24.989538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:44:21.992291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990902+0000","last_change":"2023-01-24T03:30:20.991468+0000","last_active":"2023-01-24T03:30:24.990902+0000","last_peered":"2023-01-24T03:30:24.990902+0000","last_clean":"2023-01-24T03:30:24.990902+0000","last_became_active":"2023-01-24T03:30:20.991227+0000","last_became_peered":"2023-01-24T03:30:20.991227+0000","last_unstale":"2023-01-24T03:30:24.990902+0000","last_undegraded":"2023-01-24T03:30:24.990902+0000","last_fullsized":"2023-01-24T03:30:24.990902+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:27:25.806340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992602+0000","last_change":"2023-01-24T03:30:20.989824+0000","last_active":"2023-01-24T03:30:24.992602+0000","last_peered":"2023-01-24T03:30:24.992602+0000","last_clean":"2023-01-24T03:30:24.992602+0000","last_became_active":"2023-01-24T03:30:20.989577+0000","last_became_peered":"2023-01-24T03:30:20.989577+0000","last_unstale":"2023-01-24T03:30:24.992602+0000","last_undegraded":"2023-01-24T03:30:24.992602+0000","last_fullsized":"2023-01-24T03:30:24.992602+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:48:17.948459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992452+0000","last_change":"2023-01-24T03:30:20.996266+0000","last_active":"2023-01-24T03:30:24.992452+0000","last_peered":"2023-01-24T03:30:24.992452+0000","last_clean":"2023-01-24T03:30:24.992452+0000","last_became_active":"2023-01-24T03:30:20.995610+0000","last_became_peered":"2023-01-24T03:30:20.995610+0000","last_unstale":"2023-01-24T03:30:24.992452+0000","last_undegraded":"2023-01-24T03:30:24.992452+0000","last_fullsized":"2023-01-24T03:30:24.992452+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:24:43.022530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.001943+0000","last_change":"2023-01-24T03:30:20.997081+0000","last_active":"2023-01-24T03:30:25.001943+0000","last_peered":"2023-01-24T03:30:25.001943+0000","last_clean":"2023-01-24T03:30:25.001943+0000","last_became_active":"2023-01-24T03:30:20.996837+0000","last_became_peered":"2023-01-24T03:30:20.996837+0000","last_unstale":"2023-01-24T03:30:25.001943+0000","last_undegraded":"2023-01-24T03:30:25.001943+0000","last_fullsized":"2023-01-24T03:30:25.001943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:42:06.134817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":4483,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":8966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64574,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9180,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":8832,"kb_avail":94362660,"statfs":{"total":96636764160,"available":96627363840,"internally_reserved":0,"allocated":348160,"data_stored":698731,"data_compressed":32287,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":9043968},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8576,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94363264,"statfs":{"total":96636764160,"available":96627982336,"internally_reserved":0,"allocated":253952,"data_stored":104434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9152,"kb_used_data":312,"kb_used_omap":0,"kb_used_meta":8832,"kb_avail":94362688,"statfs":{"total":96636764160,"available":96627392512,"internally_reserved":0,"allocated":319488,"data_stored":694802,"data_compressed":32287,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":9043968},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]}]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8640,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94363200,"statfs":{"total":96636764160,"available":96627916800,"internally_reserved":0,"allocated":253952,"data_stored":104382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.255}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8512,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94363328,"statfs":{"total":96636764160,"available":96628047872,"internally_reserved":0,"allocated":253952,"data_stored":104434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.254}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8580,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94363260,"statfs":{"total":96636764160,"available":96627978240,"internally_reserved":0,"allocated":258048,"data_stored":104524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8660,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94363180,"statfs":{"total":96636764160,"available":96627896320,"internally_reserved":0,"allocated":274432,"data_stored":108203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]}]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8580,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94363260,"statfs":{"total":96636764160,"available":96627978240,"internally_reserved":0,"allocated":258048,"data_stored":103705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32287,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32287,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":3929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-01-24T03:30:43.552 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-01-24T03:30:43.552 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-01-24T03:30:43.553 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-01-24T03:30:43.553 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-01-24T03:30:43.695 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.693+0000 7f36ffe91700 1 Processor -- start 2023-01-24T03:30:43.696 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.693+0000 7f36ffe91700 1 -- start start 2023-01-24T03:30:43.697 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.694+0000 7f36ffe91700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f36f814c290 0x7f36f806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:43.697 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.694+0000 7f36ffe91700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36f80c18f0 con 0x7f36f814c660 2023-01-24T03:30:43.697 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.694+0000 7f36ffe91700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36f80c1030 con 0x7f36f80c2490 2023-01-24T03:30:43.698 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.694+0000 7f36ffe91700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36f806bad0 con 0x7f36f814c290 2023-01-24T03:30:43.698 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.694+0000 7f36fdc2d700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f36f814c290 0x7f36f806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:43.698 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.694+0000 7f36fdc2d700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f36f814c290 0x7f36f806ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50832/0 (socket says 172.21.15.112:50832) 2023-01-24T03:30:43.698 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.694+0000 7f36fdc2d700 1 -- 172.21.15.112:0/262987113 learned_addr learned my addr 172.21.15.112:0/262987113 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:43.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.694+0000 7f36fdc2d700 1 -- 172.21.15.112:0/262987113 >> v1:172.21.15.142:6789/0 conn(0x7f36f80c2490 legacy=0x7f36f8155710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:43.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.694+0000 7f36fdc2d700 1 -- 172.21.15.112:0/262987113 >> v1:172.21.15.112:6789/0 conn(0x7f36f814c660 legacy=0x7f36f80c1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:43.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.694+0000 7f36fdc2d700 1 -- 172.21.15.112:0/262987113 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f806bdb0 con 0x7f36f814c290 2023-01-24T03:30:43.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.694+0000 7f36fdc2d700 1 --2- 172.21.15.112:0/262987113 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f36f814c290 0x7f36f806ec80 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f36f4006fd0 tx=0x7f36f40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae2c77ae9a1ba3ca server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:43.700 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.696+0000 7f36fcc2b700 1 -- 172.21.15.112:0/262987113 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f4011630 con 0x7f36f814c290 2023-01-24T03:30:43.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.696+0000 7f36fcc2b700 1 -- 172.21.15.112:0/262987113 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36f40117b0 con 0x7f36f814c290 2023-01-24T03:30:43.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.696+0000 7f36fcc2b700 1 -- 172.21.15.112:0/262987113 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f4011b70 con 0x7f36f814c290 2023-01-24T03:30:43.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.697+0000 7f36ffe91700 1 -- 172.21.15.112:0/262987113 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f36f814c290 msgr2=0x7f36f806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:43.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.697+0000 7f36ffe91700 1 --2- 172.21.15.112:0/262987113 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f36f814c290 0x7f36f806ec80 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f36f4006fd0 tx=0x7f36f40090f0 comp rx=0 tx=0).stop 2023-01-24T03:30:43.702 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.697+0000 7f36ffe91700 1 -- 172.21.15.112:0/262987113 shutdown_connections 2023-01-24T03:30:43.702 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.697+0000 7f36ffe91700 1 --2- 172.21.15.112:0/262987113 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f36f814c290 0x7f36f806ec80 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:43.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.697+0000 7f36ffe91700 1 -- 172.21.15.112:0/262987113 >> 172.21.15.112:0/262987113 conn(0x7f36f8067f20 msgr2=0x7f36f8068320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:43.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.697+0000 7f36ffe91700 1 -- 172.21.15.112:0/262987113 shutdown_connections 2023-01-24T03:30:43.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.697+0000 7f36ffe91700 1 -- 172.21.15.112:0/262987113 wait complete. 2023-01-24T03:30:43.707 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.698+0000 7f36ffe91700 1 Processor -- start 2023-01-24T03:30:43.707 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.698+0000 7f36ffe91700 1 -- start start 2023-01-24T03:30:43.707 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.699+0000 7f36ffe91700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36f80c2490 0x7f36f8066fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:43.707 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.699+0000 7f36ffe91700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f36f814c290 0x7f36f80bce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:43.708 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.699+0000 7f36ffe91700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f36f814c660 0x7f36f80bd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:43.708 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.699+0000 7f36ffe91700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f80be2f0 con 0x7f36f814c660 2023-01-24T03:30:43.708 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.699+0000 7f36ffe91700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f80c7f30 con 0x7f36f814c290 2023-01-24T03:30:43.708 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.699+0000 7f36ffe91700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36f80c5230 con 0x7f36f80c2490 2023-01-24T03:30:43.708 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.699+0000 7f36fe42e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f36f814c660 0x7f36f80bd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:43.709 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.699+0000 7f36fe42e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f36f814c660 0x7f36f80bd3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50842/0 (socket says 172.21.15.112:50842) 2023-01-24T03:30:43.709 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.699+0000 7f36fe42e700 1 -- 172.21.15.112:0/920491866 learned_addr learned my addr 172.21.15.112:0/920491866 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:43.709 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.699+0000 7f36fd42c700 1 --2- 172.21.15.112:0/920491866 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f36f814c290 0x7f36f80bce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:43.709 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.699+0000 7f36fdc2d700 1 --2- 172.21.15.112:0/920491866 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36f80c2490 0x7f36f8066fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:43.709 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.699+0000 7f36fe42e700 1 -- 172.21.15.112:0/920491866 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36f80c2490 msgr2=0x7f36f8066fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:43.710 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.699+0000 7f36fe42e700 1 --2- 172.21.15.112:0/920491866 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36f80c2490 0x7f36f8066fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:43.710 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.699+0000 7f36fe42e700 1 -- 172.21.15.112:0/920491866 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f36f814c290 msgr2=0x7f36f80bce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:43.710 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.699+0000 7f36fe42e700 1 --2- 172.21.15.112:0/920491866 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f36f814c290 0x7f36f80bce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:43.710 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.699+0000 7f36fe42e700 1 -- 172.21.15.112:0/920491866 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f4009370 con 0x7f36f814c660 2023-01-24T03:30:43.710 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.699+0000 7f36fe42e700 1 --2- 172.21.15.112:0/920491866 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f36f814c660 0x7f36f80bd3b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f36f000c790 tx=0x7f36f00015e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:43.711 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.699+0000 7f36e5ffb700 1 -- 172.21.15.112:0/920491866 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f0012630 con 0x7f36f814c660 2023-01-24T03:30:43.711 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.699+0000 7f36e5ffb700 1 -- 172.21.15.112:0/920491866 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36f0012790 con 0x7f36f814c660 2023-01-24T03:30:43.711 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.699+0000 7f36ffe91700 1 -- 172.21.15.112:0/920491866 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36f80bf040 con 0x7f36f814c660 2023-01-24T03:30:43.711 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.699+0000 7f36e5ffb700 1 -- 172.21.15.112:0/920491866 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f0012b10 con 0x7f36f814c660 2023-01-24T03:30:43.711 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.700+0000 7f36ffe91700 1 -- 172.21.15.112:0/920491866 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36f80c3e30 con 0x7f36f814c660 2023-01-24T03:30:43.712 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.701+0000 7f36ffe91700 1 -- 172.21.15.112:0/920491866 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36f804bb80 con 0x7f36f814c660 2023-01-24T03:30:43.712 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.701+0000 7f36e5ffb700 1 -- 172.21.15.112:0/920491866 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f36f0018020 con 0x7f36f814c660 2023-01-24T03:30:43.712 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.701+0000 7f36e5ffb700 1 --2- 172.21.15.112:0/920491866 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f36d80611c0 0x7f36d8063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:43.712 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.701+0000 7f36e5ffb700 1 -- 172.21.15.112:0/920491866 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f36f007d8f0 con 0x7f36f814c660 2023-01-24T03:30:43.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.701+0000 7f36fdc2d700 1 --2- 172.21.15.112:0/920491866 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f36d80611c0 0x7f36d8063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:43.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.701+0000 7f36fdc2d700 1 --2- 172.21.15.112:0/920491866 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f36d80611c0 0x7f36d8063670 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f36f4006fd0 tx=0x7f36f4016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:43.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.705+0000 7f36e5ffb700 1 -- 172.21.15.112:0/920491866 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f36f004d730 con 0x7f36f814c660 2023-01-24T03:30:43.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.958+0000 7f36ffe91700 1 -- 172.21.15.112:0/920491866 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f36f806c150 con 0x7f36f814c660 2023-01-24T03:30:43.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.959+0000 7f36e5ffb700 1 -- 172.21.15.112:0/920491866 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f36f0050d50 con 0x7f36f814c660 2023-01-24T03:30:43.964 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:30:43.964 INFO:teuthology.orchestra.run.smithi112.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-01-24T03:30:43.965 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.961+0000 7f36ffe91700 1 -- 172.21.15.112:0/920491866 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f36d80611c0 msgr2=0x7f36d8063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:43.966 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.961+0000 7f36ffe91700 1 --2- 172.21.15.112:0/920491866 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f36d80611c0 0x7f36d8063670 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f36f4006fd0 tx=0x7f36f4016000 comp rx=0 tx=0).stop 2023-01-24T03:30:43.966 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.961+0000 7f36ffe91700 1 -- 172.21.15.112:0/920491866 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f36f814c660 msgr2=0x7f36f80bd3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:43.966 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.961+0000 7f36ffe91700 1 --2- 172.21.15.112:0/920491866 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f36f814c660 0x7f36f80bd3b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f36f000c790 tx=0x7f36f00015e0 comp rx=0 tx=0).stop 2023-01-24T03:30:43.966 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.961+0000 7f36ffe91700 1 -- 172.21.15.112:0/920491866 shutdown_connections 2023-01-24T03:30:43.967 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.961+0000 7f36ffe91700 1 --2- 172.21.15.112:0/920491866 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f36d80611c0 0x7f36d8063670 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:43.967 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.961+0000 7f36ffe91700 1 --2- 172.21.15.112:0/920491866 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f36f814c660 0x7f36f80bd3b0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:43.967 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.961+0000 7f36ffe91700 1 --2- 172.21.15.112:0/920491866 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f36f80c2490 0x7f36f8066fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:43.968 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.961+0000 7f36ffe91700 1 --2- 172.21.15.112:0/920491866 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f36f814c290 0x7f36f80bce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:43.968 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.961+0000 7f36ffe91700 1 -- 172.21.15.112:0/920491866 >> 172.21.15.112:0/920491866 conn(0x7f36f8067f20 msgr2=0x7f36f80682f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:43.968 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.961+0000 7f36ffe91700 1 -- 172.21.15.112:0/920491866 shutdown_connections 2023-01-24T03:30:43.968 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:43.961+0000 7f36ffe91700 1 -- 172.21.15.112:0/920491866 wait complete. 2023-01-24T03:30:43.974 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-01-24T03:30:43.975 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T03:30:44.087 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.084+0000 7fefb39f5700 1 Processor -- start 2023-01-24T03:30:44.088 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.084+0000 7fefb39f5700 1 -- start start 2023-01-24T03:30:44.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.085+0000 7fefb39f5700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fefac14f2c0 0x7fefac0c4f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:44.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.085+0000 7fefb39f5700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefac05df40 con 0x7fefac0c4790 2023-01-24T03:30:44.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.085+0000 7fefb39f5700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefac155920 con 0x7fefac0c40e0 2023-01-24T03:30:44.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.085+0000 7fefb39f5700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefac1552f0 con 0x7fefac14f2c0 2023-01-24T03:30:44.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.085+0000 7fefb1f92700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fefac0c4790 0x7fefac15fc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:60068/0 (socket says 172.21.15.112:60068) 2023-01-24T03:30:44.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.085+0000 7fefb1f92700 1 -- 172.21.15.112:0/3086819584 learned_addr learned my addr 172.21.15.112:0/3086819584 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:44.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.086+0000 7fefb1791700 1 --2- 172.21.15.112:0/3086819584 >> v2:172.21.15.142:3300/0 conn(0x7fefac14f2c0 0x7fefac0c4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:44.095 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.086+0000 7fefabfff700 1 -- 172.21.15.112:0/3086819584 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2174200176 0 0) 0x7fefac05df40 con 0x7fefac0c4790 2023-01-24T03:30:44.095 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.086+0000 7fefabfff700 1 -- 172.21.15.112:0/3086819584 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef980033f0 con 0x7fefac0c4790 2023-01-24T03:30:44.095 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.086+0000 7fefabfff700 1 -- 172.21.15.112:0/3086819584 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4242213754 0 0) 0x7fef980033f0 con 0x7fefac0c4790 2023-01-24T03:30:44.095 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.086+0000 7fefabfff700 1 -- 172.21.15.112:0/3086819584 >> v2:172.21.15.142:3300/0 conn(0x7fefac14f2c0 msgr2=0x7fefac0c4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:44.095 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.086+0000 7fefabfff700 1 --2- 172.21.15.112:0/3086819584 >> v2:172.21.15.142:3300/0 conn(0x7fefac14f2c0 0x7fefac0c4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:44.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.086+0000 7fefabfff700 1 -- 172.21.15.112:0/3086819584 >> v1:172.21.15.142:6789/0 conn(0x7fefac0c40e0 legacy=0x7fefac067f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:44.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.086+0000 7fefabfff700 1 -- 172.21.15.112:0/3086819584 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefac05e190 con 0x7fefac0c4790 2023-01-24T03:30:44.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.086+0000 7fefabfff700 1 -- 172.21.15.112:0/3086819584 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7fefa4001470 con 0x7fefac0c4790 2023-01-24T03:30:44.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.086+0000 7fefabfff700 1 -- 172.21.15.112:0/3086819584 >> v1:172.21.15.112:6789/0 conn(0x7fefac0c4790 legacy=0x7fefac15fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:44.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.086+0000 7fefabfff700 1 --2- 172.21.15.112:0/3086819584 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef980038e0 0x7fef98003cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:44.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.086+0000 7fefabfff700 1 --2- 172.21.15.112:0/3086819584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fef98004690 0x7fef98004b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:44.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.086+0000 7fefabfff700 1 --2- 172.21.15.112:0/3086819584 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fef98005040 0x7fef9800b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:44.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.086+0000 7fefabfff700 1 -- 172.21.15.112:0/3086819584 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefac1552f0 con 0x7fef980038e0 2023-01-24T03:30:44.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.086+0000 7fefabfff700 1 -- 172.21.15.112:0/3086819584 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9800bf50 con 0x7fef98004690 2023-01-24T03:30:44.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.086+0000 7fefabfff700 1 -- 172.21.15.112:0/3086819584 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef9800c110 con 0x7fef98005040 2023-01-24T03:30:44.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.086+0000 7fefb1791700 1 --2- 172.21.15.112:0/3086819584 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef980038e0 0x7fef98003cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:44.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.086+0000 7fefb0f90700 1 --2- 172.21.15.112:0/3086819584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fef98004690 0x7fef98004b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:44.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.086+0000 7fefb1f92700 1 --2- 172.21.15.112:0/3086819584 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fef98005040 0x7fef9800b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:44.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.086+0000 7fefb1791700 1 -- 172.21.15.112:0/3086819584 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fef98005040 msgr2=0x7fef9800b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:44.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.086+0000 7fefb1791700 1 --2- 172.21.15.112:0/3086819584 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fef98005040 0x7fef9800b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:44.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.086+0000 7fefb1791700 1 -- 172.21.15.112:0/3086819584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fef98004690 msgr2=0x7fef98004b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:44.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.086+0000 7fefb1791700 1 --2- 172.21.15.112:0/3086819584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fef98004690 0x7fef98004b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:44.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.086+0000 7fefb1f92700 1 -- 172.21.15.112:0/3086819584 reap_dead start 2023-01-24T03:30:44.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.086+0000 7fefb1791700 1 -- 172.21.15.112:0/3086819584 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef980033f0 con 0x7fef980038e0 2023-01-24T03:30:44.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.086+0000 7fefb0f90700 1 --2- 172.21.15.112:0/3086819584 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fef98004690 0x7fef98004b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:30:44.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.086+0000 7fefb1791700 1 --2- 172.21.15.112:0/3086819584 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef980038e0 0x7fef98003cf0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fef9c00e210 tx=0x7fef9c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aedd712adcee77fd server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:44.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.087+0000 7fefabfff700 1 -- 172.21.15.112:0/3086819584 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9c00eac0 con 0x7fef980038e0 2023-01-24T03:30:44.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.087+0000 7fefabfff700 1 -- 172.21.15.112:0/3086819584 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef9c00f040 con 0x7fef980038e0 2023-01-24T03:30:44.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.087+0000 7fefabfff700 1 -- 172.21.15.112:0/3086819584 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9c00ee50 con 0x7fef980038e0 2023-01-24T03:30:44.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.087+0000 7fefb39f5700 1 -- 172.21.15.112:0/3086819584 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef980038e0 msgr2=0x7fef98003cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:44.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.087+0000 7fefb39f5700 1 --2- 172.21.15.112:0/3086819584 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef980038e0 0x7fef98003cf0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fef9c00e210 tx=0x7fef9c00b040 comp rx=0 tx=0).stop 2023-01-24T03:30:44.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.087+0000 7fefb39f5700 1 -- 172.21.15.112:0/3086819584 shutdown_connections 2023-01-24T03:30:44.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.087+0000 7fefb39f5700 1 --2- 172.21.15.112:0/3086819584 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef980038e0 0x7fef98003cf0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:44.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.087+0000 7fefb39f5700 1 -- 172.21.15.112:0/3086819584 >> 172.21.15.112:0/3086819584 conn(0x7fefac06daf0 msgr2=0x7fefac05d830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:44.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.087+0000 7fefb39f5700 1 -- 172.21.15.112:0/3086819584 shutdown_connections 2023-01-24T03:30:44.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.087+0000 7fefb39f5700 1 -- 172.21.15.112:0/3086819584 wait complete. 2023-01-24T03:30:44.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.088+0000 7fefb39f5700 1 Processor -- start 2023-01-24T03:30:44.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.088+0000 7fefb39f5700 1 -- start start 2023-01-24T03:30:44.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.089+0000 7fefb39f5700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef980038e0 0x7fefac141fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:44.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.089+0000 7fefb39f5700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fefac14c200 0x7fefac1578f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:44.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.089+0000 7fefb39f5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fefac157e30 0x7fefac165410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:44.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.089+0000 7fefb39f5700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefac157040 con 0x7fef980038e0 2023-01-24T03:30:44.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.089+0000 7fefb39f5700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefac147f20 con 0x7fefac14c200 2023-01-24T03:30:44.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.089+0000 7fefb39f5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefac1688e0 con 0x7fefac157e30 2023-01-24T03:30:44.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.089+0000 7fefb1791700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef980038e0 0x7fefac141fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:44.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.089+0000 7fefb1791700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef980038e0 0x7fefac141fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50858/0 (socket says 172.21.15.112:50858) 2023-01-24T03:30:44.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.089+0000 7fefb1791700 1 -- 172.21.15.112:0/2751819601 learned_addr learned my addr 172.21.15.112:0/2751819601 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:44.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.089+0000 7fefb1f92700 1 --2- 172.21.15.112:0/2751819601 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fefac157e30 0x7fefac165410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:44.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.089+0000 7fefb0f90700 1 --2- 172.21.15.112:0/2751819601 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fefac14c200 0x7fefac1578f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:44.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.089+0000 7fefb1791700 1 -- 172.21.15.112:0/2751819601 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fefac157e30 msgr2=0x7fefac165410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:44.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.089+0000 7fefb1791700 1 --2- 172.21.15.112:0/2751819601 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fefac157e30 0x7fefac165410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:44.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.089+0000 7fefb1791700 1 -- 172.21.15.112:0/2751819601 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fefac14c200 msgr2=0x7fefac1578f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:44.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.089+0000 7fefb1791700 1 --2- 172.21.15.112:0/2751819601 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fefac14c200 0x7fefac1578f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:44.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.089+0000 7fefb1791700 1 -- 172.21.15.112:0/2751819601 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefac168bc0 con 0x7fef980038e0 2023-01-24T03:30:44.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.090+0000 7fefb1791700 1 --2- 172.21.15.112:0/2751819601 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef980038e0 0x7fefac141fd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fef9800bf20 tx=0x7fef9c01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:44.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.090+0000 7fefa97fa700 1 -- 172.21.15.112:0/2751819601 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9c01d450 con 0x7fef980038e0 2023-01-24T03:30:44.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.090+0000 7fefa97fa700 1 -- 172.21.15.112:0/2751819601 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef9c01d5b0 con 0x7fef980038e0 2023-01-24T03:30:44.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.090+0000 7fefb39f5700 1 -- 172.21.15.112:0/2751819601 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefac168ea0 con 0x7fef980038e0 2023-01-24T03:30:44.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.090+0000 7fefa97fa700 1 -- 172.21.15.112:0/2751819601 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9c01d970 con 0x7fef980038e0 2023-01-24T03:30:44.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.090+0000 7fefb39f5700 1 -- 172.21.15.112:0/2751819601 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefac142510 con 0x7fef980038e0 2023-01-24T03:30:44.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.091+0000 7fefb39f5700 1 -- 172.21.15.112:0/2751819601 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefac169340 con 0x7fef980038e0 2023-01-24T03:30:44.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.091+0000 7fefa97fa700 1 -- 172.21.15.112:0/2751819601 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fef9c02e020 con 0x7fef980038e0 2023-01-24T03:30:44.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.091+0000 7fefa97fa700 1 --2- 172.21.15.112:0/2751819601 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fef8c0611c0 0x7fef8c063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:44.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.091+0000 7fefa97fa700 1 -- 172.21.15.112:0/2751819601 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fef9c07fdb0 con 0x7fef980038e0 2023-01-24T03:30:44.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.091+0000 7fefb0f90700 1 --2- 172.21.15.112:0/2751819601 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fef8c0611c0 0x7fef8c063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:44.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.092+0000 7fefb0f90700 1 --2- 172.21.15.112:0/2751819601 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fef8c0611c0 0x7fef8c063670 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fefa0007a40 tx=0x7fefa0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:44.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.095+0000 7fefa97fa700 1 -- 172.21.15.112:0/2751819601 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fef9c04fb00 con 0x7fef980038e0 2023-01-24T03:30:44.276 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.273+0000 7fefb39f5700 1 -- 172.21.15.112:0/2751819601 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fefac1694f0 con 0x7fef980038e0 2023-01-24T03:30:44.277 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.274+0000 7fefa97fa700 1 -- 172.21.15.112:0/2751819601 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18755 (secure 0 0 0) 0x7fef9c053120 con 0x7fef980038e0 2023-01-24T03:30:44.278 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:30:44.278 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":24,"fsid":"054b9ac9-b7b0-40e5-b932-6c8a955c5a72","created":"2023-01-24T03:30:03.798533+0000","modified":"2023-01-24T03:30:25.373340+0000","last_up_change":"2023-01-24T03:30:13.945316+0000","last_in_change":"2023-01-24T03:30:06.198433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-01-24T03:30:14.931166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-01-24T03:30:15.023730+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-01-24T03:30:19.060597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-01-24T03:30:19.943686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-01-24T03:30:22.999036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"fe6ad231-a39a-42eb-9a3a-49d008dd35e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6809","nonce":84512},{"type":"v1","addr":"172.21.15.112:6811","nonce":84512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":84512},{"type":"v1","addr":"172.21.15.112:6817","nonce":84512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6825","nonce":84512},{"type":"v1","addr":"172.21.15.112:6828","nonce":84512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6819","nonce":84512},{"type":"v1","addr":"172.21.15.112:6822","nonce":84512}]},"public_addr":"172.21.15.112:6811/84512","cluster_addr":"172.21.15.112:6817/84512","heartbeat_back_addr":"172.21.15.112:6828/84512","heartbeat_front_addr":"172.21.15.112:6822/84512","state":["exists","up"]},{"osd":1,"uuid":"8526a1d2-d540-4021-9766-c4bbb30317ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":84513},{"type":"v1","addr":"172.21.15.112:6812","nonce":84513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6815","nonce":84513},{"type":"v1","addr":"172.21.15.112:6818","nonce":84513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6827","nonce":84513},{"type":"v1","addr":"172.21.15.112:6830","nonce":84513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6821","nonce":84513},{"type":"v1","addr":"172.21.15.112:6824","nonce":84513}]},"public_addr":"172.21.15.112:6812/84513","cluster_addr":"172.21.15.112:6818/84513","heartbeat_back_addr":"172.21.15.112:6830/84513","heartbeat_front_addr":"172.21.15.112:6824/84513","state":["exists","up"]},{"osd":2,"uuid":"041ef8e4-4858-432d-9476-cbdb3a0dc47c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":84511},{"type":"v1","addr":"172.21.15.112:6813","nonce":84511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":84511},{"type":"v1","addr":"172.21.15.112:6820","nonce":84511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6829","nonce":84511},{"type":"v1","addr":"172.21.15.112:6831","nonce":84511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6823","nonce":84511},{"type":"v1","addr":"172.21.15.112:6826","nonce":84511}]},"public_addr":"172.21.15.112:6813/84511","cluster_addr":"172.21.15.112:6820/84511","heartbeat_back_addr":"172.21.15.112:6831/84511","heartbeat_front_addr":"172.21.15.112:6826/84511","state":["exists","up"]},{"osd":3,"uuid":"987587cf-1357-4a70-ad85-67b189c67729","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":84510},{"type":"v1","addr":"172.21.15.112:6801","nonce":84510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":84510},{"type":"v1","addr":"172.21.15.112:6803","nonce":84510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":84510},{"type":"v1","addr":"172.21.15.112:6807","nonce":84510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":84510},{"type":"v1","addr":"172.21.15.112:6805","nonce":84510}]},"public_addr":"172.21.15.112:6801/84510","cluster_addr":"172.21.15.112:6803/84510","heartbeat_back_addr":"172.21.15.112:6807/84510","heartbeat_front_addr":"172.21.15.112:6805/84510","state":["exists","up"]},{"osd":4,"uuid":"8503512b-43db-40b8-8b2b-c86cf2e54e17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":84614},{"type":"v1","addr":"172.21.15.142:6801","nonce":84614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":84614},{"type":"v1","addr":"172.21.15.142:6803","nonce":84614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":84614},{"type":"v1","addr":"172.21.15.142:6807","nonce":84614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":84614},{"type":"v1","addr":"172.21.15.142:6805","nonce":84614}]},"public_addr":"172.21.15.142:6801/84614","cluster_addr":"172.21.15.142:6803/84614","heartbeat_back_addr":"172.21.15.142:6807/84614","heartbeat_front_addr":"172.21.15.142:6805/84614","state":["exists","up"]},{"osd":5,"uuid":"d3bc0030-8b08-4645-9ec2-1f8a4e29ef36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":84638},{"type":"v1","addr":"172.21.15.142:6817","nonce":84638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":84638},{"type":"v1","addr":"172.21.15.142:6819","nonce":84638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":84638},{"type":"v1","addr":"172.21.15.142:6823","nonce":84638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":84638},{"type":"v1","addr":"172.21.15.142:6821","nonce":84638}]},"public_addr":"172.21.15.142:6817/84638","cluster_addr":"172.21.15.142:6819/84638","heartbeat_back_addr":"172.21.15.142:6823/84638","heartbeat_front_addr":"172.21.15.142:6821/84638","state":["exists","up"]},{"osd":6,"uuid":"c7649e3b-832d-4e6f-8866-b15e5d814476","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":84645},{"type":"v1","addr":"172.21.15.142:6825","nonce":84645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":84645},{"type":"v1","addr":"172.21.15.142:6827","nonce":84645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":84645},{"type":"v1","addr":"172.21.15.142:6831","nonce":84645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":84645},{"type":"v1","addr":"172.21.15.142:6829","nonce":84645}]},"public_addr":"172.21.15.142:6825/84645","cluster_addr":"172.21.15.142:6827/84645","heartbeat_back_addr":"172.21.15.142:6831/84645","heartbeat_front_addr":"172.21.15.142:6829/84645","state":["exists","up"]},{"osd":7,"uuid":"2ea5b3bf-884e-4b20-b572-1443efdc7477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":84615},{"type":"v1","addr":"172.21.15.142:6809","nonce":84615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":84615},{"type":"v1","addr":"172.21.15.142:6811","nonce":84615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":84615},{"type":"v1","addr":"172.21.15.142:6815","nonce":84615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":84615},{"type":"v1","addr":"172.21.15.142:6813","nonce":84615}]},"public_addr":"172.21.15.142:6809/84615","cluster_addr":"172.21.15.142:6811/84615","heartbeat_back_addr":"172.21.15.142:6815/84615","heartbeat_front_addr":"172.21.15.142:6813/84615","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.370027+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.335683+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.350667+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.350794+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.391119+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.422933+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.400439+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.354018+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T03:30:44.280 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.276+0000 7fefb39f5700 1 -- 172.21.15.112:0/2751819601 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fef8c0611c0 msgr2=0x7fef8c063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:44.280 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.276+0000 7fefb39f5700 1 --2- 172.21.15.112:0/2751819601 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fef8c0611c0 0x7fef8c063670 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fefa0007a40 tx=0x7fefa0009040 comp rx=0 tx=0).stop 2023-01-24T03:30:44.280 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.276+0000 7fefb39f5700 1 -- 172.21.15.112:0/2751819601 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef980038e0 msgr2=0x7fefac141fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:44.281 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.276+0000 7fefb39f5700 1 --2- 172.21.15.112:0/2751819601 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef980038e0 0x7fefac141fd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fef9800bf20 tx=0x7fef9c01a000 comp rx=0 tx=0).stop 2023-01-24T03:30:44.281 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.276+0000 7fefb39f5700 1 -- 172.21.15.112:0/2751819601 shutdown_connections 2023-01-24T03:30:44.282 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.276+0000 7fefb39f5700 1 --2- 172.21.15.112:0/2751819601 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fef8c0611c0 0x7fef8c063670 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:44.282 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.276+0000 7fefb39f5700 1 --2- 172.21.15.112:0/2751819601 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef980038e0 0x7fefac141fd0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:44.282 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.276+0000 7fefb39f5700 1 --2- 172.21.15.112:0/2751819601 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fefac157e30 0x7fefac165410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:44.282 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.276+0000 7fefb39f5700 1 --2- 172.21.15.112:0/2751819601 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fefac14c200 0x7fefac1578f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:44.282 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.276+0000 7fefb39f5700 1 -- 172.21.15.112:0/2751819601 >> 172.21.15.112:0/2751819601 conn(0x7fefac06daf0 msgr2=0x7fefac14ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:44.283 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.276+0000 7fefb39f5700 1 -- 172.21.15.112:0/2751819601 shutdown_connections 2023-01-24T03:30:44.283 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.276+0000 7fefb39f5700 1 -- 172.21.15.112:0/2751819601 wait complete. 2023-01-24T03:30:44.291 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-01-24T03:30:14.931166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-01-24T03:30:15.023730+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-01-24T03:30:19.060597+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-01-24T03:30:19.943686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-01-24T03:30:22.999036+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-01-24T03:30:44.292 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-01-24T03:30:44.408 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.405+0000 7f703104f700 1 Processor -- start 2023-01-24T03:30:44.408 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.405+0000 7f703104f700 1 -- start start 2023-01-24T03:30:44.410 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.406+0000 7f703104f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f702c14f3f0 0x7f702c0c4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:44.411 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.406+0000 7f703104f700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f702c0c6420 0x7f702c155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:44.411 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.406+0000 7f703104f700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f702c0c82e0 con 0x7f702c0c8770 2023-01-24T03:30:44.411 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.406+0000 7f703104f700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f702c1560e0 con 0x7f702c0c6420 2023-01-24T03:30:44.412 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.406+0000 7f703104f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f702c0c2980 con 0x7f702c14f3f0 2023-01-24T03:30:44.412 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.406+0000 7f702b7fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f702c0c6420 0x7f702c155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:44.412 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.406+0000 7f702bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f702c14f3f0 0x7f702c0c4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:44.412 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.406+0000 7f702b7fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f702c0c6420 0x7f702c155ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.112:43360/0 (socket says 172.21.15.112:43360) 2023-01-24T03:30:44.413 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.406+0000 7f702b7fe700 1 -- 172.21.15.112:0/3514917160 learned_addr learned my addr 172.21.15.112:0/3514917160 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:44.413 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.406+0000 7f702b7fe700 1 -- 172.21.15.112:0/3514917160 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f702c14f3f0 msgr2=0x7f702c0c4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:44.413 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.406+0000 7f702b7fe700 1 --2- 172.21.15.112:0/3514917160 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f702c14f3f0 0x7f702c0c4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:44.413 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.406+0000 7f702b7fe700 1 -- 172.21.15.112:0/3514917160 >> v1:172.21.15.142:6789/0 conn(0x7f702c0c8770 legacy=0x7f702c0c2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:44.413 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.406+0000 7f702b7fe700 1 -- 172.21.15.112:0/3514917160 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f702c0c8c10 con 0x7f702c0c6420 2023-01-24T03:30:44.414 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.407+0000 7f702b7fe700 1 --2- 172.21.15.112:0/3514917160 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f702c0c6420 0x7f702c155ba0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7020006fd0 tx=0x7f70200090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b3bddc881cd19a3 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:44.414 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.407+0000 7f702affd700 1 -- 172.21.15.112:0/3514917160 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7020011630 con 0x7f702c0c6420 2023-01-24T03:30:44.414 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.407+0000 7f702affd700 1 -- 172.21.15.112:0/3514917160 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70200117d0 con 0x7f702c0c6420 2023-01-24T03:30:44.414 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.407+0000 7f702affd700 1 -- 172.21.15.112:0/3514917160 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7020011b90 con 0x7f702c0c6420 2023-01-24T03:30:44.414 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.407+0000 7f703104f700 1 -- 172.21.15.112:0/3514917160 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f702c0c6420 msgr2=0x7f702c155ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:44.414 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.407+0000 7f703104f700 1 --2- 172.21.15.112:0/3514917160 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f702c0c6420 0x7f702c155ba0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7020006fd0 tx=0x7f70200090f0 comp rx=0 tx=0).stop 2023-01-24T03:30:44.415 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.407+0000 7f703104f700 1 -- 172.21.15.112:0/3514917160 shutdown_connections 2023-01-24T03:30:44.415 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.407+0000 7f703104f700 1 --2- 172.21.15.112:0/3514917160 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f702c14f3f0 0x7f702c0c4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:44.415 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.407+0000 7f703104f700 1 --2- 172.21.15.112:0/3514917160 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f702c0c6420 0x7f702c155ba0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:44.415 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.407+0000 7f703104f700 1 -- 172.21.15.112:0/3514917160 >> 172.21.15.112:0/3514917160 conn(0x7f702c05d830 msgr2=0x7f702c05dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:44.415 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.407+0000 7f703104f700 1 -- 172.21.15.112:0/3514917160 shutdown_connections 2023-01-24T03:30:44.416 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.407+0000 7f703104f700 1 -- 172.21.15.112:0/3514917160 wait complete. 2023-01-24T03:30:44.418 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.408+0000 7f703104f700 1 Processor -- start 2023-01-24T03:30:44.418 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.408+0000 7f703104f700 1 -- start start 2023-01-24T03:30:44.419 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.409+0000 7f703104f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f702c0c6420 0x7f702c146d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:44.419 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.409+0000 7f703104f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f702c0c8770 0x7f702c166a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:44.419 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.409+0000 7f703104f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f702c14f3f0 0x7f702c166fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:44.419 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.409+0000 7f703104f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f702c0cba00 con 0x7f702c0c6420 2023-01-24T03:30:44.419 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.409+0000 7f703104f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f702c0cae40 con 0x7f702c0c8770 2023-01-24T03:30:44.420 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.409+0000 7f703104f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f702c0ca230 con 0x7f702c14f3f0 2023-01-24T03:30:44.420 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.409+0000 7f702bfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f702c0c6420 0x7f702c146d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:44.420 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.409+0000 7f702bfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f702c0c6420 0x7f702c146d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50874/0 (socket says 172.21.15.112:50874) 2023-01-24T03:30:44.420 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.409+0000 7f702bfff700 1 -- 172.21.15.112:0/2424878035 learned_addr learned my addr 172.21.15.112:0/2424878035 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:44.420 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.409+0000 7f702b7fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f702c0c8770 0x7f702c166a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:44.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.409+0000 7f703084e700 1 --2- 172.21.15.112:0/2424878035 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f702c14f3f0 0x7f702c166fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:44.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.410+0000 7f702bfff700 1 -- 172.21.15.112:0/2424878035 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f702c14f3f0 msgr2=0x7f702c166fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:44.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.410+0000 7f702bfff700 1 --2- 172.21.15.112:0/2424878035 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f702c14f3f0 0x7f702c166fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:44.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.410+0000 7f702bfff700 1 -- 172.21.15.112:0/2424878035 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f702c0c8770 msgr2=0x7f702c166a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:44.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.410+0000 7f702bfff700 1 --2- 172.21.15.112:0/2424878035 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f702c0c8770 0x7f702c166a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:44.422 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.410+0000 7f702bfff700 1 -- 172.21.15.112:0/2424878035 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7020009370 con 0x7f702c0c6420 2023-01-24T03:30:44.422 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.410+0000 7f702b7fe700 1 --2- 172.21.15.112:0/2424878035 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f702c0c8770 0x7f702c166a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:30:44.422 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.410+0000 7f702bfff700 1 --2- 172.21.15.112:0/2424878035 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f702c0c6420 0x7f702c146d40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f701c00e930 tx=0x7f701c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:44.422 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.410+0000 7f7013fff700 1 -- 172.21.15.112:0/2424878035 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f701c013680 con 0x7f702c0c6420 2023-01-24T03:30:44.422 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.410+0000 7f7013fff700 1 -- 172.21.15.112:0/2424878035 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f701c0137e0 con 0x7f702c0c6420 2023-01-24T03:30:44.423 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.410+0000 7f703104f700 1 -- 172.21.15.112:0/2424878035 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f702c0ca510 con 0x7f702c0c6420 2023-01-24T03:30:44.423 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.410+0000 7f7013fff700 1 -- 172.21.15.112:0/2424878035 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f701c013b60 con 0x7f702c0c6420 2023-01-24T03:30:44.423 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.410+0000 7f703104f700 1 -- 172.21.15.112:0/2424878035 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f702c0caad0 con 0x7f702c0c6420 2023-01-24T03:30:44.423 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.411+0000 7f703104f700 1 -- 172.21.15.112:0/2424878035 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f702c05ea20 con 0x7f702c0c6420 2023-01-24T03:30:44.423 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.411+0000 7f7013fff700 1 -- 172.21.15.112:0/2424878035 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f701c019030 con 0x7f702c0c6420 2023-01-24T03:30:44.424 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.411+0000 7f7013fff700 1 --2- 172.21.15.112:0/2424878035 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f700c0611f0 0x7f700c0636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:44.424 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.412+0000 7f7013fff700 1 -- 172.21.15.112:0/2424878035 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f701c07d740 con 0x7f702c0c6420 2023-01-24T03:30:44.424 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.412+0000 7f702b7fe700 1 --2- 172.21.15.112:0/2424878035 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f700c0611f0 0x7f700c0636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:44.424 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.412+0000 7f702b7fe700 1 --2- 172.21.15.112:0/2424878035 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f700c0611f0 0x7f700c0636a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7020009340 tx=0x7f7020016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:44.424 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.416+0000 7f7013fff700 1 -- 172.21.15.112:0/2424878035 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f701c04d580 con 0x7f702c0c6420 2023-01-24T03:30:44.595 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.592+0000 7f703104f700 1 -- 172.21.15.112:0/2424878035 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f702c0c5290 con 0x7f702c0c6420 2023-01-24T03:30:44.596 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.592+0000 7f7013fff700 1 -- 172.21.15.112:0/2424878035 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v24) v1 ==== 92+0+10 (secure 0 0 0) 0x7f701c050ba0 con 0x7f702c0c6420 2023-01-24T03:30:44.596 INFO:teuthology.orchestra.run.smithi112.stdout:pg_num: 8 2023-01-24T03:30:44.597 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.594+0000 7f703104f700 1 -- 172.21.15.112:0/2424878035 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f700c0611f0 msgr2=0x7f700c0636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:44.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.594+0000 7f703104f700 1 --2- 172.21.15.112:0/2424878035 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f700c0611f0 0x7f700c0636a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7020009340 tx=0x7f7020016000 comp rx=0 tx=0).stop 2023-01-24T03:30:44.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.594+0000 7f703104f700 1 -- 172.21.15.112:0/2424878035 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f702c0c6420 msgr2=0x7f702c146d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:44.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.594+0000 7f703104f700 1 --2- 172.21.15.112:0/2424878035 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f702c0c6420 0x7f702c146d40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f701c00e930 tx=0x7f701c009060 comp rx=0 tx=0).stop 2023-01-24T03:30:44.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.594+0000 7f703104f700 1 -- 172.21.15.112:0/2424878035 shutdown_connections 2023-01-24T03:30:44.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.594+0000 7f703104f700 1 --2- 172.21.15.112:0/2424878035 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f700c0611f0 0x7f700c0636a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:44.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.594+0000 7f703104f700 1 --2- 172.21.15.112:0/2424878035 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f702c0c6420 0x7f702c146d40 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:44.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.594+0000 7f703104f700 1 --2- 172.21.15.112:0/2424878035 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f702c14f3f0 0x7f702c166fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:44.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.594+0000 7f703104f700 1 --2- 172.21.15.112:0/2424878035 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f702c0c8770 0x7f702c166a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:44.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.594+0000 7f703104f700 1 -- 172.21.15.112:0/2424878035 >> 172.21.15.112:0/2424878035 conn(0x7f702c05d830 msgr2=0x7f702c14b230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:44.600 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.594+0000 7f703104f700 1 -- 172.21.15.112:0/2424878035 shutdown_connections 2023-01-24T03:30:44.600 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.594+0000 7f703104f700 1 -- 172.21.15.112:0/2424878035 wait complete. 2023-01-24T03:30:44.609 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-01-24T03:30:44.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.722+0000 7f8800aab700 1 Processor -- start 2023-01-24T03:30:44.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.722+0000 7f8800aab700 1 -- start start 2023-01-24T03:30:44.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.723+0000 7f8800aab700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f87fc157120 0x7f87fc0c6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:44.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.723+0000 7f8800aab700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87fc0c5880 con 0x7f87fc06c4a0 2023-01-24T03:30:44.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.723+0000 7f8800aab700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87fc0c37c0 con 0x7f87fc156d50 2023-01-24T03:30:44.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.723+0000 7f8800aab700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87fc06c0f0 con 0x7f87fc157120 2023-01-24T03:30:44.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.723+0000 7f87faffd700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f87fc157120 0x7f87fc0c6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:44.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.723+0000 7f87fbfff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f87fc06c4a0 0x7f87fc15d840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:60102/0 (socket says 172.21.15.112:60102) 2023-01-24T03:30:44.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.723+0000 7f87faffd700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f87fc157120 0x7f87fc0c6090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50880/0 (socket says 172.21.15.112:50880) 2023-01-24T03:30:44.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.723+0000 7f87fbfff700 1 -- 172.21.15.112:0/865094557 learned_addr learned my addr 172.21.15.112:0/865094557 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:44.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.723+0000 7f87faffd700 1 -- 172.21.15.112:0/865094557 >> v1:172.21.15.142:6789/0 conn(0x7f87fc156d50 legacy=0x7f87fc0c30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:44.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.723+0000 7f87faffd700 1 -- 172.21.15.112:0/865094557 >> v1:172.21.15.112:6789/0 conn(0x7f87fc06c4a0 legacy=0x7f87fc15d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:44.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.723+0000 7f87faffd700 1 -- 172.21.15.112:0/865094557 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87fc0bd360 con 0x7f87fc157120 2023-01-24T03:30:44.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.723+0000 7f87fa7fc700 1 -- 172.21.15.112:0/865094557 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2541327303 0 0) 0x7f87fc0c5880 con 0x7f87fc06c4a0 2023-01-24T03:30:44.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.723+0000 7f87faffd700 1 --2- 172.21.15.112:0/865094557 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f87fc157120 0x7f87fc0c6090 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f87f0009760 tx=0x7f87f0009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4baab705bb020c4d server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:44.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.723+0000 7f87fa7fc700 1 -- 172.21.15.112:0/865094557 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f0011630 con 0x7f87fc157120 2023-01-24T03:30:44.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.723+0000 7f87fa7fc700 1 -- 172.21.15.112:0/865094557 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f0011790 con 0x7f87fc157120 2023-01-24T03:30:44.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.723+0000 7f87fa7fc700 1 -- 172.21.15.112:0/865094557 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f0011b30 con 0x7f87fc157120 2023-01-24T03:30:44.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.724+0000 7f8800aab700 1 -- 172.21.15.112:0/865094557 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f87fc157120 msgr2=0x7f87fc0c6090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:44.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.724+0000 7f8800aab700 1 --2- 172.21.15.112:0/865094557 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f87fc157120 0x7f87fc0c6090 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f87f0009760 tx=0x7f87f0009060 comp rx=0 tx=0).stop 2023-01-24T03:30:44.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.724+0000 7f8800aab700 1 -- 172.21.15.112:0/865094557 shutdown_connections 2023-01-24T03:30:44.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.724+0000 7f8800aab700 1 --2- 172.21.15.112:0/865094557 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f87fc157120 0x7f87fc0c6090 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:44.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.724+0000 7f8800aab700 1 -- 172.21.15.112:0/865094557 >> 172.21.15.112:0/865094557 conn(0x7f87fc06ce00 msgr2=0x7f87fc06d200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:44.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.724+0000 7f8800aab700 1 -- 172.21.15.112:0/865094557 shutdown_connections 2023-01-24T03:30:44.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.724+0000 7f8800aab700 1 -- 172.21.15.112:0/865094557 wait complete. 2023-01-24T03:30:44.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.725+0000 7f8800aab700 1 Processor -- start 2023-01-24T03:30:44.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.725+0000 7f8800aab700 1 -- start start 2023-01-24T03:30:44.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.726+0000 7f8800aab700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f87fc06c4a0 0x7f87fc0c03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:44.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.726+0000 7f8800aab700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f87fc156d50 0x7f87fc1647f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:44.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.726+0000 7f8800aab700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f87fc157120 0x7f87fc164d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:44.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.726+0000 7f8800aab700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc0a6260 con 0x7f87fc157120 2023-01-24T03:30:44.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.726+0000 7f8800aab700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc0aa970 con 0x7f87fc156d50 2023-01-24T03:30:44.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.726+0000 7f8800aab700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc0a5620 con 0x7f87fc06c4a0 2023-01-24T03:30:44.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.726+0000 7f87fbfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f87fc157120 0x7f87fc164d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:44.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.726+0000 7f87fbfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f87fc157120 0x7f87fc164d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50888/0 (socket says 172.21.15.112:50888) 2023-01-24T03:30:44.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.726+0000 7f87fbfff700 1 -- 172.21.15.112:0/800484736 learned_addr learned my addr 172.21.15.112:0/800484736 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:44.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.726+0000 7f87faffd700 1 --2- 172.21.15.112:0/800484736 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f87fc156d50 0x7f87fc1647f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:44.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.726+0000 7f87fb7fe700 1 --2- 172.21.15.112:0/800484736 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f87fc06c4a0 0x7f87fc0c03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:44.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.726+0000 7f87fbfff700 1 -- 172.21.15.112:0/800484736 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f87fc06c4a0 msgr2=0x7f87fc0c03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:44.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.726+0000 7f87fbfff700 1 --2- 172.21.15.112:0/800484736 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f87fc06c4a0 0x7f87fc0c03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:44.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.726+0000 7f87fbfff700 1 -- 172.21.15.112:0/800484736 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f87fc156d50 msgr2=0x7f87fc1647f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:44.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.726+0000 7f87fbfff700 1 --2- 172.21.15.112:0/800484736 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f87fc156d50 0x7f87fc1647f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:44.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.726+0000 7f87fbfff700 1 -- 172.21.15.112:0/800484736 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f00092a0 con 0x7f87fc157120 2023-01-24T03:30:44.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.726+0000 7f87fbfff700 1 --2- 172.21.15.112:0/800484736 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f87fc157120 0x7f87fc164d30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f87f400c840 tx=0x7f87f4007f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:44.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.726+0000 7f87e37fe700 1 -- 172.21.15.112:0/800484736 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f4005d30 con 0x7f87fc157120 2023-01-24T03:30:44.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.727+0000 7f87e37fe700 1 -- 172.21.15.112:0/800484736 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f4005e90 con 0x7f87fc157120 2023-01-24T03:30:44.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.727+0000 7f8800aab700 1 -- 172.21.15.112:0/800484736 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87fc0a58c0 con 0x7f87fc157120 2023-01-24T03:30:44.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.727+0000 7f87e37fe700 1 -- 172.21.15.112:0/800484736 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f4007340 con 0x7f87fc157120 2023-01-24T03:30:44.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.727+0000 7f8800aab700 1 -- 172.21.15.112:0/800484736 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87fc0a5e80 con 0x7f87fc157120 2023-01-24T03:30:44.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.727+0000 7f8800aab700 1 -- 172.21.15.112:0/800484736 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87fc0a63a0 con 0x7f87fc157120 2023-01-24T03:30:44.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.728+0000 7f87e37fe700 1 -- 172.21.15.112:0/800484736 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f87f4010070 con 0x7f87fc157120 2023-01-24T03:30:44.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.728+0000 7f87e37fe700 1 --2- 172.21.15.112:0/800484736 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f87dc0611f0 0x7f87dc0636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:44.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.728+0000 7f87e37fe700 1 -- 172.21.15.112:0/800484736 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f87f407d200 con 0x7f87fc157120 2023-01-24T03:30:44.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.728+0000 7f87fb7fe700 1 --2- 172.21.15.112:0/800484736 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f87dc0611f0 0x7f87dc0636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:44.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.729+0000 7f87fb7fe700 1 --2- 172.21.15.112:0/800484736 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f87dc0611f0 0x7f87dc0636a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f87ec0038b0 tx=0x7f87ec007d20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:44.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.732+0000 7f87e37fe700 1 -- 172.21.15.112:0/800484736 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f87fc0a63a0 con 0x7f87fc157120 2023-01-24T03:30:44.912 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.909+0000 7f8800aab700 1 -- 172.21.15.112:0/800484736 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f87fc0a63a0 con 0x7f87fc157120 2023-01-24T03:30:44.913 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.909+0000 7f87e37fe700 1 -- 172.21.15.112:0/800484736 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f87fc0a63a0 con 0x7f87fc157120 2023-01-24T03:30:44.913 INFO:teuthology.orchestra.run.smithi112.stdout:pg_num: 1 2023-01-24T03:30:44.915 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.912+0000 7f8800aab700 1 -- 172.21.15.112:0/800484736 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f87dc0611f0 msgr2=0x7f87dc0636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:44.916 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.912+0000 7f8800aab700 1 --2- 172.21.15.112:0/800484736 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f87dc0611f0 0x7f87dc0636a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f87ec0038b0 tx=0x7f87ec007d20 comp rx=0 tx=0).stop 2023-01-24T03:30:44.916 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.912+0000 7f8800aab700 1 -- 172.21.15.112:0/800484736 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f87fc157120 msgr2=0x7f87fc164d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:44.916 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.912+0000 7f8800aab700 1 --2- 172.21.15.112:0/800484736 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f87fc157120 0x7f87fc164d30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f87f400c840 tx=0x7f87f4007f30 comp rx=0 tx=0).stop 2023-01-24T03:30:44.916 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.912+0000 7f8800aab700 1 -- 172.21.15.112:0/800484736 shutdown_connections 2023-01-24T03:30:44.917 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.912+0000 7f8800aab700 1 --2- 172.21.15.112:0/800484736 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f87dc0611f0 0x7f87dc0636a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:44.917 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.912+0000 7f8800aab700 1 --2- 172.21.15.112:0/800484736 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f87fc157120 0x7f87fc164d30 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:44.917 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.912+0000 7f8800aab700 1 --2- 172.21.15.112:0/800484736 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f87fc06c4a0 0x7f87fc0c03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:44.917 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.912+0000 7f8800aab700 1 --2- 172.21.15.112:0/800484736 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f87fc156d50 0x7f87fc1647f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:44.917 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.912+0000 7f8800aab700 1 -- 172.21.15.112:0/800484736 >> 172.21.15.112:0/800484736 conn(0x7f87fc06ce00 msgr2=0x7f87fc0c26a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:44.918 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.912+0000 7f8800aab700 1 -- 172.21.15.112:0/800484736 shutdown_connections 2023-01-24T03:30:44.918 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:44.912+0000 7f8800aab700 1 -- 172.21.15.112:0/800484736 wait complete. 2023-01-24T03:30:44.927 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-01-24T03:30:45.049 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.043+0000 7ff18759e700 1 Processor -- start 2023-01-24T03:30:45.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.043+0000 7ff18759e700 1 -- start start 2023-01-24T03:30:45.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.044+0000 7ff18759e700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7ff1880c6250 0x7ff1880c5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:45.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.044+0000 7ff18759e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff1880c5bb0 0x7ff1881559a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:45.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.044+0000 7ff18759e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7ff18806c4a0 0x7ff188155ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:45.051 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.044+0000 7ff18759e700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff188068930 con 0x7ff1880c6250 2023-01-24T03:30:45.051 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.044+0000 7ff18759e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff188068580 con 0x7ff18806c4a0 2023-01-24T03:30:45.051 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.044+0000 7ff18759e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff18806bb70 con 0x7ff1880c5bb0 2023-01-24T03:30:45.052 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.044+0000 7ff18659c700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7ff1880c6250 0x7ff1880c5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:45.052 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.044+0000 7ff18659c700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7ff1880c6250 0x7ff1880c5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50892/0 (socket says 172.21.15.112:50892) 2023-01-24T03:30:45.052 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.044+0000 7ff18659c700 1 -- 172.21.15.112:0/2029456960 learned_addr learned my addr 172.21.15.112:0/2029456960 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:45.053 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.044+0000 7ff185d9b700 1 --2- 172.21.15.112:0/2029456960 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff1880c5bb0 0x7ff1881559a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:45.053 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.044+0000 7ff186d9d700 1 --2- 172.21.15.112:0/2029456960 >> v2:172.21.15.142:3300/0 conn(0x7ff18806c4a0 0x7ff188155ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:45.054 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.044+0000 7ff18659c700 1 -- 172.21.15.112:0/2029456960 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff1880c5bb0 msgr2=0x7ff1881559a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:45.054 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.044+0000 7ff18659c700 1 --2- 172.21.15.112:0/2029456960 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff1880c5bb0 0x7ff1881559a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:45.054 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.044+0000 7ff18659c700 1 -- 172.21.15.112:0/2029456960 >> v2:172.21.15.142:3300/0 conn(0x7ff18806c4a0 msgr2=0x7ff188155ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:45.054 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.044+0000 7ff18659c700 1 --2- 172.21.15.112:0/2029456960 >> v2:172.21.15.142:3300/0 conn(0x7ff18806c4a0 0x7ff188155ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:45.055 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.044+0000 7ff18659c700 1 -- 172.21.15.112:0/2029456960 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff188069770 con 0x7ff1880c6250 2023-01-24T03:30:45.055 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.044+0000 7ff18659c700 1 --2- 172.21.15.112:0/2029456960 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1880c6250 0x7ff1880c5670 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff18806e220 tx=0x7ff180009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae42be1906947e2a server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:45.055 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.044+0000 7ff18559a700 1 -- 172.21.15.112:0/2029456960 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff18000e070 con 0x7ff1880c6250 2023-01-24T03:30:45.055 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.044+0000 7ff18559a700 1 -- 172.21.15.112:0/2029456960 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff18000ba50 con 0x7ff1880c6250 2023-01-24T03:30:45.055 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.044+0000 7ff18559a700 1 -- 172.21.15.112:0/2029456960 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1800092e0 con 0x7ff1880c6250 2023-01-24T03:30:45.056 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.045+0000 7ff18759e700 1 -- 172.21.15.112:0/2029456960 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1880c6250 msgr2=0x7ff1880c5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:45.056 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.045+0000 7ff18759e700 1 --2- 172.21.15.112:0/2029456960 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1880c6250 0x7ff1880c5670 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff18806e220 tx=0x7ff180009840 comp rx=0 tx=0).stop 2023-01-24T03:30:45.056 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.045+0000 7ff18759e700 1 -- 172.21.15.112:0/2029456960 shutdown_connections 2023-01-24T03:30:45.056 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.045+0000 7ff18759e700 1 --2- 172.21.15.112:0/2029456960 >> v2:172.21.15.142:3300/0 conn(0x7ff18806c4a0 0x7ff188155ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:45.057 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.045+0000 7ff18759e700 1 --2- 172.21.15.112:0/2029456960 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1880c6250 0x7ff1880c5670 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:45.057 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.045+0000 7ff18759e700 1 --2- 172.21.15.112:0/2029456960 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff1880c5bb0 0x7ff1881559a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:45.057 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.045+0000 7ff18759e700 1 -- 172.21.15.112:0/2029456960 >> 172.21.15.112:0/2029456960 conn(0x7ff18806ce00 msgr2=0x7ff18806d200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:45.057 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.045+0000 7ff18759e700 1 -- 172.21.15.112:0/2029456960 shutdown_connections 2023-01-24T03:30:45.057 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.045+0000 7ff18759e700 1 -- 172.21.15.112:0/2029456960 wait complete. 2023-01-24T03:30:45.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.046+0000 7ff18759e700 1 Processor -- start 2023-01-24T03:30:45.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.046+0000 7ff18759e700 1 -- start start 2023-01-24T03:30:45.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.047+0000 7ff18759e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff18806c4a0 0x7ff188165470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:45.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.047+0000 7ff18759e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1880c5bb0 0x7ff1880c9780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:45.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.047+0000 7ff18759e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1880c6250 0x7ff188166940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:45.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.047+0000 7ff18759e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1880aaa40 con 0x7ff1880c5bb0 2023-01-24T03:30:45.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.047+0000 7ff18759e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1880a98b0 con 0x7ff1880c6250 2023-01-24T03:30:45.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.047+0000 7ff18759e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1880ae760 con 0x7ff18806c4a0 2023-01-24T03:30:45.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.047+0000 7ff185d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1880c5bb0 0x7ff1880c9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:45.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.047+0000 7ff185d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1880c5bb0 0x7ff1880c9780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50900/0 (socket says 172.21.15.112:50900) 2023-01-24T03:30:45.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.047+0000 7ff185d9b700 1 -- 172.21.15.112:0/1830762897 learned_addr learned my addr 172.21.15.112:0/1830762897 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:45.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.047+0000 7ff186d9d700 1 --2- 172.21.15.112:0/1830762897 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1880c6250 0x7ff188166940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:45.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.047+0000 7ff18659c700 1 --2- 172.21.15.112:0/1830762897 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff18806c4a0 0x7ff188165470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:45.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.047+0000 7ff185d9b700 1 -- 172.21.15.112:0/1830762897 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff18806c4a0 msgr2=0x7ff188165470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:45.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.047+0000 7ff185d9b700 1 --2- 172.21.15.112:0/1830762897 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff18806c4a0 0x7ff188165470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:45.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.047+0000 7ff185d9b700 1 -- 172.21.15.112:0/1830762897 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1880c6250 msgr2=0x7ff188166940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:45.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.047+0000 7ff185d9b700 1 --2- 172.21.15.112:0/1830762897 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1880c6250 0x7ff188166940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:45.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.047+0000 7ff185d9b700 1 -- 172.21.15.112:0/1830762897 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff180009ae0 con 0x7ff1880c5bb0 2023-01-24T03:30:45.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.047+0000 7ff185d9b700 1 --2- 172.21.15.112:0/1830762897 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1880c5bb0 0x7ff1880c9780 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff17c00c850 tx=0x7ff17c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:45.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.047+0000 7ff1767fc700 1 -- 172.21.15.112:0/1830762897 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17c011600 con 0x7ff1880c5bb0 2023-01-24T03:30:45.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.047+0000 7ff1767fc700 1 -- 172.21.15.112:0/1830762897 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff17c011760 con 0x7ff1880c5bb0 2023-01-24T03:30:45.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.047+0000 7ff18759e700 1 -- 172.21.15.112:0/1830762897 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1880a9b50 con 0x7ff1880c5bb0 2023-01-24T03:30:45.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.047+0000 7ff1767fc700 1 -- 172.21.15.112:0/1830762897 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17c011ae0 con 0x7ff1880c5bb0 2023-01-24T03:30:45.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.047+0000 7ff18759e700 1 -- 172.21.15.112:0/1830762897 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1880aa110 con 0x7ff1880c5bb0 2023-01-24T03:30:45.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.048+0000 7ff18759e700 1 -- 172.21.15.112:0/1830762897 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff18805ea20 con 0x7ff1880c5bb0 2023-01-24T03:30:45.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.049+0000 7ff1767fc700 1 -- 172.21.15.112:0/1830762897 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7ff17c0223a0 con 0x7ff1880c5bb0 2023-01-24T03:30:45.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.049+0000 7ff1767fc700 1 --2- 172.21.15.112:0/1830762897 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff1640611f0 0x7ff1640636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:45.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.049+0000 7ff1767fc700 1 -- 172.21.15.112:0/1830762897 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7ff17c07d7e0 con 0x7ff1880c5bb0 2023-01-24T03:30:45.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.049+0000 7ff18659c700 1 --2- 172.21.15.112:0/1830762897 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff1640611f0 0x7ff1640636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:45.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.049+0000 7ff18659c700 1 --2- 172.21.15.112:0/1830762897 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff1640611f0 0x7ff1640636a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff180009880 tx=0x7ff180018000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:45.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.053+0000 7ff1767fc700 1 -- 172.21.15.112:0/1830762897 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7ff17c04d5c0 con 0x7ff1880c5bb0 2023-01-24T03:30:45.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.229+0000 7ff18759e700 1 -- 172.21.15.112:0/1830762897 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff1880692a0 con 0x7ff1880c5bb0 2023-01-24T03:30:45.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.230+0000 7ff1767fc700 1 -- 172.21.15.112:0/1830762897 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff17c01e020 con 0x7ff1880c5bb0 2023-01-24T03:30:45.234 INFO:teuthology.orchestra.run.smithi112.stdout:pg_num: 32 2023-01-24T03:30:45.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.232+0000 7ff18759e700 1 -- 172.21.15.112:0/1830762897 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff1640611f0 msgr2=0x7ff1640636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:45.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.232+0000 7ff18759e700 1 --2- 172.21.15.112:0/1830762897 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff1640611f0 0x7ff1640636a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff180009880 tx=0x7ff180018000 comp rx=0 tx=0).stop 2023-01-24T03:30:45.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.232+0000 7ff18759e700 1 -- 172.21.15.112:0/1830762897 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1880c5bb0 msgr2=0x7ff1880c9780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:45.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.232+0000 7ff18759e700 1 --2- 172.21.15.112:0/1830762897 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1880c5bb0 0x7ff1880c9780 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff17c00c850 tx=0x7ff17c008040 comp rx=0 tx=0).stop 2023-01-24T03:30:45.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.232+0000 7ff18759e700 1 -- 172.21.15.112:0/1830762897 shutdown_connections 2023-01-24T03:30:45.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.232+0000 7ff18759e700 1 --2- 172.21.15.112:0/1830762897 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff1640611f0 0x7ff1640636a0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:45.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.232+0000 7ff18759e700 1 --2- 172.21.15.112:0/1830762897 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1880c6250 0x7ff188166940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:45.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.232+0000 7ff18759e700 1 --2- 172.21.15.112:0/1830762897 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff18806c4a0 0x7ff188165470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:45.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.232+0000 7ff18759e700 1 --2- 172.21.15.112:0/1830762897 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1880c5bb0 0x7ff1880c9780 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:45.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.232+0000 7ff18759e700 1 -- 172.21.15.112:0/1830762897 >> 172.21.15.112:0/1830762897 conn(0x7ff18806ce00 msgr2=0x7ff18806d380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:45.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.232+0000 7ff18759e700 1 -- 172.21.15.112:0/1830762897 shutdown_connections 2023-01-24T03:30:45.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.232+0000 7ff18759e700 1 -- 172.21.15.112:0/1830762897 wait complete. 2023-01-24T03:30:45.247 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-01-24T03:30:45.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.364+0000 7f2cfd0a6700 1 Processor -- start 2023-01-24T03:30:45.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.364+0000 7f2cfd0a6700 1 -- start start 2023-01-24T03:30:45.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.365+0000 7f2cfd0a6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2cf805e420 0x7f2cf8069890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:45.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.365+0000 7f2cfd0a6700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f2cf8068d20 0x7f2cf8155f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:45.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.365+0000 7f2cfd0a6700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cf8155ab0 con 0x7f2cf80694c0 2023-01-24T03:30:45.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.365+0000 7f2cfd0a6700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cf8155550 con 0x7f2cf8068d20 2023-01-24T03:30:45.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.365+0000 7f2cfd0a6700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf8156480 con 0x7f2cf805e420 2023-01-24T03:30:45.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.365+0000 7f2cf759e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f2cf8068d20 0x7f2cf8155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:45.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.365+0000 7f2cf659c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2cf805e420 0x7f2cf8069890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:45.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.365+0000 7f2cf759e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f2cf8068d20 0x7f2cf8155f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.112:43404/0 (socket says 172.21.15.112:43404) 2023-01-24T03:30:45.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.365+0000 7f2cf759e700 1 -- 172.21.15.112:0/3455888016 learned_addr learned my addr 172.21.15.112:0/3455888016 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:45.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.365+0000 7f2cf6d9d700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f2cf80694c0 0x7f2cf80c68d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.112:45684/0 (socket says 172.21.15.112:45684) 2023-01-24T03:30:45.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.365+0000 7f2cf759e700 1 -- 172.21.15.112:0/3455888016 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2cf805e420 msgr2=0x7f2cf8069890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:45.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.365+0000 7f2cf759e700 1 --2- 172.21.15.112:0/3455888016 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2cf805e420 0x7f2cf8069890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:45.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.365+0000 7f2cf759e700 1 -- 172.21.15.112:0/3455888016 >> v1:172.21.15.142:6789/0 conn(0x7f2cf80694c0 legacy=0x7f2cf80c68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:45.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.365+0000 7f2cf759e700 1 -- 172.21.15.112:0/3455888016 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf80c70e0 con 0x7f2cf8068d20 2023-01-24T03:30:45.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.366+0000 7f2cf759e700 1 --2- 172.21.15.112:0/3455888016 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2cf8068d20 0x7f2cf8155f40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8006fd0 tx=0x7f2ce80090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a775004346f71553 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:45.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.366+0000 7f2cf5d9b700 1 -- 172.21.15.112:0/3455888016 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce8011630 con 0x7f2cf8068d20 2023-01-24T03:30:45.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.366+0000 7f2cf5d9b700 1 -- 172.21.15.112:0/3455888016 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ce80117d0 con 0x7f2cf8068d20 2023-01-24T03:30:45.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.366+0000 7f2cf5d9b700 1 -- 172.21.15.112:0/3455888016 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce8011b90 con 0x7f2cf8068d20 2023-01-24T03:30:45.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.366+0000 7f2cfd0a6700 1 -- 172.21.15.112:0/3455888016 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2cf8068d20 msgr2=0x7f2cf8155f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:45.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.366+0000 7f2cfd0a6700 1 --2- 172.21.15.112:0/3455888016 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2cf8068d20 0x7f2cf8155f40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8006fd0 tx=0x7f2ce80090f0 comp rx=0 tx=0).stop 2023-01-24T03:30:45.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.366+0000 7f2cfd0a6700 1 -- 172.21.15.112:0/3455888016 shutdown_connections 2023-01-24T03:30:45.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.366+0000 7f2cfd0a6700 1 --2- 172.21.15.112:0/3455888016 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2cf8068d20 0x7f2cf8155f40 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:45.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.366+0000 7f2cfd0a6700 1 --2- 172.21.15.112:0/3455888016 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2cf805e420 0x7f2cf8069890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:45.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.366+0000 7f2cfd0a6700 1 -- 172.21.15.112:0/3455888016 >> 172.21.15.112:0/3455888016 conn(0x7f2cf805d830 msgr2=0x7f2cf805dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:45.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.366+0000 7f2cfd0a6700 1 -- 172.21.15.112:0/3455888016 shutdown_connections 2023-01-24T03:30:45.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.366+0000 7f2cfd0a6700 1 -- 172.21.15.112:0/3455888016 wait complete. 2023-01-24T03:30:45.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.367+0000 7f2cfd0a6700 1 Processor -- start 2023-01-24T03:30:45.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.367+0000 7f2cfd0a6700 1 -- start start 2023-01-24T03:30:45.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.368+0000 7f2cfd0a6700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2cf805e420 0x7f2cf814cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:45.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.368+0000 7f2cfd0a6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2cf8068d20 0x7f2cf81658d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:45.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.368+0000 7f2cfd0a6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2cf80694c0 0x7f2cf8165e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:45.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.368+0000 7f2cfd0a6700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf8165440 con 0x7f2cf80694c0 2023-01-24T03:30:45.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.368+0000 7f2cfd0a6700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf814ff10 con 0x7f2cf805e420 2023-01-24T03:30:45.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.368+0000 7f2cfd0a6700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf81501b0 con 0x7f2cf8068d20 2023-01-24T03:30:45.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.368+0000 7f2cf759e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2cf80694c0 0x7f2cf8165e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:45.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.368+0000 7f2cf759e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2cf80694c0 0x7f2cf8165e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50916/0 (socket says 172.21.15.112:50916) 2023-01-24T03:30:45.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.368+0000 7f2cf759e700 1 -- 172.21.15.112:0/1144218326 learned_addr learned my addr 172.21.15.112:0/1144218326 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:45.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.368+0000 7f2cf659c700 1 --2- 172.21.15.112:0/1144218326 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2cf8068d20 0x7f2cf81658d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:45.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.368+0000 7f2cf6d9d700 1 --2- 172.21.15.112:0/1144218326 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2cf805e420 0x7f2cf814cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:45.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.369+0000 7f2cf759e700 1 -- 172.21.15.112:0/1144218326 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2cf8068d20 msgr2=0x7f2cf81658d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:45.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.369+0000 7f2cf759e700 1 --2- 172.21.15.112:0/1144218326 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2cf8068d20 0x7f2cf81658d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:45.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.369+0000 7f2cf759e700 1 -- 172.21.15.112:0/1144218326 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2cf805e420 msgr2=0x7f2cf814cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:45.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.369+0000 7f2cf759e700 1 --2- 172.21.15.112:0/1144218326 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2cf805e420 0x7f2cf814cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:45.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.369+0000 7f2cf759e700 1 -- 172.21.15.112:0/1144218326 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce8009370 con 0x7f2cf80694c0 2023-01-24T03:30:45.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.369+0000 7f2cf759e700 1 --2- 172.21.15.112:0/1144218326 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2cf80694c0 0x7f2cf8165e10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8008000 tx=0x7f2ce8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:45.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.369+0000 7f2ceeffd700 1 -- 172.21.15.112:0/1144218326 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce8011b50 con 0x7f2cf80694c0 2023-01-24T03:30:45.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.369+0000 7f2ceeffd700 1 -- 172.21.15.112:0/1144218326 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ce8020070 con 0x7f2cf80694c0 2023-01-24T03:30:45.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.369+0000 7f2cfd0a6700 1 -- 172.21.15.112:0/1144218326 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cf8150490 con 0x7f2cf80694c0 2023-01-24T03:30:45.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.369+0000 7f2ceeffd700 1 -- 172.21.15.112:0/1144218326 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce80276d0 con 0x7f2cf80694c0 2023-01-24T03:30:45.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.369+0000 7f2cfd0a6700 1 -- 172.21.15.112:0/1144218326 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cf8150a50 con 0x7f2cf80694c0 2023-01-24T03:30:45.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.370+0000 7f2cfd0a6700 1 -- 172.21.15.112:0/1144218326 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cf81655b0 con 0x7f2cf80694c0 2023-01-24T03:30:45.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.370+0000 7f2ceeffd700 1 -- 172.21.15.112:0/1144218326 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f2ce8017020 con 0x7f2cf80694c0 2023-01-24T03:30:45.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.370+0000 7f2ceeffd700 1 --2- 172.21.15.112:0/1144218326 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2cdc0611c0 0x7f2cdc063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:45.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.371+0000 7f2ceeffd700 1 -- 172.21.15.112:0/1144218326 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f2ce8022070 con 0x7f2cf80694c0 2023-01-24T03:30:45.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.371+0000 7f2cf6d9d700 1 --2- 172.21.15.112:0/1144218326 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2cdc0611c0 0x7f2cdc063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:45.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.371+0000 7f2cf6d9d700 1 --2- 172.21.15.112:0/1144218326 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2cdc0611c0 0x7f2cdc063670 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0001a10 tx=0x7f2cf0001560 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:45.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.375+0000 7f2ceeffd700 1 -- 172.21.15.112:0/1144218326 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f2ce804f660 con 0x7f2cf80694c0 2023-01-24T03:30:45.556 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.553+0000 7f2cfd0a6700 1 -- 172.21.15.112:0/1144218326 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2cf806f3a0 con 0x7f2cf80694c0 2023-01-24T03:30:45.557 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.554+0000 7f2ceeffd700 1 -- 172.21.15.112:0/1144218326 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2ce8052c80 con 0x7f2cf80694c0 2023-01-24T03:30:45.557 INFO:teuthology.orchestra.run.smithi112.stdout:pg_num: 64 2023-01-24T03:30:45.560 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.556+0000 7f2cfd0a6700 1 -- 172.21.15.112:0/1144218326 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2cdc0611c0 msgr2=0x7f2cdc063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:45.560 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.556+0000 7f2cfd0a6700 1 --2- 172.21.15.112:0/1144218326 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2cdc0611c0 0x7f2cdc063670 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0001a10 tx=0x7f2cf0001560 comp rx=0 tx=0).stop 2023-01-24T03:30:45.560 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.556+0000 7f2cfd0a6700 1 -- 172.21.15.112:0/1144218326 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2cf80694c0 msgr2=0x7f2cf8165e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:45.561 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.556+0000 7f2cfd0a6700 1 --2- 172.21.15.112:0/1144218326 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2cf80694c0 0x7f2cf8165e10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8008000 tx=0x7f2ce8016000 comp rx=0 tx=0).stop 2023-01-24T03:30:45.561 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.556+0000 7f2cfd0a6700 1 -- 172.21.15.112:0/1144218326 shutdown_connections 2023-01-24T03:30:45.561 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.556+0000 7f2cfd0a6700 1 --2- 172.21.15.112:0/1144218326 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2cdc0611c0 0x7f2cdc063670 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:45.561 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.556+0000 7f2cfd0a6700 1 --2- 172.21.15.112:0/1144218326 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2cf80694c0 0x7f2cf8165e10 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:45.561 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.556+0000 7f2cfd0a6700 1 --2- 172.21.15.112:0/1144218326 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2cf805e420 0x7f2cf814cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:45.562 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.556+0000 7f2cfd0a6700 1 --2- 172.21.15.112:0/1144218326 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2cf8068d20 0x7f2cf81658d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:45.562 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.556+0000 7f2cfd0a6700 1 -- 172.21.15.112:0/1144218326 >> 172.21.15.112:0/1144218326 conn(0x7f2cf805d830 msgr2=0x7f2cf81548e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:45.562 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.556+0000 7f2cfd0a6700 1 -- 172.21.15.112:0/1144218326 shutdown_connections 2023-01-24T03:30:45.562 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.556+0000 7f2cfd0a6700 1 -- 172.21.15.112:0/1144218326 wait complete. 2023-01-24T03:30:45.571 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-01-24T03:30:45.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.683+0000 7f2e16287700 1 Processor -- start 2023-01-24T03:30:45.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.683+0000 7f2e16287700 1 -- start start 2023-01-24T03:30:45.688 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.683+0000 7f2e16287700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f2e100694c0 0x7f2e10069890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:45.688 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.683+0000 7f2e16287700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2e1005e420 0x7f2e10155f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:45.689 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.683+0000 7f2e16287700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f2e10068d20 0x7f2e100c68d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:45.689 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.683+0000 7f2e16287700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e10155550 con 0x7f2e100694c0 2023-01-24T03:30:45.689 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.683+0000 7f2e16287700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e10155ab0 con 0x7f2e10068d20 2023-01-24T03:30:45.689 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.683+0000 7f2e16287700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e10156480 con 0x7f2e1005e420 2023-01-24T03:30:45.690 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.683+0000 7f2e0ffff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f2e100694c0 0x7f2e10069890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:45.690 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.684+0000 7f2e0ffff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f2e100694c0 0x7f2e10069890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50920/0 (socket says 172.21.15.112:50920) 2023-01-24T03:30:45.690 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.684+0000 7f2e0ffff700 1 -- 172.21.15.112:0/2425347314 learned_addr learned my addr 172.21.15.112:0/2425347314 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:45.690 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.684+0000 7f2e0f7fe700 1 --2- 172.21.15.112:0/2425347314 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2e1005e420 0x7f2e10155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:45.690 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.684+0000 7f2e14824700 1 --2- 172.21.15.112:0/2425347314 >> v2:172.21.15.142:3300/0 conn(0x7f2e10068d20 0x7f2e100c68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:45.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.684+0000 7f2e0ffff700 1 -- 172.21.15.112:0/2425347314 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2e1005e420 msgr2=0x7f2e10155f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:45.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.684+0000 7f2e0ffff700 1 --2- 172.21.15.112:0/2425347314 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2e1005e420 0x7f2e10155f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:45.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.684+0000 7f2e0ffff700 1 -- 172.21.15.112:0/2425347314 >> v2:172.21.15.142:3300/0 conn(0x7f2e10068d20 msgr2=0x7f2e100c68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:45.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.684+0000 7f2e0ffff700 1 --2- 172.21.15.112:0/2425347314 >> v2:172.21.15.142:3300/0 conn(0x7f2e10068d20 0x7f2e100c68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:45.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.684+0000 7f2e0ffff700 1 -- 172.21.15.112:0/2425347314 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e100c6e10 con 0x7f2e100694c0 2023-01-24T03:30:45.692 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.684+0000 7f2e14824700 1 --2- 172.21.15.112:0/2425347314 >> v2:172.21.15.142:3300/0 conn(0x7f2e10068d20 0x7f2e100c68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:30:45.692 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.684+0000 7f2e0f7fe700 1 --2- 172.21.15.112:0/2425347314 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2e1005e420 0x7f2e10155f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:30:45.692 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.684+0000 7f2e0ffff700 1 --2- 172.21.15.112:0/2425347314 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2e100694c0 0x7f2e10069890 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2e1005e980 tx=0x7f2e000090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34fb9ad190f4aeb4 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:45.692 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.684+0000 7f2e0effd700 1 -- 172.21.15.112:0/2425347314 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e00011630 con 0x7f2e100694c0 2023-01-24T03:30:45.692 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.684+0000 7f2e0effd700 1 -- 172.21.15.112:0/2425347314 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e000117d0 con 0x7f2e100694c0 2023-01-24T03:30:45.693 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.684+0000 7f2e0effd700 1 -- 172.21.15.112:0/2425347314 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e00011b90 con 0x7f2e100694c0 2023-01-24T03:30:45.693 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.685+0000 7f2e16287700 1 -- 172.21.15.112:0/2425347314 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2e100694c0 msgr2=0x7f2e10069890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:45.693 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.685+0000 7f2e16287700 1 --2- 172.21.15.112:0/2425347314 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2e100694c0 0x7f2e10069890 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2e1005e980 tx=0x7f2e000090f0 comp rx=0 tx=0).stop 2023-01-24T03:30:45.693 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.685+0000 7f2e16287700 1 -- 172.21.15.112:0/2425347314 shutdown_connections 2023-01-24T03:30:45.693 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.685+0000 7f2e16287700 1 --2- 172.21.15.112:0/2425347314 >> v2:172.21.15.142:3300/0 conn(0x7f2e10068d20 0x7f2e100c68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:45.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.685+0000 7f2e16287700 1 --2- 172.21.15.112:0/2425347314 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2e100694c0 0x7f2e10069890 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:45.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.685+0000 7f2e16287700 1 --2- 172.21.15.112:0/2425347314 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2e1005e420 0x7f2e10155f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:45.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.685+0000 7f2e16287700 1 -- 172.21.15.112:0/2425347314 >> 172.21.15.112:0/2425347314 conn(0x7f2e1005d830 msgr2=0x7f2e1005dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:45.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.685+0000 7f2e16287700 1 -- 172.21.15.112:0/2425347314 shutdown_connections 2023-01-24T03:30:45.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.685+0000 7f2e16287700 1 -- 172.21.15.112:0/2425347314 wait complete. 2023-01-24T03:30:45.698 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.686+0000 7f2e16287700 1 Processor -- start 2023-01-24T03:30:45.698 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.686+0000 7f2e16287700 1 -- start start 2023-01-24T03:30:45.698 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.687+0000 7f2e16287700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2e1005e420 0x7f2e10165fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:45.698 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.687+0000 7f2e16287700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2e10068d20 0x7f2e10153320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:45.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.687+0000 7f2e16287700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e100694c0 0x7f2e100cb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:45.700 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.687+0000 7f2e16287700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e10166500 con 0x7f2e1005e420 2023-01-24T03:30:45.700 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.687+0000 7f2e16287700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e100c0300 con 0x7f2e100694c0 2023-01-24T03:30:45.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.687+0000 7f2e16287700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e10153c90 con 0x7f2e10068d20 2023-01-24T03:30:45.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.687+0000 7f2e0ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2e1005e420 0x7f2e10165fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:45.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.687+0000 7f2e0ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2e1005e420 0x7f2e10165fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50926/0 (socket says 172.21.15.112:50926) 2023-01-24T03:30:45.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.687+0000 7f2e0ffff700 1 -- 172.21.15.112:0/206580395 learned_addr learned my addr 172.21.15.112:0/206580395 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:45.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.687+0000 7f2e0f7fe700 1 --2- 172.21.15.112:0/206580395 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2e10068d20 0x7f2e10153320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:45.702 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.687+0000 7f2e14824700 1 --2- 172.21.15.112:0/206580395 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e100694c0 0x7f2e100cb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:45.702 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.687+0000 7f2e0ffff700 1 -- 172.21.15.112:0/206580395 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2e10068d20 msgr2=0x7f2e10153320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:45.702 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.687+0000 7f2e0ffff700 1 --2- 172.21.15.112:0/206580395 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2e10068d20 0x7f2e10153320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:45.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.687+0000 7f2e0ffff700 1 -- 172.21.15.112:0/206580395 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e100694c0 msgr2=0x7f2e100cb7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:45.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.687+0000 7f2e0ffff700 1 --2- 172.21.15.112:0/206580395 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e100694c0 0x7f2e100cb7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:45.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.687+0000 7f2e0ffff700 1 -- 172.21.15.112:0/206580395 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e00009370 con 0x7f2e1005e420 2023-01-24T03:30:45.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.687+0000 7f2e0f7fe700 1 --2- 172.21.15.112:0/206580395 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2e10068d20 0x7f2e10153320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-01-24T03:30:45.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.687+0000 7f2e14824700 1 --2- 172.21.15.112:0/206580395 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e100694c0 0x7f2e100cb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-01-24T03:30:45.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.687+0000 7f2e0ffff700 1 --2- 172.21.15.112:0/206580395 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2e1005e420 0x7f2e10165fc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2e10066330 tx=0x7f2e00016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:45.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.687+0000 7f2df7fff700 1 -- 172.21.15.112:0/206580395 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e00011b50 con 0x7f2e1005e420 2023-01-24T03:30:45.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.687+0000 7f2df7fff700 1 -- 172.21.15.112:0/206580395 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e00020070 con 0x7f2e1005e420 2023-01-24T03:30:45.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.687+0000 7f2df7fff700 1 -- 172.21.15.112:0/206580395 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e000276f0 con 0x7f2e1005e420 2023-01-24T03:30:45.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.687+0000 7f2e16287700 1 -- 172.21.15.112:0/206580395 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e100c0060 con 0x7f2e1005e420 2023-01-24T03:30:45.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.687+0000 7f2e16287700 1 -- 172.21.15.112:0/206580395 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e100c0fc0 con 0x7f2e1005e420 2023-01-24T03:30:45.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.688+0000 7f2e16287700 1 -- 172.21.15.112:0/206580395 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e1006f3a0 con 0x7f2e1005e420 2023-01-24T03:30:45.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.689+0000 7f2df7fff700 1 -- 172.21.15.112:0/206580395 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f2e00017020 con 0x7f2e1005e420 2023-01-24T03:30:45.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.689+0000 7f2df7fff700 1 --2- 172.21.15.112:0/206580395 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2df00611c0 0x7f2df0063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:45.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.689+0000 7f2e0f7fe700 1 --2- 172.21.15.112:0/206580395 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2df00611c0 0x7f2df0063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:45.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.689+0000 7f2df7fff700 1 -- 172.21.15.112:0/206580395 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f2e00022070 con 0x7f2e1005e420 2023-01-24T03:30:45.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.690+0000 7f2e0f7fe700 1 --2- 172.21.15.112:0/206580395 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2df00611c0 0x7f2df0063670 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2e04006d80 tx=0x7f2e04008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:45.707 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.693+0000 7f2df7fff700 1 -- 172.21.15.112:0/206580395 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f2e0004f420 con 0x7f2e1005e420 2023-01-24T03:30:45.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.870+0000 7f2e16287700 1 -- 172.21.15.112:0/206580395 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f2e100b1970 con 0x7f2e1005e420 2023-01-24T03:30:45.874 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.871+0000 7f2df7fff700 1 -- 172.21.15.112:0/206580395 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7f2e0002c020 con 0x7f2e1005e420 2023-01-24T03:30:45.874 INFO:teuthology.orchestra.run.smithi112.stdout:pg_num: 32 2023-01-24T03:30:45.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.873+0000 7f2e16287700 1 -- 172.21.15.112:0/206580395 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2df00611c0 msgr2=0x7f2df0063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:45.877 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.873+0000 7f2e16287700 1 --2- 172.21.15.112:0/206580395 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2df00611c0 0x7f2df0063670 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2e04006d80 tx=0x7f2e04008040 comp rx=0 tx=0).stop 2023-01-24T03:30:45.877 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.873+0000 7f2e16287700 1 -- 172.21.15.112:0/206580395 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2e1005e420 msgr2=0x7f2e10165fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:45.877 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.873+0000 7f2e16287700 1 --2- 172.21.15.112:0/206580395 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2e1005e420 0x7f2e10165fc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2e10066330 tx=0x7f2e00016000 comp rx=0 tx=0).stop 2023-01-24T03:30:45.877 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.873+0000 7f2e16287700 1 -- 172.21.15.112:0/206580395 shutdown_connections 2023-01-24T03:30:45.877 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.873+0000 7f2e16287700 1 --2- 172.21.15.112:0/206580395 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2df00611c0 0x7f2df0063670 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:45.878 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.873+0000 7f2e16287700 1 --2- 172.21.15.112:0/206580395 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2e1005e420 0x7f2e10165fc0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:45.878 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.873+0000 7f2e16287700 1 --2- 172.21.15.112:0/206580395 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2e100694c0 0x7f2e100cb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:45.878 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.873+0000 7f2e16287700 1 --2- 172.21.15.112:0/206580395 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2e10068d20 0x7f2e10153320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:45.878 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.873+0000 7f2e16287700 1 -- 172.21.15.112:0/206580395 >> 172.21.15.112:0/206580395 conn(0x7f2e1005d830 msgr2=0x7f2e100c8df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:45.879 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.873+0000 7f2e16287700 1 -- 172.21.15.112:0/206580395 shutdown_connections 2023-01-24T03:30:45.879 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.873+0000 7f2e16287700 1 -- 172.21.15.112:0/206580395 wait complete. 2023-01-24T03:30:45.888 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:30:46.000 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.997+0000 7f4331f6f700 1 Processor -- start 2023-01-24T03:30:46.000 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.997+0000 7f4331f6f700 1 -- start start 2023-01-24T03:30:46.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.998+0000 7f4331f6f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f432c0c1580 0x7f432c06cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:46.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.998+0000 7f4331f6f700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f432c0c3490 0x7f432c06d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:46.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.998+0000 7f4331f6f700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f432c0c3af0 con 0x7f432c0c30c0 2023-01-24T03:30:46.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.998+0000 7f4331f6f700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f432c153da0 con 0x7f432c0c3490 2023-01-24T03:30:46.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.998+0000 7f4331f6f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f432c05e650 con 0x7f432c0c1580 2023-01-24T03:30:46.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.998+0000 7f4330f6d700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f432c0c30c0 0x7f432c05df40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:60154/0 (socket says 172.21.15.112:60154) 2023-01-24T03:30:46.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.998+0000 7f4330f6d700 1 -- 172.21.15.112:0/485502337 learned_addr learned my addr 172.21.15.112:0/485502337 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:46.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.998+0000 7f432b7fe700 1 -- 172.21.15.112:0/485502337 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 792188654 0 0) 0x7f432c0c3af0 con 0x7f432c0c30c0 2023-01-24T03:30:46.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.998+0000 7f432b7fe700 1 -- 172.21.15.112:0/485502337 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43180033f0 con 0x7f432c0c30c0 2023-01-24T03:30:46.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.998+0000 7f432bfff700 1 --2- 172.21.15.112:0/485502337 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f432c0c1580 0x7f432c06cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:46.006 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.998+0000 7f433176e700 1 --2- 172.21.15.112:0/485502337 >> v2:172.21.15.142:3300/0 conn(0x7f432c0c3490 0x7f432c06d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:46.006 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.998+0000 7f432b7fe700 1 -- 172.21.15.112:0/485502337 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3844151216 0 0) 0x7f43180033f0 con 0x7f432c0c30c0 2023-01-24T03:30:46.006 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.998+0000 7f432b7fe700 1 -- 172.21.15.112:0/485502337 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f432c0c1580 msgr2=0x7f432c06cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:46.006 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.998+0000 7f432b7fe700 1 --2- 172.21.15.112:0/485502337 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f432c0c1580 0x7f432c06cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:46.006 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.998+0000 7f432b7fe700 1 -- 172.21.15.112:0/485502337 >> v2:172.21.15.142:3300/0 conn(0x7f432c0c3490 msgr2=0x7f432c06d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:46.007 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.998+0000 7f432b7fe700 1 --2- 172.21.15.112:0/485502337 >> v2:172.21.15.142:3300/0 conn(0x7f432c0c3490 0x7f432c06d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:46.007 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.998+0000 7f432b7fe700 1 -- 172.21.15.112:0/485502337 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f432c152f40 con 0x7f432c0c30c0 2023-01-24T03:30:46.007 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.998+0000 7f432b7fe700 1 -- 172.21.15.112:0/485502337 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7f431c001470 con 0x7f432c0c30c0 2023-01-24T03:30:46.007 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.998+0000 7f432b7fe700 1 -- 172.21.15.112:0/485502337 >> v1:172.21.15.112:6789/0 conn(0x7f432c0c30c0 legacy=0x7f432c05df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:46.007 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.998+0000 7f432b7fe700 1 --2- 172.21.15.112:0/485502337 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43180038e0 0x7f4318003cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:46.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.999+0000 7f432b7fe700 1 --2- 172.21.15.112:0/485502337 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4318004690 0x7f4318004b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:46.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.999+0000 7f432b7fe700 1 --2- 172.21.15.112:0/485502337 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f432c0c1580 0x7f4318005040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:46.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.999+0000 7f432b7fe700 1 -- 172.21.15.112:0/485502337 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432c153da0 con 0x7f4318004690 2023-01-24T03:30:46.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.999+0000 7f432b7fe700 1 -- 172.21.15.112:0/485502337 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432c05e650 con 0x7f43180038e0 2023-01-24T03:30:46.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.999+0000 7f432b7fe700 1 -- 172.21.15.112:0/485502337 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f431800b8e0 con 0x7f432c0c1580 2023-01-24T03:30:46.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.999+0000 7f432bfff700 1 --2- 172.21.15.112:0/485502337 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4318004690 0x7f4318004b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:46.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.999+0000 7f4330f6d700 1 --2- 172.21.15.112:0/485502337 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43180038e0 0x7f4318003cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:46.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.999+0000 7f433176e700 1 --2- 172.21.15.112:0/485502337 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f432c0c1580 0x7f4318005040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:46.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.999+0000 7f432bfff700 1 -- 172.21.15.112:0/485502337 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f432c0c1580 msgr2=0x7f4318005040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:46.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.999+0000 7f432bfff700 1 --2- 172.21.15.112:0/485502337 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f432c0c1580 0x7f4318005040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:46.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.999+0000 7f432bfff700 1 -- 172.21.15.112:0/485502337 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43180038e0 msgr2=0x7f4318003cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:46.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.999+0000 7f432bfff700 1 --2- 172.21.15.112:0/485502337 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43180038e0 0x7f4318003cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:46.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.999+0000 7f432bfff700 1 -- 172.21.15.112:0/485502337 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43180033f0 con 0x7f4318004690 2023-01-24T03:30:46.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.999+0000 7f432bfff700 1 --2- 172.21.15.112:0/485502337 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4318004690 0x7f4318004b00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4320009ab0 tx=0x7f4320009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=821776b61c89c398 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:46.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.999+0000 7f432b7fe700 1 -- 172.21.15.112:0/485502337 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4320009430 con 0x7f4318004690 2023-01-24T03:30:46.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.999+0000 7f432b7fe700 1 -- 172.21.15.112:0/485502337 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f432000e640 con 0x7f4318004690 2023-01-24T03:30:46.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:45.999+0000 7f432b7fe700 1 -- 172.21.15.112:0/485502337 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432000ea00 con 0x7f4318004690 2023-01-24T03:30:46.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.000+0000 7f4331f6f700 1 -- 172.21.15.112:0/485502337 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4318004690 msgr2=0x7f4318004b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:46.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.000+0000 7f4331f6f700 1 --2- 172.21.15.112:0/485502337 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4318004690 0x7f4318004b00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4320009ab0 tx=0x7f4320009060 comp rx=0 tx=0).stop 2023-01-24T03:30:46.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.000+0000 7f433176e700 1 -- 172.21.15.112:0/485502337 reap_dead start 2023-01-24T03:30:46.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.000+0000 7f4331f6f700 1 -- 172.21.15.112:0/485502337 shutdown_connections 2023-01-24T03:30:46.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.000+0000 7f4331f6f700 1 -- 172.21.15.112:0/485502337 >> 172.21.15.112:0/485502337 conn(0x7f432c06c360 msgr2=0x7f432c05d830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:46.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.000+0000 7f4331f6f700 1 -- 172.21.15.112:0/485502337 shutdown_connections 2023-01-24T03:30:46.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.000+0000 7f4331f6f700 1 -- 172.21.15.112:0/485502337 wait complete. 2023-01-24T03:30:46.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.001+0000 7f4331f6f700 1 Processor -- start 2023-01-24T03:30:46.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.001+0000 7f4331f6f700 1 -- start start 2023-01-24T03:30:46.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.002+0000 7f4331f6f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4318004690 0x7f432c14d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:46.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.002+0000 7f4331f6f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f432c067f20 0x7f432c14e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:46.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.002+0000 7f4331f6f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f432c1661d0 0x7f432c068940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:46.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.002+0000 7f4331f6f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432c14f550 con 0x7f4318004690 2023-01-24T03:30:46.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.002+0000 7f4331f6f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432c165db0 con 0x7f432c067f20 2023-01-24T03:30:46.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.002+0000 7f4331f6f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f432c165470 con 0x7f432c1661d0 2023-01-24T03:30:46.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.002+0000 7f4330f6d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4318004690 0x7f432c14d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:46.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.002+0000 7f4330f6d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4318004690 0x7f432c14d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50946/0 (socket says 172.21.15.112:50946) 2023-01-24T03:30:46.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.002+0000 7f4330f6d700 1 -- 172.21.15.112:0/2045386728 learned_addr learned my addr 172.21.15.112:0/2045386728 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:46.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.002+0000 7f432bfff700 1 --2- 172.21.15.112:0/2045386728 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f432c067f20 0x7f432c14e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:46.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.002+0000 7f433176e700 1 --2- 172.21.15.112:0/2045386728 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f432c1661d0 0x7f432c068940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:46.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.002+0000 7f4330f6d700 1 -- 172.21.15.112:0/2045386728 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f432c1661d0 msgr2=0x7f432c068940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:46.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.002+0000 7f4330f6d700 1 --2- 172.21.15.112:0/2045386728 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f432c1661d0 0x7f432c068940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:46.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.002+0000 7f4330f6d700 1 -- 172.21.15.112:0/2045386728 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f432c067f20 msgr2=0x7f432c14e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:46.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.002+0000 7f4330f6d700 1 --2- 172.21.15.112:0/2045386728 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f432c067f20 0x7f432c14e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:46.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.002+0000 7f4330f6d700 1 -- 172.21.15.112:0/2045386728 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f432c069aa0 con 0x7f4318004690 2023-01-24T03:30:46.020 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.002+0000 7f433176e700 1 --2- 172.21.15.112:0/2045386728 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f432c1661d0 0x7f432c068940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:30:46.020 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.002+0000 7f4330f6d700 1 --2- 172.21.15.112:0/2045386728 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4318004690 0x7f432c14d5c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f431c004990 tx=0x7f432c152e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:46.020 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.002+0000 7f4328ff9700 1 -- 172.21.15.112:0/2045386728 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f431c018030 con 0x7f4318004690 2023-01-24T03:30:46.020 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.002+0000 7f4328ff9700 1 -- 172.21.15.112:0/2045386728 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f431c00f8d0 con 0x7f4318004690 2023-01-24T03:30:46.020 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.002+0000 7f4331f6f700 1 -- 172.21.15.112:0/2045386728 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f432c069d80 con 0x7f4318004690 2023-01-24T03:30:46.021 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.002+0000 7f4328ff9700 1 -- 172.21.15.112:0/2045386728 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f431c00fc70 con 0x7f4318004690 2023-01-24T03:30:46.021 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.002+0000 7f4331f6f700 1 -- 172.21.15.112:0/2045386728 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f432c1659d0 con 0x7f4318004690 2023-01-24T03:30:46.021 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.003+0000 7f4331f6f700 1 -- 172.21.15.112:0/2045386728 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f432c05ea20 con 0x7f4318004690 2023-01-24T03:30:46.021 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.004+0000 7f4328ff9700 1 -- 172.21.15.112:0/2045386728 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f431c01e470 con 0x7f4318004690 2023-01-24T03:30:46.021 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.004+0000 7f4328ff9700 1 --2- 172.21.15.112:0/2045386728 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f430c0611c0 0x7f430c063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:46.021 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.004+0000 7f4328ff9700 1 -- 172.21.15.112:0/2045386728 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f431c019070 con 0x7f4318004690 2023-01-24T03:30:46.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.004+0000 7f432bfff700 1 --2- 172.21.15.112:0/2045386728 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f430c0611c0 0x7f430c063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:46.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.004+0000 7f432bfff700 1 --2- 172.21.15.112:0/2045386728 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f430c0611c0 0x7f430c063670 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f432c1525e0 tx=0x7f432001a000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:46.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.008+0000 7f4328ff9700 1 -- 172.21.15.112:0/2045386728 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f431c04d850 con 0x7f4318004690 2023-01-24T03:30:46.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.224+0000 7f4331f6f700 1 -- 172.21.15.112:0/2045386728 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f432c068720 con 0x7f4318004690 2023-01-24T03:30:46.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.225+0000 7f4328ff9700 1 -- 172.21.15.112:0/2045386728 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6114 (secure 0 0 0) 0x7f431c050e70 con 0x7f4318004690 2023-01-24T03:30:46.228 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:30:46.229 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/869267413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":869267413},{"type":"v1","addr":"172.21.15.142:6834","nonce":869267413}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:30:37.600288+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.142:6835/2762621058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2762621058},{"type":"v1","addr":"172.21.15.142:6835","nonce":2762621058}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:30:46.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.227+0000 7f4331f6f700 1 -- 172.21.15.112:0/2045386728 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f430c0611c0 msgr2=0x7f430c063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:46.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.227+0000 7f4331f6f700 1 --2- 172.21.15.112:0/2045386728 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f430c0611c0 0x7f430c063670 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f432c1525e0 tx=0x7f432001a000 comp rx=0 tx=0).stop 2023-01-24T03:30:46.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.227+0000 7f4331f6f700 1 -- 172.21.15.112:0/2045386728 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4318004690 msgr2=0x7f432c14d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:46.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.227+0000 7f4331f6f700 1 --2- 172.21.15.112:0/2045386728 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4318004690 0x7f432c14d5c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f431c004990 tx=0x7f432c152e30 comp rx=0 tx=0).stop 2023-01-24T03:30:46.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.227+0000 7f4331f6f700 1 -- 172.21.15.112:0/2045386728 shutdown_connections 2023-01-24T03:30:46.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.227+0000 7f4331f6f700 1 --2- 172.21.15.112:0/2045386728 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f430c0611c0 0x7f430c063670 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:46.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.227+0000 7f4331f6f700 1 --2- 172.21.15.112:0/2045386728 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4318004690 0x7f432c14d5c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:46.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.227+0000 7f4331f6f700 1 --2- 172.21.15.112:0/2045386728 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f432c1661d0 0x7f432c068940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:46.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.227+0000 7f4331f6f700 1 --2- 172.21.15.112:0/2045386728 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f432c067f20 0x7f432c14e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:46.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.227+0000 7f4331f6f700 1 -- 172.21.15.112:0/2045386728 >> 172.21.15.112:0/2045386728 conn(0x7f432c06c360 msgr2=0x7f432c05d830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:46.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.227+0000 7f4331f6f700 1 -- 172.21.15.112:0/2045386728 shutdown_connections 2023-01-24T03:30:46.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.227+0000 7f4331f6f700 1 -- 172.21.15.112:0/2045386728 wait complete. 2023-01-24T03:30:46.233 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 10 2023-01-24T03:30:46.845 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 10, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-01-24T03:30:20.971710+0000', 'modified': '2023-01-24T03:30:37.600288+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4293}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4293': {'gid': 4293, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.142:6835/2762621058', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6833', 'nonce': 2762621058}, {'type': 'v1', 'addr': '172.21.15.142:6835', 'nonce': 2762621058}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [4, 5], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-01-24T03:30:46.846 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-01-24T03:30:46.846 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-01-24T03:30:46.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.960+0000 7f7b55567700 1 Processor -- start 2023-01-24T03:30:46.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.960+0000 7f7b55567700 1 -- start start 2023-01-24T03:30:46.966 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.961+0000 7f7b55567700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f7b500c82a0 0x7f7b50159320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:46.966 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.961+0000 7f7b55567700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7b500c7800 0x7f7b50155380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:46.966 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.961+0000 7f7b55567700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b501588d0 con 0x7f7b500c6e00 2023-01-24T03:30:46.966 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.961+0000 7f7b55567700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b50158e90 con 0x7f7b500c82a0 2023-01-24T03:30:46.966 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.961+0000 7f7b55567700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b500c72d0 con 0x7f7b500c7800 2023-01-24T03:30:46.967 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.961+0000 7f7b4ffff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f7b500c82a0 0x7f7b50159320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:46.967 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.961+0000 7f7b4ffff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f7b500c82a0 0x7f7b50159320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50954/0 (socket says 172.21.15.112:50954) 2023-01-24T03:30:46.967 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.961+0000 7f7b4ffff700 1 -- 172.21.15.112:0/2711463183 learned_addr learned my addr 172.21.15.112:0/2711463183 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:46.967 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.961+0000 7f7b4f7fe700 1 --2- 172.21.15.112:0/2711463183 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7b500c7800 0x7f7b50155380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:46.967 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.961+0000 7f7b4ffff700 1 -- 172.21.15.112:0/2711463183 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7b500c7800 msgr2=0x7f7b50155380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:46.968 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.961+0000 7f7b4ffff700 1 --2- 172.21.15.112:0/2711463183 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7b500c7800 0x7f7b50155380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:46.968 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.961+0000 7f7b4ffff700 1 -- 172.21.15.112:0/2711463183 >> v1:172.21.15.142:6789/0 conn(0x7f7b500c6e00 legacy=0x7f7b500c2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:46.968 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.961+0000 7f7b4ffff700 1 -- 172.21.15.112:0/2711463183 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b500c7590 con 0x7f7b500c82a0 2023-01-24T03:30:46.968 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.961+0000 7f7b4ffff700 1 --2- 172.21.15.112:0/2711463183 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b500c82a0 0x7f7b50159320 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7b38006fd0 tx=0x7f7b380090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=646b4ee070635da5 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:46.968 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.961+0000 7f7b4effd700 1 -- 172.21.15.112:0/2711463183 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b38011630 con 0x7f7b500c82a0 2023-01-24T03:30:46.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.961+0000 7f7b4effd700 1 -- 172.21.15.112:0/2711463183 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b380117d0 con 0x7f7b500c82a0 2023-01-24T03:30:46.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.961+0000 7f7b4effd700 1 -- 172.21.15.112:0/2711463183 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b38011b90 con 0x7f7b500c82a0 2023-01-24T03:30:46.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.962+0000 7f7b55567700 1 -- 172.21.15.112:0/2711463183 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b500c82a0 msgr2=0x7f7b50159320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:46.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.962+0000 7f7b55567700 1 --2- 172.21.15.112:0/2711463183 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b500c82a0 0x7f7b50159320 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7b38006fd0 tx=0x7f7b380090f0 comp rx=0 tx=0).stop 2023-01-24T03:30:46.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.962+0000 7f7b55567700 1 -- 172.21.15.112:0/2711463183 shutdown_connections 2023-01-24T03:30:46.970 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.962+0000 7f7b55567700 1 --2- 172.21.15.112:0/2711463183 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b500c82a0 0x7f7b50159320 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:46.970 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.962+0000 7f7b55567700 1 --2- 172.21.15.112:0/2711463183 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7b500c7800 0x7f7b50155380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:46.970 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.962+0000 7f7b55567700 1 -- 172.21.15.112:0/2711463183 >> 172.21.15.112:0/2711463183 conn(0x7f7b5005d830 msgr2=0x7f7b5005dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:46.970 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.962+0000 7f7b55567700 1 -- 172.21.15.112:0/2711463183 shutdown_connections 2023-01-24T03:30:46.970 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.962+0000 7f7b55567700 1 -- 172.21.15.112:0/2711463183 wait complete. 2023-01-24T03:30:46.971 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.963+0000 7f7b55567700 1 Processor -- start 2023-01-24T03:30:46.971 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.963+0000 7f7b55567700 1 -- start start 2023-01-24T03:30:46.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.964+0000 7f7b55567700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7b500c6e00 0x7f7b5015a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:46.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.964+0000 7f7b55567700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b500c7800 0x7f7b501669d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:46.980 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.964+0000 7f7b55567700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7b500c82a0 0x7f7b5014cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:46.980 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.964+0000 7f7b55567700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b5015b890 con 0x7f7b500c7800 2023-01-24T03:30:46.980 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.964+0000 7f7b55567700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b5014d4b0 con 0x7f7b500c6e00 2023-01-24T03:30:46.980 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.964+0000 7f7b55567700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b5015b080 con 0x7f7b500c82a0 2023-01-24T03:30:46.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.964+0000 7f7b4f7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b500c7800 0x7f7b501669d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:46.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.964+0000 7f7b4f7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b500c7800 0x7f7b501669d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50960/0 (socket says 172.21.15.112:50960) 2023-01-24T03:30:46.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.964+0000 7f7b4f7fe700 1 -- 172.21.15.112:0/3123798691 learned_addr learned my addr 172.21.15.112:0/3123798691 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:46.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.964+0000 7f7b4ffff700 1 --2- 172.21.15.112:0/3123798691 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7b500c6e00 0x7f7b5015a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:46.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.964+0000 7f7b54d66700 1 --2- 172.21.15.112:0/3123798691 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7b500c82a0 0x7f7b5014cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:46.982 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.964+0000 7f7b4f7fe700 1 -- 172.21.15.112:0/3123798691 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7b500c82a0 msgr2=0x7f7b5014cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:46.982 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.964+0000 7f7b4f7fe700 1 --2- 172.21.15.112:0/3123798691 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7b500c82a0 0x7f7b5014cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:46.982 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.964+0000 7f7b4f7fe700 1 -- 172.21.15.112:0/3123798691 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7b500c6e00 msgr2=0x7f7b5015a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:46.982 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.964+0000 7f7b4f7fe700 1 --2- 172.21.15.112:0/3123798691 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7b500c6e00 0x7f7b5015a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:46.982 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.964+0000 7f7b4f7fe700 1 -- 172.21.15.112:0/3123798691 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b38009370 con 0x7f7b500c7800 2023-01-24T03:30:46.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.964+0000 7f7b4f7fe700 1 --2- 172.21.15.112:0/3123798691 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b500c7800 0x7f7b501669d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7b4000e960 tx=0x7f7b40009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:46.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.964+0000 7f7b47fff700 1 -- 172.21.15.112:0/3123798691 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b40013680 con 0x7f7b500c7800 2023-01-24T03:30:46.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.965+0000 7f7b47fff700 1 -- 172.21.15.112:0/3123798691 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b400137e0 con 0x7f7b500c7800 2023-01-24T03:30:46.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.965+0000 7f7b55567700 1 -- 172.21.15.112:0/3123798691 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b5015a140 con 0x7f7b500c7800 2023-01-24T03:30:46.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.965+0000 7f7b47fff700 1 -- 172.21.15.112:0/3123798691 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b40013b60 con 0x7f7b500c7800 2023-01-24T03:30:46.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.965+0000 7f7b55567700 1 -- 172.21.15.112:0/3123798691 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b5015b5b0 con 0x7f7b500c7800 2023-01-24T03:30:46.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.966+0000 7f7b47fff700 1 -- 172.21.15.112:0/3123798691 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f7b40019030 con 0x7f7b500c7800 2023-01-24T03:30:46.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.966+0000 7f7b47fff700 1 --2- 172.21.15.112:0/3123798691 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7b340611c0 0x7f7b34063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:46.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.966+0000 7f7b47fff700 1 -- 172.21.15.112:0/3123798691 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f7b4007d560 con 0x7f7b500c7800 2023-01-24T03:30:46.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.966+0000 7f7b4ffff700 1 --2- 172.21.15.112:0/3123798691 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7b340611c0 0x7f7b34063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:46.985 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.967+0000 7f7b4ffff700 1 --2- 172.21.15.112:0/3123798691 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7b340611c0 0x7f7b34063670 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7b38009340 tx=0x7f7b38016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:46.985 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.968+0000 7f7b5fab2b80 1 Processor -- start 2023-01-24T03:30:46.985 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.968+0000 7f7b5fab2b80 1 -- start start 2023-01-24T03:30:46.985 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.969+0000 7f7b5fab2b80 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f84bffd860 0x55f84bfaf940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:46.985 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.969+0000 7f7b5fab2b80 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x55f84bfb5d90 0x55f84bffdc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:46.986 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.969+0000 7f7b5fab2b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f84bffe170 0x55f84bfee020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:46.986 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.969+0000 7f7b5fab2b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f84bf3d0a0 con 0x55f84bffe170 2023-01-24T03:30:46.986 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.969+0000 7f7b5fab2b80 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f84bec1460 con 0x55f84bffd860 2023-01-24T03:30:46.986 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.969+0000 7f7b5fab2b80 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f84bf0a270 con 0x55f84bfb5d90 2023-01-24T03:30:46.986 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.969+0000 7f7b54d66700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f84bffe170 0x55f84bfee020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:46.987 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.969+0000 7f7b54d66700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f84bffe170 0x55f84bfee020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50972/0 (socket says 172.21.15.112:50972) 2023-01-24T03:30:46.987 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.969+0000 7f7b54d66700 1 -- 172.21.15.112:0/2100660787 learned_addr learned my addr 172.21.15.112:0/2100660787 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:46.987 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.969+0000 7f7b54d66700 1 -- 172.21.15.112:0/2100660787 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x55f84bfb5d90 msgr2=0x55f84bffdc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:46.987 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.969+0000 7f7b54d66700 1 --2- 172.21.15.112:0/2100660787 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x55f84bfb5d90 0x55f84bffdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:46.987 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.969+0000 7f7b4ffff700 1 --2- 172.21.15.112:0/2100660787 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f84bffd860 0x55f84bfaf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:46.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.969+0000 7f7b54d66700 1 -- 172.21.15.112:0/2100660787 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f84bffd860 msgr2=0x55f84bfaf940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:46.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.969+0000 7f7b54d66700 1 --2- 172.21.15.112:0/2100660787 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f84bffd860 0x55f84bfaf940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:46.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.969+0000 7f7b54d66700 1 -- 172.21.15.112:0/2100660787 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f84bd9f270 con 0x55f84bffe170 2023-01-24T03:30:46.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.969+0000 7f7b54d66700 1 --2- 172.21.15.112:0/2100660787 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f84bffe170 0x55f84bfee020 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7b48002970 tx=0x7f7b48001460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=674869bb690b34e3 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:46.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.969+0000 7f7b457fa700 1 -- 172.21.15.112:0/2100660787 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b48002280 con 0x55f84bffe170 2023-01-24T03:30:46.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.969+0000 7f7b457fa700 1 -- 172.21.15.112:0/2100660787 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b480023e0 con 0x55f84bffe170 2023-01-24T03:30:46.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.969+0000 7f7b457fa700 1 -- 172.21.15.112:0/2100660787 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b48001720 con 0x55f84bffe170 2023-01-24T03:30:46.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.970+0000 7f7b5fab2b80 1 -- 172.21.15.112:0/2100660787 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f84bffe170 msgr2=0x55f84bfee020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:46.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.970+0000 7f7b5fab2b80 1 --2- 172.21.15.112:0/2100660787 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f84bffe170 0x55f84bfee020 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7b48002970 tx=0x7f7b48001460 comp rx=0 tx=0).stop 2023-01-24T03:30:46.990 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.970+0000 7f7b5fab2b80 1 -- 172.21.15.112:0/2100660787 shutdown_connections 2023-01-24T03:30:46.990 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.970+0000 7f7b5fab2b80 1 --2- 172.21.15.112:0/2100660787 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f84bffe170 0x55f84bfee020 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:46.990 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.970+0000 7f7b5fab2b80 1 --2- 172.21.15.112:0/2100660787 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f84bffd860 0x55f84bfaf940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:46.990 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.970+0000 7f7b5fab2b80 1 --2- 172.21.15.112:0/2100660787 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x55f84bfb5d90 0x55f84bffdc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:46.990 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.970+0000 7f7b5fab2b80 1 -- 172.21.15.112:0/2100660787 >> 172.21.15.112:0/2100660787 conn(0x55f84bfb22e0 msgr2=0x55f84bfb2930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:46.991 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.970+0000 7f7b5fab2b80 1 -- 172.21.15.112:0/2100660787 shutdown_connections 2023-01-24T03:30:46.991 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.970+0000 7f7b5fab2b80 1 -- 172.21.15.112:0/2100660787 wait complete. 2023-01-24T03:30:46.991 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.971+0000 7f7b5fab2b80 1 -- start start 2023-01-24T03:30:46.991 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.971+0000 7f7b5fab2b80 1 Processor -- start 2023-01-24T03:30:46.991 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.971+0000 7f7b5fab2b80 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f84bfb5d90 0x55f84bfb2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:46.991 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.971+0000 7f7b5fab2b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f84bffd860 0x55f84c007d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:46.992 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.971+0000 7f7b5fab2b80 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x55f84bffe170 0x55f84bfef470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:46.992 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.971+0000 7f7b5fab2b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f84bf03ae0 con 0x55f84bffd860 2023-01-24T03:30:46.992 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.971+0000 7f7b5fab2b80 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f84bd87e30 con 0x55f84bfb5d90 2023-01-24T03:30:46.992 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.971+0000 7f7b5fab2b80 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f84bfb31e0 con 0x55f84bffe170 2023-01-24T03:30:46.992 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.972+0000 7f7b5fab2b80 10 client.0 _ll_register_callbacks cb 0x55f84bf75ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-01-24T03:30:46.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.972+0000 7f7b4f7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f84bffd860 0x55f84c007d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:46.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.972+0000 7f7b4f7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f84bffd860 0x55f84c007d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50976/0 (socket says 172.21.15.112:50976) 2023-01-24T03:30:46.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.972+0000 7f7b4f7fe700 1 -- 172.21.15.112:0/1109800450 learned_addr learned my addr 172.21.15.112:0/1109800450 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:46.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.972+0000 7f7b4ffff700 1 --2- 172.21.15.112:0/1109800450 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f84bfb5d90 0x55f84bfb2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:46.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.972+0000 7f7b54d66700 1 --2- 172.21.15.112:0/1109800450 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x55f84bffe170 0x55f84bfef470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:46.994 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.972+0000 7f7b4f7fe700 1 -- 172.21.15.112:0/1109800450 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x55f84bffe170 msgr2=0x55f84bfef470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:46.994 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.972+0000 7f7b4f7fe700 1 --2- 172.21.15.112:0/1109800450 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x55f84bffe170 0x55f84bfef470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:46.994 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.972+0000 7f7b4f7fe700 1 -- 172.21.15.112:0/1109800450 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f84bfb5d90 msgr2=0x55f84bfb2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:46.994 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.972+0000 7f7b4f7fe700 1 --2- 172.21.15.112:0/1109800450 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f84bfb5d90 0x55f84bfb2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:46.994 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.972+0000 7f7b4f7fe700 1 -- 172.21.15.112:0/1109800450 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b4800f040 con 0x55f84bffd860 2023-01-24T03:30:46.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.972+0000 7f7b4f7fe700 1 --2- 172.21.15.112:0/1109800450 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f84bffd860 0x55f84c007d60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7b40050f40 tx=0x7f7b40051690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2c8c088e890872e server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:46.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.972+0000 7f7b2e7fc700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2023-01-24T03:30:46.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.972+0000 7f7b2e7fc700 1 -- 172.21.15.112:0/1109800450 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b40057880 con 0x55f84bffd860 2023-01-24T03:30:46.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.972+0000 7f7b2e7fc700 1 -- 172.21.15.112:0/1109800450 <== mon.0 v2:172.21.15.112:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f7b400573f0 con 0x55f84bffd860 2023-01-24T03:30:46.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.972+0000 7f7b5fab2b80 1 -- 172.21.15.112:0/1109800450 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f84bfa7980 con 0x55f84bffd860 2023-01-24T03:30:46.996 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.972+0000 7f7b2e7fc700 1 -- 172.21.15.112:0/1109800450 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=10) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b4007d0f0 con 0x55f84bffd860 2023-01-24T03:30:46.996 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.972+0000 7f7b5fab2b80 10 client.4725 fetch_fsmap learned FSMap version 10 2023-01-24T03:30:46.996 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.972+0000 7f7b5fab2b80 1 -- 172.21.15.112:0/1109800450 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=10,monmap=2+}) v3 -- 0x55f84bd41fe0 con 0x55f84bffd860 2023-01-24T03:30:46.996 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.972+0000 7f7b2e7fc700 1 -- 172.21.15.112:0/1109800450 <== mon.0 v2:172.21.15.112:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b4006a080 con 0x55f84bffd860 2023-01-24T03:30:46.996 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.972+0000 7f7b2e7fc700 1 -- 172.21.15.112:0/1109800450 <== mon.0 v2:172.21.15.112:3300/0 5 ==== fsmap(e 10) v1 ==== 4198+0+0 (secure 0 0 0) 0x7f7b4007d250 con 0x55f84bffd860 2023-01-24T03:30:46.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.972+0000 7f7b5fab2b80 10 client.4725 fetch_fsmap finished waiting for FSMap version 10 2023-01-24T03:30:46.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.972+0000 7f7b5fab2b80 10 client.4725 resolve_mds: resolved name 'd' to daemon mds.d 2023-01-24T03:30:46.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.972+0000 7f7b5fab2b80 20 client.4725 populate_metadata read hostname 'smithi112' 2023-01-24T03:30:46.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.972+0000 7f7b5fab2b80 1 --2- 172.21.15.112:0/1109800450 >> [v2:172.21.15.142:6833/2762621058,v1:172.21.15.142:6835/2762621058] conn(0x55f84bfe55c0 0x55f84bfe5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:46.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.973+0000 7f7b5fab2b80 4 client.4725 mds_command: new command op to 4293 tid=0[{"prefix": "get_command_descriptions"}] 2023-01-24T03:30:46.998 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.973+0000 7f7b5fab2b80 1 -- 172.21.15.112:0/1109800450 --> [v2:172.21.15.142:6833/2762621058,v1:172.21.15.142:6835/2762621058] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f84bea8480 con 0x55f84bfe55c0 2023-01-24T03:30:46.998 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.973+0000 7f7b4ffff700 1 --2- 172.21.15.112:0/1109800450 >> [v2:172.21.15.142:6833/2762621058,v1:172.21.15.142:6835/2762621058] conn(0x55f84bfe55c0 0x55f84bfe5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:46.998 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.973+0000 7f7b2e7fc700 10 client.4725 ms_handle_connect on v2:172.21.15.142:6833/2762621058 2023-01-24T03:30:46.998 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.973+0000 7f7b4ffff700 1 --2- 172.21.15.112:0/1109800450 >> [v2:172.21.15.142:6833/2762621058,v1:172.21.15.142:6835/2762621058] conn(0x55f84bfe55c0 0x55f84bfe5990 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77926d1ffe2cae25 server_cookie=d8d8c02e9902c7dc in_seq=0 out_seq=0 2023-01-24T03:30:46.998 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.974+0000 7f7b2e7fc700 1 -- 172.21.15.112:0/1109800450 <== mds.0 v2:172.21.15.142:6833/2762621058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17090 (crc 0 0 0) 0x55f84bea8480 con 0x55f84bfe55c0 2023-01-24T03:30:46.999 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.974+0000 7f7b2e7fc700 10 client.4725 handle_command_reply: tid=0 2023-01-24T03:30:46.999 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.974+0000 7f7b5fab2b80 1 client.4725 shutdown 2023-01-24T03:30:46.999 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.974+0000 7f7b5fab2b80 1 -- 172.21.15.112:0/1109800450 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f84bffd860 msgr2=0x55f84c007d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:46.999 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.974+0000 7f7b5fab2b80 1 --2- 172.21.15.112:0/1109800450 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f84bffd860 0x55f84c007d60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7b40050f40 tx=0x7f7b40051690 comp rx=0 tx=0).stop 2023-01-24T03:30:46.999 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.974+0000 7f7b5fab2b80 1 -- 172.21.15.112:0/1109800450 shutdown_connections 2023-01-24T03:30:47.000 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.974+0000 7f7b5fab2b80 1 --2- 172.21.15.112:0/1109800450 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x55f84bffe170 0x55f84bfef470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:47.000 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.974+0000 7f7b5fab2b80 1 --2- 172.21.15.112:0/1109800450 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f84bfb5d90 0x55f84bfb2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:47.000 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.974+0000 7f7b5fab2b80 1 --2- 172.21.15.112:0/1109800450 >> [v2:172.21.15.142:6833/2762621058,v1:172.21.15.142:6835/2762621058] conn(0x55f84bfe55c0 0x55f84bfe5990 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:47.000 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.974+0000 7f7b5fab2b80 1 --2- 172.21.15.112:0/1109800450 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f84bffd860 0x55f84c007d60 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:47.000 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.974+0000 7f7b2e7fc700 0 client.4725 ms_handle_reset on v2:172.21.15.142:6833/2762621058 2023-01-24T03:30:47.001 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.974+0000 7f7b5fab2b80 1 -- 172.21.15.112:0/1109800450 >> 172.21.15.112:0/1109800450 conn(0x55f84bfb22e0 msgr2=0x55f84bfaeff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:47.001 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.974+0000 7f7b5fab2b80 1 -- 172.21.15.112:0/1109800450 shutdown_connections 2023-01-24T03:30:47.001 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.974+0000 7f7b5fab2b80 1 -- 172.21.15.112:0/1109800450 wait complete. 2023-01-24T03:30:47.001 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.974+0000 7f7b5fab2b80 20 client.4725 trim_cache size 0 max 16384 2023-01-24T03:30:47.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.991+0000 7f7b5fab2b80 1 Processor -- start 2023-01-24T03:30:47.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.991+0000 7f7b5fab2b80 1 -- start start 2023-01-24T03:30:47.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.992+0000 7f7b5fab2b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f84bfb5d90 0x55f84bffc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:47.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.992+0000 7f7b5fab2b80 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x55f84bffd860 0x55f84bff4c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:47.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.992+0000 7f7b5fab2b80 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f84bffe170 0x55f84bff51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:47.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.992+0000 7f7b5fab2b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f84bff22d0 con 0x55f84bfb5d90 2023-01-24T03:30:47.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.992+0000 7f7b5fab2b80 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f84bfef580 con 0x55f84bffe170 2023-01-24T03:30:47.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.992+0000 7f7b5fab2b80 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f84bffe030 con 0x55f84bffd860 2023-01-24T03:30:47.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.992+0000 7f7b4ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f84bfb5d90 0x55f84bffc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:47.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.992+0000 7f7b4ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f84bfb5d90 0x55f84bffc8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50982/0 (socket says 172.21.15.112:50982) 2023-01-24T03:30:47.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.992+0000 7f7b4ffff700 1 -- 172.21.15.112:0/2181775434 learned_addr learned my addr 172.21.15.112:0/2181775434 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:47.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.992+0000 7f7b54d66700 1 --2- 172.21.15.112:0/2181775434 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f84bffe170 0x55f84bff51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:47.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.992+0000 7f7b4f7fe700 1 --2- 172.21.15.112:0/2181775434 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x55f84bffd860 0x55f84bff4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:47.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.992+0000 7f7b4ffff700 1 -- 172.21.15.112:0/2181775434 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x55f84bffd860 msgr2=0x55f84bff4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:47.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.992+0000 7f7b4ffff700 1 --2- 172.21.15.112:0/2181775434 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x55f84bffd860 0x55f84bff4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:47.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.992+0000 7f7b4ffff700 1 -- 172.21.15.112:0/2181775434 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f84bffe170 msgr2=0x55f84bff51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:47.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.992+0000 7f7b4ffff700 1 --2- 172.21.15.112:0/2181775434 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f84bffe170 0x55f84bff51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:47.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.992+0000 7f7b4ffff700 1 -- 172.21.15.112:0/2181775434 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f84bd69f80 con 0x55f84bfb5d90 2023-01-24T03:30:47.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.992+0000 7f7b4ffff700 1 --2- 172.21.15.112:0/2181775434 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f84bfb5d90 0x55f84bffc8e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7b38009ea0 tx=0x7f7b38013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca52130a7b7b408e server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:47.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.992+0000 7f7b45ffb700 1 -- 172.21.15.112:0/2181775434 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b38021070 con 0x55f84bfb5d90 2023-01-24T03:30:47.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.992+0000 7f7b45ffb700 1 -- 172.21.15.112:0/2181775434 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b3802f040 con 0x55f84bfb5d90 2023-01-24T03:30:47.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.993+0000 7f7b45ffb700 1 -- 172.21.15.112:0/2181775434 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3801b950 con 0x55f84bfb5d90 2023-01-24T03:30:47.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.993+0000 7f7b5fab2b80 1 -- 172.21.15.112:0/2181775434 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f84bfb5d90 msgr2=0x55f84bffc8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:47.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.993+0000 7f7b5fab2b80 1 --2- 172.21.15.112:0/2181775434 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f84bfb5d90 0x55f84bffc8e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7b38009ea0 tx=0x7f7b38013000 comp rx=0 tx=0).stop 2023-01-24T03:30:47.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.993+0000 7f7b5fab2b80 1 -- 172.21.15.112:0/2181775434 shutdown_connections 2023-01-24T03:30:47.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.993+0000 7f7b5fab2b80 1 --2- 172.21.15.112:0/2181775434 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f84bfb5d90 0x55f84bffc8e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:47.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.993+0000 7f7b5fab2b80 1 --2- 172.21.15.112:0/2181775434 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f84bffe170 0x55f84bff51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:47.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.993+0000 7f7b5fab2b80 1 --2- 172.21.15.112:0/2181775434 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x55f84bffd860 0x55f84bff4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:47.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.993+0000 7f7b5fab2b80 1 -- 172.21.15.112:0/2181775434 >> 172.21.15.112:0/2181775434 conn(0x55f84bfb22e0 msgr2=0x55f84bfb5680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:47.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.993+0000 7f7b5fab2b80 1 -- 172.21.15.112:0/2181775434 shutdown_connections 2023-01-24T03:30:47.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.993+0000 7f7b5fab2b80 1 -- 172.21.15.112:0/2181775434 wait complete. 2023-01-24T03:30:47.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.994+0000 7f7b5fab2b80 1 -- start start 2023-01-24T03:30:47.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.994+0000 7f7b5fab2b80 1 Processor -- start 2023-01-24T03:30:47.015 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.994+0000 7f7b5fab2b80 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f84bfb5d90 0x55f84bfed440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:47.015 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.994+0000 7f7b5fab2b80 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x55f84bffd860 0x55f84bff8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:47.015 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.994+0000 7f7b5fab2b80 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f84bffe170 0x55f84c053280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:47.015 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.994+0000 7f7b5fab2b80 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f84bfe2f10 con 0x55f84bffe170 2023-01-24T03:30:47.015 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.994+0000 7f7b5fab2b80 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f84bfed980 con 0x55f84bfb5d90 2023-01-24T03:30:47.015 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.994+0000 7f7b5fab2b80 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f84bff86d0 con 0x55f84bffd860 2023-01-24T03:30:47.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.994+0000 7f7b5fab2b80 10 client.0 _ll_register_callbacks cb 0x55f84bf75ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-01-24T03:30:47.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.995+0000 7f7b54d66700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f84bffe170 0x55f84c053280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:47.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.995+0000 7f7b54d66700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f84bffe170 0x55f84c053280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:50992/0 (socket says 172.21.15.112:50992) 2023-01-24T03:30:47.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.995+0000 7f7b54d66700 1 -- 172.21.15.112:0/3120551250 learned_addr learned my addr 172.21.15.112:0/3120551250 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:47.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.995+0000 7f7b4f7fe700 1 --2- 172.21.15.112:0/3120551250 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x55f84bffd860 0x55f84bff8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:47.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.995+0000 7f7b4ffff700 1 --2- 172.21.15.112:0/3120551250 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f84bfb5d90 0x55f84bfed440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:47.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.995+0000 7f7b54d66700 1 -- 172.21.15.112:0/3120551250 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x55f84bffd860 msgr2=0x55f84bff8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:47.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.995+0000 7f7b54d66700 1 --2- 172.21.15.112:0/3120551250 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x55f84bffd860 0x55f84bff8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:47.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.995+0000 7f7b54d66700 1 -- 172.21.15.112:0/3120551250 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f84bfb5d90 msgr2=0x55f84bfed440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:47.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.995+0000 7f7b54d66700 1 --2- 172.21.15.112:0/3120551250 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f84bfb5d90 0x55f84bfed440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:47.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.995+0000 7f7b54d66700 1 -- 172.21.15.112:0/3120551250 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b500c7590 con 0x55f84bffe170 2023-01-24T03:30:47.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.995+0000 7f7b54d66700 1 --2- 172.21.15.112:0/3120551250 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f84bffe170 0x55f84c053280 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x55f84beb2f30 tx=0x7f7b48014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adb78cd8a4243f76 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:47.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.995+0000 7f7b2e7fc700 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2023-01-24T03:30:47.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.995+0000 7f7b2e7fc700 1 -- 172.21.15.112:0/3120551250 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b48001ca0 con 0x55f84bffe170 2023-01-24T03:30:47.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.995+0000 7f7b2e7fc700 1 -- 172.21.15.112:0/3120551250 <== mon.0 v2:172.21.15.112:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f7b48001e00 con 0x55f84bffe170 2023-01-24T03:30:47.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.995+0000 7f7b5fab2b80 1 -- 172.21.15.112:0/3120551250 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7b480023e0 con 0x55f84bffe170 2023-01-24T03:30:47.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.995+0000 7f7b2e7fc700 1 -- 172.21.15.112:0/3120551250 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=10) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b4800e860 con 0x55f84bffe170 2023-01-24T03:30:47.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.995+0000 7f7b5fab2b80 10 client.4731 fetch_fsmap learned FSMap version 10 2023-01-24T03:30:47.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.995+0000 7f7b5fab2b80 1 -- 172.21.15.112:0/3120551250 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=10,monmap=2+}) v3 -- 0x55f84bd43b10 con 0x55f84bffe170 2023-01-24T03:30:47.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.995+0000 7f7b2e7fc700 1 -- 172.21.15.112:0/3120551250 <== mon.0 v2:172.21.15.112:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b480023e0 con 0x55f84bffe170 2023-01-24T03:30:47.020 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.995+0000 7f7b2e7fc700 1 -- 172.21.15.112:0/3120551250 <== mon.0 v2:172.21.15.112:3300/0 5 ==== fsmap(e 10) v1 ==== 4198+0+0 (secure 0 0 0) 0x7f7b4800e9c0 con 0x55f84bffe170 2023-01-24T03:30:47.020 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.995+0000 7f7b5fab2b80 10 client.4731 fetch_fsmap finished waiting for FSMap version 10 2023-01-24T03:30:47.020 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.995+0000 7f7b5fab2b80 10 client.4731 resolve_mds: resolved name 'd' to daemon mds.d 2023-01-24T03:30:47.020 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.995+0000 7f7b5fab2b80 20 client.4731 populate_metadata read hostname 'smithi112' 2023-01-24T03:30:47.020 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.995+0000 7f7b5fab2b80 1 --2- 172.21.15.112:0/3120551250 >> [v2:172.21.15.142:6833/2762621058,v1:172.21.15.142:6835/2762621058] conn(0x55f84bff3690 0x55f84bff3a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:47.021 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.995+0000 7f7b5fab2b80 4 client.4731 mds_command: new command op to 4293 tid=0[{"prefix": "status"}] 2023-01-24T03:30:47.021 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.995+0000 7f7b5fab2b80 1 -- 172.21.15.112:0/3120551250 --> [v2:172.21.15.142:6833/2762621058,v1:172.21.15.142:6835/2762621058] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f84becbac0 con 0x55f84bff3690 2023-01-24T03:30:47.021 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.996+0000 7f7b4ffff700 1 --2- 172.21.15.112:0/3120551250 >> [v2:172.21.15.142:6833/2762621058,v1:172.21.15.142:6835/2762621058] conn(0x55f84bff3690 0x55f84bff3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:47.021 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.996+0000 7f7b2e7fc700 10 client.4731 ms_handle_connect on v2:172.21.15.142:6833/2762621058 2023-01-24T03:30:47.021 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.996+0000 7f7b4ffff700 1 --2- 172.21.15.112:0/3120551250 >> [v2:172.21.15.142:6833/2762621058,v1:172.21.15.142:6835/2762621058] conn(0x55f84bff3690 0x55f84bff3a60 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4503e0edef3a1c30 server_cookie=fad94a9733fee9bd in_seq=0 out_seq=0 2023-01-24T03:30:47.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.996+0000 7f7b2e7fc700 1 -- 172.21.15.112:0/3120551250 <== mds.0 v2:172.21.15.142:6833/2762621058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55f84becbac0 con 0x55f84bff3690 2023-01-24T03:30:47.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.996+0000 7f7b2e7fc700 10 client.4731 handle_command_reply: tid=0 2023-01-24T03:30:47.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.996+0000 7f7b5fab2b80 1 client.4731 shutdown 2023-01-24T03:30:47.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.997+0000 7f7b5fab2b80 1 -- 172.21.15.112:0/3120551250 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f84bffe170 msgr2=0x55f84c053280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:47.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.997+0000 7f7b5fab2b80 1 --2- 172.21.15.112:0/3120551250 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f84bffe170 0x55f84c053280 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x55f84beb2f30 tx=0x7f7b48014000 comp rx=0 tx=0).stop 2023-01-24T03:30:47.023 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.997+0000 7f7b5fab2b80 1 -- 172.21.15.112:0/3120551250 shutdown_connections 2023-01-24T03:30:47.023 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.997+0000 7f7b5fab2b80 1 --2- 172.21.15.112:0/3120551250 >> [v2:172.21.15.142:6833/2762621058,v1:172.21.15.142:6835/2762621058] conn(0x55f84bff3690 0x55f84bff3a60 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:47.023 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.997+0000 7f7b5fab2b80 1 --2- 172.21.15.112:0/3120551250 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f84bffe170 0x55f84c053280 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:47.023 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.997+0000 7f7b5fab2b80 1 --2- 172.21.15.112:0/3120551250 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x55f84bfb5d90 0x55f84bfed440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:47.023 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.997+0000 7f7b5fab2b80 1 --2- 172.21.15.112:0/3120551250 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x55f84bffd860 0x55f84bff8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:47.024 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.997+0000 7f7b5fab2b80 1 -- 172.21.15.112:0/3120551250 >> 172.21.15.112:0/3120551250 conn(0x55f84bfb22e0 msgr2=0x55f84bffc8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:47.024 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.997+0000 7f7b2e7fc700 0 client.4731 ms_handle_reset on v2:172.21.15.142:6833/2762621058 2023-01-24T03:30:47.024 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.997+0000 7f7b5fab2b80 1 -- 172.21.15.112:0/3120551250 shutdown_connections 2023-01-24T03:30:47.024 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.997+0000 7f7b5fab2b80 1 -- 172.21.15.112:0/3120551250 wait complete. 2023-01-24T03:30:47.024 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.997+0000 7f7b5fab2b80 20 client.4731 trim_cache size 0 max 16384 2023-01-24T03:30:47.025 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-01-24T03:30:47.025 INFO:teuthology.orchestra.run.smithi112.stdout: "cluster_fsid": "054b9ac9-b7b0-40e5-b932-6c8a955c5a72", 2023-01-24T03:30:47.025 INFO:teuthology.orchestra.run.smithi112.stdout: "whoami": 0, 2023-01-24T03:30:47.026 INFO:teuthology.orchestra.run.smithi112.stdout: "id": 4293, 2023-01-24T03:30:47.026 INFO:teuthology.orchestra.run.smithi112.stdout: "want_state": "up:active", 2023-01-24T03:30:47.026 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "up:active", 2023-01-24T03:30:47.026 INFO:teuthology.orchestra.run.smithi112.stdout: "fs_name": "cephfs", 2023-01-24T03:30:47.026 INFO:teuthology.orchestra.run.smithi112.stdout: "rank_uptime": 26.024746604000001, 2023-01-24T03:30:47.027 INFO:teuthology.orchestra.run.smithi112.stdout: "mdsmap_epoch": 10, 2023-01-24T03:30:47.027 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap_epoch": 24, 2023-01-24T03:30:47.027 INFO:teuthology.orchestra.run.smithi112.stdout: "osdmap_epoch_barrier": 24, 2023-01-24T03:30:47.027 INFO:teuthology.orchestra.run.smithi112.stdout: "uptime": 30.830646473000002 2023-01-24T03:30:47.027 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-01-24T03:30:47.029 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.997+0000 7f7b55567700 1 -- 172.21.15.112:0/3123798691 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7b340611c0 msgr2=0x7f7b34063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:47.029 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.997+0000 7f7b55567700 1 --2- 172.21.15.112:0/3123798691 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7b340611c0 0x7f7b34063670 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7b38009340 tx=0x7f7b38016000 comp rx=0 tx=0).stop 2023-01-24T03:30:47.029 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.997+0000 7f7b55567700 1 -- 172.21.15.112:0/3123798691 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b500c7800 msgr2=0x7f7b501669d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:47.030 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.997+0000 7f7b55567700 1 --2- 172.21.15.112:0/3123798691 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b500c7800 0x7f7b501669d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7b4000e960 tx=0x7f7b40009060 comp rx=0 tx=0).stop 2023-01-24T03:30:47.030 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.997+0000 7f7b55567700 1 -- 172.21.15.112:0/3123798691 shutdown_connections 2023-01-24T03:30:47.030 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.997+0000 7f7b55567700 1 --2- 172.21.15.112:0/3123798691 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7b340611c0 0x7f7b34063670 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:47.030 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.997+0000 7f7b55567700 1 --2- 172.21.15.112:0/3123798691 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7b500c82a0 0x7f7b5014cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:47.030 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.997+0000 7f7b55567700 1 --2- 172.21.15.112:0/3123798691 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7b500c6e00 0x7f7b5015a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:47.031 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.997+0000 7f7b55567700 1 --2- 172.21.15.112:0/3123798691 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b500c7800 0x7f7b501669d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:47.031 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.998+0000 7f7b55567700 1 -- 172.21.15.112:0/3123798691 >> 172.21.15.112:0/3123798691 conn(0x7f7b5005d830 msgr2=0x7f7b50153220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:47.031 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.998+0000 7f7b55567700 1 -- 172.21.15.112:0/3123798691 shutdown_connections 2023-01-24T03:30:47.031 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:46.998+0000 7f7b55567700 1 -- 172.21.15.112:0/3123798691 wait complete. 2023-01-24T03:30:47.032 INFO:teuthology.run_tasks:Running task kclient... 2023-01-24T03:30:47.048 INFO:tasks.kclient:Mounting kernel clients... 2023-01-24T03:30:47.049 INFO:tasks.kclient:config is {'client.0': {}} 2023-01-24T03:30:47.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-01-24T03:30:47.049 DEBUG:teuthology.orchestra.run.smithi112:> ip netns list 2023-01-24T03:30:47.063 INFO:teuthology.orchestra.run.smithi112.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2023-01-24T03:30:47.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-01-24T03:30:47.064 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-01-24T03:30:47.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-01-24T03:30:47.138 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip link delete ceph-brx 2023-01-24T03:30:47.235 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2023-01-24T03:30:47.235 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-01-24T03:30:47.348 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.345+0000 7fdf623d0700 1 Processor -- start 2023-01-24T03:30:47.348 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.345+0000 7fdf623d0700 1 -- start start 2023-01-24T03:30:47.351 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.345+0000 7fdf623d0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf5c0c7e40 0x7fdf5c0b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:47.351 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.345+0000 7fdf623d0700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fdf5c155da0 0x7fdf5c0ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:47.351 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.345+0000 7fdf623d0700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf5c156210 con 0x7fdf5c0aef20 2023-01-24T03:30:47.352 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.346+0000 7fdf623d0700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf5c0b1170 con 0x7fdf5c155da0 2023-01-24T03:30:47.352 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.346+0000 7fdf623d0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5c0b1450 con 0x7fdf5c0c7e40 2023-01-24T03:30:47.352 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.346+0000 7fdf6096d700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fdf5c0aef20 0x7fdf5c0ca3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:60220/0 (socket says 172.21.15.112:60220) 2023-01-24T03:30:47.352 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.346+0000 7fdf6096d700 1 -- 172.21.15.112:0/1395851730 learned_addr learned my addr 172.21.15.112:0/1395851730 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:47.352 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.346+0000 7fdf5bfff700 1 --2- 172.21.15.112:0/1395851730 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf5c0c7e40 0x7fdf5c0b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:47.353 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.346+0000 7fdf5b7fe700 1 --2- 172.21.15.112:0/1395851730 >> v2:172.21.15.142:3300/0 conn(0x7fdf5c155da0 0x7fdf5c0ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:47.353 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.346+0000 7fdf5affd700 1 -- 172.21.15.112:0/1395851730 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2285256206 0 0) 0x7fdf5c156210 con 0x7fdf5c0aef20 2023-01-24T03:30:47.353 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.346+0000 7fdf5affd700 1 -- 172.21.15.112:0/1395851730 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf480033f0 con 0x7fdf5c0aef20 2023-01-24T03:30:47.353 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.346+0000 7fdf5affd700 1 -- 172.21.15.112:0/1395851730 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4155015932 0 0) 0x7fdf480033f0 con 0x7fdf5c0aef20 2023-01-24T03:30:47.353 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.346+0000 7fdf5b7fe700 1 -- 172.21.15.112:0/1395851730 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf5c0c7e40 msgr2=0x7fdf5c0b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:47.354 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.346+0000 7fdf5b7fe700 1 --2- 172.21.15.112:0/1395851730 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf5c0c7e40 0x7fdf5c0b0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:47.354 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.346+0000 7fdf5b7fe700 1 -- 172.21.15.112:0/1395851730 >> v1:172.21.15.112:6789/0 conn(0x7fdf5c0aef20 legacy=0x7fdf5c0ca3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:47.354 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.346+0000 7fdf5b7fe700 1 -- 172.21.15.112:0/1395851730 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf5c0b1730 con 0x7fdf5c155da0 2023-01-24T03:30:47.354 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.346+0000 7fdf5bfff700 1 --2- 172.21.15.112:0/1395851730 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf5c0c7e40 0x7fdf5c0b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:30:47.355 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.346+0000 7fdf5b7fe700 1 --2- 172.21.15.112:0/1395851730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf5c155da0 0x7fdf5c0ae9e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdf50009760 tx=0x7fdf50009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=782cd0c52bcb2d1b server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:47.355 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.346+0000 7fdf5affd700 1 -- 172.21.15.112:0/1395851730 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf540017f0 con 0x7fdf5c155da0 2023-01-24T03:30:47.355 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.347+0000 7fdf5affd700 1 -- 172.21.15.112:0/1395851730 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf50011440 con 0x7fdf5c155da0 2023-01-24T03:30:47.355 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.347+0000 7fdf5affd700 1 -- 172.21.15.112:0/1395851730 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf500117e0 con 0x7fdf5c155da0 2023-01-24T03:30:47.355 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.347+0000 7fdf623d0700 1 -- 172.21.15.112:0/1395851730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf5c155da0 msgr2=0x7fdf5c0ae9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:47.355 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.347+0000 7fdf623d0700 1 --2- 172.21.15.112:0/1395851730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf5c155da0 0x7fdf5c0ae9e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdf50009760 tx=0x7fdf50009060 comp rx=0 tx=0).stop 2023-01-24T03:30:47.356 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.347+0000 7fdf623d0700 1 -- 172.21.15.112:0/1395851730 shutdown_connections 2023-01-24T03:30:47.356 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.347+0000 7fdf623d0700 1 --2- 172.21.15.112:0/1395851730 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf5c0c7e40 0x7fdf5c0b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:47.356 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.347+0000 7fdf623d0700 1 --2- 172.21.15.112:0/1395851730 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf5c155da0 0x7fdf5c0ae9e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:47.356 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.347+0000 7fdf623d0700 1 -- 172.21.15.112:0/1395851730 >> 172.21.15.112:0/1395851730 conn(0x7fdf5c05e2c0 msgr2=0x7fdf5c0cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:47.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.347+0000 7fdf623d0700 1 -- 172.21.15.112:0/1395851730 shutdown_connections 2023-01-24T03:30:47.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.347+0000 7fdf623d0700 1 -- 172.21.15.112:0/1395851730 wait complete. 2023-01-24T03:30:47.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.348+0000 7fdf623d0700 1 Processor -- start 2023-01-24T03:30:47.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.348+0000 7fdf623d0700 1 -- start start 2023-01-24T03:30:47.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.349+0000 7fdf623d0700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf5c0aef20 0x7fdf5c152490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:47.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.349+0000 7fdf623d0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdf5c0c7e40 0x7fdf5c0c4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:47.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.349+0000 7fdf623d0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf5c155da0 0x7fdf5c14edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:47.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.349+0000 7fdf623d0700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5c0c3210 con 0x7fdf5c0c7e40 2023-01-24T03:30:47.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.349+0000 7fdf623d0700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5c152070 con 0x7fdf5c0aef20 2023-01-24T03:30:47.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.349+0000 7fdf623d0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5c0c5580 con 0x7fdf5c155da0 2023-01-24T03:30:47.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.349+0000 7fdf5b7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdf5c0c7e40 0x7fdf5c0c4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:47.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.349+0000 7fdf5b7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdf5c0c7e40 0x7fdf5c0c4990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51002/0 (socket says 172.21.15.112:51002) 2023-01-24T03:30:47.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.349+0000 7fdf5b7fe700 1 -- 172.21.15.112:0/3446002623 learned_addr learned my addr 172.21.15.112:0/3446002623 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:47.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.349+0000 7fdf5bfff700 1 --2- 172.21.15.112:0/3446002623 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf5c0aef20 0x7fdf5c152490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:47.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.349+0000 7fdf6096d700 1 --2- 172.21.15.112:0/3446002623 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf5c155da0 0x7fdf5c14edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:47.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.349+0000 7fdf5b7fe700 1 -- 172.21.15.112:0/3446002623 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf5c155da0 msgr2=0x7fdf5c14edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:47.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.349+0000 7fdf5b7fe700 1 --2- 172.21.15.112:0/3446002623 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf5c155da0 0x7fdf5c14edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:47.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.349+0000 7fdf5b7fe700 1 -- 172.21.15.112:0/3446002623 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf5c0aef20 msgr2=0x7fdf5c152490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:47.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.349+0000 7fdf5b7fe700 1 --2- 172.21.15.112:0/3446002623 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf5c0aef20 0x7fdf5c152490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:47.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.349+0000 7fdf5b7fe700 1 -- 172.21.15.112:0/3446002623 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf4c0092a0 con 0x7fdf5c0c7e40 2023-01-24T03:30:47.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.349+0000 7fdf5b7fe700 1 --2- 172.21.15.112:0/3446002623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdf5c0c7e40 0x7fdf5c0c4990 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c089340 tx=0x7fdf50015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:47.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.350+0000 7fdf43fff700 1 -- 172.21.15.112:0/3446002623 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf500108b0 con 0x7fdf5c0c7e40 2023-01-24T03:30:47.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.350+0000 7fdf43fff700 1 -- 172.21.15.112:0/3446002623 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf50010a30 con 0x7fdf5c0c7e40 2023-01-24T03:30:47.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.350+0000 7fdf623d0700 1 -- 172.21.15.112:0/3446002623 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf500092a0 con 0x7fdf5c0c7e40 2023-01-24T03:30:47.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.350+0000 7fdf43fff700 1 -- 172.21.15.112:0/3446002623 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf50010e30 con 0x7fdf5c0c7e40 2023-01-24T03:30:47.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.350+0000 7fdf623d0700 1 -- 172.21.15.112:0/3446002623 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf5c0c2c50 con 0x7fdf5c0c7e40 2023-01-24T03:30:47.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.350+0000 7fdf623d0700 1 -- 172.21.15.112:0/3446002623 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf5c151dd0 con 0x7fdf5c0c7e40 2023-01-24T03:30:47.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.351+0000 7fdf43fff700 1 -- 172.21.15.112:0/3446002623 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fdf5000f070 con 0x7fdf5c0c7e40 2023-01-24T03:30:47.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.351+0000 7fdf43fff700 1 --2- 172.21.15.112:0/3446002623 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fdf3c0611c0 0x7fdf3c063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:47.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.351+0000 7fdf43fff700 1 -- 172.21.15.112:0/3446002623 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fdf5007ecf0 con 0x7fdf5c0c7e40 2023-01-24T03:30:47.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.351+0000 7fdf5bfff700 1 --2- 172.21.15.112:0/3446002623 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fdf3c0611c0 0x7fdf3c063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:47.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.352+0000 7fdf5bfff700 1 --2- 172.21.15.112:0/3446002623 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fdf3c0611c0 0x7fdf3c063670 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c009dc0 tx=0x7fdf4c009060 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:47.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.355+0000 7fdf43fff700 1 -- 172.21.15.112:0/3446002623 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fdf5c151dd0 con 0x7fdf5c0c7e40 2023-01-24T03:30:47.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.575+0000 7fdf623d0700 1 -- 172.21.15.112:0/3446002623 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fdf5c1521b0 con 0x7fdf5c0c7e40 2023-01-24T03:30:47.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.575+0000 7fdf43fff700 1 -- 172.21.15.112:0/3446002623 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v10) v1 ==== 53+0+88 (secure 0 0 0) 0x7fdf5c1521b0 con 0x7fdf5c0c7e40 2023-01-24T03:30:47.579 INFO:teuthology.orchestra.run.smithi112.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-01-24T03:30:47.581 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.577+0000 7fdf623d0700 1 -- 172.21.15.112:0/3446002623 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fdf3c0611c0 msgr2=0x7fdf3c063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:47.581 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.577+0000 7fdf623d0700 1 --2- 172.21.15.112:0/3446002623 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fdf3c0611c0 0x7fdf3c063670 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c009dc0 tx=0x7fdf4c009060 comp rx=0 tx=0).stop 2023-01-24T03:30:47.581 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.577+0000 7fdf623d0700 1 -- 172.21.15.112:0/3446002623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdf5c0c7e40 msgr2=0x7fdf5c0c4990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:47.582 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.577+0000 7fdf623d0700 1 --2- 172.21.15.112:0/3446002623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdf5c0c7e40 0x7fdf5c0c4990 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c089340 tx=0x7fdf50015000 comp rx=0 tx=0).stop 2023-01-24T03:30:47.582 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.577+0000 7fdf623d0700 1 -- 172.21.15.112:0/3446002623 shutdown_connections 2023-01-24T03:30:47.582 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.577+0000 7fdf623d0700 1 --2- 172.21.15.112:0/3446002623 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fdf3c0611c0 0x7fdf3c063670 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:47.582 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.577+0000 7fdf623d0700 1 --2- 172.21.15.112:0/3446002623 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf5c155da0 0x7fdf5c14edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:47.582 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.577+0000 7fdf623d0700 1 --2- 172.21.15.112:0/3446002623 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf5c0aef20 0x7fdf5c152490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:47.583 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.577+0000 7fdf623d0700 1 --2- 172.21.15.112:0/3446002623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdf5c0c7e40 0x7fdf5c0c4990 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:47.583 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.577+0000 7fdf623d0700 1 -- 172.21.15.112:0/3446002623 >> 172.21.15.112:0/3446002623 conn(0x7fdf5c05e2c0 msgr2=0x7fdf5c1551b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:47.583 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.577+0000 7fdf623d0700 1 -- 172.21.15.112:0/3446002623 shutdown_connections 2023-01-24T03:30:47.583 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:47.577+0000 7fdf623d0700 1 -- 172.21.15.112:0/3446002623 wait complete. 2023-01-24T03:30:47.593 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-01-24T03:30:47.593 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-01-24T03:30:47.593 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi112.front.sepia.ceph.com 2023-01-24T03:30:47.593 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-01-24T03:30:47.594 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-01-24T03:30:47.594 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-01-24T03:30:47.594 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-01-24T03:30:47.594 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-01-24T03:30:47.595 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-01-24T03:30:47.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-01-24T03:30:47.595 DEBUG:teuthology.orchestra.run.smithi112:> ip addr 2023-01-24T03:30:47.610 INFO:teuthology.orchestra.run.smithi112.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-01-24T03:30:47.610 INFO:teuthology.orchestra.run.smithi112.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-01-24T03:30:47.610 INFO:teuthology.orchestra.run.smithi112.stdout: inet 127.0.0.1/8 scope host lo 2023-01-24T03:30:47.611 INFO:teuthology.orchestra.run.smithi112.stdout: valid_lft forever preferred_lft forever 2023-01-24T03:30:47.611 INFO:teuthology.orchestra.run.smithi112.stdout: inet6 ::1/128 scope host 2023-01-24T03:30:47.611 INFO:teuthology.orchestra.run.smithi112.stdout: valid_lft forever preferred_lft forever 2023-01-24T03:30:47.611 INFO:teuthology.orchestra.run.smithi112.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-01-24T03:30:47.611 INFO:teuthology.orchestra.run.smithi112.stdout: link/ether 0c:c4:7a:d8:a5:2c brd ff:ff:ff:ff:ff:ff 2023-01-24T03:30:47.612 INFO:teuthology.orchestra.run.smithi112.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-01-24T03:30:47.612 INFO:teuthology.orchestra.run.smithi112.stdout: link/ether 0c:c4:7a:d8:a5:2d brd ff:ff:ff:ff:ff:ff 2023-01-24T03:30:47.612 INFO:teuthology.orchestra.run.smithi112.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-01-24T03:30:47.612 INFO:teuthology.orchestra.run.smithi112.stdout: link/ether 0c:c4:7a:8f:cb:68 brd ff:ff:ff:ff:ff:ff 2023-01-24T03:30:47.612 INFO:teuthology.orchestra.run.smithi112.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-01-24T03:30:47.613 INFO:teuthology.orchestra.run.smithi112.stdout: link/ether 0c:c4:7a:8f:cb:69 brd ff:ff:ff:ff:ff:ff 2023-01-24T03:30:47.613 INFO:teuthology.orchestra.run.smithi112.stdout: inet 172.21.15.112/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-01-24T03:30:47.613 INFO:teuthology.orchestra.run.smithi112.stdout: valid_lft 42591sec preferred_lft 42591sec 2023-01-24T03:30:47.613 INFO:teuthology.orchestra.run.smithi112.stdout: inet6 fe80::ec4:7aff:fe8f:cb69/64 scope link 2023-01-24T03:30:47.613 INFO:teuthology.orchestra.run.smithi112.stdout: valid_lft forever preferred_lft forever 2023-01-24T03:30:47.614 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-01-24T03:30:47.614 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-01-24T03:30:47.614 DEBUG:teuthology.orchestra.run.smithi112:> (cd / && exec bash -c ' 2023-01-24T03:30:47.615 DEBUG:teuthology.orchestra.run.smithi112:> set -e 2023-01-24T03:30:47.615 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip link add name ceph-brx type bridge 2023-01-24T03:30:47.615 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip addr flush dev ceph-brx 2023-01-24T03:30:47.616 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip link set ceph-brx up 2023-01-24T03:30:47.616 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-01-24T03:30:47.616 DEBUG:teuthology.orchestra.run.smithi112:> ') 2023-01-24T03:30:47.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-01-24T03:30:47.717 DEBUG:teuthology.orchestra.run.smithi112:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-01-24T03:30:47.740 INFO:teuthology.orchestra.run.smithi112.stdout:1 2023-01-24T03:30:47.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-01-24T03:30:47.741 DEBUG:teuthology.orchestra.run.smithi112:> route 2023-01-24T03:30:47.796 INFO:teuthology.orchestra.run.smithi112.stdout:Kernel IP routing table 2023-01-24T03:30:47.797 INFO:teuthology.orchestra.run.smithi112.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-01-24T03:30:47.797 INFO:teuthology.orchestra.run.smithi112.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-01-24T03:30:47.797 INFO:teuthology.orchestra.run.smithi112.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-01-24T03:30:47.797 INFO:teuthology.orchestra.run.smithi112.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-01-24T03:30:47.798 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-01-24T03:30:47.798 DEBUG:teuthology.orchestra.run.smithi112:> (cd / && exec bash -c ' 2023-01-24T03:30:47.798 DEBUG:teuthology.orchestra.run.smithi112:> set -e 2023-01-24T03:30:47.799 DEBUG:teuthology.orchestra.run.smithi112:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-01-24T03:30:47.799 DEBUG:teuthology.orchestra.run.smithi112:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-01-24T03:30:47.799 DEBUG:teuthology.orchestra.run.smithi112:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-01-24T03:30:47.800 DEBUG:teuthology.orchestra.run.smithi112:> ') 2023-01-24T03:30:47.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-01-24T03:30:47.896 DEBUG:teuthology.orchestra.run.smithi112:> ip netns list 2023-01-24T03:30:47.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-01-24T03:30:47.911 DEBUG:teuthology.orchestra.run.smithi112:> ip netns list-id 2023-01-24T03:30:47.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-01-24T03:30:47.965 DEBUG:teuthology.orchestra.run.smithi112:> (cd / && exec bash -c ' 2023-01-24T03:30:47.965 DEBUG:teuthology.orchestra.run.smithi112:> set -e 2023-01-24T03:30:47.965 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-01-24T03:30:47.966 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-01-24T03:30:47.966 DEBUG:teuthology.orchestra.run.smithi112:> ') 2023-01-24T03:30:48.051 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-01-24T03:30:48.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-01-24T03:30:48.052 DEBUG:teuthology.orchestra.run.smithi112:> (cd / && exec bash -c ' 2023-01-24T03:30:48.052 DEBUG:teuthology.orchestra.run.smithi112:> set -e 2023-01-24T03:30:48.053 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-01-24T03:30:48.053 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-01-24T03:30:48.053 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-01-24T03:30:48.053 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-01-24T03:30:48.054 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-01-24T03:30:48.054 DEBUG:teuthology.orchestra.run.smithi112:> ') 2023-01-24T03:30:48.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-01-24T03:30:48.245 DEBUG:teuthology.orchestra.run.smithi112:> (cd / && exec bash -c ' 2023-01-24T03:30:48.246 DEBUG:teuthology.orchestra.run.smithi112:> set -e 2023-01-24T03:30:48.246 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip link set brx.0 up 2023-01-24T03:30:48.246 DEBUG:teuthology.orchestra.run.smithi112:> sudo ip link set dev brx.0 master ceph-brx 2023-01-24T03:30:48.247 DEBUG:teuthology.orchestra.run.smithi112:> ') 2023-01-24T03:30:48.337 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-01-24T03:30:48.338 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-01-24T03:30:48.352 INFO:teuthology.orchestra.run.smithi112.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-01-24T03:30:48.353 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-01-24T03:30:48.354 DEBUG:teuthology.orchestra.run.smithi112:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-01-24T03:30:48.406 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-01-24T03:30:48.407 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-01-24T03:30:48.407 DEBUG:teuthology.orchestra.run.smithi112:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-01-24T03:30:48.485 INFO:teuthology.orchestra.run.smithi112.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-01-24T03:30:48.485 INFO:teuthology.orchestra.run.smithi112.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-01-24T03:30:48.485 INFO:teuthology.orchestra.run.smithi112.stdout:invalid new device string format 2023-01-24T03:30:48.578 INFO:teuthology.orchestra.run.smithi112.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-01-24T03:30:48.578 INFO:teuthology.orchestra.run.smithi112.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-01-24T03:30:48.579 INFO:teuthology.orchestra.run.smithi112.stdout:invalid new device string format 2023-01-24T03:30:48.579 INFO:teuthology.orchestra.run.smithi112.stdout:mount.ceph: resolved to: "172.21.15.112:3300,172.21.15.142:3300,172.21.15.142:3301" 2023-01-24T03:30:48.579 INFO:teuthology.orchestra.run.smithi112.stdout:mount.ceph: switching to using v1 address with old syntax 2023-01-24T03:30:48.618 INFO:teuthology.orchestra.run.smithi112.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-01-24T03:30:48.619 INFO:teuthology.orchestra.run.smithi112.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-01-24T03:30:48.619 INFO:teuthology.orchestra.run.smithi112.stdout:invalid new device string format 2023-01-24T03:30:48.619 INFO:teuthology.orchestra.run.smithi112.stdout:mount.ceph: resolved to: "172.21.15.112:3300,172.21.15.142:3300,172.21.15.142:3301" 2023-01-24T03:30:48.619 INFO:teuthology.orchestra.run.smithi112.stdout:mount.ceph: switching to using v1 address with old syntax 2023-01-24T03:30:48.620 INFO:teuthology.orchestra.run.smithi112.stdout:mount.ceph: resolved to: "172.21.15.112:6789,172.21.15.142:6789,172.21.15.142:6790" 2023-01-24T03:30:48.620 INFO:teuthology.orchestra.run.smithi112.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.112:6789,172.21.15.142:6789,172.21.15.142:6790:/ 2023-01-24T03:30:48.620 INFO:teuthology.orchestra.run.smithi112.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=054b9ac9-b7b0-40e5-b932-6c8a955c5a72" will pass to kernel 2023-01-24T03:30:48.621 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-01-24T03:30:48.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-01-24T03:30:48.622 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-01-24T03:30:48.685 INFO:teuthology.run_tasks:Running task mds_thrash... 2023-01-24T03:30:48.698 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T03:30:48.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.835+0000 7fa84359e700 1 Processor -- start 2023-01-24T03:30:48.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.835+0000 7fa84359e700 1 -- start start 2023-01-24T03:30:48.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.836+0000 7fa84359e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa844156c10 0x7fa84405df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:48.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.836+0000 7fa84359e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa84405e5b0 con 0x7fa8440c3730 2023-01-24T03:30:48.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.836+0000 7fa84359e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa84406cd70 con 0x7fa8440c3360 2023-01-24T03:30:48.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.836+0000 7fa84359e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa84406d050 con 0x7fa844156c10 2023-01-24T03:30:48.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.836+0000 7fa83bfff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fa8440c3730 0x7fa844156500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:60242/0 (socket says 172.21.15.112:60242) 2023-01-24T03:30:48.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.836+0000 7fa83bfff700 1 -- 172.21.15.112:0/2562169752 learned_addr learned my addr 172.21.15.112:0/2562169752 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:48.843 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.836+0000 7fa842d9d700 1 --2- 172.21.15.112:0/2562169752 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa844156c10 0x7fa84405df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:48.843 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.836+0000 7fa841d9b700 1 -- 172.21.15.112:0/2562169752 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 400034106 0 0) 0x7fa84405e5b0 con 0x7fa8440c3730 2023-01-24T03:30:48.843 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.836+0000 7fa841d9b700 1 -- 172.21.15.112:0/2562169752 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8240033f0 con 0x7fa8440c3730 2023-01-24T03:30:48.844 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.836+0000 7fa841d9b700 1 -- 172.21.15.112:0/2562169752 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1110232244 0 0) 0x7fa84406cd70 con 0x7fa8440c3360 2023-01-24T03:30:48.844 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.836+0000 7fa841d9b700 1 -- 172.21.15.112:0/2562169752 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa84405e5b0 con 0x7fa8440c3360 2023-01-24T03:30:48.844 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.836+0000 7fa841d9b700 1 -- 172.21.15.112:0/2562169752 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 204373904 0 0) 0x7fa8240033f0 con 0x7fa8440c3730 2023-01-24T03:30:48.844 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.836+0000 7fa842d9d700 1 -- 172.21.15.112:0/2562169752 >> v1:172.21.15.142:6789/0 conn(0x7fa8440c3360 legacy=0x7fa844067f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:48.845 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.836+0000 7fa842d9d700 1 -- 172.21.15.112:0/2562169752 >> v1:172.21.15.112:6789/0 conn(0x7fa8440c3730 legacy=0x7fa844156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:48.845 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.836+0000 7fa842d9d700 1 -- 172.21.15.112:0/2562169752 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa84406d330 con 0x7fa844156c10 2023-01-24T03:30:48.845 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.837+0000 7fa842d9d700 1 --2- 172.21.15.112:0/2562169752 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa844156c10 0x7fa84405df40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa83c009e90 tx=0x7fa83c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=784fa2c58751e13e server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:48.846 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.837+0000 7fa841d9b700 1 -- 172.21.15.112:0/2562169752 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83c00e070 con 0x7fa844156c10 2023-01-24T03:30:48.847 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.837+0000 7fa841d9b700 1 -- 172.21.15.112:0/2562169752 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa83c00b870 con 0x7fa844156c10 2023-01-24T03:30:48.847 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.837+0000 7fa841d9b700 1 -- 172.21.15.112:0/2562169752 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82c001470 con 0x7fa844156c10 2023-01-24T03:30:48.847 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.837+0000 7fa84359e700 1 -- 172.21.15.112:0/2562169752 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa844156c10 msgr2=0x7fa84405df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:48.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.837+0000 7fa84359e700 1 --2- 172.21.15.112:0/2562169752 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa844156c10 0x7fa84405df40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa83c009e90 tx=0x7fa83c009780 comp rx=0 tx=0).stop 2023-01-24T03:30:48.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.837+0000 7fa84359e700 1 -- 172.21.15.112:0/2562169752 shutdown_connections 2023-01-24T03:30:48.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.837+0000 7fa84359e700 1 --2- 172.21.15.112:0/2562169752 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa844156c10 0x7fa84405df40 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:48.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.837+0000 7fa84359e700 1 -- 172.21.15.112:0/2562169752 >> 172.21.15.112:0/2562169752 conn(0x7fa84406c360 msgr2=0x7fa84405d830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:48.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.837+0000 7fa84359e700 1 -- 172.21.15.112:0/2562169752 shutdown_connections 2023-01-24T03:30:48.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.837+0000 7fa84359e700 1 -- 172.21.15.112:0/2562169752 wait complete. 2023-01-24T03:30:48.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.838+0000 7fa84359e700 1 Processor -- start 2023-01-24T03:30:48.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.838+0000 7fa84359e700 1 -- start start 2023-01-24T03:30:48.853 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.839+0000 7fa84359e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8440c3360 0x7fa8441658b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:48.853 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.839+0000 7fa84359e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa844156c10 0x7fa8440bdde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:48.854 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.839+0000 7fa84359e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa84414ef80 0x7fa844158b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:48.854 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.839+0000 7fa84359e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8440bef30 con 0x7fa844156c10 2023-01-24T03:30:48.854 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.839+0000 7fa84359e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8440bd950 con 0x7fa84414ef80 2023-01-24T03:30:48.854 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.839+0000 7fa84359e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8440c2840 con 0x7fa8440c3360 2023-01-24T03:30:48.855 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.839+0000 7fa83bfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa844156c10 0x7fa8440bdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:48.856 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.839+0000 7fa83bfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa844156c10 0x7fa8440bdde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51026/0 (socket says 172.21.15.112:51026) 2023-01-24T03:30:48.856 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.839+0000 7fa83bfff700 1 -- 172.21.15.112:0/3102462000 learned_addr learned my addr 172.21.15.112:0/3102462000 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:48.856 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.839+0000 7fa842d9d700 1 --2- 172.21.15.112:0/3102462000 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa84414ef80 0x7fa844158b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:48.856 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.839+0000 7fa84259c700 1 --2- 172.21.15.112:0/3102462000 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8440c3360 0x7fa8441658b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:48.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.839+0000 7fa83bfff700 1 -- 172.21.15.112:0/3102462000 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8440c3360 msgr2=0x7fa8441658b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:48.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.839+0000 7fa83bfff700 1 --2- 172.21.15.112:0/3102462000 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8440c3360 0x7fa8441658b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:48.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.839+0000 7fa83bfff700 1 -- 172.21.15.112:0/3102462000 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa84414ef80 msgr2=0x7fa844158b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:48.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.839+0000 7fa83bfff700 1 --2- 172.21.15.112:0/3102462000 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa84414ef80 0x7fa844158b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:48.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.839+0000 7fa83bfff700 1 -- 172.21.15.112:0/3102462000 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa83c009a20 con 0x7fa844156c10 2023-01-24T03:30:48.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.840+0000 7fa83bfff700 1 --2- 172.21.15.112:0/3102462000 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa844156c10 0x7fa8440bdde0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa82c00c800 tx=0x7fa82c001b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:48.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.840+0000 7fa83a7fc700 1 -- 172.21.15.112:0/3102462000 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82c0073c0 con 0x7fa844156c10 2023-01-24T03:30:48.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.840+0000 7fa83a7fc700 1 -- 172.21.15.112:0/3102462000 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa82c007520 con 0x7fa844156c10 2023-01-24T03:30:48.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.840+0000 7fa84359e700 1 -- 172.21.15.112:0/3102462000 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8440bfad0 con 0x7fa844156c10 2023-01-24T03:30:48.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.840+0000 7fa83a7fc700 1 -- 172.21.15.112:0/3102462000 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82c0048e0 con 0x7fa844156c10 2023-01-24T03:30:48.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.840+0000 7fa84359e700 1 -- 172.21.15.112:0/3102462000 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8440c0090 con 0x7fa844156c10 2023-01-24T03:30:48.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.841+0000 7fa84359e700 1 -- 172.21.15.112:0/3102462000 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8440bdb00 con 0x7fa844156c10 2023-01-24T03:30:48.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.841+0000 7fa83a7fc700 1 -- 172.21.15.112:0/3102462000 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fa82c01a020 con 0x7fa844156c10 2023-01-24T03:30:48.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.841+0000 7fa83a7fc700 1 --2- 172.21.15.112:0/3102462000 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa8280611c0 0x7fa828063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:48.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.841+0000 7fa83a7fc700 1 -- 172.21.15.112:0/3102462000 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fa82c07d500 con 0x7fa844156c10 2023-01-24T03:30:48.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.841+0000 7fa84259c700 1 --2- 172.21.15.112:0/3102462000 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa8280611c0 0x7fa828063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:48.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.842+0000 7fa84259c700 1 --2- 172.21.15.112:0/3102462000 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa8280611c0 0x7fa828063670 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa834003930 tx=0x7fa834007bd0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:48.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:48.845+0000 7fa83a7fc700 1 -- 172.21.15.112:0/3102462000 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fa8440bdb00 con 0x7fa844156c10 2023-01-24T03:30:49.033 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.030+0000 7fa84359e700 1 -- 172.21.15.112:0/3102462000 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8440bdac0 con 0x7fa844156c10 2023-01-24T03:30:49.034 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.031+0000 7fa83a7fc700 1 -- 172.21.15.112:0/3102462000 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18755 (secure 0 0 0) 0x7fa82c050df0 con 0x7fa844156c10 2023-01-24T03:30:49.035 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:30:49.035 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":24,"fsid":"054b9ac9-b7b0-40e5-b932-6c8a955c5a72","created":"2023-01-24T03:30:03.798533+0000","modified":"2023-01-24T03:30:25.373340+0000","last_up_change":"2023-01-24T03:30:13.945316+0000","last_in_change":"2023-01-24T03:30:06.198433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-01-24T03:30:14.931166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-01-24T03:30:15.023730+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-01-24T03:30:19.060597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-01-24T03:30:19.943686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-01-24T03:30:22.999036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"fe6ad231-a39a-42eb-9a3a-49d008dd35e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6809","nonce":84512},{"type":"v1","addr":"172.21.15.112:6811","nonce":84512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":84512},{"type":"v1","addr":"172.21.15.112:6817","nonce":84512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6825","nonce":84512},{"type":"v1","addr":"172.21.15.112:6828","nonce":84512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6819","nonce":84512},{"type":"v1","addr":"172.21.15.112:6822","nonce":84512}]},"public_addr":"172.21.15.112:6811/84512","cluster_addr":"172.21.15.112:6817/84512","heartbeat_back_addr":"172.21.15.112:6828/84512","heartbeat_front_addr":"172.21.15.112:6822/84512","state":["exists","up"]},{"osd":1,"uuid":"8526a1d2-d540-4021-9766-c4bbb30317ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":84513},{"type":"v1","addr":"172.21.15.112:6812","nonce":84513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6815","nonce":84513},{"type":"v1","addr":"172.21.15.112:6818","nonce":84513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6827","nonce":84513},{"type":"v1","addr":"172.21.15.112:6830","nonce":84513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6821","nonce":84513},{"type":"v1","addr":"172.21.15.112:6824","nonce":84513}]},"public_addr":"172.21.15.112:6812/84513","cluster_addr":"172.21.15.112:6818/84513","heartbeat_back_addr":"172.21.15.112:6830/84513","heartbeat_front_addr":"172.21.15.112:6824/84513","state":["exists","up"]},{"osd":2,"uuid":"041ef8e4-4858-432d-9476-cbdb3a0dc47c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":84511},{"type":"v1","addr":"172.21.15.112:6813","nonce":84511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":84511},{"type":"v1","addr":"172.21.15.112:6820","nonce":84511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6829","nonce":84511},{"type":"v1","addr":"172.21.15.112:6831","nonce":84511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6823","nonce":84511},{"type":"v1","addr":"172.21.15.112:6826","nonce":84511}]},"public_addr":"172.21.15.112:6813/84511","cluster_addr":"172.21.15.112:6820/84511","heartbeat_back_addr":"172.21.15.112:6831/84511","heartbeat_front_addr":"172.21.15.112:6826/84511","state":["exists","up"]},{"osd":3,"uuid":"987587cf-1357-4a70-ad85-67b189c67729","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":84510},{"type":"v1","addr":"172.21.15.112:6801","nonce":84510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":84510},{"type":"v1","addr":"172.21.15.112:6803","nonce":84510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":84510},{"type":"v1","addr":"172.21.15.112:6807","nonce":84510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":84510},{"type":"v1","addr":"172.21.15.112:6805","nonce":84510}]},"public_addr":"172.21.15.112:6801/84510","cluster_addr":"172.21.15.112:6803/84510","heartbeat_back_addr":"172.21.15.112:6807/84510","heartbeat_front_addr":"172.21.15.112:6805/84510","state":["exists","up"]},{"osd":4,"uuid":"8503512b-43db-40b8-8b2b-c86cf2e54e17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":84614},{"type":"v1","addr":"172.21.15.142:6801","nonce":84614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":84614},{"type":"v1","addr":"172.21.15.142:6803","nonce":84614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":84614},{"type":"v1","addr":"172.21.15.142:6807","nonce":84614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":84614},{"type":"v1","addr":"172.21.15.142:6805","nonce":84614}]},"public_addr":"172.21.15.142:6801/84614","cluster_addr":"172.21.15.142:6803/84614","heartbeat_back_addr":"172.21.15.142:6807/84614","heartbeat_front_addr":"172.21.15.142:6805/84614","state":["exists","up"]},{"osd":5,"uuid":"d3bc0030-8b08-4645-9ec2-1f8a4e29ef36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":84638},{"type":"v1","addr":"172.21.15.142:6817","nonce":84638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":84638},{"type":"v1","addr":"172.21.15.142:6819","nonce":84638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":84638},{"type":"v1","addr":"172.21.15.142:6823","nonce":84638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":84638},{"type":"v1","addr":"172.21.15.142:6821","nonce":84638}]},"public_addr":"172.21.15.142:6817/84638","cluster_addr":"172.21.15.142:6819/84638","heartbeat_back_addr":"172.21.15.142:6823/84638","heartbeat_front_addr":"172.21.15.142:6821/84638","state":["exists","up"]},{"osd":6,"uuid":"c7649e3b-832d-4e6f-8866-b15e5d814476","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":84645},{"type":"v1","addr":"172.21.15.142:6825","nonce":84645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":84645},{"type":"v1","addr":"172.21.15.142:6827","nonce":84645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":84645},{"type":"v1","addr":"172.21.15.142:6831","nonce":84645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":84645},{"type":"v1","addr":"172.21.15.142:6829","nonce":84645}]},"public_addr":"172.21.15.142:6825/84645","cluster_addr":"172.21.15.142:6827/84645","heartbeat_back_addr":"172.21.15.142:6831/84645","heartbeat_front_addr":"172.21.15.142:6829/84645","state":["exists","up"]},{"osd":7,"uuid":"2ea5b3bf-884e-4b20-b572-1443efdc7477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":84615},{"type":"v1","addr":"172.21.15.142:6809","nonce":84615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":84615},{"type":"v1","addr":"172.21.15.142:6811","nonce":84615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":84615},{"type":"v1","addr":"172.21.15.142:6815","nonce":84615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":84615},{"type":"v1","addr":"172.21.15.142:6813","nonce":84615}]},"public_addr":"172.21.15.142:6809/84615","cluster_addr":"172.21.15.142:6811/84615","heartbeat_back_addr":"172.21.15.142:6815/84615","heartbeat_front_addr":"172.21.15.142:6813/84615","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.370027+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.335683+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.350667+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.350794+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.391119+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.422933+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.400439+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.354018+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T03:30:49.036 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.033+0000 7fa84359e700 1 -- 172.21.15.112:0/3102462000 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa8280611c0 msgr2=0x7fa828063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:49.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.033+0000 7fa84359e700 1 --2- 172.21.15.112:0/3102462000 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa8280611c0 0x7fa828063670 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa834003930 tx=0x7fa834007bd0 comp rx=0 tx=0).stop 2023-01-24T03:30:49.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.033+0000 7fa84359e700 1 -- 172.21.15.112:0/3102462000 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa844156c10 msgr2=0x7fa8440bdde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:49.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.033+0000 7fa84359e700 1 --2- 172.21.15.112:0/3102462000 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa844156c10 0x7fa8440bdde0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa82c00c800 tx=0x7fa82c001b50 comp rx=0 tx=0).stop 2023-01-24T03:30:49.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.033+0000 7fa84359e700 1 -- 172.21.15.112:0/3102462000 shutdown_connections 2023-01-24T03:30:49.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.033+0000 7fa84359e700 1 --2- 172.21.15.112:0/3102462000 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa8280611c0 0x7fa828063670 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:49.038 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.033+0000 7fa84359e700 1 --2- 172.21.15.112:0/3102462000 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa84414ef80 0x7fa844158b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:49.038 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.033+0000 7fa84359e700 1 --2- 172.21.15.112:0/3102462000 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8440c3360 0x7fa8441658b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:49.038 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.033+0000 7fa84359e700 1 --2- 172.21.15.112:0/3102462000 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa844156c10 0x7fa8440bdde0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:49.038 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.033+0000 7fa84359e700 1 -- 172.21.15.112:0/3102462000 >> 172.21.15.112:0/3102462000 conn(0x7fa84406c360 msgr2=0x7fa84405d830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:49.039 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.033+0000 7fa84359e700 1 -- 172.21.15.112:0/3102462000 shutdown_connections 2023-01-24T03:30:49.039 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.033+0000 7fa84359e700 1 -- 172.21.15.112:0/3102462000 wait complete. 2023-01-24T03:30:49.048 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-01-24T03:30:14.931166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-01-24T03:30:15.023730+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-01-24T03:30:19.060597+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-01-24T03:30:19.943686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-01-24T03:30:22.999036+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-01-24T03:30:49.049 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-01-24T03:30:49.164 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.161+0000 7fa557fff700 1 Processor -- start 2023-01-24T03:30:49.164 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.161+0000 7fa557fff700 1 -- start start 2023-01-24T03:30:49.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.162+0000 7fa557fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa5580c6420 0x7fa5580c4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:49.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.162+0000 7fa557fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa558155ba0 0x7fa5580c8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:49.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.162+0000 7fa557fff700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5580c82e0 con 0x7fa55814f3f0 2023-01-24T03:30:49.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.162+0000 7fa557fff700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5580c8cb0 con 0x7fa558155ba0 2023-01-24T03:30:49.172 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.162+0000 7fa557fff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5580c2980 con 0x7fa5580c6420 2023-01-24T03:30:49.172 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.162+0000 7fa54ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa5580c6420 0x7fa5580c4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:49.172 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.162+0000 7fa5577fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa558155ba0 0x7fa5580c8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:49.172 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.162+0000 7fa54ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa5580c6420 0x7fa5580c4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.112:38340/0 (socket says 172.21.15.112:38340) 2023-01-24T03:30:49.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.162+0000 7fa54ffff700 1 -- 172.21.15.112:0/160070176 learned_addr learned my addr 172.21.15.112:0/160070176 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:49.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.162+0000 7fa5577fe700 1 -- 172.21.15.112:0/160070176 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa5580c6420 msgr2=0x7fa5580c4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:49.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.162+0000 7fa5577fe700 1 --2- 172.21.15.112:0/160070176 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa5580c6420 0x7fa5580c4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:49.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.162+0000 7fa5577fe700 1 -- 172.21.15.112:0/160070176 >> v1:172.21.15.142:6789/0 conn(0x7fa55814f3f0 legacy=0x7fa5580c2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:49.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.162+0000 7fa5577fe700 1 -- 172.21.15.112:0/160070176 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5580c5290 con 0x7fa558155ba0 2023-01-24T03:30:49.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.162+0000 7fa54ffff700 1 --2- 172.21.15.112:0/160070176 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa5580c6420 0x7fa5580c4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:30:49.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.163+0000 7fa5577fe700 1 --2- 172.21.15.112:0/160070176 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa558155ba0 0x7fa5580c8770 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa550009e60 tx=0x7fa550009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7beb5b55c1c73037 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:49.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.163+0000 7fa5567fc700 1 -- 172.21.15.112:0/160070176 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa55000e070 con 0x7fa558155ba0 2023-01-24T03:30:49.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.163+0000 7fa5567fc700 1 -- 172.21.15.112:0/160070176 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa55000bb60 con 0x7fa558155ba0 2023-01-24T03:30:49.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.163+0000 7fa5567fc700 1 -- 172.21.15.112:0/160070176 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa550012430 con 0x7fa558155ba0 2023-01-24T03:30:49.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.163+0000 7fa557fff700 1 -- 172.21.15.112:0/160070176 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa558155ba0 msgr2=0x7fa5580c8770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:49.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.163+0000 7fa557fff700 1 --2- 172.21.15.112:0/160070176 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa558155ba0 0x7fa5580c8770 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa550009e60 tx=0x7fa550009780 comp rx=0 tx=0).stop 2023-01-24T03:30:49.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.163+0000 7fa557fff700 1 -- 172.21.15.112:0/160070176 shutdown_connections 2023-01-24T03:30:49.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.163+0000 7fa557fff700 1 --2- 172.21.15.112:0/160070176 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa558155ba0 0x7fa5580c8770 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:49.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.163+0000 7fa557fff700 1 --2- 172.21.15.112:0/160070176 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa5580c6420 0x7fa5580c4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:49.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.163+0000 7fa557fff700 1 -- 172.21.15.112:0/160070176 >> 172.21.15.112:0/160070176 conn(0x7fa55805d830 msgr2=0x7fa55805dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:49.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.163+0000 7fa557fff700 1 -- 172.21.15.112:0/160070176 shutdown_connections 2023-01-24T03:30:49.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.163+0000 7fa557fff700 1 -- 172.21.15.112:0/160070176 wait complete. 2023-01-24T03:30:49.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.164+0000 7fa557fff700 1 Processor -- start 2023-01-24T03:30:49.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.164+0000 7fa557fff700 1 -- start start 2023-01-24T03:30:49.178 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.165+0000 7fa557fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa5580c6420 0x7fa55814d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:49.178 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.165+0000 7fa557fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa55814f3f0 0x7fa558153820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:49.178 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.165+0000 7fa557fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa558155ba0 0x7fa558148170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:49.178 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.165+0000 7fa557fff700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5580ca980 con 0x7fa558155ba0 2023-01-24T03:30:49.178 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.165+0000 7fa557fff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5580ca170 con 0x7fa5580c6420 2023-01-24T03:30:49.179 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.165+0000 7fa557fff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa558154f00 con 0x7fa55814f3f0 2023-01-24T03:30:49.179 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.165+0000 7fa5577fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa558155ba0 0x7fa558148170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:49.179 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.165+0000 7fa5577fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa558155ba0 0x7fa558148170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51044/0 (socket says 172.21.15.112:51044) 2023-01-24T03:30:49.179 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.165+0000 7fa5577fe700 1 -- 172.21.15.112:0/3004512235 learned_addr learned my addr 172.21.15.112:0/3004512235 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:49.179 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.165+0000 7fa556ffd700 1 --2- 172.21.15.112:0/3004512235 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa5580c6420 0x7fa55814d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:49.180 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.165+0000 7fa54ffff700 1 --2- 172.21.15.112:0/3004512235 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa55814f3f0 0x7fa558153820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:49.180 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.165+0000 7fa5577fe700 1 -- 172.21.15.112:0/3004512235 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa55814f3f0 msgr2=0x7fa558153820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:49.180 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.165+0000 7fa5577fe700 1 --2- 172.21.15.112:0/3004512235 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa55814f3f0 0x7fa558153820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:49.180 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.165+0000 7fa5577fe700 1 -- 172.21.15.112:0/3004512235 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa5580c6420 msgr2=0x7fa55814d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:49.181 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.166+0000 7fa5577fe700 1 --2- 172.21.15.112:0/3004512235 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa5580c6420 0x7fa55814d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:49.181 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.166+0000 7fa5577fe700 1 -- 172.21.15.112:0/3004512235 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa540009370 con 0x7fa558155ba0 2023-01-24T03:30:49.181 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.166+0000 7fa556ffd700 1 --2- 172.21.15.112:0/3004512235 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa5580c6420 0x7fa55814d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:30:49.181 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.166+0000 7fa5577fe700 1 --2- 172.21.15.112:0/3004512235 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa558155ba0 0x7fa558148170 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa5580891c0 tx=0x7fa550019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:49.181 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.166+0000 7fa54effd700 1 -- 172.21.15.112:0/3004512235 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa550027470 con 0x7fa558155ba0 2023-01-24T03:30:49.181 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.166+0000 7fa54effd700 1 -- 172.21.15.112:0/3004512235 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5500275f0 con 0x7fa558155ba0 2023-01-24T03:30:49.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.166+0000 7fa557fff700 1 -- 172.21.15.112:0/3004512235 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa550009a20 con 0x7fa558155ba0 2023-01-24T03:30:49.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.166+0000 7fa54effd700 1 -- 172.21.15.112:0/3004512235 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5500279b0 con 0x7fa558155ba0 2023-01-24T03:30:49.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.166+0000 7fa557fff700 1 -- 172.21.15.112:0/3004512235 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5580ca3c0 con 0x7fa558155ba0 2023-01-24T03:30:49.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.167+0000 7fa557fff700 1 -- 172.21.15.112:0/3004512235 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5580c3b80 con 0x7fa558155ba0 2023-01-24T03:30:49.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.167+0000 7fa54effd700 1 -- 172.21.15.112:0/3004512235 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fa55001d420 con 0x7fa558155ba0 2023-01-24T03:30:49.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.167+0000 7fa54effd700 1 --2- 172.21.15.112:0/3004512235 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa53c0611c0 0x7fa53c063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:49.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.167+0000 7fa54effd700 1 -- 172.21.15.112:0/3004512235 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fa550080660 con 0x7fa558155ba0 2023-01-24T03:30:49.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.167+0000 7fa556ffd700 1 --2- 172.21.15.112:0/3004512235 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa53c0611c0 0x7fa53c063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:49.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.168+0000 7fa556ffd700 1 --2- 172.21.15.112:0/3004512235 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa53c0611c0 0x7fa53c063670 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa548001c40 tx=0x7fa548009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:49.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.172+0000 7fa54effd700 1 -- 172.21.15.112:0/3004512235 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fa550050440 con 0x7fa558155ba0 2023-01-24T03:30:49.352 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.349+0000 7fa557fff700 1 -- 172.21.15.112:0/3004512235 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fa55805ea20 con 0x7fa558155ba0 2023-01-24T03:30:49.352 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.349+0000 7fa54effd700 1 -- 172.21.15.112:0/3004512235 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v24) v1 ==== 92+0+10 (secure 0 0 0) 0x7fa55002c020 con 0x7fa558155ba0 2023-01-24T03:30:49.353 INFO:teuthology.orchestra.run.smithi112.stdout:pg_num: 8 2023-01-24T03:30:49.354 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.351+0000 7fa557fff700 1 -- 172.21.15.112:0/3004512235 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa53c0611c0 msgr2=0x7fa53c063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:49.355 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.351+0000 7fa557fff700 1 --2- 172.21.15.112:0/3004512235 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa53c0611c0 0x7fa53c063670 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa548001c40 tx=0x7fa548009040 comp rx=0 tx=0).stop 2023-01-24T03:30:49.355 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.351+0000 7fa557fff700 1 -- 172.21.15.112:0/3004512235 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa558155ba0 msgr2=0x7fa558148170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:49.355 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.351+0000 7fa557fff700 1 --2- 172.21.15.112:0/3004512235 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa558155ba0 0x7fa558148170 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa5580891c0 tx=0x7fa550019000 comp rx=0 tx=0).stop 2023-01-24T03:30:49.355 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.351+0000 7fa557fff700 1 -- 172.21.15.112:0/3004512235 shutdown_connections 2023-01-24T03:30:49.355 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.351+0000 7fa557fff700 1 --2- 172.21.15.112:0/3004512235 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa53c0611c0 0x7fa53c063670 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:49.356 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.351+0000 7fa557fff700 1 --2- 172.21.15.112:0/3004512235 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa558155ba0 0x7fa558148170 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:49.356 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.351+0000 7fa557fff700 1 --2- 172.21.15.112:0/3004512235 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa5580c6420 0x7fa55814d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:49.356 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.351+0000 7fa557fff700 1 --2- 172.21.15.112:0/3004512235 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa55814f3f0 0x7fa558153820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:49.356 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.351+0000 7fa557fff700 1 -- 172.21.15.112:0/3004512235 >> 172.21.15.112:0/3004512235 conn(0x7fa55805d830 msgr2=0x7fa5580c7260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:49.356 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.351+0000 7fa557fff700 1 -- 172.21.15.112:0/3004512235 shutdown_connections 2023-01-24T03:30:49.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.351+0000 7fa557fff700 1 -- 172.21.15.112:0/3004512235 wait complete. 2023-01-24T03:30:49.366 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-01-24T03:30:49.480 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.477+0000 7f2d7a0d0700 1 Processor -- start 2023-01-24T03:30:49.481 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.477+0000 7f2d7a0d0700 1 -- start start 2023-01-24T03:30:49.483 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.478+0000 7f2d7a0d0700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f2d7405e420 0x7f2d74069890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:49.483 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.478+0000 7f2d7a0d0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2d74068d20 0x7f2d74155f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:49.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.478+0000 7f2d7a0d0700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d74155ab0 con 0x7f2d740694c0 2023-01-24T03:30:49.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.478+0000 7f2d7a0d0700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d74155550 con 0x7f2d7405e420 2023-01-24T03:30:49.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.478+0000 7f2d7a0d0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d74156480 con 0x7f2d74068d20 2023-01-24T03:30:49.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.478+0000 7f2d72ffd700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f2d7405e420 0x7f2d74069890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:49.485 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.478+0000 7f2d72ffd700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f2d7405e420 0x7f2d74069890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51052/0 (socket says 172.21.15.112:51052) 2023-01-24T03:30:49.485 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.478+0000 7f2d72ffd700 1 -- 172.21.15.112:0/3977985504 learned_addr learned my addr 172.21.15.112:0/3977985504 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:49.485 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.478+0000 7f2d73fff700 1 --2- 172.21.15.112:0/3977985504 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2d74068d20 0x7f2d74155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:49.485 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.479+0000 7f2d72ffd700 1 -- 172.21.15.112:0/3977985504 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2d74068d20 msgr2=0x7f2d74155f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:49.486 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.479+0000 7f2d72ffd700 1 --2- 172.21.15.112:0/3977985504 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2d74068d20 0x7f2d74155f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:49.486 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.479+0000 7f2d72ffd700 1 -- 172.21.15.112:0/3977985504 >> v1:172.21.15.142:6789/0 conn(0x7f2d740694c0 legacy=0x7f2d740c68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:49.486 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.479+0000 7f2d72ffd700 1 -- 172.21.15.112:0/3977985504 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d740c70e0 con 0x7f2d7405e420 2023-01-24T03:30:49.486 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.479+0000 7f2d72ffd700 1 --2- 172.21.15.112:0/3977985504 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2d7405e420 0x7f2d74069890 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2d68006fd0 tx=0x7f2d680090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e861a7b2a50e9199 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:49.486 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.479+0000 7f2d727fc700 1 -- 172.21.15.112:0/3977985504 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d68011630 con 0x7f2d7405e420 2023-01-24T03:30:49.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.479+0000 7f2d727fc700 1 -- 172.21.15.112:0/3977985504 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d68011790 con 0x7f2d7405e420 2023-01-24T03:30:49.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.479+0000 7f2d727fc700 1 -- 172.21.15.112:0/3977985504 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d68011b50 con 0x7f2d7405e420 2023-01-24T03:30:49.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.479+0000 7f2d7a0d0700 1 -- 172.21.15.112:0/3977985504 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2d7405e420 msgr2=0x7f2d74069890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:49.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.479+0000 7f2d7a0d0700 1 --2- 172.21.15.112:0/3977985504 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2d7405e420 0x7f2d74069890 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2d68006fd0 tx=0x7f2d680090f0 comp rx=0 tx=0).stop 2023-01-24T03:30:49.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.479+0000 7f2d7a0d0700 1 -- 172.21.15.112:0/3977985504 shutdown_connections 2023-01-24T03:30:49.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.479+0000 7f2d7a0d0700 1 --2- 172.21.15.112:0/3977985504 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2d74068d20 0x7f2d74155f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:49.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.479+0000 7f2d7a0d0700 1 --2- 172.21.15.112:0/3977985504 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2d7405e420 0x7f2d74069890 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:49.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.479+0000 7f2d7a0d0700 1 -- 172.21.15.112:0/3977985504 >> 172.21.15.112:0/3977985504 conn(0x7f2d7405d830 msgr2=0x7f2d7405dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:49.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.479+0000 7f2d7a0d0700 1 -- 172.21.15.112:0/3977985504 shutdown_connections 2023-01-24T03:30:49.489 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.479+0000 7f2d7a0d0700 1 -- 172.21.15.112:0/3977985504 wait complete. 2023-01-24T03:30:49.489 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.481+0000 7f2d7a0d0700 1 Processor -- start 2023-01-24T03:30:49.489 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.481+0000 7f2d7a0d0700 1 -- start start 2023-01-24T03:30:49.492 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.481+0000 7f2d7a0d0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2d7405e420 0x7f2d7414d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:49.494 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.481+0000 7f2d7a0d0700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2d74068d20 0x7f2d740c54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:49.494 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.481+0000 7f2d7a0d0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2d740694c0 0x7f2d740c5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:49.495 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.481+0000 7f2d7a0d0700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d74154ae0 con 0x7f2d740694c0 2023-01-24T03:30:49.495 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.481+0000 7f2d7a0d0700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d74155060 con 0x7f2d74068d20 2023-01-24T03:30:49.495 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.481+0000 7f2d7a0d0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d74151220 con 0x7f2d7405e420 2023-01-24T03:30:49.495 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.481+0000 7f2d73fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2d740694c0 0x7f2d740c5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:49.495 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.481+0000 7f2d73fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2d740694c0 0x7f2d740c5a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51058/0 (socket says 172.21.15.112:51058) 2023-01-24T03:30:49.496 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.481+0000 7f2d73fff700 1 -- 172.21.15.112:0/269365525 learned_addr learned my addr 172.21.15.112:0/269365525 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:49.496 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.481+0000 7f2d737fe700 1 --2- 172.21.15.112:0/269365525 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2d7405e420 0x7f2d7414d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:49.496 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.481+0000 7f2d72ffd700 1 --2- 172.21.15.112:0/269365525 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2d74068d20 0x7f2d740c54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:49.496 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.482+0000 7f2d73fff700 1 -- 172.21.15.112:0/269365525 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2d7405e420 msgr2=0x7f2d7414d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:49.497 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.482+0000 7f2d73fff700 1 --2- 172.21.15.112:0/269365525 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2d7405e420 0x7f2d7414d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:49.497 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.482+0000 7f2d73fff700 1 -- 172.21.15.112:0/269365525 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2d74068d20 msgr2=0x7f2d740c54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:49.497 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.482+0000 7f2d73fff700 1 --2- 172.21.15.112:0/269365525 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2d74068d20 0x7f2d740c54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:49.497 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.482+0000 7f2d73fff700 1 -- 172.21.15.112:0/269365525 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d68009370 con 0x7f2d740694c0 2023-01-24T03:30:49.497 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.482+0000 7f2d72ffd700 1 --2- 172.21.15.112:0/269365525 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2d74068d20 0x7f2d740c54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:30:49.497 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.482+0000 7f2d73fff700 1 --2- 172.21.15.112:0/269365525 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2d740694c0 0x7f2d740c5a20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c00d890 tx=0x7f2d6c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:49.498 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.482+0000 7f2d5b7fe700 1 -- 172.21.15.112:0/269365525 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c0125f0 con 0x7f2d740694c0 2023-01-24T03:30:49.498 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.482+0000 7f2d5b7fe700 1 -- 172.21.15.112:0/269365525 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d6c012750 con 0x7f2d740694c0 2023-01-24T03:30:49.498 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.482+0000 7f2d5b7fe700 1 -- 172.21.15.112:0/269365525 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c012b10 con 0x7f2d740694c0 2023-01-24T03:30:49.498 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.482+0000 7f2d7a0d0700 1 -- 172.21.15.112:0/269365525 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d741514c0 con 0x7f2d740694c0 2023-01-24T03:30:49.498 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.482+0000 7f2d7a0d0700 1 -- 172.21.15.112:0/269365525 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d74152e40 con 0x7f2d740694c0 2023-01-24T03:30:49.499 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.483+0000 7f2d7a0d0700 1 -- 172.21.15.112:0/269365525 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d7406f3a0 con 0x7f2d740694c0 2023-01-24T03:30:49.499 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.483+0000 7f2d5b7fe700 1 -- 172.21.15.112:0/269365525 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f2d6c01e460 con 0x7f2d740694c0 2023-01-24T03:30:49.499 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.483+0000 7f2d5b7fe700 1 --2- 172.21.15.112:0/269365525 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2d540611c0 0x7f2d54063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:49.499 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.484+0000 7f2d5b7fe700 1 -- 172.21.15.112:0/269365525 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f2d6c07ddf0 con 0x7f2d740694c0 2023-01-24T03:30:49.499 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.484+0000 7f2d737fe700 1 --2- 172.21.15.112:0/269365525 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2d540611c0 0x7f2d54063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:49.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.484+0000 7f2d737fe700 1 --2- 172.21.15.112:0/269365525 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2d540611c0 0x7f2d54063670 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2d64004290 tx=0x7f2d64001560 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:49.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.488+0000 7f2d5b7fe700 1 -- 172.21.15.112:0/269365525 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f2d6c04dbd0 con 0x7f2d740694c0 2023-01-24T03:30:49.666 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.663+0000 7f2d7a0d0700 1 -- 172.21.15.112:0/269365525 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2d7404b340 con 0x7f2d740694c0 2023-01-24T03:30:49.667 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.663+0000 7f2d5b7fe700 1 -- 172.21.15.112:0/269365525 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2d6c0511f0 con 0x7f2d740694c0 2023-01-24T03:30:49.667 INFO:teuthology.orchestra.run.smithi112.stdout:pg_num: 1 2023-01-24T03:30:49.669 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.666+0000 7f2d7a0d0700 1 -- 172.21.15.112:0/269365525 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2d540611c0 msgr2=0x7f2d54063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:49.669 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.666+0000 7f2d7a0d0700 1 --2- 172.21.15.112:0/269365525 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2d540611c0 0x7f2d54063670 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2d64004290 tx=0x7f2d64001560 comp rx=0 tx=0).stop 2023-01-24T03:30:49.669 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.666+0000 7f2d7a0d0700 1 -- 172.21.15.112:0/269365525 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2d740694c0 msgr2=0x7f2d740c5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:49.670 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.666+0000 7f2d7a0d0700 1 --2- 172.21.15.112:0/269365525 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2d740694c0 0x7f2d740c5a20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c00d890 tx=0x7f2d6c008040 comp rx=0 tx=0).stop 2023-01-24T03:30:49.670 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.666+0000 7f2d7a0d0700 1 -- 172.21.15.112:0/269365525 shutdown_connections 2023-01-24T03:30:49.670 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.666+0000 7f2d7a0d0700 1 --2- 172.21.15.112:0/269365525 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2d540611c0 0x7f2d54063670 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:49.670 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.666+0000 7f2d7a0d0700 1 --2- 172.21.15.112:0/269365525 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2d740694c0 0x7f2d740c5a20 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:49.670 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.666+0000 7f2d7a0d0700 1 --2- 172.21.15.112:0/269365525 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2d7405e420 0x7f2d7414d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:49.671 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.666+0000 7f2d7a0d0700 1 --2- 172.21.15.112:0/269365525 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2d74068d20 0x7f2d740c54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:49.671 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.666+0000 7f2d7a0d0700 1 -- 172.21.15.112:0/269365525 >> 172.21.15.112:0/269365525 conn(0x7f2d7405d830 msgr2=0x7f2d7414f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:49.671 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.666+0000 7f2d7a0d0700 1 -- 172.21.15.112:0/269365525 shutdown_connections 2023-01-24T03:30:49.671 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.666+0000 7f2d7a0d0700 1 -- 172.21.15.112:0/269365525 wait complete. 2023-01-24T03:30:49.680 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-01-24T03:30:49.797 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.794+0000 7f9e7893d700 1 Processor -- start 2023-01-24T03:30:49.798 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.794+0000 7f9e7893d700 1 -- start start 2023-01-24T03:30:49.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.795+0000 7f9e7893d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f9e740694c0 0x7f9e74069890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:49.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.795+0000 7f9e7893d700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e74155ab0 con 0x7f9e74068d20 2023-01-24T03:30:49.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.795+0000 7f9e7893d700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e740c70e0 con 0x7f9e7405e420 2023-01-24T03:30:49.808 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.795+0000 7f9e7893d700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e74155550 con 0x7f9e740694c0 2023-01-24T03:30:49.808 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.795+0000 7f9e72d9d700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f9e74068d20 0x7f9e740ae9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:60284/0 (socket says 172.21.15.112:60284) 2023-01-24T03:30:49.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.795+0000 7f9e72d9d700 1 -- 172.21.15.112:0/1477224099 learned_addr learned my addr 172.21.15.112:0/1477224099 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:49.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.796+0000 7f9e7259c700 1 --2- 172.21.15.112:0/1477224099 >> v2:172.21.15.142:3300/0 conn(0x7f9e740694c0 0x7f9e74069890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:49.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.796+0000 7f9e7159a700 1 -- 172.21.15.112:0/1477224099 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3333285002 0 0) 0x7f9e74155ab0 con 0x7f9e74068d20 2023-01-24T03:30:49.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.796+0000 7f9e7159a700 1 -- 172.21.15.112:0/1477224099 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e540033f0 con 0x7f9e74068d20 2023-01-24T03:30:49.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.796+0000 7f9e7159a700 1 -- 172.21.15.112:0/1477224099 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 567257430 0 0) 0x7f9e540033f0 con 0x7f9e74068d20 2023-01-24T03:30:49.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.796+0000 7f9e7259c700 1 -- 172.21.15.112:0/1477224099 >> v1:172.21.15.142:6789/0 conn(0x7f9e7405e420 legacy=0x7f9e740c68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:49.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.796+0000 7f9e7259c700 1 -- 172.21.15.112:0/1477224099 >> v1:172.21.15.112:6789/0 conn(0x7f9e74068d20 legacy=0x7f9e740ae9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:49.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.796+0000 7f9e7259c700 1 -- 172.21.15.112:0/1477224099 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e740c73c0 con 0x7f9e740694c0 2023-01-24T03:30:49.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.796+0000 7f9e7259c700 1 --2- 172.21.15.112:0/1477224099 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e740694c0 0x7f9e74069890 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9e64009e60 tx=0x7f9e64009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c476475950c7f4e server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:49.812 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.796+0000 7f9e7159a700 1 -- 172.21.15.112:0/1477224099 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6400e070 con 0x7f9e740694c0 2023-01-24T03:30:49.812 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.796+0000 7f9e7159a700 1 -- 172.21.15.112:0/1477224099 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e6400b850 con 0x7f9e740694c0 2023-01-24T03:30:49.813 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.796+0000 7f9e7159a700 1 -- 172.21.15.112:0/1477224099 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c001470 con 0x7f9e740694c0 2023-01-24T03:30:49.813 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.797+0000 7f9e7893d700 1 -- 172.21.15.112:0/1477224099 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e740694c0 msgr2=0x7f9e74069890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:49.813 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.797+0000 7f9e7893d700 1 --2- 172.21.15.112:0/1477224099 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e740694c0 0x7f9e74069890 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9e64009e60 tx=0x7f9e64009780 comp rx=0 tx=0).stop 2023-01-24T03:30:49.814 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.797+0000 7f9e7893d700 1 -- 172.21.15.112:0/1477224099 shutdown_connections 2023-01-24T03:30:49.814 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.797+0000 7f9e7893d700 1 --2- 172.21.15.112:0/1477224099 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e740694c0 0x7f9e74069890 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:49.814 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.797+0000 7f9e7893d700 1 -- 172.21.15.112:0/1477224099 >> 172.21.15.112:0/1477224099 conn(0x7f9e7405d830 msgr2=0x7f9e7405dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:49.815 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.797+0000 7f9e7893d700 1 -- 172.21.15.112:0/1477224099 shutdown_connections 2023-01-24T03:30:49.815 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.797+0000 7f9e7893d700 1 -- 172.21.15.112:0/1477224099 wait complete. 2023-01-24T03:30:49.815 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.798+0000 7f9e7893d700 1 Processor -- start 2023-01-24T03:30:49.816 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.798+0000 7f9e7893d700 1 -- start start 2023-01-24T03:30:49.816 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.799+0000 7f9e7893d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e7405e420 0x7f9e74168c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:49.817 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.799+0000 7f9e7893d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9e74068d20 0x7f9e740bfdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:49.817 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.799+0000 7f9e7893d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e740694c0 0x7f9e740c0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:49.817 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.799+0000 7f9e7893d700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e74152b70 con 0x7f9e74068d20 2023-01-24T03:30:49.818 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.799+0000 7f9e7893d700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e74166120 con 0x7f9e7405e420 2023-01-24T03:30:49.818 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.799+0000 7f9e7893d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e741665b0 con 0x7f9e740694c0 2023-01-24T03:30:49.818 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.799+0000 7f9e71d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9e74068d20 0x7f9e740bfdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:49.820 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.799+0000 7f9e71d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9e74068d20 0x7f9e740bfdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51070/0 (socket says 172.21.15.112:51070) 2023-01-24T03:30:49.820 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.799+0000 7f9e71d9b700 1 -- 172.21.15.112:0/3748498698 learned_addr learned my addr 172.21.15.112:0/3748498698 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:49.820 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.799+0000 7f9e7259c700 1 --2- 172.21.15.112:0/3748498698 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e7405e420 0x7f9e74168c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:49.820 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.799+0000 7f9e72d9d700 1 --2- 172.21.15.112:0/3748498698 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e740694c0 0x7f9e740c0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:49.821 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.799+0000 7f9e71d9b700 1 -- 172.21.15.112:0/3748498698 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e740694c0 msgr2=0x7f9e740c0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:49.821 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.799+0000 7f9e71d9b700 1 --2- 172.21.15.112:0/3748498698 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e740694c0 0x7f9e740c0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:49.822 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.799+0000 7f9e71d9b700 1 -- 172.21.15.112:0/3748498698 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e7405e420 msgr2=0x7f9e74168c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:49.822 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.799+0000 7f9e71d9b700 1 --2- 172.21.15.112:0/3748498698 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e7405e420 0x7f9e74168c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:49.822 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.799+0000 7f9e71d9b700 1 -- 172.21.15.112:0/3748498698 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e64009a20 con 0x7f9e74068d20 2023-01-24T03:30:49.823 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.799+0000 7f9e71d9b700 1 --2- 172.21.15.112:0/3748498698 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9e74068d20 0x7f9e740bfdc0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c00c470 tx=0x7f9e5c007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:49.823 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.799+0000 7f9e6a7fc700 1 -- 172.21.15.112:0/3748498698 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c012420 con 0x7f9e74068d20 2023-01-24T03:30:49.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.799+0000 7f9e6a7fc700 1 -- 172.21.15.112:0/3748498698 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e5c0125a0 con 0x7f9e74068d20 2023-01-24T03:30:49.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.799+0000 7f9e7893d700 1 -- 172.21.15.112:0/3748498698 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e74152890 con 0x7f9e74068d20 2023-01-24T03:30:49.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.800+0000 7f9e6a7fc700 1 -- 172.21.15.112:0/3748498698 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c012960 con 0x7f9e74068d20 2023-01-24T03:30:49.825 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.800+0000 7f9e7893d700 1 -- 172.21.15.112:0/3748498698 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e74165ad0 con 0x7f9e74068d20 2023-01-24T03:30:49.825 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.801+0000 7f9e7893d700 1 -- 172.21.15.112:0/3748498698 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e740c0840 con 0x7f9e74068d20 2023-01-24T03:30:49.825 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.801+0000 7f9e6a7fc700 1 -- 172.21.15.112:0/3748498698 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f9e5c019030 con 0x7f9e74068d20 2023-01-24T03:30:49.826 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.801+0000 7f9e6a7fc700 1 --2- 172.21.15.112:0/3748498698 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9e580611c0 0x7f9e58063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:49.826 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.801+0000 7f9e6a7fc700 1 -- 172.21.15.112:0/3748498698 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f9e5c07cbd0 con 0x7f9e74068d20 2023-01-24T03:30:49.827 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.801+0000 7f9e7259c700 1 --2- 172.21.15.112:0/3748498698 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9e580611c0 0x7f9e58063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:49.827 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.802+0000 7f9e7259c700 1 --2- 172.21.15.112:0/3748498698 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9e580611c0 0x7f9e58063670 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9e640097c0 tx=0x7f9e64017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:49.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.805+0000 7f9e6a7fc700 1 -- 172.21.15.112:0/3748498698 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f9e5c04c9b0 con 0x7f9e74068d20 2023-01-24T03:30:49.986 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.982+0000 7f9e7893d700 1 -- 172.21.15.112:0/3748498698 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9e7404b340 con 0x7f9e74068d20 2023-01-24T03:30:49.986 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.983+0000 7f9e6a7fc700 1 -- 172.21.15.112:0/3748498698 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9e5c0170a0 con 0x7f9e74068d20 2023-01-24T03:30:49.986 INFO:teuthology.orchestra.run.smithi112.stdout:pg_num: 32 2023-01-24T03:30:49.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.985+0000 7f9e7893d700 1 -- 172.21.15.112:0/3748498698 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9e580611c0 msgr2=0x7f9e58063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:49.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.985+0000 7f9e7893d700 1 --2- 172.21.15.112:0/3748498698 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9e580611c0 0x7f9e58063670 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9e640097c0 tx=0x7f9e64017000 comp rx=0 tx=0).stop 2023-01-24T03:30:49.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.985+0000 7f9e7893d700 1 -- 172.21.15.112:0/3748498698 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9e74068d20 msgr2=0x7f9e740bfdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:49.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.985+0000 7f9e7893d700 1 --2- 172.21.15.112:0/3748498698 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9e74068d20 0x7f9e740bfdc0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c00c470 tx=0x7f9e5c007dd0 comp rx=0 tx=0).stop 2023-01-24T03:30:49.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.985+0000 7f9e7893d700 1 -- 172.21.15.112:0/3748498698 shutdown_connections 2023-01-24T03:30:49.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.985+0000 7f9e7893d700 1 --2- 172.21.15.112:0/3748498698 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9e580611c0 0x7f9e58063670 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:49.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.985+0000 7f9e7893d700 1 --2- 172.21.15.112:0/3748498698 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e740694c0 0x7f9e740c0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:49.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.985+0000 7f9e7893d700 1 --2- 172.21.15.112:0/3748498698 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e7405e420 0x7f9e74168c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:49.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.985+0000 7f9e7893d700 1 --2- 172.21.15.112:0/3748498698 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9e74068d20 0x7f9e740bfdc0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:49.990 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.985+0000 7f9e7893d700 1 -- 172.21.15.112:0/3748498698 >> 172.21.15.112:0/3748498698 conn(0x7f9e7405d830 msgr2=0x7f9e7414efa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:49.990 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.985+0000 7f9e7893d700 1 -- 172.21.15.112:0/3748498698 shutdown_connections 2023-01-24T03:30:49.990 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:49.985+0000 7f9e7893d700 1 -- 172.21.15.112:0/3748498698 wait complete. 2023-01-24T03:30:49.999 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-01-24T03:30:50.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.111+0000 7ff2798cb700 1 Processor -- start 2023-01-24T03:30:50.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.111+0000 7ff2798cb700 1 -- start start 2023-01-24T03:30:50.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.112+0000 7ff2798cb700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7ff27414f3f0 0x7ff2740c4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:50.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.112+0000 7ff2798cb700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff2740c6420 0x7ff274155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:50.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.112+0000 7ff2798cb700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2740c82e0 con 0x7ff2740c8770 2023-01-24T03:30:50.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.112+0000 7ff2798cb700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2741560e0 con 0x7ff27414f3f0 2023-01-24T03:30:50.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.112+0000 7ff2798cb700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2740c2980 con 0x7ff2740c6420 2023-01-24T03:30:50.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.112+0000 7ff2788c9700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7ff27414f3f0 0x7ff2740c4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:50.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.112+0000 7ff2788c9700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7ff27414f3f0 0x7ff2740c4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51076/0 (socket says 172.21.15.112:51076) 2023-01-24T03:30:50.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.112+0000 7ff2788c9700 1 -- 172.21.15.112:0/1230565015 learned_addr learned my addr 172.21.15.112:0/1230565015 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:50.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.112+0000 7ff273fff700 1 --2- 172.21.15.112:0/1230565015 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff2740c6420 0x7ff274155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:50.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.112+0000 7ff2788c9700 1 -- 172.21.15.112:0/1230565015 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff2740c6420 msgr2=0x7ff274155ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:50.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.112+0000 7ff2788c9700 1 --2- 172.21.15.112:0/1230565015 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff2740c6420 0x7ff274155ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:50.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.112+0000 7ff2788c9700 1 -- 172.21.15.112:0/1230565015 >> v1:172.21.15.142:6789/0 conn(0x7ff2740c8770 legacy=0x7ff2740c2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:50.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.112+0000 7ff2788c9700 1 -- 172.21.15.112:0/1230565015 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2740c5290 con 0x7ff27414f3f0 2023-01-24T03:30:50.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.112+0000 7ff2788c9700 1 --2- 172.21.15.112:0/1230565015 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff27414f3f0 0x7ff2740c4ca0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff268006fd0 tx=0x7ff2680090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e718fee717ada2f5 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:50.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.112+0000 7ff2737fe700 1 -- 172.21.15.112:0/1230565015 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff268011630 con 0x7ff27414f3f0 2023-01-24T03:30:50.124 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.112+0000 7ff2737fe700 1 -- 172.21.15.112:0/1230565015 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2680117b0 con 0x7ff27414f3f0 2023-01-24T03:30:50.124 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.112+0000 7ff2737fe700 1 -- 172.21.15.112:0/1230565015 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff268011b70 con 0x7ff27414f3f0 2023-01-24T03:30:50.124 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.113+0000 7ff2798cb700 1 -- 172.21.15.112:0/1230565015 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff27414f3f0 msgr2=0x7ff2740c4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:50.124 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.113+0000 7ff2798cb700 1 --2- 172.21.15.112:0/1230565015 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff27414f3f0 0x7ff2740c4ca0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff268006fd0 tx=0x7ff2680090f0 comp rx=0 tx=0).stop 2023-01-24T03:30:50.125 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.113+0000 7ff2798cb700 1 -- 172.21.15.112:0/1230565015 shutdown_connections 2023-01-24T03:30:50.125 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.113+0000 7ff2798cb700 1 --2- 172.21.15.112:0/1230565015 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff27414f3f0 0x7ff2740c4ca0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:50.125 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.113+0000 7ff2798cb700 1 --2- 172.21.15.112:0/1230565015 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff2740c6420 0x7ff274155ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:50.125 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.113+0000 7ff2798cb700 1 -- 172.21.15.112:0/1230565015 >> 172.21.15.112:0/1230565015 conn(0x7ff27405d830 msgr2=0x7ff27405dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:50.126 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.113+0000 7ff2798cb700 1 -- 172.21.15.112:0/1230565015 shutdown_connections 2023-01-24T03:30:50.126 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.113+0000 7ff2798cb700 1 -- 172.21.15.112:0/1230565015 wait complete. 2023-01-24T03:30:50.126 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.114+0000 7ff2798cb700 1 Processor -- start 2023-01-24T03:30:50.126 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.114+0000 7ff2798cb700 1 -- start start 2023-01-24T03:30:50.126 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.115+0000 7ff2798cb700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff2740c6420 0x7ff2740c0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:50.127 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.115+0000 7ff2798cb700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff2740c8770 0x7ff2740c0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:50.127 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.115+0000 7ff2798cb700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff27414f3f0 0x7ff27414de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:50.127 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.115+0000 7ff2798cb700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2740c1c20 con 0x7ff27414f3f0 2023-01-24T03:30:50.128 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.115+0000 7ff2798cb700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff274154fd0 con 0x7ff2740c8770 2023-01-24T03:30:50.128 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.115+0000 7ff2798cb700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff274154d80 con 0x7ff2740c6420 2023-01-24T03:30:50.128 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.115+0000 7ff2790ca700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff27414f3f0 0x7ff27414de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:50.128 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.115+0000 7ff2790ca700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff27414f3f0 0x7ff27414de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51086/0 (socket says 172.21.15.112:51086) 2023-01-24T03:30:50.129 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.115+0000 7ff2790ca700 1 -- 172.21.15.112:0/1901393993 learned_addr learned my addr 172.21.15.112:0/1901393993 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:50.129 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.115+0000 7ff273fff700 1 --2- 172.21.15.112:0/1901393993 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff2740c8770 0x7ff2740c0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:50.129 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.115+0000 7ff2788c9700 1 --2- 172.21.15.112:0/1901393993 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff2740c6420 0x7ff2740c0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:50.129 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.115+0000 7ff2790ca700 1 -- 172.21.15.112:0/1901393993 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff2740c6420 msgr2=0x7ff2740c0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:50.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.115+0000 7ff2790ca700 1 --2- 172.21.15.112:0/1901393993 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff2740c6420 0x7ff2740c0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:50.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.115+0000 7ff2790ca700 1 -- 172.21.15.112:0/1901393993 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff2740c8770 msgr2=0x7ff2740c0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:50.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.115+0000 7ff2790ca700 1 --2- 172.21.15.112:0/1901393993 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff2740c8770 0x7ff2740c0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:50.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.115+0000 7ff2790ca700 1 -- 172.21.15.112:0/1901393993 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff268009370 con 0x7ff27414f3f0 2023-01-24T03:30:50.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.115+0000 7ff2790ca700 1 --2- 172.21.15.112:0/1901393993 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff27414f3f0 0x7ff27414de70 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff26c00c870 tx=0x7ff26c002560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:50.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.116+0000 7ff270ff9700 1 -- 172.21.15.112:0/1901393993 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26c011590 con 0x7ff27414f3f0 2023-01-24T03:30:50.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.116+0000 7ff270ff9700 1 -- 172.21.15.112:0/1901393993 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff26c011710 con 0x7ff27414f3f0 2023-01-24T03:30:50.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.116+0000 7ff2798cb700 1 -- 172.21.15.112:0/1901393993 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2740c02d0 con 0x7ff27414f3f0 2023-01-24T03:30:50.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.116+0000 7ff270ff9700 1 -- 172.21.15.112:0/1901393993 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26c011ad0 con 0x7ff27414f3f0 2023-01-24T03:30:50.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.116+0000 7ff2798cb700 1 -- 172.21.15.112:0/1901393993 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff274152c30 con 0x7ff27414f3f0 2023-01-24T03:30:50.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.116+0000 7ff2798cb700 1 -- 172.21.15.112:0/1901393993 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2740c00a0 con 0x7ff27414f3f0 2023-01-24T03:30:50.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.117+0000 7ff270ff9700 1 -- 172.21.15.112:0/1901393993 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7ff26c010050 con 0x7ff27414f3f0 2023-01-24T03:30:50.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.117+0000 7ff270ff9700 1 --2- 172.21.15.112:0/1901393993 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff25c0611c0 0x7ff25c063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:50.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.117+0000 7ff270ff9700 1 -- 172.21.15.112:0/1901393993 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7ff26c07d380 con 0x7ff27414f3f0 2023-01-24T03:30:50.133 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.117+0000 7ff2788c9700 1 --2- 172.21.15.112:0/1901393993 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff25c0611c0 0x7ff25c063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:50.133 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.118+0000 7ff2788c9700 1 --2- 172.21.15.112:0/1901393993 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff25c0611c0 0x7ff25c063670 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff268006fd0 tx=0x7ff268017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:50.133 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.121+0000 7ff270ff9700 1 -- 172.21.15.112:0/1901393993 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7ff2740c00a0 con 0x7ff27414f3f0 2023-01-24T03:30:50.301 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.298+0000 7ff2798cb700 1 -- 172.21.15.112:0/1901393993 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff2740c0460 con 0x7ff27414f3f0 2023-01-24T03:30:50.301 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.298+0000 7ff270ff9700 1 -- 172.21.15.112:0/1901393993 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff26c04d160 con 0x7ff27414f3f0 2023-01-24T03:30:50.302 INFO:teuthology.orchestra.run.smithi112.stdout:pg_num: 64 2023-01-24T03:30:50.303 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.300+0000 7ff2798cb700 1 -- 172.21.15.112:0/1901393993 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff25c0611c0 msgr2=0x7ff25c063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:50.304 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.300+0000 7ff2798cb700 1 --2- 172.21.15.112:0/1901393993 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff25c0611c0 0x7ff25c063670 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff268006fd0 tx=0x7ff268017000 comp rx=0 tx=0).stop 2023-01-24T03:30:50.304 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.300+0000 7ff2798cb700 1 -- 172.21.15.112:0/1901393993 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff27414f3f0 msgr2=0x7ff27414de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:50.304 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.300+0000 7ff2798cb700 1 --2- 172.21.15.112:0/1901393993 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff27414f3f0 0x7ff27414de70 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff26c00c870 tx=0x7ff26c002560 comp rx=0 tx=0).stop 2023-01-24T03:30:50.304 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.300+0000 7ff2798cb700 1 -- 172.21.15.112:0/1901393993 shutdown_connections 2023-01-24T03:30:50.304 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.300+0000 7ff2798cb700 1 --2- 172.21.15.112:0/1901393993 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff25c0611c0 0x7ff25c063670 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:50.305 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.300+0000 7ff2798cb700 1 --2- 172.21.15.112:0/1901393993 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff27414f3f0 0x7ff27414de70 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:50.305 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.300+0000 7ff2798cb700 1 --2- 172.21.15.112:0/1901393993 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff2740c6420 0x7ff2740c0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:50.305 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.300+0000 7ff2798cb700 1 --2- 172.21.15.112:0/1901393993 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff2740c8770 0x7ff2740c0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:50.305 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.300+0000 7ff2798cb700 1 -- 172.21.15.112:0/1901393993 >> 172.21.15.112:0/1901393993 conn(0x7ff27405d830 msgr2=0x7ff274068250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:50.305 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.300+0000 7ff2798cb700 1 -- 172.21.15.112:0/1901393993 shutdown_connections 2023-01-24T03:30:50.306 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.300+0000 7ff2798cb700 1 -- 172.21.15.112:0/1901393993 wait complete. 2023-01-24T03:30:50.315 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-01-24T03:30:50.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.427+0000 7f12da21a700 1 Processor -- start 2023-01-24T03:30:50.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.427+0000 7f12da21a700 1 -- start start 2023-01-24T03:30:50.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.428+0000 7f12da21a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f12d414d4c0 0x7f12d40c4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:50.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.428+0000 7f12da21a700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f12d405dff0 0x7f12d406d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:50.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.428+0000 7f12da21a700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f12d4153240 0x7f12d40c32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:50.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.428+0000 7f12da21a700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12d406d4f0 con 0x7f12d405dff0 2023-01-24T03:30:50.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.428+0000 7f12da21a700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12d406cee0 con 0x7f12d4153240 2023-01-24T03:30:50.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.428+0000 7f12da21a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12d4153af0 con 0x7f12d414d4c0 2023-01-24T03:30:50.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.428+0000 7f12d2ffd700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f12d405dff0 0x7f12d406d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:50.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.428+0000 7f12d2ffd700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f12d405dff0 0x7f12d406d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51092/0 (socket says 172.21.15.112:51092) 2023-01-24T03:30:50.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.428+0000 7f12d2ffd700 1 -- 172.21.15.112:0/117304451 learned_addr learned my addr 172.21.15.112:0/117304451 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:50.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.428+0000 7f12d3fff700 1 --2- 172.21.15.112:0/117304451 >> v2:172.21.15.142:3300/0 conn(0x7f12d4153240 0x7f12d40c32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:50.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.428+0000 7f12d2ffd700 1 -- 172.21.15.112:0/117304451 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f12d414d4c0 msgr2=0x7f12d40c4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:50.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.428+0000 7f12d37fe700 1 --2- 172.21.15.112:0/117304451 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f12d414d4c0 0x7f12d40c4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:50.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.428+0000 7f12d2ffd700 1 --2- 172.21.15.112:0/117304451 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f12d414d4c0 0x7f12d40c4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:50.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.428+0000 7f12d2ffd700 1 -- 172.21.15.112:0/117304451 >> v2:172.21.15.142:3300/0 conn(0x7f12d4153240 msgr2=0x7f12d40c32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:50.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.428+0000 7f12d2ffd700 1 --2- 172.21.15.112:0/117304451 >> v2:172.21.15.142:3300/0 conn(0x7f12d4153240 0x7f12d40c32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:50.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.428+0000 7f12d2ffd700 1 -- 172.21.15.112:0/117304451 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12d4153dd0 con 0x7f12d405dff0 2023-01-24T03:30:50.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.428+0000 7f12d3fff700 1 --2- 172.21.15.112:0/117304451 >> v2:172.21.15.142:3300/0 conn(0x7f12d4153240 0x7f12d40c32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-01-24T03:30:50.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.428+0000 7f12d2ffd700 1 --2- 172.21.15.112:0/117304451 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f12d405dff0 0x7f12d406d980 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f12d40bd300 tx=0x7f12c80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c3ad057619eb70e server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:50.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.429+0000 7f12d27fc700 1 -- 172.21.15.112:0/117304451 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c8011630 con 0x7f12d405dff0 2023-01-24T03:30:50.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.429+0000 7f12d27fc700 1 -- 172.21.15.112:0/117304451 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12c80117d0 con 0x7f12d405dff0 2023-01-24T03:30:50.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.429+0000 7f12d27fc700 1 -- 172.21.15.112:0/117304451 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c8011b90 con 0x7f12d405dff0 2023-01-24T03:30:50.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.429+0000 7f12da21a700 1 -- 172.21.15.112:0/117304451 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f12d405dff0 msgr2=0x7f12d406d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:50.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.429+0000 7f12da21a700 1 --2- 172.21.15.112:0/117304451 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f12d405dff0 0x7f12d406d980 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f12d40bd300 tx=0x7f12c80090f0 comp rx=0 tx=0).stop 2023-01-24T03:30:50.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.429+0000 7f12da21a700 1 -- 172.21.15.112:0/117304451 shutdown_connections 2023-01-24T03:30:50.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.429+0000 7f12da21a700 1 --2- 172.21.15.112:0/117304451 >> v2:172.21.15.142:3300/0 conn(0x7f12d4153240 0x7f12d40c32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:50.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.429+0000 7f12da21a700 1 --2- 172.21.15.112:0/117304451 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f12d414d4c0 0x7f12d40c4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:50.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.429+0000 7f12da21a700 1 --2- 172.21.15.112:0/117304451 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f12d405dff0 0x7f12d406d980 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:50.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.429+0000 7f12da21a700 1 -- 172.21.15.112:0/117304451 >> 172.21.15.112:0/117304451 conn(0x7f12d406c4f0 msgr2=0x7f12d405d830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:50.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.429+0000 7f12da21a700 1 -- 172.21.15.112:0/117304451 shutdown_connections 2023-01-24T03:30:50.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.429+0000 7f12da21a700 1 -- 172.21.15.112:0/117304451 wait complete. 2023-01-24T03:30:50.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.430+0000 7f12da21a700 1 Processor -- start 2023-01-24T03:30:50.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.430+0000 7f12da21a700 1 -- start start 2023-01-24T03:30:50.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.431+0000 7f12da21a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f12d405dff0 0x7f12d4157d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:50.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.431+0000 7f12da21a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12d414d4c0 0x7f12d4163b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:50.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.431+0000 7f12da21a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f12d4153240 0x7f12d4145d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:50.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.431+0000 7f12da21a700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d4158290 con 0x7f12d405dff0 2023-01-24T03:30:50.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.431+0000 7f12da21a700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d414db00 con 0x7f12d414d4c0 2023-01-24T03:30:50.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.431+0000 7f12da21a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12d4163690 con 0x7f12d4153240 2023-01-24T03:30:50.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.431+0000 7f12d37fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f12d405dff0 0x7f12d4157d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:50.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.431+0000 7f12d37fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f12d405dff0 0x7f12d4157d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51096/0 (socket says 172.21.15.112:51096) 2023-01-24T03:30:50.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.431+0000 7f12d37fe700 1 -- 172.21.15.112:0/437075699 learned_addr learned my addr 172.21.15.112:0/437075699 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:50.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.431+0000 7f12d2ffd700 1 --2- 172.21.15.112:0/437075699 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12d414d4c0 0x7f12d4163b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:50.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.431+0000 7f12d3fff700 1 --2- 172.21.15.112:0/437075699 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f12d4153240 0x7f12d4145d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:50.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.431+0000 7f12d37fe700 1 -- 172.21.15.112:0/437075699 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f12d4153240 msgr2=0x7f12d4145d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:50.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.431+0000 7f12d37fe700 1 --2- 172.21.15.112:0/437075699 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f12d4153240 0x7f12d4145d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:50.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.431+0000 7f12d37fe700 1 -- 172.21.15.112:0/437075699 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12d414d4c0 msgr2=0x7f12d4163b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:50.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.431+0000 7f12d37fe700 1 --2- 172.21.15.112:0/437075699 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12d414d4c0 0x7f12d4163b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:50.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.431+0000 7f12d37fe700 1 -- 172.21.15.112:0/437075699 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c8009370 con 0x7f12d405dff0 2023-01-24T03:30:50.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.431+0000 7f12d3fff700 1 --2- 172.21.15.112:0/437075699 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f12d4153240 0x7f12d4145d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-01-24T03:30:50.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.432+0000 7f12d37fe700 1 --2- 172.21.15.112:0/437075699 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f12d405dff0 0x7f12d4157d50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f12c400c850 tx=0x7f12c4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:50.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.432+0000 7f12bb7fe700 1 -- 172.21.15.112:0/437075699 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c4011600 con 0x7f12d405dff0 2023-01-24T03:30:50.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.432+0000 7f12bb7fe700 1 -- 172.21.15.112:0/437075699 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12c4011760 con 0x7f12d405dff0 2023-01-24T03:30:50.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.432+0000 7f12bb7fe700 1 -- 172.21.15.112:0/437075699 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12c4011ae0 con 0x7f12d405dff0 2023-01-24T03:30:50.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.432+0000 7f12da21a700 1 -- 172.21.15.112:0/437075699 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12d414e4b0 con 0x7f12d405dff0 2023-01-24T03:30:50.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.432+0000 7f12da21a700 1 -- 172.21.15.112:0/437075699 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12d414dff0 con 0x7f12d405dff0 2023-01-24T03:30:50.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.433+0000 7f12da21a700 1 -- 172.21.15.112:0/437075699 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12d4163970 con 0x7f12d405dff0 2023-01-24T03:30:50.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.433+0000 7f12bb7fe700 1 -- 172.21.15.112:0/437075699 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f12c4020070 con 0x7f12d405dff0 2023-01-24T03:30:50.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.433+0000 7f12bb7fe700 1 --2- 172.21.15.112:0/437075699 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f12b40611f0 0x7f12b40636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:50.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.433+0000 7f12bb7fe700 1 -- 172.21.15.112:0/437075699 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f12c407d490 con 0x7f12d405dff0 2023-01-24T03:30:50.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.433+0000 7f12d2ffd700 1 --2- 172.21.15.112:0/437075699 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f12b40611f0 0x7f12b40636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:50.450 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.434+0000 7f12d2ffd700 1 --2- 172.21.15.112:0/437075699 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f12b40611f0 0x7f12b40636a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f12c8009340 tx=0x7f12c8016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:50.450 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.437+0000 7f12bb7fe700 1 -- 172.21.15.112:0/437075699 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f12c404d1e0 con 0x7f12d405dff0 2023-01-24T03:30:50.616 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.613+0000 7f12da21a700 1 -- 172.21.15.112:0/437075699 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f12d406d180 con 0x7f12d405dff0 2023-01-24T03:30:50.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.614+0000 7f12bb7fe700 1 -- 172.21.15.112:0/437075699 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7f12c40150a0 con 0x7f12d405dff0 2023-01-24T03:30:50.617 INFO:teuthology.orchestra.run.smithi112.stdout:pg_num: 32 2023-01-24T03:30:50.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.615+0000 7f12da21a700 1 -- 172.21.15.112:0/437075699 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f12b40611f0 msgr2=0x7f12b40636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:50.619 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.615+0000 7f12da21a700 1 --2- 172.21.15.112:0/437075699 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f12b40611f0 0x7f12b40636a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f12c8009340 tx=0x7f12c8016000 comp rx=0 tx=0).stop 2023-01-24T03:30:50.619 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.615+0000 7f12da21a700 1 -- 172.21.15.112:0/437075699 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f12d405dff0 msgr2=0x7f12d4157d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:50.619 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.615+0000 7f12da21a700 1 --2- 172.21.15.112:0/437075699 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f12d405dff0 0x7f12d4157d50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f12c400c850 tx=0x7f12c4008040 comp rx=0 tx=0).stop 2023-01-24T03:30:50.619 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.615+0000 7f12da21a700 1 -- 172.21.15.112:0/437075699 shutdown_connections 2023-01-24T03:30:50.619 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.615+0000 7f12da21a700 1 --2- 172.21.15.112:0/437075699 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f12b40611f0 0x7f12b40636a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:50.620 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.615+0000 7f12da21a700 1 --2- 172.21.15.112:0/437075699 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f12d405dff0 0x7f12d4157d50 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:50.620 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.615+0000 7f12da21a700 1 --2- 172.21.15.112:0/437075699 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f12d4153240 0x7f12d4145d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:50.620 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.615+0000 7f12da21a700 1 --2- 172.21.15.112:0/437075699 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12d414d4c0 0x7f12d4163b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:50.620 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.615+0000 7f12da21a700 1 -- 172.21.15.112:0/437075699 >> 172.21.15.112:0/437075699 conn(0x7f12d406c4f0 msgr2=0x7f12d4146c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:50.620 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.615+0000 7f12da21a700 1 -- 172.21.15.112:0/437075699 shutdown_connections 2023-01-24T03:30:50.621 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.615+0000 7f12da21a700 1 -- 172.21.15.112:0/437075699 wait complete. 2023-01-24T03:30:50.631 INFO:tasks.mds_thrash:mds thrasher using random seed: 1674531050 2023-01-24T03:30:50.631 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T03:30:50.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.743+0000 7ffa8b84b700 1 Processor -- start 2023-01-24T03:30:50.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.743+0000 7ffa8b84b700 1 -- start start 2023-01-24T03:30:50.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.744+0000 7ffa8b84b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7ffa840c7e40 0x7ffa840b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:50.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.744+0000 7ffa8b84b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa840ca3e0 0x7ffa840ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:50.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.744+0000 7ffa8b84b700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa84156240 con 0x7ffa84155da0 2023-01-24T03:30:50.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.744+0000 7ffa8b84b700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa840b1170 con 0x7ffa840c7e40 2023-01-24T03:30:50.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.744+0000 7ffa8b84b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa840b1450 con 0x7ffa840ca3e0 2023-01-24T03:30:50.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.744+0000 7ffa895e7700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7ffa840c7e40 0x7ffa840b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:50.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.744+0000 7ffa89de8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa840ca3e0 0x7ffa840ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:50.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.744+0000 7ffa895e7700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7ffa840c7e40 0x7ffa840b0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.112:43602/0 (socket says 172.21.15.112:43602) 2023-01-24T03:30:50.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.744+0000 7ffa88de6700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7ffa84155da0 0x7ffa840ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.112:45888/0 (socket says 172.21.15.112:45888) 2023-01-24T03:30:50.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.744+0000 7ffa895e7700 1 -- 172.21.15.112:0/4279068640 learned_addr learned my addr 172.21.15.112:0/4279068640 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:50.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.744+0000 7ffa89de8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa840ca3e0 0x7ffa840ca7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.112:38414/0 (socket says 172.21.15.112:38414) 2023-01-24T03:30:50.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.744+0000 7ffa895e7700 1 -- 172.21.15.112:0/4279068640 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa840ca3e0 msgr2=0x7ffa840ca7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:50.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.744+0000 7ffa895e7700 1 --2- 172.21.15.112:0/4279068640 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa840ca3e0 0x7ffa840ca7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:50.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.744+0000 7ffa895e7700 1 -- 172.21.15.112:0/4279068640 >> v1:172.21.15.142:6789/0 conn(0x7ffa84155da0 legacy=0x7ffa840ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:50.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.744+0000 7ffa895e7700 1 -- 172.21.15.112:0/4279068640 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa840b1730 con 0x7ffa840c7e40 2023-01-24T03:30:50.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.744+0000 7ffa77fff700 1 -- 172.21.15.112:0/4279068640 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1076370957 0 0) 0x7ffa84156240 con 0x7ffa84155da0 2023-01-24T03:30:50.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.744+0000 7ffa89de8700 1 --2- 172.21.15.112:0/4279068640 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa840ca3e0 0x7ffa840ca7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:30:50.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.744+0000 7ffa895e7700 1 --2- 172.21.15.112:0/4279068640 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa840c7e40 0x7ffa840b0aa0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c009f20 tx=0x7ffa7c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ba57af0abbf3aad server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:50.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.745+0000 7ffa77fff700 1 -- 172.21.15.112:0/4279068640 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7c00e070 con 0x7ffa840c7e40 2023-01-24T03:30:50.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.745+0000 7ffa77fff700 1 -- 172.21.15.112:0/4279068640 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa7c00b9d0 con 0x7ffa840c7e40 2023-01-24T03:30:50.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.745+0000 7ffa77fff700 1 -- 172.21.15.112:0/4279068640 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7c0092e0 con 0x7ffa840c7e40 2023-01-24T03:30:50.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.745+0000 7ffa8b84b700 1 -- 172.21.15.112:0/4279068640 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa840c7e40 msgr2=0x7ffa840b0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:50.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.745+0000 7ffa8b84b700 1 --2- 172.21.15.112:0/4279068640 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa840c7e40 0x7ffa840b0aa0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c009f20 tx=0x7ffa7c009840 comp rx=0 tx=0).stop 2023-01-24T03:30:50.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.745+0000 7ffa8b84b700 1 -- 172.21.15.112:0/4279068640 shutdown_connections 2023-01-24T03:30:50.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.745+0000 7ffa8b84b700 1 --2- 172.21.15.112:0/4279068640 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa840ca3e0 0x7ffa840ca7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:50.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.745+0000 7ffa8b84b700 1 --2- 172.21.15.112:0/4279068640 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa840c7e40 0x7ffa840b0aa0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:50.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.745+0000 7ffa8b84b700 1 -- 172.21.15.112:0/4279068640 >> 172.21.15.112:0/4279068640 conn(0x7ffa8405e2c0 msgr2=0x7ffa840cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:50.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.745+0000 7ffa8b84b700 1 -- 172.21.15.112:0/4279068640 shutdown_connections 2023-01-24T03:30:50.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.745+0000 7ffa8b84b700 1 -- 172.21.15.112:0/4279068640 wait complete. 2023-01-24T03:30:50.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.746+0000 7ffa8b84b700 1 Processor -- start 2023-01-24T03:30:50.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.746+0000 7ffa8b84b700 1 -- start start 2023-01-24T03:30:50.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.747+0000 7ffa8b84b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffa840c7e40 0x7ffa8414bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:50.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.747+0000 7ffa8b84b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa840ca3e0 0x7ffa840c4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:50.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.747+0000 7ffa8b84b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa84155da0 0x7ffa840c54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:50.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.747+0000 7ffa8b84b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa840c4310 con 0x7ffa840c7e40 2023-01-24T03:30:50.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.747+0000 7ffa8b84b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa840c4740 con 0x7ffa840ca3e0 2023-01-24T03:30:50.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.747+0000 7ffa8b84b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa841583d0 con 0x7ffa84155da0 2023-01-24T03:30:50.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.747+0000 7ffa895e7700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffa840c7e40 0x7ffa8414bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:50.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.747+0000 7ffa895e7700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffa840c7e40 0x7ffa8414bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51114/0 (socket says 172.21.15.112:51114) 2023-01-24T03:30:50.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.747+0000 7ffa895e7700 1 -- 172.21.15.112:0/82333474 learned_addr learned my addr 172.21.15.112:0/82333474 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:50.759 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.747+0000 7ffa88de6700 1 --2- 172.21.15.112:0/82333474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa840ca3e0 0x7ffa840c4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:50.759 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.747+0000 7ffa89de8700 1 --2- 172.21.15.112:0/82333474 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa84155da0 0x7ffa840c54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:50.759 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.748+0000 7ffa895e7700 1 -- 172.21.15.112:0/82333474 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa84155da0 msgr2=0x7ffa840c54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:50.759 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.748+0000 7ffa895e7700 1 --2- 172.21.15.112:0/82333474 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa84155da0 0x7ffa840c54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:50.759 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.748+0000 7ffa895e7700 1 -- 172.21.15.112:0/82333474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa840ca3e0 msgr2=0x7ffa840c4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:50.760 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.748+0000 7ffa895e7700 1 --2- 172.21.15.112:0/82333474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa840ca3e0 0x7ffa840c4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:50.760 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.748+0000 7ffa895e7700 1 -- 172.21.15.112:0/82333474 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa800092a0 con 0x7ffa840c7e40 2023-01-24T03:30:50.760 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.748+0000 7ffa895e7700 1 --2- 172.21.15.112:0/82333474 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffa840c7e40 0x7ffa8414bfe0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c00b610 tx=0x7ffa7c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:50.760 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.748+0000 7ffa757fa700 1 -- 172.21.15.112:0/82333474 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7c008870 con 0x7ffa840c7e40 2023-01-24T03:30:50.760 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.748+0000 7ffa757fa700 1 -- 172.21.15.112:0/82333474 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa7c0089d0 con 0x7ffa840c7e40 2023-01-24T03:30:50.761 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.748+0000 7ffa8b84b700 1 -- 172.21.15.112:0/82333474 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa7c009ae0 con 0x7ffa840c7e40 2023-01-24T03:30:50.761 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.748+0000 7ffa757fa700 1 -- 172.21.15.112:0/82333474 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7c008d90 con 0x7ffa840c7e40 2023-01-24T03:30:50.761 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.748+0000 7ffa8b84b700 1 -- 172.21.15.112:0/82333474 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa840c4990 con 0x7ffa840c7e40 2023-01-24T03:30:50.761 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.749+0000 7ffa8b84b700 1 -- 172.21.15.112:0/82333474 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa840c4450 con 0x7ffa840c7e40 2023-01-24T03:30:50.761 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.749+0000 7ffa757fa700 1 -- 172.21.15.112:0/82333474 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7ffa7c01c620 con 0x7ffa840c7e40 2023-01-24T03:30:50.761 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.750+0000 7ffa757fa700 1 --2- 172.21.15.112:0/82333474 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ffa640611c0 0x7ffa64063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:50.762 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.750+0000 7ffa757fa700 1 -- 172.21.15.112:0/82333474 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7ffa7c07f0b0 con 0x7ffa840c7e40 2023-01-24T03:30:50.762 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.750+0000 7ffa88de6700 1 --2- 172.21.15.112:0/82333474 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ffa640611c0 0x7ffa64063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:50.762 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.750+0000 7ffa88de6700 1 --2- 172.21.15.112:0/82333474 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ffa640611c0 0x7ffa64063670 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ffa840686c0 tx=0x7ffa78007bb0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:50.762 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.754+0000 7ffa757fa700 1 -- 172.21.15.112:0/82333474 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7ffa840c4450 con 0x7ffa840c7e40 2023-01-24T03:30:50.934 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.931+0000 7ffa8b84b700 1 -- 172.21.15.112:0/82333474 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa840c5df0 con 0x7ffa840c7e40 2023-01-24T03:30:50.935 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.932+0000 7ffa757fa700 1 -- 172.21.15.112:0/82333474 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18755 (secure 0 0 0) 0x7ffa7c04efb0 con 0x7ffa840c7e40 2023-01-24T03:30:50.936 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:30:50.936 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":24,"fsid":"054b9ac9-b7b0-40e5-b932-6c8a955c5a72","created":"2023-01-24T03:30:03.798533+0000","modified":"2023-01-24T03:30:25.373340+0000","last_up_change":"2023-01-24T03:30:13.945316+0000","last_in_change":"2023-01-24T03:30:06.198433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-01-24T03:30:14.931166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-01-24T03:30:15.023730+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-01-24T03:30:19.060597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-01-24T03:30:19.943686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-01-24T03:30:22.999036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"fe6ad231-a39a-42eb-9a3a-49d008dd35e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6809","nonce":84512},{"type":"v1","addr":"172.21.15.112:6811","nonce":84512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":84512},{"type":"v1","addr":"172.21.15.112:6817","nonce":84512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6825","nonce":84512},{"type":"v1","addr":"172.21.15.112:6828","nonce":84512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6819","nonce":84512},{"type":"v1","addr":"172.21.15.112:6822","nonce":84512}]},"public_addr":"172.21.15.112:6811/84512","cluster_addr":"172.21.15.112:6817/84512","heartbeat_back_addr":"172.21.15.112:6828/84512","heartbeat_front_addr":"172.21.15.112:6822/84512","state":["exists","up"]},{"osd":1,"uuid":"8526a1d2-d540-4021-9766-c4bbb30317ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":84513},{"type":"v1","addr":"172.21.15.112:6812","nonce":84513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6815","nonce":84513},{"type":"v1","addr":"172.21.15.112:6818","nonce":84513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6827","nonce":84513},{"type":"v1","addr":"172.21.15.112:6830","nonce":84513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6821","nonce":84513},{"type":"v1","addr":"172.21.15.112:6824","nonce":84513}]},"public_addr":"172.21.15.112:6812/84513","cluster_addr":"172.21.15.112:6818/84513","heartbeat_back_addr":"172.21.15.112:6830/84513","heartbeat_front_addr":"172.21.15.112:6824/84513","state":["exists","up"]},{"osd":2,"uuid":"041ef8e4-4858-432d-9476-cbdb3a0dc47c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":84511},{"type":"v1","addr":"172.21.15.112:6813","nonce":84511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":84511},{"type":"v1","addr":"172.21.15.112:6820","nonce":84511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6829","nonce":84511},{"type":"v1","addr":"172.21.15.112:6831","nonce":84511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6823","nonce":84511},{"type":"v1","addr":"172.21.15.112:6826","nonce":84511}]},"public_addr":"172.21.15.112:6813/84511","cluster_addr":"172.21.15.112:6820/84511","heartbeat_back_addr":"172.21.15.112:6831/84511","heartbeat_front_addr":"172.21.15.112:6826/84511","state":["exists","up"]},{"osd":3,"uuid":"987587cf-1357-4a70-ad85-67b189c67729","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":84510},{"type":"v1","addr":"172.21.15.112:6801","nonce":84510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":84510},{"type":"v1","addr":"172.21.15.112:6803","nonce":84510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":84510},{"type":"v1","addr":"172.21.15.112:6807","nonce":84510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":84510},{"type":"v1","addr":"172.21.15.112:6805","nonce":84510}]},"public_addr":"172.21.15.112:6801/84510","cluster_addr":"172.21.15.112:6803/84510","heartbeat_back_addr":"172.21.15.112:6807/84510","heartbeat_front_addr":"172.21.15.112:6805/84510","state":["exists","up"]},{"osd":4,"uuid":"8503512b-43db-40b8-8b2b-c86cf2e54e17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":84614},{"type":"v1","addr":"172.21.15.142:6801","nonce":84614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":84614},{"type":"v1","addr":"172.21.15.142:6803","nonce":84614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":84614},{"type":"v1","addr":"172.21.15.142:6807","nonce":84614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":84614},{"type":"v1","addr":"172.21.15.142:6805","nonce":84614}]},"public_addr":"172.21.15.142:6801/84614","cluster_addr":"172.21.15.142:6803/84614","heartbeat_back_addr":"172.21.15.142:6807/84614","heartbeat_front_addr":"172.21.15.142:6805/84614","state":["exists","up"]},{"osd":5,"uuid":"d3bc0030-8b08-4645-9ec2-1f8a4e29ef36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":84638},{"type":"v1","addr":"172.21.15.142:6817","nonce":84638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":84638},{"type":"v1","addr":"172.21.15.142:6819","nonce":84638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":84638},{"type":"v1","addr":"172.21.15.142:6823","nonce":84638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":84638},{"type":"v1","addr":"172.21.15.142:6821","nonce":84638}]},"public_addr":"172.21.15.142:6817/84638","cluster_addr":"172.21.15.142:6819/84638","heartbeat_back_addr":"172.21.15.142:6823/84638","heartbeat_front_addr":"172.21.15.142:6821/84638","state":["exists","up"]},{"osd":6,"uuid":"c7649e3b-832d-4e6f-8866-b15e5d814476","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":84645},{"type":"v1","addr":"172.21.15.142:6825","nonce":84645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":84645},{"type":"v1","addr":"172.21.15.142:6827","nonce":84645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":84645},{"type":"v1","addr":"172.21.15.142:6831","nonce":84645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":84645},{"type":"v1","addr":"172.21.15.142:6829","nonce":84645}]},"public_addr":"172.21.15.142:6825/84645","cluster_addr":"172.21.15.142:6827/84645","heartbeat_back_addr":"172.21.15.142:6831/84645","heartbeat_front_addr":"172.21.15.142:6829/84645","state":["exists","up"]},{"osd":7,"uuid":"2ea5b3bf-884e-4b20-b572-1443efdc7477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":84615},{"type":"v1","addr":"172.21.15.142:6809","nonce":84615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":84615},{"type":"v1","addr":"172.21.15.142:6811","nonce":84615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":84615},{"type":"v1","addr":"172.21.15.142:6815","nonce":84615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":84615},{"type":"v1","addr":"172.21.15.142:6813","nonce":84615}]},"public_addr":"172.21.15.142:6809/84615","cluster_addr":"172.21.15.142:6811/84615","heartbeat_back_addr":"172.21.15.142:6815/84615","heartbeat_front_addr":"172.21.15.142:6813/84615","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.370027+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.335683+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.350667+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.350794+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.391119+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.422933+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.400439+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.354018+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T03:30:50.937 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.934+0000 7ffa8b84b700 1 -- 172.21.15.112:0/82333474 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ffa640611c0 msgr2=0x7ffa64063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:50.938 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.934+0000 7ffa8b84b700 1 --2- 172.21.15.112:0/82333474 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ffa640611c0 0x7ffa64063670 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ffa840686c0 tx=0x7ffa78007bb0 comp rx=0 tx=0).stop 2023-01-24T03:30:50.938 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.934+0000 7ffa8b84b700 1 -- 172.21.15.112:0/82333474 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffa840c7e40 msgr2=0x7ffa8414bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:50.938 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.934+0000 7ffa8b84b700 1 --2- 172.21.15.112:0/82333474 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffa840c7e40 0x7ffa8414bfe0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c00b610 tx=0x7ffa7c018000 comp rx=0 tx=0).stop 2023-01-24T03:30:50.938 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.935+0000 7ffa8b84b700 1 -- 172.21.15.112:0/82333474 shutdown_connections 2023-01-24T03:30:50.939 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.935+0000 7ffa8b84b700 1 --2- 172.21.15.112:0/82333474 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ffa640611c0 0x7ffa64063670 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:50.939 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.935+0000 7ffa8b84b700 1 --2- 172.21.15.112:0/82333474 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffa840c7e40 0x7ffa8414bfe0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:50.939 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.935+0000 7ffa8b84b700 1 --2- 172.21.15.112:0/82333474 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffa84155da0 0x7ffa840c54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:50.939 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.935+0000 7ffa8b84b700 1 --2- 172.21.15.112:0/82333474 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa840ca3e0 0x7ffa840c4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:50.939 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.935+0000 7ffa8b84b700 1 -- 172.21.15.112:0/82333474 >> 172.21.15.112:0/82333474 conn(0x7ffa8405e2c0 msgr2=0x7ffa840689d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:50.940 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.935+0000 7ffa8b84b700 1 -- 172.21.15.112:0/82333474 shutdown_connections 2023-01-24T03:30:50.940 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:50.935+0000 7ffa8b84b700 1 -- 172.21.15.112:0/82333474 wait complete. 2023-01-24T03:30:50.949 INFO:tasks.mds_thrash.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-01-24T03:30:14.931166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-01-24T03:30:15.023730+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-01-24T03:30:19.060597+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-01-24T03:30:19.943686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-01-24T03:30:22.999036+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-01-24T03:30:50.950 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-01-24T03:30:51.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.062+0000 7f33d3af5700 1 Processor -- start 2023-01-24T03:30:51.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.062+0000 7f33d3af5700 1 -- start start 2023-01-24T03:30:51.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.063+0000 7f33d3af5700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f33cc0694c0 0x7f33cc069890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:51.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.063+0000 7f33d3af5700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f33cc05e420 0x7f33cc155f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:51.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.063+0000 7f33d3af5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f33cc068d20 0x7f33cc0c68d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:51.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.063+0000 7f33d3af5700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33cc155550 con 0x7f33cc05e420 2023-01-24T03:30:51.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.063+0000 7f33d3af5700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33cc155ab0 con 0x7f33cc0694c0 2023-01-24T03:30:51.069 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.063+0000 7f33d3af5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc156480 con 0x7f33cc068d20 2023-01-24T03:30:51.069 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.063+0000 7f33d1090700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f33cc05e420 0x7f33cc155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:51.069 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.063+0000 7f33d1090700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f33cc05e420 0x7f33cc155f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51126/0 (socket says 172.21.15.112:51126) 2023-01-24T03:30:51.069 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.063+0000 7f33d1090700 1 -- 172.21.15.112:0/1484131554 learned_addr learned my addr 172.21.15.112:0/1484131554 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:51.069 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.063+0000 7f33d1891700 1 --2- 172.21.15.112:0/1484131554 >> v2:172.21.15.142:3300/0 conn(0x7f33cc0694c0 0x7f33cc069890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:51.070 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.063+0000 7f33d2092700 1 --2- 172.21.15.112:0/1484131554 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f33cc068d20 0x7f33cc0c68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:51.070 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.063+0000 7f33d1090700 1 -- 172.21.15.112:0/1484131554 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f33cc068d20 msgr2=0x7f33cc0c68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:51.070 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.063+0000 7f33d1090700 1 --2- 172.21.15.112:0/1484131554 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f33cc068d20 0x7f33cc0c68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:51.071 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.063+0000 7f33d1090700 1 -- 172.21.15.112:0/1484131554 >> v2:172.21.15.142:3300/0 conn(0x7f33cc0694c0 msgr2=0x7f33cc069890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:51.071 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.063+0000 7f33d1090700 1 --2- 172.21.15.112:0/1484131554 >> v2:172.21.15.142:3300/0 conn(0x7f33cc0694c0 0x7f33cc069890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:51.071 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.063+0000 7f33d1090700 1 -- 172.21.15.112:0/1484131554 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33cc0c6e10 con 0x7f33cc05e420 2023-01-24T03:30:51.071 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.063+0000 7f33d1891700 1 --2- 172.21.15.112:0/1484131554 >> v2:172.21.15.142:3300/0 conn(0x7f33cc0694c0 0x7f33cc069890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:30:51.071 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.063+0000 7f33d1090700 1 --2- 172.21.15.112:0/1484131554 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33cc05e420 0x7f33cc155f40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f33cc066360 tx=0x7f33bc0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cbf2a9f82144c85 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:51.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.064+0000 7f33d088f700 1 -- 172.21.15.112:0/1484131554 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33bc011630 con 0x7f33cc05e420 2023-01-24T03:30:51.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.064+0000 7f33d088f700 1 -- 172.21.15.112:0/1484131554 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33bc0117d0 con 0x7f33cc05e420 2023-01-24T03:30:51.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.064+0000 7f33d088f700 1 -- 172.21.15.112:0/1484131554 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33bc011b90 con 0x7f33cc05e420 2023-01-24T03:30:51.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.064+0000 7f33d3af5700 1 -- 172.21.15.112:0/1484131554 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33cc05e420 msgr2=0x7f33cc155f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:51.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.064+0000 7f33d3af5700 1 --2- 172.21.15.112:0/1484131554 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33cc05e420 0x7f33cc155f40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f33cc066360 tx=0x7f33bc0090f0 comp rx=0 tx=0).stop 2023-01-24T03:30:51.073 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.064+0000 7f33d3af5700 1 -- 172.21.15.112:0/1484131554 shutdown_connections 2023-01-24T03:30:51.073 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.064+0000 7f33d3af5700 1 --2- 172.21.15.112:0/1484131554 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f33cc068d20 0x7f33cc0c68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:51.073 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.064+0000 7f33d3af5700 1 --2- 172.21.15.112:0/1484131554 >> v2:172.21.15.142:3300/0 conn(0x7f33cc0694c0 0x7f33cc069890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:51.073 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.064+0000 7f33d3af5700 1 --2- 172.21.15.112:0/1484131554 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33cc05e420 0x7f33cc155f40 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:51.073 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.064+0000 7f33d3af5700 1 -- 172.21.15.112:0/1484131554 >> 172.21.15.112:0/1484131554 conn(0x7f33cc05d830 msgr2=0x7f33cc05dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:51.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.064+0000 7f33d3af5700 1 -- 172.21.15.112:0/1484131554 shutdown_connections 2023-01-24T03:30:51.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.064+0000 7f33d3af5700 1 -- 172.21.15.112:0/1484131554 wait complete. 2023-01-24T03:30:51.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.065+0000 7f33d3af5700 1 Processor -- start 2023-01-24T03:30:51.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.065+0000 7f33d3af5700 1 -- start start 2023-01-24T03:30:51.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.066+0000 7f33d3af5700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33cc05e420 0x7f33cc0c9b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:51.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.066+0000 7f33d3af5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f33cc068d20 0x7f33cc0cc440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:51.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.066+0000 7f33d3af5700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33cc0694c0 0x7f33cc14cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:51.078 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.066+0000 7f33d3af5700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc0c3070 con 0x7f33cc0694c0 2023-01-24T03:30:51.078 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.066+0000 7f33d3af5700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc1662c0 con 0x7f33cc05e420 2023-01-24T03:30:51.078 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.066+0000 7f33d3af5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc0c60b0 con 0x7f33cc068d20 2023-01-24T03:30:51.078 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.066+0000 7f33d2092700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33cc0694c0 0x7f33cc14cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:51.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.066+0000 7f33d2092700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33cc0694c0 0x7f33cc14cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51132/0 (socket says 172.21.15.112:51132) 2023-01-24T03:30:51.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.066+0000 7f33d2092700 1 -- 172.21.15.112:0/1135535053 learned_addr learned my addr 172.21.15.112:0/1135535053 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:51.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.066+0000 7f33d1891700 1 --2- 172.21.15.112:0/1135535053 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33cc05e420 0x7f33cc0c9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:51.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.066+0000 7f33d1090700 1 --2- 172.21.15.112:0/1135535053 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f33cc068d20 0x7f33cc0cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:51.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.066+0000 7f33d2092700 1 -- 172.21.15.112:0/1135535053 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f33cc068d20 msgr2=0x7f33cc0cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:51.080 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.066+0000 7f33d2092700 1 --2- 172.21.15.112:0/1135535053 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f33cc068d20 0x7f33cc0cc440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:51.080 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.066+0000 7f33d2092700 1 -- 172.21.15.112:0/1135535053 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33cc05e420 msgr2=0x7f33cc0c9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:51.080 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.066+0000 7f33d2092700 1 --2- 172.21.15.112:0/1135535053 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33cc05e420 0x7f33cc0c9b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:51.080 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.066+0000 7f33d2092700 1 -- 172.21.15.112:0/1135535053 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33bc009370 con 0x7f33cc0694c0 2023-01-24T03:30:51.080 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.066+0000 7f33d2092700 1 --2- 172.21.15.112:0/1135535053 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33cc0694c0 0x7f33cc14cce0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f33c4009fa0 tx=0x7f33c4009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:51.081 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.066+0000 7f33b9ffb700 1 -- 172.21.15.112:0/1135535053 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c400ebc0 con 0x7f33cc0694c0 2023-01-24T03:30:51.081 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.067+0000 7f33b9ffb700 1 -- 172.21.15.112:0/1135535053 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33c400ed20 con 0x7f33cc0694c0 2023-01-24T03:30:51.083 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.067+0000 7f33b9ffb700 1 -- 172.21.15.112:0/1135535053 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c4014670 con 0x7f33cc0694c0 2023-01-24T03:30:51.083 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.067+0000 7f33d3af5700 1 -- 172.21.15.112:0/1135535053 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33cc165fe0 con 0x7f33cc0694c0 2023-01-24T03:30:51.083 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.067+0000 7f33d3af5700 1 -- 172.21.15.112:0/1135535053 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33cc0cbc40 con 0x7f33cc0694c0 2023-01-24T03:30:51.083 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.068+0000 7f33d3af5700 1 -- 172.21.15.112:0/1135535053 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33cc0cb7c0 con 0x7f33cc0694c0 2023-01-24T03:30:51.084 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.068+0000 7f33b9ffb700 1 -- 172.21.15.112:0/1135535053 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f33c4012050 con 0x7f33cc0694c0 2023-01-24T03:30:51.084 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.068+0000 7f33b9ffb700 1 --2- 172.21.15.112:0/1135535053 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f33ac0611f0 0x7f33ac0636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:51.084 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.068+0000 7f33b9ffb700 1 -- 172.21.15.112:0/1135535053 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f33c407dd20 con 0x7f33cc0694c0 2023-01-24T03:30:51.084 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.068+0000 7f33d1891700 1 --2- 172.21.15.112:0/1135535053 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f33ac0611f0 0x7f33ac0636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:51.084 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.069+0000 7f33d1891700 1 --2- 172.21.15.112:0/1135535053 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f33ac0611f0 0x7f33ac0636a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f33c8006d80 tx=0x7f33c8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:51.085 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.072+0000 7f33b9ffb700 1 -- 172.21.15.112:0/1135535053 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f33cc0cb7c0 con 0x7f33cc0694c0 2023-01-24T03:30:51.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.250+0000 7f33d3af5700 1 -- 172.21.15.112:0/1135535053 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f33cc0cb7c0 con 0x7f33cc0694c0 2023-01-24T03:30:51.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.250+0000 7f33b9ffb700 1 -- 172.21.15.112:0/1135535053 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v24) v1 ==== 92+0+10 (secure 0 0 0) 0x7f33cc0cb7c0 con 0x7f33cc0694c0 2023-01-24T03:30:51.254 INFO:teuthology.orchestra.run.smithi112.stdout:pg_num: 8 2023-01-24T03:30:51.255 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.252+0000 7f33d3af5700 1 -- 172.21.15.112:0/1135535053 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f33ac0611f0 msgr2=0x7f33ac0636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:51.255 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.252+0000 7f33d3af5700 1 --2- 172.21.15.112:0/1135535053 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f33ac0611f0 0x7f33ac0636a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f33c8006d80 tx=0x7f33c8008040 comp rx=0 tx=0).stop 2023-01-24T03:30:51.256 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.252+0000 7f33d3af5700 1 -- 172.21.15.112:0/1135535053 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33cc0694c0 msgr2=0x7f33cc14cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:51.256 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.252+0000 7f33d3af5700 1 --2- 172.21.15.112:0/1135535053 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33cc0694c0 0x7f33cc14cce0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f33c4009fa0 tx=0x7f33c4009900 comp rx=0 tx=0).stop 2023-01-24T03:30:51.256 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.252+0000 7f33d3af5700 1 -- 172.21.15.112:0/1135535053 shutdown_connections 2023-01-24T03:30:51.256 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.252+0000 7f33d3af5700 1 --2- 172.21.15.112:0/1135535053 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f33ac0611f0 0x7f33ac0636a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:51.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.252+0000 7f33d3af5700 1 --2- 172.21.15.112:0/1135535053 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33cc0694c0 0x7f33cc14cce0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:51.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.252+0000 7f33d3af5700 1 --2- 172.21.15.112:0/1135535053 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33cc05e420 0x7f33cc0c9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:51.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.252+0000 7f33d3af5700 1 --2- 172.21.15.112:0/1135535053 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f33cc068d20 0x7f33cc0cc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:51.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.252+0000 7f33d3af5700 1 -- 172.21.15.112:0/1135535053 >> 172.21.15.112:0/1135535053 conn(0x7f33cc05d830 msgr2=0x7f33cc14f140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:51.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.252+0000 7f33d3af5700 1 -- 172.21.15.112:0/1135535053 shutdown_connections 2023-01-24T03:30:51.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.252+0000 7f33d3af5700 1 -- 172.21.15.112:0/1135535053 wait complete. 2023-01-24T03:30:51.267 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-01-24T03:30:51.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.378+0000 7ff1a4dea700 1 Processor -- start 2023-01-24T03:30:51.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.378+0000 7ff1a4dea700 1 -- start start 2023-01-24T03:30:51.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.379+0000 7ff1a4dea700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7ff1a00b84d0 0x7ff1a0155970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:51.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.379+0000 7ff1a4dea700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff1a00c93b0 0x7ff1a014f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:51.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.379+0000 7ff1a4dea700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7ff1a00b88a0 0x7ff1a014fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:51.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.379+0000 7ff1a4dea700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1a006ec60 con 0x7ff1a00b88a0 2023-01-24T03:30:51.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.379+0000 7ff1a4dea700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1a00c3650 con 0x7ff1a00b84d0 2023-01-24T03:30:51.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.379+0000 7ff1a4dea700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a006bb10 con 0x7ff1a00c93b0 2023-01-24T03:30:51.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.379+0000 7ff19ed9d700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7ff1a00b88a0 0x7ff1a014fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:51.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.380+0000 7ff19ed9d700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7ff1a00b88a0 0x7ff1a014fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51138/0 (socket says 172.21.15.112:51138) 2023-01-24T03:30:51.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.380+0000 7ff19ed9d700 1 -- 172.21.15.112:0/2469585147 learned_addr learned my addr 172.21.15.112:0/2469585147 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:51.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.380+0000 7ff19e59c700 1 --2- 172.21.15.112:0/2469585147 >> v2:172.21.15.142:3300/0 conn(0x7ff1a00b84d0 0x7ff1a0155970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:51.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.380+0000 7ff19dd9b700 1 --2- 172.21.15.112:0/2469585147 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff1a00c93b0 0x7ff1a014f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:51.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.380+0000 7ff19ed9d700 1 -- 172.21.15.112:0/2469585147 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff1a00c93b0 msgr2=0x7ff1a014f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:51.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.380+0000 7ff19ed9d700 1 --2- 172.21.15.112:0/2469585147 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff1a00c93b0 0x7ff1a014f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:51.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.380+0000 7ff19ed9d700 1 -- 172.21.15.112:0/2469585147 >> v2:172.21.15.142:3300/0 conn(0x7ff1a00b84d0 msgr2=0x7ff1a0155970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:51.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.380+0000 7ff19ed9d700 1 --2- 172.21.15.112:0/2469585147 >> v2:172.21.15.142:3300/0 conn(0x7ff1a00b84d0 0x7ff1a0155970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:51.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.380+0000 7ff19ed9d700 1 -- 172.21.15.112:0/2469585147 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a00c4e90 con 0x7ff1a00b88a0 2023-01-24T03:30:51.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.380+0000 7ff19e59c700 1 --2- 172.21.15.112:0/2469585147 >> v2:172.21.15.142:3300/0 conn(0x7ff1a00b84d0 0x7ff1a0155970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:30:51.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.380+0000 7ff19ed9d700 1 --2- 172.21.15.112:0/2469585147 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1a00b88a0 0x7ff1a014fb80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0063d60 tx=0x7ff1980090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b0e6183ae0f1990 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:51.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.380+0000 7ff19d59a700 1 -- 172.21.15.112:0/2469585147 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff198011630 con 0x7ff1a00b88a0 2023-01-24T03:30:51.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.380+0000 7ff19d59a700 1 -- 172.21.15.112:0/2469585147 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1980117d0 con 0x7ff1a00b88a0 2023-01-24T03:30:51.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.380+0000 7ff19d59a700 1 -- 172.21.15.112:0/2469585147 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff198011b90 con 0x7ff1a00b88a0 2023-01-24T03:30:51.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.381+0000 7ff1a4dea700 1 -- 172.21.15.112:0/2469585147 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1a00b88a0 msgr2=0x7ff1a014fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:51.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.381+0000 7ff1a4dea700 1 --2- 172.21.15.112:0/2469585147 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1a00b88a0 0x7ff1a014fb80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0063d60 tx=0x7ff1980090f0 comp rx=0 tx=0).stop 2023-01-24T03:30:51.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.381+0000 7ff1a4dea700 1 -- 172.21.15.112:0/2469585147 shutdown_connections 2023-01-24T03:30:51.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.381+0000 7ff1a4dea700 1 --2- 172.21.15.112:0/2469585147 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1a00b88a0 0x7ff1a014fb80 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:51.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.381+0000 7ff1a4dea700 1 --2- 172.21.15.112:0/2469585147 >> v2:172.21.15.142:3300/0 conn(0x7ff1a00b84d0 0x7ff1a0155970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:51.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.381+0000 7ff1a4dea700 1 --2- 172.21.15.112:0/2469585147 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff1a00c93b0 0x7ff1a014f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:51.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.381+0000 7ff1a4dea700 1 -- 172.21.15.112:0/2469585147 >> 172.21.15.112:0/2469585147 conn(0x7ff1a00689f0 msgr2=0x7ff1a00c8720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:51.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.381+0000 7ff1a4dea700 1 -- 172.21.15.112:0/2469585147 shutdown_connections 2023-01-24T03:30:51.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.381+0000 7ff1a4dea700 1 -- 172.21.15.112:0/2469585147 wait complete. 2023-01-24T03:30:51.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.382+0000 7ff1a4dea700 1 Processor -- start 2023-01-24T03:30:51.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.382+0000 7ff1a4dea700 1 -- start start 2023-01-24T03:30:51.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.382+0000 7ff1a4dea700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff1a00b84d0 0x7ff1a0157480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:51.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.382+0000 7ff1a4dea700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1a00b88a0 0x7ff1a01520e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:51.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.382+0000 7ff1a4dea700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1a00c93b0 0x7ff1a0158910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:51.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.382+0000 7ff1a4dea700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a0151650 con 0x7ff1a00c93b0 2023-01-24T03:30:51.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.382+0000 7ff1a4dea700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a00c7a80 con 0x7ff1a00b88a0 2023-01-24T03:30:51.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.382+0000 7ff1a4dea700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a0152cd0 con 0x7ff1a00b84d0 2023-01-24T03:30:51.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.383+0000 7ff19ed9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1a00c93b0 0x7ff1a0158910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:51.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.383+0000 7ff19ed9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1a00c93b0 0x7ff1a0158910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51146/0 (socket says 172.21.15.112:51146) 2023-01-24T03:30:51.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.383+0000 7ff19ed9d700 1 -- 172.21.15.112:0/2524995930 learned_addr learned my addr 172.21.15.112:0/2524995930 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:51.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.383+0000 7ff19dd9b700 1 --2- 172.21.15.112:0/2524995930 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1a00b88a0 0x7ff1a01520e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:51.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.383+0000 7ff19e59c700 1 --2- 172.21.15.112:0/2524995930 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff1a00b84d0 0x7ff1a0157480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:51.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.383+0000 7ff19ed9d700 1 -- 172.21.15.112:0/2524995930 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff1a00b84d0 msgr2=0x7ff1a0157480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:51.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.383+0000 7ff19ed9d700 1 --2- 172.21.15.112:0/2524995930 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff1a00b84d0 0x7ff1a0157480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:51.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.383+0000 7ff19ed9d700 1 -- 172.21.15.112:0/2524995930 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1a00b88a0 msgr2=0x7ff1a01520e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:51.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.383+0000 7ff19ed9d700 1 --2- 172.21.15.112:0/2524995930 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1a00b88a0 0x7ff1a01520e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:51.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.383+0000 7ff19ed9d700 1 -- 172.21.15.112:0/2524995930 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff198009370 con 0x7ff1a00c93b0 2023-01-24T03:30:51.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.383+0000 7ff19ed9d700 1 --2- 172.21.15.112:0/2524995930 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1a00c93b0 0x7ff1a0158910 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff1a006d1b0 tx=0x7ff198016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:51.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.383+0000 7ff1967fc700 1 -- 172.21.15.112:0/2524995930 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff198006d90 con 0x7ff1a00c93b0 2023-01-24T03:30:51.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.383+0000 7ff1967fc700 1 -- 172.21.15.112:0/2524995930 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff198020070 con 0x7ff1a00c93b0 2023-01-24T03:30:51.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.383+0000 7ff1967fc700 1 -- 172.21.15.112:0/2524995930 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1980276b0 con 0x7ff1a00c93b0 2023-01-24T03:30:51.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.383+0000 7ff1a4dea700 1 -- 172.21.15.112:0/2524995930 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1a0158480 con 0x7ff1a00c93b0 2023-01-24T03:30:51.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.383+0000 7ff1a4dea700 1 -- 172.21.15.112:0/2524995930 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1a01531e0 con 0x7ff1a00c93b0 2023-01-24T03:30:51.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.384+0000 7ff1a4dea700 1 -- 172.21.15.112:0/2524995930 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1a005e820 con 0x7ff1a00c93b0 2023-01-24T03:30:51.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.384+0000 7ff1967fc700 1 -- 172.21.15.112:0/2524995930 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7ff198017020 con 0x7ff1a00c93b0 2023-01-24T03:30:51.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.385+0000 7ff1967fc700 1 --2- 172.21.15.112:0/2524995930 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff1840611c0 0x7ff184063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:51.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.385+0000 7ff1967fc700 1 -- 172.21.15.112:0/2524995930 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7ff198022070 con 0x7ff1a00c93b0 2023-01-24T03:30:51.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.385+0000 7ff19e59c700 1 --2- 172.21.15.112:0/2524995930 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff1840611c0 0x7ff184063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:51.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.385+0000 7ff19e59c700 1 --2- 172.21.15.112:0/2524995930 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff1840611c0 0x7ff184063670 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff190007740 tx=0x7ff1900099d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:51.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.389+0000 7ff1967fc700 1 -- 172.21.15.112:0/2524995930 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7ff19804f4a0 con 0x7ff1a00c93b0 2023-01-24T03:30:51.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.564+0000 7ff1a4dea700 1 -- 172.21.15.112:0/2524995930 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff1a0158610 con 0x7ff1a00c93b0 2023-01-24T03:30:51.568 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.565+0000 7ff1967fc700 1 -- 172.21.15.112:0/2524995930 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff198011b50 con 0x7ff1a00c93b0 2023-01-24T03:30:51.568 INFO:teuthology.orchestra.run.smithi112.stdout:pg_num: 1 2023-01-24T03:30:51.570 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.567+0000 7ff1a4dea700 1 -- 172.21.15.112:0/2524995930 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff1840611c0 msgr2=0x7ff184063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:51.570 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.567+0000 7ff1a4dea700 1 --2- 172.21.15.112:0/2524995930 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff1840611c0 0x7ff184063670 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff190007740 tx=0x7ff1900099d0 comp rx=0 tx=0).stop 2023-01-24T03:30:51.570 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.567+0000 7ff1a4dea700 1 -- 172.21.15.112:0/2524995930 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1a00c93b0 msgr2=0x7ff1a0158910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:51.570 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.567+0000 7ff1a4dea700 1 --2- 172.21.15.112:0/2524995930 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1a00c93b0 0x7ff1a0158910 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff1a006d1b0 tx=0x7ff198016000 comp rx=0 tx=0).stop 2023-01-24T03:30:51.571 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.567+0000 7ff1a4dea700 1 -- 172.21.15.112:0/2524995930 shutdown_connections 2023-01-24T03:30:51.571 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.567+0000 7ff1a4dea700 1 --2- 172.21.15.112:0/2524995930 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff1840611c0 0x7ff184063670 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:51.571 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.567+0000 7ff1a4dea700 1 --2- 172.21.15.112:0/2524995930 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1a00c93b0 0x7ff1a0158910 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:51.571 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.567+0000 7ff1a4dea700 1 --2- 172.21.15.112:0/2524995930 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff1a00b84d0 0x7ff1a0157480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:51.571 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.567+0000 7ff1a4dea700 1 --2- 172.21.15.112:0/2524995930 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff1a00b88a0 0x7ff1a01520e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:51.572 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.567+0000 7ff1a4dea700 1 -- 172.21.15.112:0/2524995930 >> 172.21.15.112:0/2524995930 conn(0x7ff1a00689f0 msgr2=0x7ff1a00c58b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:51.572 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.567+0000 7ff1a4dea700 1 -- 172.21.15.112:0/2524995930 shutdown_connections 2023-01-24T03:30:51.572 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.567+0000 7ff1a4dea700 1 -- 172.21.15.112:0/2524995930 wait complete. 2023-01-24T03:30:51.583 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-01-24T03:30:51.697 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.694+0000 7f3abf0e8700 1 Processor -- start 2023-01-24T03:30:51.697 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.694+0000 7f3abf0e8700 1 -- start start 2023-01-24T03:30:51.700 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.695+0000 7f3abf0e8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ab8068d20 0x7f3ab8069890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:51.700 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.695+0000 7f3abf0e8700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ab8155ab0 con 0x7f3ab80694c0 2023-01-24T03:30:51.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.695+0000 7f3abf0e8700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ab80c70e0 con 0x7f3ab805e420 2023-01-24T03:30:51.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.695+0000 7f3abf0e8700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab8155550 con 0x7f3ab8068d20 2023-01-24T03:30:51.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.695+0000 7f3abce84700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f3ab80694c0 0x7f3ab80c68d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:60370/0 (socket says 172.21.15.112:60370) 2023-01-24T03:30:51.702 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.695+0000 7f3abce84700 1 -- 172.21.15.112:0/2504537608 learned_addr learned my addr 172.21.15.112:0/2504537608 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:51.702 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.695+0000 7f3abd685700 1 --2- 172.21.15.112:0/2504537608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ab8068d20 0x7f3ab8069890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:51.702 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.695+0000 7f3ab77fe700 1 -- 172.21.15.112:0/2504537608 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 404406868 0 0) 0x7f3ab8155ab0 con 0x7f3ab80694c0 2023-01-24T03:30:51.702 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.695+0000 7f3ab77fe700 1 -- 172.21.15.112:0/2504537608 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a980033f0 con 0x7f3ab80694c0 2023-01-24T03:30:51.702 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.695+0000 7f3ab77fe700 1 -- 172.21.15.112:0/2504537608 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2743136795 0 0) 0x7f3ab80c70e0 con 0x7f3ab805e420 2023-01-24T03:30:51.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.695+0000 7f3ab77fe700 1 -- 172.21.15.112:0/2504537608 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ab8155ab0 con 0x7f3ab805e420 2023-01-24T03:30:51.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.695+0000 7f3abd685700 1 -- 172.21.15.112:0/2504537608 >> v1:172.21.15.142:6789/0 conn(0x7f3ab805e420 legacy=0x7f3ab80ae9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:51.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.695+0000 7f3abd685700 1 -- 172.21.15.112:0/2504537608 >> v1:172.21.15.112:6789/0 conn(0x7f3ab80694c0 legacy=0x7f3ab80c68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:51.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.695+0000 7f3abd685700 1 -- 172.21.15.112:0/2504537608 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab80c73c0 con 0x7f3ab8068d20 2023-01-24T03:30:51.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.696+0000 7f3abd685700 1 --2- 172.21.15.112:0/2504537608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ab8068d20 0x7f3ab8069890 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3ab0009790 tx=0x7f3ab0009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca2c8532b90ab7f7 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:51.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.696+0000 7f3ab77fe700 1 -- 172.21.15.112:0/2504537608 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab0011630 con 0x7f3ab8068d20 2023-01-24T03:30:51.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.696+0000 7f3ab77fe700 1 -- 172.21.15.112:0/2504537608 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ab0011790 con 0x7f3ab8068d20 2023-01-24T03:30:51.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.696+0000 7f3ab77fe700 1 -- 172.21.15.112:0/2504537608 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab0011b50 con 0x7f3ab8068d20 2023-01-24T03:30:51.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.696+0000 7f3abf0e8700 1 -- 172.21.15.112:0/2504537608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ab8068d20 msgr2=0x7f3ab8069890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:51.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.696+0000 7f3abf0e8700 1 --2- 172.21.15.112:0/2504537608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ab8068d20 0x7f3ab8069890 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3ab0009790 tx=0x7f3ab0009060 comp rx=0 tx=0).stop 2023-01-24T03:30:51.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.696+0000 7f3abf0e8700 1 -- 172.21.15.112:0/2504537608 shutdown_connections 2023-01-24T03:30:51.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.696+0000 7f3abf0e8700 1 --2- 172.21.15.112:0/2504537608 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ab8068d20 0x7f3ab8069890 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:51.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.696+0000 7f3abf0e8700 1 -- 172.21.15.112:0/2504537608 >> 172.21.15.112:0/2504537608 conn(0x7f3ab805d830 msgr2=0x7f3ab805dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:51.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.696+0000 7f3abf0e8700 1 -- 172.21.15.112:0/2504537608 shutdown_connections 2023-01-24T03:30:51.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.696+0000 7f3abf0e8700 1 -- 172.21.15.112:0/2504537608 wait complete. 2023-01-24T03:30:51.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.697+0000 7f3abf0e8700 1 Processor -- start 2023-01-24T03:30:51.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.698+0000 7f3abf0e8700 1 -- start start 2023-01-24T03:30:51.710 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.698+0000 7f3abf0e8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ab805e420 0x7f3ab816d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:51.710 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.698+0000 7f3abf0e8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3ab8068d20 0x7f3ab80c5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:51.710 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.698+0000 7f3abf0e8700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3ab80694c0 0x7f3ab81521a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:51.710 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.698+0000 7f3abf0e8700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab816dcd0 con 0x7f3ab80694c0 2023-01-24T03:30:51.710 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.698+0000 7f3abf0e8700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab80c89e0 con 0x7f3ab8068d20 2023-01-24T03:30:51.711 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.698+0000 7f3abf0e8700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab8150c60 con 0x7f3ab805e420 2023-01-24T03:30:51.711 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.698+0000 7f3abd685700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3ab80694c0 0x7f3ab81521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:51.711 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.698+0000 7f3abd685700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3ab80694c0 0x7f3ab81521a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51160/0 (socket says 172.21.15.112:51160) 2023-01-24T03:30:51.711 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.698+0000 7f3abd685700 1 -- 172.21.15.112:0/4194423343 learned_addr learned my addr 172.21.15.112:0/4194423343 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:51.711 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.698+0000 7f3abce84700 1 --2- 172.21.15.112:0/4194423343 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ab805e420 0x7f3ab816d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:51.712 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.698+0000 7f3ab7fff700 1 --2- 172.21.15.112:0/4194423343 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3ab8068d20 0x7f3ab80c5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:51.712 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.699+0000 7f3abd685700 1 -- 172.21.15.112:0/4194423343 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ab805e420 msgr2=0x7f3ab816d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:51.712 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.699+0000 7f3abd685700 1 --2- 172.21.15.112:0/4194423343 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ab805e420 0x7f3ab816d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:51.712 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.699+0000 7f3abd685700 1 -- 172.21.15.112:0/4194423343 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3ab8068d20 msgr2=0x7f3ab80c5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:51.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.699+0000 7f3abd685700 1 --2- 172.21.15.112:0/4194423343 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3ab8068d20 0x7f3ab80c5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:51.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.699+0000 7f3abd685700 1 -- 172.21.15.112:0/4194423343 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab00092a0 con 0x7f3ab80694c0 2023-01-24T03:30:51.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.699+0000 7f3abd685700 1 --2- 172.21.15.112:0/4194423343 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3ab80694c0 0x7f3ab81521a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3ab0013000 tx=0x7f3ab0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:51.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.699+0000 7f3ab4ff9700 1 -- 172.21.15.112:0/4194423343 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab0002f70 con 0x7f3ab80694c0 2023-01-24T03:30:51.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.699+0000 7f3ab4ff9700 1 -- 172.21.15.112:0/4194423343 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ab0011900 con 0x7f3ab80694c0 2023-01-24T03:30:51.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.699+0000 7f3ab4ff9700 1 -- 172.21.15.112:0/4194423343 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab0025420 con 0x7f3ab80694c0 2023-01-24T03:30:51.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.699+0000 7f3abf0e8700 1 -- 172.21.15.112:0/4194423343 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ab81503e0 con 0x7f3ab80694c0 2023-01-24T03:30:51.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.699+0000 7f3abf0e8700 1 -- 172.21.15.112:0/4194423343 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ab8150980 con 0x7f3ab80694c0 2023-01-24T03:30:51.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.700+0000 7f3abf0e8700 1 -- 172.21.15.112:0/4194423343 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ab806f3a0 con 0x7f3ab80694c0 2023-01-24T03:30:51.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.700+0000 7f3ab4ff9700 1 -- 172.21.15.112:0/4194423343 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f3ab002c030 con 0x7f3ab80694c0 2023-01-24T03:30:51.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.700+0000 7f3ab4ff9700 1 --2- 172.21.15.112:0/4194423343 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3a9c0611c0 0x7f3a9c063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:51.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.701+0000 7f3ab4ff9700 1 -- 172.21.15.112:0/4194423343 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f3ab007ece0 con 0x7f3ab80694c0 2023-01-24T03:30:51.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.701+0000 7f3abce84700 1 --2- 172.21.15.112:0/4194423343 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3a9c0611c0 0x7f3a9c063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:51.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.701+0000 7f3abce84700 1 --2- 172.21.15.112:0/4194423343 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3a9c0611c0 0x7f3a9c063670 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3aa8005750 tx=0x7f3aa8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:51.716 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.705+0000 7f3ab4ff9700 1 -- 172.21.15.112:0/4194423343 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f3ab004eac0 con 0x7f3ab80694c0 2023-01-24T03:30:51.886 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.883+0000 7f3abf0e8700 1 -- 172.21.15.112:0/4194423343 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3ab804b340 con 0x7f3ab80694c0 2023-01-24T03:30:51.886 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.883+0000 7f3ab4ff9700 1 -- 172.21.15.112:0/4194423343 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3ab00520e0 con 0x7f3ab80694c0 2023-01-24T03:30:51.886 INFO:teuthology.orchestra.run.smithi112.stdout:pg_num: 32 2023-01-24T03:30:51.888 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.885+0000 7f3abf0e8700 1 -- 172.21.15.112:0/4194423343 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3a9c0611c0 msgr2=0x7f3a9c063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:51.889 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.885+0000 7f3abf0e8700 1 --2- 172.21.15.112:0/4194423343 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3a9c0611c0 0x7f3a9c063670 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3aa8005750 tx=0x7f3aa8009040 comp rx=0 tx=0).stop 2023-01-24T03:30:51.889 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.885+0000 7f3abf0e8700 1 -- 172.21.15.112:0/4194423343 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3ab80694c0 msgr2=0x7f3ab81521a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:51.889 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.885+0000 7f3abf0e8700 1 --2- 172.21.15.112:0/4194423343 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3ab80694c0 0x7f3ab81521a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3ab0013000 tx=0x7f3ab0017000 comp rx=0 tx=0).stop 2023-01-24T03:30:51.889 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.885+0000 7f3abf0e8700 1 -- 172.21.15.112:0/4194423343 shutdown_connections 2023-01-24T03:30:51.889 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.885+0000 7f3abf0e8700 1 --2- 172.21.15.112:0/4194423343 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3a9c0611c0 0x7f3a9c063670 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:51.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.885+0000 7f3abf0e8700 1 --2- 172.21.15.112:0/4194423343 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3ab80694c0 0x7f3ab81521a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:51.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.885+0000 7f3abf0e8700 1 --2- 172.21.15.112:0/4194423343 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3ab805e420 0x7f3ab816d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:51.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.885+0000 7f3abf0e8700 1 --2- 172.21.15.112:0/4194423343 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3ab8068d20 0x7f3ab80c5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:51.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.885+0000 7f3abf0e8700 1 -- 172.21.15.112:0/4194423343 >> 172.21.15.112:0/4194423343 conn(0x7f3ab805d830 msgr2=0x7f3ab81540c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:51.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.885+0000 7f3abf0e8700 1 -- 172.21.15.112:0/4194423343 shutdown_connections 2023-01-24T03:30:51.891 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:51.885+0000 7f3abf0e8700 1 -- 172.21.15.112:0/4194423343 wait complete. 2023-01-24T03:30:51.900 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-01-24T03:30:52.015 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.012+0000 7fbf8e257700 1 Processor -- start 2023-01-24T03:30:52.015 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.012+0000 7fbf8e257700 1 -- start start 2023-01-24T03:30:52.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.013+0000 7fbf8e257700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fbf88069c70 0x7fbf881552b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:52.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.013+0000 7fbf8e257700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf880677e0 con 0x7fbf88068380 2023-01-24T03:30:52.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.013+0000 7fbf8e257700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf88155e40 con 0x7fbf880c2750 2023-01-24T03:30:52.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.013+0000 7fbf8e257700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf8806f4c0 con 0x7fbf88069c70 2023-01-24T03:30:52.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.013+0000 7fbf8d255700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fbf88069c70 0x7fbf881552b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:52.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.013+0000 7fbf8da56700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fbf88068380 0x7fbf8814ebe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:60388/0 (socket says 172.21.15.112:60388) 2023-01-24T03:30:52.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.013+0000 7fbf8da56700 1 -- 172.21.15.112:0/308878738 learned_addr learned my addr 172.21.15.112:0/308878738 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:52.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.013+0000 7fbf8d255700 1 -- 172.21.15.112:0/308878738 >> v1:172.21.15.142:6789/0 conn(0x7fbf880c2750 legacy=0x7fbf88067c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:52.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.013+0000 7fbf8d255700 1 -- 172.21.15.112:0/308878738 >> v1:172.21.15.112:6789/0 conn(0x7fbf88068380 legacy=0x7fbf8814ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:52.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.013+0000 7fbf77fff700 1 -- 172.21.15.112:0/308878738 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1507147181 0 0) 0x7fbf880677e0 con 0x7fbf88068380 2023-01-24T03:30:52.020 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.013+0000 7fbf8d255700 1 -- 172.21.15.112:0/308878738 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf88156120 con 0x7fbf88069c70 2023-01-24T03:30:52.020 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.013+0000 7fbf8d255700 1 --2- 172.21.15.112:0/308878738 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf88069c70 0x7fbf881552b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbf84009760 tx=0x7fbf84009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8dc51380d42e32e server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:52.020 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.014+0000 7fbf77fff700 1 -- 172.21.15.112:0/308878738 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf84011630 con 0x7fbf88069c70 2023-01-24T03:30:52.020 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.014+0000 7fbf77fff700 1 -- 172.21.15.112:0/308878738 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf84011790 con 0x7fbf88069c70 2023-01-24T03:30:52.020 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.014+0000 7fbf77fff700 1 -- 172.21.15.112:0/308878738 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf84011b10 con 0x7fbf88069c70 2023-01-24T03:30:52.021 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.014+0000 7fbf8e257700 1 -- 172.21.15.112:0/308878738 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf88069c70 msgr2=0x7fbf881552b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:52.021 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.014+0000 7fbf8e257700 1 --2- 172.21.15.112:0/308878738 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf88069c70 0x7fbf881552b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbf84009760 tx=0x7fbf84009060 comp rx=0 tx=0).stop 2023-01-24T03:30:52.021 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.014+0000 7fbf8e257700 1 -- 172.21.15.112:0/308878738 shutdown_connections 2023-01-24T03:30:52.021 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.014+0000 7fbf8e257700 1 --2- 172.21.15.112:0/308878738 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf88069c70 0x7fbf881552b0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:52.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.014+0000 7fbf8e257700 1 -- 172.21.15.112:0/308878738 >> 172.21.15.112:0/308878738 conn(0x7fbf880c4920 msgr2=0x7fbf880c4d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:52.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.014+0000 7fbf8e257700 1 -- 172.21.15.112:0/308878738 shutdown_connections 2023-01-24T03:30:52.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.014+0000 7fbf8e257700 1 -- 172.21.15.112:0/308878738 wait complete. 2023-01-24T03:30:52.025 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.015+0000 7fbf8e257700 1 Processor -- start 2023-01-24T03:30:52.025 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.015+0000 7fbf8e257700 1 -- start start 2023-01-24T03:30:52.025 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.016+0000 7fbf8e257700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf88068380 0x7fbf8806a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:52.026 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.016+0000 7fbf8e257700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf88069c70 0x7fbf880a98b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:52.026 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.016+0000 7fbf8e257700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf880c2750 0x7fbf880a9df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:52.026 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.016+0000 7fbf8e257700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf880ab450 con 0x7fbf88068380 2023-01-24T03:30:52.026 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.016+0000 7fbf8e257700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf88165410 con 0x7fbf880c2750 2023-01-24T03:30:52.026 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.016+0000 7fbf8e257700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf88168b80 con 0x7fbf88069c70 2023-01-24T03:30:52.027 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.016+0000 7fbf8d255700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf88068380 0x7fbf8806a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:52.027 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.016+0000 7fbf8d255700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf88068380 0x7fbf8806a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51170/0 (socket says 172.21.15.112:51170) 2023-01-24T03:30:52.027 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.016+0000 7fbf8d255700 1 -- 172.21.15.112:0/1792020582 learned_addr learned my addr 172.21.15.112:0/1792020582 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:52.027 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.016+0000 7fbf8da56700 1 --2- 172.21.15.112:0/1792020582 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf880c2750 0x7fbf880a9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:52.027 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.016+0000 7fbf8ca54700 1 --2- 172.21.15.112:0/1792020582 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf88069c70 0x7fbf880a98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:52.028 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.016+0000 7fbf8d255700 1 -- 172.21.15.112:0/1792020582 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf88069c70 msgr2=0x7fbf880a98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:52.028 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.016+0000 7fbf8d255700 1 --2- 172.21.15.112:0/1792020582 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf88069c70 0x7fbf880a98b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:52.028 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.016+0000 7fbf8d255700 1 -- 172.21.15.112:0/1792020582 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf880c2750 msgr2=0x7fbf880a9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:52.028 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.016+0000 7fbf8d255700 1 --2- 172.21.15.112:0/1792020582 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf880c2750 0x7fbf880a9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:52.028 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.016+0000 7fbf8d255700 1 -- 172.21.15.112:0/1792020582 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf840092a0 con 0x7fbf88068380 2023-01-24T03:30:52.029 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.016+0000 7fbf8da56700 1 --2- 172.21.15.112:0/1792020582 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf880c2750 0x7fbf880a9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-01-24T03:30:52.029 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.017+0000 7fbf8d255700 1 --2- 172.21.15.112:0/1792020582 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf88068380 0x7fbf8806a200 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fbf8805e090 tx=0x7fbf84017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:52.029 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.017+0000 7fbf757fa700 1 -- 172.21.15.112:0/1792020582 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf84000c90 con 0x7fbf88068380 2023-01-24T03:30:52.029 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.017+0000 7fbf757fa700 1 -- 172.21.15.112:0/1792020582 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf84011900 con 0x7fbf88068380 2023-01-24T03:30:52.029 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.017+0000 7fbf8e257700 1 -- 172.21.15.112:0/1792020582 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf88168e60 con 0x7fbf88068380 2023-01-24T03:30:52.030 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.017+0000 7fbf757fa700 1 -- 172.21.15.112:0/1792020582 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf84025670 con 0x7fbf88068380 2023-01-24T03:30:52.030 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.017+0000 7fbf8e257700 1 -- 172.21.15.112:0/1792020582 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf88169420 con 0x7fbf88068380 2023-01-24T03:30:52.030 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.018+0000 7fbf8e257700 1 -- 172.21.15.112:0/1792020582 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf8805ea20 con 0x7fbf88068380 2023-01-24T03:30:52.030 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.018+0000 7fbf757fa700 1 -- 172.21.15.112:0/1792020582 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fbf8402c030 con 0x7fbf88068380 2023-01-24T03:30:52.031 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.018+0000 7fbf757fa700 1 --2- 172.21.15.112:0/1792020582 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fbf680611c0 0x7fbf68063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:52.031 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.019+0000 7fbf757fa700 1 -- 172.21.15.112:0/1792020582 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fbf8407eeb0 con 0x7fbf88068380 2023-01-24T03:30:52.031 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.019+0000 7fbf8ca54700 1 --2- 172.21.15.112:0/1792020582 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fbf680611c0 0x7fbf68063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:52.031 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.019+0000 7fbf8ca54700 1 --2- 172.21.15.112:0/1792020582 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fbf680611c0 0x7fbf68063670 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fbf88067f00 tx=0x7fbf78009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:52.032 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.023+0000 7fbf757fa700 1 -- 172.21.15.112:0/1792020582 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fbf8404ecf0 con 0x7fbf88068380 2023-01-24T03:30:52.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.200+0000 7fbf8e257700 1 -- 172.21.15.112:0/1792020582 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbf8805d960 con 0x7fbf88068380 2023-01-24T03:30:52.204 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.201+0000 7fbf757fa700 1 -- 172.21.15.112:0/1792020582 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbf8402a020 con 0x7fbf88068380 2023-01-24T03:30:52.204 INFO:teuthology.orchestra.run.smithi112.stdout:pg_num: 64 2023-01-24T03:30:52.206 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.202+0000 7fbf8e257700 1 -- 172.21.15.112:0/1792020582 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fbf680611c0 msgr2=0x7fbf68063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:52.206 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.202+0000 7fbf8e257700 1 --2- 172.21.15.112:0/1792020582 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fbf680611c0 0x7fbf68063670 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fbf88067f00 tx=0x7fbf78009040 comp rx=0 tx=0).stop 2023-01-24T03:30:52.206 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.202+0000 7fbf8e257700 1 -- 172.21.15.112:0/1792020582 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf88068380 msgr2=0x7fbf8806a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:52.206 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.202+0000 7fbf8e257700 1 --2- 172.21.15.112:0/1792020582 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf88068380 0x7fbf8806a200 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fbf8805e090 tx=0x7fbf84017000 comp rx=0 tx=0).stop 2023-01-24T03:30:52.206 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.203+0000 7fbf8e257700 1 -- 172.21.15.112:0/1792020582 shutdown_connections 2023-01-24T03:30:52.207 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.203+0000 7fbf8e257700 1 --2- 172.21.15.112:0/1792020582 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fbf680611c0 0x7fbf68063670 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:52.207 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.203+0000 7fbf8e257700 1 --2- 172.21.15.112:0/1792020582 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbf88068380 0x7fbf8806a200 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:52.207 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.203+0000 7fbf8e257700 1 --2- 172.21.15.112:0/1792020582 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbf880c2750 0x7fbf880a9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:52.207 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.203+0000 7fbf8e257700 1 --2- 172.21.15.112:0/1792020582 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbf88069c70 0x7fbf880a98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:52.207 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.203+0000 7fbf8e257700 1 -- 172.21.15.112:0/1792020582 >> 172.21.15.112:0/1792020582 conn(0x7fbf880c4920 msgr2=0x7fbf880c1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:52.207 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.203+0000 7fbf8e257700 1 -- 172.21.15.112:0/1792020582 shutdown_connections 2023-01-24T03:30:52.208 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.203+0000 7fbf8e257700 1 -- 172.21.15.112:0/1792020582 wait complete. 2023-01-24T03:30:52.217 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-01-24T03:30:52.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.329+0000 7f530b7ad700 1 Processor -- start 2023-01-24T03:30:52.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.329+0000 7f530b7ad700 1 -- start start 2023-01-24T03:30:52.335 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.330+0000 7f530b7ad700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f530406b060 0x7f53040688f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:52.336 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.330+0000 7f530b7ad700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f530406d360 0x7f530406d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:52.336 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.330+0000 7f530b7ad700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53040c3c10 con 0x7f53040c3840 2023-01-24T03:30:52.336 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.330+0000 7f530b7ad700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f530414f820 con 0x7f530406b060 2023-01-24T03:30:52.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.330+0000 7f530b7ad700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5304069750 con 0x7f530406d360 2023-01-24T03:30:52.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.330+0000 7f5308d48700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f530406b060 0x7f53040688f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:52.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.330+0000 7f5309549700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f53040c3840 0x7f53040c97e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:60398/0 (socket says 172.21.15.112:60398) 2023-01-24T03:30:52.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.330+0000 7f5309549700 1 -- 172.21.15.112:0/3785654410 learned_addr learned my addr 172.21.15.112:0/3785654410 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:52.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.331+0000 7f5308d48700 1 -- 172.21.15.112:0/3785654410 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f530406d360 msgr2=0x7f530406d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:52.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.331+0000 7f5309d4a700 1 --2- 172.21.15.112:0/3785654410 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f530406d360 0x7f530406d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:52.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.331+0000 7f5308d48700 1 --2- 172.21.15.112:0/3785654410 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f530406d360 0x7f530406d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:52.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.331+0000 7f5308d48700 1 -- 172.21.15.112:0/3785654410 >> v1:172.21.15.112:6789/0 conn(0x7f53040c3840 legacy=0x7f53040c97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:52.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.331+0000 7f5303fff700 1 -- 172.21.15.112:0/3785654410 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1153209381 0 0) 0x7f53040c3c10 con 0x7f53040c3840 2023-01-24T03:30:52.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.331+0000 7f5308d48700 1 -- 172.21.15.112:0/3785654410 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f530406ced0 con 0x7f530406b060 2023-01-24T03:30:52.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.331+0000 7f5309d4a700 1 --2- 172.21.15.112:0/3785654410 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f530406d360 0x7f530406d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-01-24T03:30:52.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.331+0000 7f5308d48700 1 --2- 172.21.15.112:0/3785654410 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f530406b060 0x7f53040688f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f52f8009760 tx=0x7f52f8009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d96be7eb10e6db7 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:52.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.331+0000 7f5303fff700 1 -- 172.21.15.112:0/3785654410 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f8011630 con 0x7f530406b060 2023-01-24T03:30:52.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.331+0000 7f5303fff700 1 -- 172.21.15.112:0/3785654410 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52f8011790 con 0x7f530406b060 2023-01-24T03:30:52.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.331+0000 7f5303fff700 1 -- 172.21.15.112:0/3785654410 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f8011b30 con 0x7f530406b060 2023-01-24T03:30:52.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.331+0000 7f530b7ad700 1 -- 172.21.15.112:0/3785654410 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f530406b060 msgr2=0x7f53040688f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:52.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.331+0000 7f530b7ad700 1 --2- 172.21.15.112:0/3785654410 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f530406b060 0x7f53040688f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f52f8009760 tx=0x7f52f8009060 comp rx=0 tx=0).stop 2023-01-24T03:30:52.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.331+0000 7f530b7ad700 1 -- 172.21.15.112:0/3785654410 shutdown_connections 2023-01-24T03:30:52.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.331+0000 7f530b7ad700 1 --2- 172.21.15.112:0/3785654410 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f530406d360 0x7f530406d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:52.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.331+0000 7f530b7ad700 1 --2- 172.21.15.112:0/3785654410 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f530406b060 0x7f53040688f0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:52.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.331+0000 7f530b7ad700 1 -- 172.21.15.112:0/3785654410 >> 172.21.15.112:0/3785654410 conn(0x7f530405d830 msgr2=0x7f530405dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:52.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.332+0000 7f530b7ad700 1 -- 172.21.15.112:0/3785654410 shutdown_connections 2023-01-24T03:30:52.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.332+0000 7f530b7ad700 1 -- 172.21.15.112:0/3785654410 wait complete. 2023-01-24T03:30:52.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.333+0000 7f530b7ad700 1 Processor -- start 2023-01-24T03:30:52.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.333+0000 7f530b7ad700 1 -- start start 2023-01-24T03:30:52.345 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.333+0000 7f530b7ad700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f530406b060 0x7f530414ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:52.345 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.333+0000 7f530b7ad700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f530406d360 0x7f53040c8c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:52.346 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.333+0000 7f530b7ad700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53040c3840 0x7f53040bd260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:52.346 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.333+0000 7f530b7ad700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5304165af0 con 0x7f530406b060 2023-01-24T03:30:52.346 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.333+0000 7f530b7ad700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5304165f80 con 0x7f53040c3840 2023-01-24T03:30:52.346 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.333+0000 7f530b7ad700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5304152600 con 0x7f530406d360 2023-01-24T03:30:52.346 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.333+0000 7f5309549700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f530406b060 0x7f530414ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:52.347 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.333+0000 7f5309549700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f530406b060 0x7f530414ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51184/0 (socket says 172.21.15.112:51184) 2023-01-24T03:30:52.347 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.333+0000 7f5309549700 1 -- 172.21.15.112:0/1831843470 learned_addr learned my addr 172.21.15.112:0/1831843470 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:52.347 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.334+0000 7f5309549700 1 -- 172.21.15.112:0/1831843470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f530406d360 msgr2=0x7f53040c8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:52.347 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.334+0000 7f5308d48700 1 --2- 172.21.15.112:0/1831843470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f530406d360 0x7f53040c8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:52.347 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.334+0000 7f5309d4a700 1 --2- 172.21.15.112:0/1831843470 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53040c3840 0x7f53040bd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:52.348 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.334+0000 7f5309549700 1 --2- 172.21.15.112:0/1831843470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f530406d360 0x7f53040c8c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:52.348 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.334+0000 7f5309549700 1 -- 172.21.15.112:0/1831843470 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53040c3840 msgr2=0x7f53040bd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:52.348 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.334+0000 7f5309549700 1 --2- 172.21.15.112:0/1831843470 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53040c3840 0x7f53040bd260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:52.348 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.334+0000 7f5309549700 1 -- 172.21.15.112:0/1831843470 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f80092a0 con 0x7f530406b060 2023-01-24T03:30:52.349 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.334+0000 7f5309549700 1 --2- 172.21.15.112:0/1831843470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f530406b060 0x7f530414ad80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f52f400c810 tx=0x7f52f4001570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:52.349 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.334+0000 7f53017fa700 1 -- 172.21.15.112:0/1831843470 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f4005d30 con 0x7f530406b060 2023-01-24T03:30:52.349 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.334+0000 7f53017fa700 1 -- 172.21.15.112:0/1831843470 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52f40070f0 con 0x7f530406b060 2023-01-24T03:30:52.349 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.334+0000 7f530b7ad700 1 -- 172.21.15.112:0/1831843470 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53041528a0 con 0x7f530406b060 2023-01-24T03:30:52.350 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.334+0000 7f53017fa700 1 -- 172.21.15.112:0/1831843470 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f40074b0 con 0x7f530406b060 2023-01-24T03:30:52.350 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.334+0000 7f530b7ad700 1 -- 172.21.15.112:0/1831843470 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53041651c0 con 0x7f530406b060 2023-01-24T03:30:52.350 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.335+0000 7f530b7ad700 1 -- 172.21.15.112:0/1831843470 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5304165dd0 con 0x7f530406b060 2023-01-24T03:30:52.350 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.335+0000 7f53017fa700 1 -- 172.21.15.112:0/1831843470 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f52f4010070 con 0x7f530406b060 2023-01-24T03:30:52.350 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.336+0000 7f53017fa700 1 --2- 172.21.15.112:0/1831843470 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f52e40611f0 0x7f52e40636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:52.351 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.336+0000 7f53017fa700 1 -- 172.21.15.112:0/1831843470 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f52f407d2c0 con 0x7f530406b060 2023-01-24T03:30:52.351 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.336+0000 7f5308d48700 1 --2- 172.21.15.112:0/1831843470 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f52e40611f0 0x7f52e40636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:52.351 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.336+0000 7f5308d48700 1 --2- 172.21.15.112:0/1831843470 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f52e40611f0 0x7f52e40636a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f530405df50 tx=0x7f52f8017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:52.351 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.340+0000 7f53017fa700 1 -- 172.21.15.112:0/1831843470 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f52f404d100 con 0x7f530406b060 2023-01-24T03:30:52.519 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.515+0000 7f530b7ad700 1 -- 172.21.15.112:0/1831843470 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f53041551a0 con 0x7f530406b060 2023-01-24T03:30:52.519 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.516+0000 7f53017fa700 1 -- 172.21.15.112:0/1831843470 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7f52f4050720 con 0x7f530406b060 2023-01-24T03:30:52.519 INFO:teuthology.orchestra.run.smithi112.stdout:pg_num: 32 2023-01-24T03:30:52.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.518+0000 7f530b7ad700 1 -- 172.21.15.112:0/1831843470 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f52e40611f0 msgr2=0x7f52e40636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:52.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.518+0000 7f530b7ad700 1 --2- 172.21.15.112:0/1831843470 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f52e40611f0 0x7f52e40636a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f530405df50 tx=0x7f52f8017000 comp rx=0 tx=0).stop 2023-01-24T03:30:52.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.518+0000 7f530b7ad700 1 -- 172.21.15.112:0/1831843470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f530406b060 msgr2=0x7f530414ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:52.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.518+0000 7f530b7ad700 1 --2- 172.21.15.112:0/1831843470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f530406b060 0x7f530414ad80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f52f400c810 tx=0x7f52f4001570 comp rx=0 tx=0).stop 2023-01-24T03:30:52.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.518+0000 7f530b7ad700 1 -- 172.21.15.112:0/1831843470 shutdown_connections 2023-01-24T03:30:52.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.518+0000 7f530b7ad700 1 --2- 172.21.15.112:0/1831843470 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f52e40611f0 0x7f52e40636a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:52.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.518+0000 7f530b7ad700 1 --2- 172.21.15.112:0/1831843470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f530406b060 0x7f530414ad80 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:52.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.518+0000 7f530b7ad700 1 --2- 172.21.15.112:0/1831843470 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53040c3840 0x7f53040bd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:52.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.518+0000 7f530b7ad700 1 --2- 172.21.15.112:0/1831843470 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f530406d360 0x7f53040c8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:52.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.518+0000 7f530b7ad700 1 -- 172.21.15.112:0/1831843470 >> 172.21.15.112:0/1831843470 conn(0x7f530405d830 msgr2=0x7f53040c6490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:52.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.518+0000 7f530b7ad700 1 -- 172.21.15.112:0/1831843470 shutdown_connections 2023-01-24T03:30:52.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.518+0000 7f530b7ad700 1 -- 172.21.15.112:0/1831843470 wait complete. 2023-01-24T03:30:52.532 INFO:tasks.mds_thrash:Wait for all MDSs to reach steady state... 2023-01-24T03:30:52.533 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:30:52.646 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.643+0000 7f5fafdd1700 1 Processor -- start 2023-01-24T03:30:52.647 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.643+0000 7f5fafdd1700 1 -- start start 2023-01-24T03:30:52.649 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.644+0000 7f5fafdd1700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f5fa806a550 0x7f5fa806a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:52.649 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.644+0000 7f5fafdd1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fa80681f0 0x7f5fa80c3c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:52.649 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.644+0000 7f5fafdd1700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fa806d930 con 0x7f5fa8067e20 2023-01-24T03:30:52.649 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.644+0000 7f5fafdd1700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fa8153150 con 0x7f5fa806a550 2023-01-24T03:30:52.650 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.644+0000 7f5fafdd1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa81528d0 con 0x7f5fa80681f0 2023-01-24T03:30:52.650 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.644+0000 7f5fad36c700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f5fa806a550 0x7f5fa806a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:52.650 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.644+0000 7f5fad36c700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f5fa806a550 0x7f5fa806a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51194/0 (socket says 172.21.15.112:51194) 2023-01-24T03:30:52.650 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.644+0000 7f5fad36c700 1 -- 172.21.15.112:0/3540870094 learned_addr learned my addr 172.21.15.112:0/3540870094 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:52.651 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.645+0000 7f5fae36e700 1 --2- 172.21.15.112:0/3540870094 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fa80681f0 0x7f5fa80c3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:52.651 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.645+0000 7f5fad36c700 1 -- 172.21.15.112:0/3540870094 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fa80681f0 msgr2=0x7f5fa80c3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:52.651 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.645+0000 7f5fad36c700 1 --2- 172.21.15.112:0/3540870094 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fa80681f0 0x7f5fa80c3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:52.651 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.645+0000 7f5fad36c700 1 -- 172.21.15.112:0/3540870094 >> v1:172.21.15.142:6789/0 conn(0x7f5fa8067e20 legacy=0x7f5fa8152120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:52.651 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.645+0000 7f5fad36c700 1 -- 172.21.15.112:0/3540870094 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa8152b90 con 0x7f5fa806a550 2023-01-24T03:30:52.652 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.645+0000 7f5fad36c700 1 --2- 172.21.15.112:0/3540870094 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fa806a550 0x7f5fa806a9f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c006fd0 tx=0x7f5f9c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=345ec8b7a41ca48f server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:52.652 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.645+0000 7f5facb6b700 1 -- 172.21.15.112:0/3540870094 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f9c011630 con 0x7f5fa806a550 2023-01-24T03:30:52.652 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.645+0000 7f5facb6b700 1 -- 172.21.15.112:0/3540870094 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f9c011790 con 0x7f5fa806a550 2023-01-24T03:30:52.652 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.645+0000 7f5facb6b700 1 -- 172.21.15.112:0/3540870094 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f9c011b50 con 0x7f5fa806a550 2023-01-24T03:30:52.652 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.645+0000 7f5fafdd1700 1 -- 172.21.15.112:0/3540870094 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fa806a550 msgr2=0x7f5fa806a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:52.653 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.645+0000 7f5fafdd1700 1 --2- 172.21.15.112:0/3540870094 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fa806a550 0x7f5fa806a9f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c006fd0 tx=0x7f5f9c0090f0 comp rx=0 tx=0).stop 2023-01-24T03:30:52.653 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.645+0000 7f5fafdd1700 1 -- 172.21.15.112:0/3540870094 shutdown_connections 2023-01-24T03:30:52.653 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.646+0000 7f5fafdd1700 1 --2- 172.21.15.112:0/3540870094 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fa80681f0 0x7f5fa80c3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:52.653 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.646+0000 7f5fafdd1700 1 --2- 172.21.15.112:0/3540870094 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fa806a550 0x7f5fa806a9f0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:52.653 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.646+0000 7f5fafdd1700 1 -- 172.21.15.112:0/3540870094 >> 172.21.15.112:0/3540870094 conn(0x7f5fa80c8940 msgr2=0x7f5fa80c4910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:52.654 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.646+0000 7f5fafdd1700 1 -- 172.21.15.112:0/3540870094 shutdown_connections 2023-01-24T03:30:52.654 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.646+0000 7f5fafdd1700 1 -- 172.21.15.112:0/3540870094 wait complete. 2023-01-24T03:30:52.657 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.647+0000 7f5fafdd1700 1 Processor -- start 2023-01-24T03:30:52.657 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.647+0000 7f5fafdd1700 1 -- start start 2023-01-24T03:30:52.657 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.647+0000 7f5fafdd1700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fa8067e20 0x7f5fa80bf4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:52.658 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.647+0000 7f5fafdd1700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5fa80681f0 0x7f5fa814a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:52.658 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.647+0000 7f5fafdd1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fa806a550 0x7f5fa80c08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:52.658 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.647+0000 7f5fafdd1700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa81512d0 con 0x7f5fa8067e20 2023-01-24T03:30:52.658 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.647+0000 7f5fafdd1700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa81515b0 con 0x7f5fa80681f0 2023-01-24T03:30:52.658 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.647+0000 7f5fafdd1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa8151890 con 0x7f5fa806a550 2023-01-24T03:30:52.659 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.647+0000 7f5fadb6d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fa8067e20 0x7f5fa80bf4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:52.659 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.647+0000 7f5fadb6d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fa8067e20 0x7f5fa80bf4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51198/0 (socket says 172.21.15.112:51198) 2023-01-24T03:30:52.659 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.647+0000 7f5fadb6d700 1 -- 172.21.15.112:0/2588160828 learned_addr learned my addr 172.21.15.112:0/2588160828 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:52.659 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.647+0000 7f5fae36e700 1 --2- 172.21.15.112:0/2588160828 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fa806a550 0x7f5fa80c08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:52.659 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.648+0000 7f5fad36c700 1 --2- 172.21.15.112:0/2588160828 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5fa80681f0 0x7f5fa814a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:52.660 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.648+0000 7f5fadb6d700 1 -- 172.21.15.112:0/2588160828 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fa806a550 msgr2=0x7f5fa80c08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:52.660 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.648+0000 7f5fadb6d700 1 --2- 172.21.15.112:0/2588160828 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fa806a550 0x7f5fa80c08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:52.660 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.648+0000 7f5fadb6d700 1 -- 172.21.15.112:0/2588160828 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5fa80681f0 msgr2=0x7f5fa814a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:52.660 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.648+0000 7f5fadb6d700 1 --2- 172.21.15.112:0/2588160828 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5fa80681f0 0x7f5fa814a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:52.660 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.648+0000 7f5fadb6d700 1 -- 172.21.15.112:0/2588160828 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f9c009370 con 0x7f5fa8067e20 2023-01-24T03:30:52.661 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.648+0000 7f5fae36e700 1 --2- 172.21.15.112:0/2588160828 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fa806a550 0x7f5fa80c08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:30:52.661 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.648+0000 7f5fad36c700 1 --2- 172.21.15.112:0/2588160828 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5fa80681f0 0x7f5fa814a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:30:52.661 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.648+0000 7f5fadb6d700 1 --2- 172.21.15.112:0/2588160828 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fa8067e20 0x7f5fa80bf4a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5f9800c840 tx=0x7f5f98001660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:52.661 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.648+0000 7f5fa5ffb700 1 -- 172.21.15.112:0/2588160828 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f98012630 con 0x7f5fa8067e20 2023-01-24T03:30:52.661 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.648+0000 7f5fa5ffb700 1 -- 172.21.15.112:0/2588160828 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f980127b0 con 0x7f5fa8067e20 2023-01-24T03:30:52.662 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.648+0000 7f5fafdd1700 1 -- 172.21.15.112:0/2588160828 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fa8151b70 con 0x7f5fa8067e20 2023-01-24T03:30:52.662 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.648+0000 7f5fa5ffb700 1 -- 172.21.15.112:0/2588160828 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f98012b70 con 0x7f5fa8067e20 2023-01-24T03:30:52.662 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.648+0000 7f5fafdd1700 1 -- 172.21.15.112:0/2588160828 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fa80a6e60 con 0x7f5fa8067e20 2023-01-24T03:30:52.662 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.649+0000 7f5fafdd1700 1 -- 172.21.15.112:0/2588160828 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fa804bb80 con 0x7f5fa8067e20 2023-01-24T03:30:52.662 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.649+0000 7f5fa5ffb700 1 -- 172.21.15.112:0/2588160828 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f5f98018020 con 0x7f5fa8067e20 2023-01-24T03:30:52.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.650+0000 7f5fa5ffb700 1 --2- 172.21.15.112:0/2588160828 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5f880611f0 0x7f5f880636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:52.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.650+0000 7f5fad36c700 1 --2- 172.21.15.112:0/2588160828 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5f880611f0 0x7f5f880636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:52.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.650+0000 7f5fa5ffb700 1 -- 172.21.15.112:0/2588160828 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f5f9807d840 con 0x7f5fa8067e20 2023-01-24T03:30:52.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.650+0000 7f5fad36c700 1 --2- 172.21.15.112:0/2588160828 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5f880611f0 0x7f5f880636a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5fa80c2420 tx=0x7f5f9c016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:52.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.655+0000 7f5fa5ffb700 1 -- 172.21.15.112:0/2588160828 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f5f9804d680 con 0x7f5fa8067e20 2023-01-24T03:30:52.883 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.880+0000 7f5fafdd1700 1 -- 172.21.15.112:0/2588160828 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5fa8152e70 con 0x7f5fa8067e20 2023-01-24T03:30:52.884 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.880+0000 7f5fa5ffb700 1 -- 172.21.15.112:0/2588160828 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6114 (secure 0 0 0) 0x7f5f98050ca0 con 0x7f5fa8067e20 2023-01-24T03:30:52.884 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:30:52.885 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/869267413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":869267413},{"type":"v1","addr":"172.21.15.142:6834","nonce":869267413}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:30:37.600288+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.142:6835/2762621058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2762621058},{"type":"v1","addr":"172.21.15.142:6835","nonce":2762621058}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:30:52.886 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.884+0000 7f5fafdd1700 1 -- 172.21.15.112:0/2588160828 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5f880611f0 msgr2=0x7f5f880636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:52.886 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.884+0000 7f5fafdd1700 1 --2- 172.21.15.112:0/2588160828 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5f880611f0 0x7f5f880636a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5fa80c2420 tx=0x7f5f9c016000 comp rx=0 tx=0).stop 2023-01-24T03:30:52.887 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.884+0000 7f5fafdd1700 1 -- 172.21.15.112:0/2588160828 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fa8067e20 msgr2=0x7f5fa80bf4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:52.887 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.884+0000 7f5fafdd1700 1 --2- 172.21.15.112:0/2588160828 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fa8067e20 0x7f5fa80bf4a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5f9800c840 tx=0x7f5f98001660 comp rx=0 tx=0).stop 2023-01-24T03:30:52.887 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.884+0000 7f5fafdd1700 1 -- 172.21.15.112:0/2588160828 shutdown_connections 2023-01-24T03:30:52.887 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.884+0000 7f5fafdd1700 1 --2- 172.21.15.112:0/2588160828 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5f880611f0 0x7f5f880636a0 secure :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5fa80c2420 tx=0x7f5f9c016000 comp rx=0 tx=0).stop 2023-01-24T03:30:52.887 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.884+0000 7f5fafdd1700 1 --2- 172.21.15.112:0/2588160828 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fa8067e20 0x7f5fa80bf4a0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:52.888 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.884+0000 7f5fafdd1700 1 --2- 172.21.15.112:0/2588160828 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fa806a550 0x7f5fa80c08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:52.888 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.884+0000 7f5fafdd1700 1 --2- 172.21.15.112:0/2588160828 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5fa80681f0 0x7f5fa814a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:52.888 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.884+0000 7f5fafdd1700 1 -- 172.21.15.112:0/2588160828 >> 172.21.15.112:0/2588160828 conn(0x7f5fa80c8940 msgr2=0x7f5fa80c6890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:52.888 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.884+0000 7f5fafdd1700 1 -- 172.21.15.112:0/2588160828 shutdown_connections 2023-01-24T03:30:52.888 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:52.884+0000 7f5fafdd1700 1 -- 172.21.15.112:0/2588160828 wait complete. 2023-01-24T03:30:52.889 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 10 2023-01-24T03:30:52.897 INFO:tasks.mds_thrash:Ready to start thrashing 2023-01-24T03:30:52.898 INFO:tasks.mds_thrash.ceph_manager:waiting for clean 2023-01-24T03:30:52.898 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-24T03:30:53.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.012+0000 7fd16e833700 1 Processor -- start 2023-01-24T03:30:53.015 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.012+0000 7fd16e833700 1 -- start start 2023-01-24T03:30:53.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.012+0000 7fd16e833700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fd16814c660 0x7fd1680c1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:53.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.012+0000 7fd16e833700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1680c22c0 0x7fd168155710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:53.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.012+0000 7fd16e833700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1680c18f0 con 0x7fd16814c290 2023-01-24T03:30:53.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.012+0000 7fd16e833700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1680c1030 con 0x7fd16814c660 2023-01-24T03:30:53.020 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.012+0000 7fd16e833700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1680c2720 con 0x7fd1680c22c0 2023-01-24T03:30:53.020 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.012+0000 7fd1677fe700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fd16814c660 0x7fd1680c1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:53.020 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.012+0000 7fd1677fe700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fd16814c660 0x7fd1680c1d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51210/0 (socket says 172.21.15.112:51210) 2023-01-24T03:30:53.020 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.012+0000 7fd1677fe700 1 -- 172.21.15.112:0/1015450006 learned_addr learned my addr 172.21.15.112:0/1015450006 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:53.020 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.012+0000 7fd16cdd0700 1 --2- 172.21.15.112:0/1015450006 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1680c22c0 0x7fd168155710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:53.021 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.012+0000 7fd1677fe700 1 -- 172.21.15.112:0/1015450006 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1680c22c0 msgr2=0x7fd168155710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:53.021 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.012+0000 7fd1677fe700 1 --2- 172.21.15.112:0/1015450006 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1680c22c0 0x7fd168155710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:53.021 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.012+0000 7fd1677fe700 1 -- 172.21.15.112:0/1015450006 >> v1:172.21.15.142:6789/0 conn(0x7fd16814c290 legacy=0x7fd16806ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:53.021 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.012+0000 7fd1677fe700 1 -- 172.21.15.112:0/1015450006 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd16806bad0 con 0x7fd16814c660 2023-01-24T03:30:53.021 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.012+0000 7fd16cdd0700 1 --2- 172.21.15.112:0/1015450006 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1680c22c0 0x7fd168155710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-01-24T03:30:53.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.012+0000 7fd1677fe700 1 --2- 172.21.15.112:0/1015450006 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd16814c660 0x7fd1680c1d80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd15c006fd0 tx=0x7fd15c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e8c384795568465 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:53.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.012+0000 7fd166ffd700 1 -- 172.21.15.112:0/1015450006 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15c011630 con 0x7fd16814c660 2023-01-24T03:30:53.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.012+0000 7fd166ffd700 1 -- 172.21.15.112:0/1015450006 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd15c0117b0 con 0x7fd16814c660 2023-01-24T03:30:53.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.012+0000 7fd166ffd700 1 -- 172.21.15.112:0/1015450006 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15c011b70 con 0x7fd16814c660 2023-01-24T03:30:53.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.013+0000 7fd16e833700 1 -- 172.21.15.112:0/1015450006 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd16814c660 msgr2=0x7fd1680c1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:53.023 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.013+0000 7fd16e833700 1 --2- 172.21.15.112:0/1015450006 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd16814c660 0x7fd1680c1d80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd15c006fd0 tx=0x7fd15c0090f0 comp rx=0 tx=0).stop 2023-01-24T03:30:53.023 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.013+0000 7fd16e833700 1 -- 172.21.15.112:0/1015450006 shutdown_connections 2023-01-24T03:30:53.023 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.013+0000 7fd16e833700 1 --2- 172.21.15.112:0/1015450006 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1680c22c0 0x7fd168155710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:53.023 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.013+0000 7fd16e833700 1 --2- 172.21.15.112:0/1015450006 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd16814c660 0x7fd1680c1d80 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:53.024 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.013+0000 7fd16e833700 1 -- 172.21.15.112:0/1015450006 >> 172.21.15.112:0/1015450006 conn(0x7fd168067f20 msgr2=0x7fd168068320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:53.024 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.013+0000 7fd16e833700 1 -- 172.21.15.112:0/1015450006 shutdown_connections 2023-01-24T03:30:53.024 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.013+0000 7fd16e833700 1 -- 172.21.15.112:0/1015450006 wait complete. 2023-01-24T03:30:53.024 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.014+0000 7fd16e833700 1 Processor -- start 2023-01-24T03:30:53.024 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.014+0000 7fd16e833700 1 -- start start 2023-01-24T03:30:53.025 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.016+0000 7fd16e833700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1680c22c0 0x7fd16814ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:53.025 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.016+0000 7fd16e833700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd16814c290 0x7fd1680684b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:53.025 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.016+0000 7fd16e833700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd16814c660 0x7fd168165930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:53.025 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.016+0000 7fd16e833700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd168152c00 con 0x7fd16814c660 2023-01-24T03:30:53.025 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.016+0000 7fd16e833700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd168151da0 con 0x7fd16814c290 2023-01-24T03:30:53.026 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.016+0000 7fd16e833700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd168151ff0 con 0x7fd1680c22c0 2023-01-24T03:30:53.026 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.016+0000 7fd16cdd0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd16814c660 0x7fd168165930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:53.026 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.016+0000 7fd16cdd0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd16814c660 0x7fd168165930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51218/0 (socket says 172.21.15.112:51218) 2023-01-24T03:30:53.026 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.016+0000 7fd16cdd0700 1 -- 172.21.15.112:0/3055754329 learned_addr learned my addr 172.21.15.112:0/3055754329 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:53.026 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.016+0000 7fd167fff700 1 --2- 172.21.15.112:0/3055754329 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1680c22c0 0x7fd16814ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:53.027 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.016+0000 7fd1677fe700 1 --2- 172.21.15.112:0/3055754329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd16814c290 0x7fd1680684b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:53.027 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.016+0000 7fd16cdd0700 1 -- 172.21.15.112:0/3055754329 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1680c22c0 msgr2=0x7fd16814ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:53.027 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.016+0000 7fd16cdd0700 1 --2- 172.21.15.112:0/3055754329 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1680c22c0 0x7fd16814ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:53.027 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.016+0000 7fd16cdd0700 1 -- 172.21.15.112:0/3055754329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd16814c290 msgr2=0x7fd1680684b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:53.027 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.016+0000 7fd16cdd0700 1 --2- 172.21.15.112:0/3055754329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd16814c290 0x7fd1680684b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:53.028 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.016+0000 7fd16cdd0700 1 -- 172.21.15.112:0/3055754329 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd15c009370 con 0x7fd16814c660 2023-01-24T03:30:53.028 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.016+0000 7fd16cdd0700 1 --2- 172.21.15.112:0/3055754329 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd16814c660 0x7fd168165930 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd16000cef0 tx=0x7fd160007860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:53.028 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.016+0000 7fd14ffff700 1 -- 172.21.15.112:0/3055754329 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16000ad20 con 0x7fd16814c660 2023-01-24T03:30:53.028 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.016+0000 7fd14ffff700 1 -- 172.21.15.112:0/3055754329 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd160016420 con 0x7fd16814c660 2023-01-24T03:30:53.028 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.016+0000 7fd16e833700 1 -- 172.21.15.112:0/3055754329 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1681522d0 con 0x7fd16814c660 2023-01-24T03:30:53.029 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.016+0000 7fd14ffff700 1 -- 172.21.15.112:0/3055754329 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16000aea0 con 0x7fd16814c660 2023-01-24T03:30:53.029 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.016+0000 7fd16e833700 1 -- 172.21.15.112:0/3055754329 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd168152890 con 0x7fd16814c660 2023-01-24T03:30:53.030 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.016+0000 7fd16e833700 1 -- 172.21.15.112:0/3055754329 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd16804bb80 con 0x7fd16814c660 2023-01-24T03:30:53.030 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.017+0000 7fd14ffff700 1 -- 172.21.15.112:0/3055754329 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fd160016a70 con 0x7fd16814c660 2023-01-24T03:30:53.030 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.017+0000 7fd14ffff700 1 --2- 172.21.15.112:0/3055754329 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd1480611c0 0x7fd148063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:53.031 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.017+0000 7fd14ffff700 1 -- 172.21.15.112:0/3055754329 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fd160012070 con 0x7fd16814c660 2023-01-24T03:30:53.031 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.017+0000 7fd167fff700 1 --2- 172.21.15.112:0/3055754329 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd1480611c0 0x7fd148063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:53.031 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.017+0000 7fd167fff700 1 --2- 172.21.15.112:0/3055754329 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd1480611c0 0x7fd148063670 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd15800b9b0 tx=0x7fd158008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:53.031 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.022+0000 7fd14ffff700 1 -- 172.21.15.112:0/3055754329 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fd16004e000 con 0x7fd16814c660 2023-01-24T03:30:53.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.197+0000 7fd16e833700 1 -- 172.21.15.112:0/3055754329 --> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd1680c20a0 con 0x7fd1480611c0 2023-01-24T03:30:53.208 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.206+0000 7fd14ffff700 1 -- 172.21.15.112:0/3055754329 <== mgr.4104 v2:172.21.15.112:6832/84276 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+365110 (secure 0 0 0) 0x7fd1680c20a0 con 0x7fd1480611c0 2023-01-24T03:30:53.212 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:30:53.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.210+0000 7fd16e833700 1 -- 172.21.15.112:0/3055754329 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd1480611c0 msgr2=0x7fd148063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:53.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.210+0000 7fd16e833700 1 --2- 172.21.15.112:0/3055754329 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd1480611c0 0x7fd148063670 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd15800b9b0 tx=0x7fd158008040 comp rx=0 tx=0).stop 2023-01-24T03:30:53.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.210+0000 7fd16e833700 1 -- 172.21.15.112:0/3055754329 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd16814c660 msgr2=0x7fd168165930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:53.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.210+0000 7fd16e833700 1 --2- 172.21.15.112:0/3055754329 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd16814c660 0x7fd168165930 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd16000cef0 tx=0x7fd160007860 comp rx=0 tx=0).stop 2023-01-24T03:30:53.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.210+0000 7fd16e833700 1 -- 172.21.15.112:0/3055754329 shutdown_connections 2023-01-24T03:30:53.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.210+0000 7fd16e833700 1 --2- 172.21.15.112:0/3055754329 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd1480611c0 0x7fd148063670 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:53.217 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.210+0000 7fd16e833700 1 --2- 172.21.15.112:0/3055754329 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd16814c660 0x7fd168165930 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:53.217 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.210+0000 7fd16e833700 1 --2- 172.21.15.112:0/3055754329 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1680c22c0 0x7fd16814ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:53.217 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.210+0000 7fd16e833700 1 --2- 172.21.15.112:0/3055754329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd16814c290 0x7fd1680684b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:53.217 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.210+0000 7fd16e833700 1 -- 172.21.15.112:0/3055754329 >> 172.21.15.112:0/3055754329 conn(0x7fd168067f20 msgr2=0x7fd16814b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:53.217 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.210+0000 7fd16e833700 1 -- 172.21.15.112:0/3055754329 shutdown_connections 2023-01-24T03:30:53.217 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.210+0000 7fd16e833700 1 -- 172.21.15.112:0/3055754329 wait complete. 2023-01-24T03:30:53.218 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2023-01-24T03:30:53.223 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2023-01-24T03:30:53.009421+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595114,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":209,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":69880,"kb_used_data":2168,"kb_used_omap":0,"kb_used_meta":67648,"kb_avail":754904840,"statfs":{"total":773094113280,"available":773022556160,"internally_reserved":0,"allocated":2220032,"data_stored":2023478,"data_compressed":64574,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":69271552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":263,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002903"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993289+0000","last_change":"2023-01-24T03:30:20.986308+0000","last_active":"2023-01-24T03:30:24.993289+0000","last_peered":"2023-01-24T03:30:24.993289+0000","last_clean":"2023-01-24T03:30:24.993289+0000","last_became_active":"2023-01-24T03:30:20.986025+0000","last_became_peered":"2023-01-24T03:30:20.986025+0000","last_unstale":"2023-01-24T03:30:24.993289+0000","last_undegraded":"2023-01-24T03:30:24.993289+0000","last_fullsized":"2023-01-24T03:30:24.993289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:47:58.616698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.006202+0000","last_change":"2023-01-24T03:30:20.994478+0000","last_active":"2023-01-24T03:30:25.006202+0000","last_peered":"2023-01-24T03:30:25.006202+0000","last_clean":"2023-01-24T03:30:25.006202+0000","last_became_active":"2023-01-24T03:30:20.994207+0000","last_became_peered":"2023-01-24T03:30:20.994207+0000","last_unstale":"2023-01-24T03:30:25.006202+0000","last_undegraded":"2023-01-24T03:30:25.006202+0000","last_fullsized":"2023-01-24T03:30:25.006202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:02:53.230231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993177+0000","last_change":"2023-01-24T03:30:20.989651+0000","last_active":"2023-01-24T03:30:24.993177+0000","last_peered":"2023-01-24T03:30:24.993177+0000","last_clean":"2023-01-24T03:30:24.993177+0000","last_became_active":"2023-01-24T03:30:20.987585+0000","last_became_peered":"2023-01-24T03:30:20.987585+0000","last_unstale":"2023-01-24T03:30:24.993177+0000","last_undegraded":"2023-01-24T03:30:24.993177+0000","last_fullsized":"2023-01-24T03:30:24.993177+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:36:02.177544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990511+0000","last_change":"2023-01-24T03:30:20.993601+0000","last_active":"2023-01-24T03:30:24.990511+0000","last_peered":"2023-01-24T03:30:24.990511+0000","last_clean":"2023-01-24T03:30:24.990511+0000","last_became_active":"2023-01-24T03:30:20.993418+0000","last_became_peered":"2023-01-24T03:30:20.993418+0000","last_unstale":"2023-01-24T03:30:24.990511+0000","last_undegraded":"2023-01-24T03:30:24.990511+0000","last_fullsized":"2023-01-24T03:30:24.990511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:47:50.260497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996430+0000","last_change":"2023-01-24T03:30:20.991605+0000","last_active":"2023-01-24T03:30:24.996430+0000","last_peered":"2023-01-24T03:30:24.996430+0000","last_clean":"2023-01-24T03:30:24.996430+0000","last_became_active":"2023-01-24T03:30:20.991361+0000","last_became_peered":"2023-01-24T03:30:20.991361+0000","last_unstale":"2023-01-24T03:30:24.996430+0000","last_undegraded":"2023-01-24T03:30:24.996430+0000","last_fullsized":"2023-01-24T03:30:24.996430+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:31:25.418430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993288+0000","last_change":"2023-01-24T03:30:20.985058+0000","last_active":"2023-01-24T03:30:24.993288+0000","last_peered":"2023-01-24T03:30:24.993288+0000","last_clean":"2023-01-24T03:30:24.993288+0000","last_became_active":"2023-01-24T03:30:20.984754+0000","last_became_peered":"2023-01-24T03:30:20.984754+0000","last_unstale":"2023-01-24T03:30:24.993288+0000","last_undegraded":"2023-01-24T03:30:24.993288+0000","last_fullsized":"2023-01-24T03:30:24.993288+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:12:03.225109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996279+0000","last_change":"2023-01-24T03:30:21.000890+0000","last_active":"2023-01-24T03:30:24.996279+0000","last_peered":"2023-01-24T03:30:24.996279+0000","last_clean":"2023-01-24T03:30:24.996279+0000","last_became_active":"2023-01-24T03:30:21.000438+0000","last_became_peered":"2023-01-24T03:30:21.000438+0000","last_unstale":"2023-01-24T03:30:24.996279+0000","last_undegraded":"2023-01-24T03:30:24.996279+0000","last_fullsized":"2023-01-24T03:30:24.996279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:17:12.999664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.004717+0000","last_change":"2023-01-24T03:30:20.986753+0000","last_active":"2023-01-24T03:30:25.004717+0000","last_peered":"2023-01-24T03:30:25.004717+0000","last_clean":"2023-01-24T03:30:25.004717+0000","last_became_active":"2023-01-24T03:30:20.986524+0000","last_became_peered":"2023-01-24T03:30:20.986524+0000","last_unstale":"2023-01-24T03:30:25.004717+0000","last_undegraded":"2023-01-24T03:30:25.004717+0000","last_fullsized":"2023-01-24T03:30:25.004717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:24:55.699379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992350+0000","last_change":"2023-01-24T03:30:20.991136+0000","last_active":"2023-01-24T03:30:24.992350+0000","last_peered":"2023-01-24T03:30:24.992350+0000","last_clean":"2023-01-24T03:30:24.992350+0000","last_became_active":"2023-01-24T03:30:20.990925+0000","last_became_peered":"2023-01-24T03:30:20.990925+0000","last_unstale":"2023-01-24T03:30:24.992350+0000","last_undegraded":"2023-01-24T03:30:24.992350+0000","last_fullsized":"2023-01-24T03:30:24.992350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:45:37.485199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.006401+0000","last_change":"2023-01-24T03:30:20.995730+0000","last_active":"2023-01-24T03:30:25.006401+0000","last_peered":"2023-01-24T03:30:25.006401+0000","last_clean":"2023-01-24T03:30:25.006401+0000","last_became_active":"2023-01-24T03:30:20.994880+0000","last_became_peered":"2023-01-24T03:30:20.994880+0000","last_unstale":"2023-01-24T03:30:25.006401+0000","last_undegraded":"2023-01-24T03:30:25.006401+0000","last_fullsized":"2023-01-24T03:30:25.006401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:07:21.546386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993144+0000","last_change":"2023-01-24T03:30:20.993328+0000","last_active":"2023-01-24T03:30:24.993144+0000","last_peered":"2023-01-24T03:30:24.993144+0000","last_clean":"2023-01-24T03:30:24.993144+0000","last_became_active":"2023-01-24T03:30:20.992730+0000","last_became_peered":"2023-01-24T03:30:20.992730+0000","last_unstale":"2023-01-24T03:30:24.993144+0000","last_undegraded":"2023-01-24T03:30:24.993144+0000","last_fullsized":"2023-01-24T03:30:24.993144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:43:39.048550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.994605+0000","last_change":"2023-01-24T03:30:20.984993+0000","last_active":"2023-01-24T03:30:24.994605+0000","last_peered":"2023-01-24T03:30:24.994605+0000","last_clean":"2023-01-24T03:30:24.994605+0000","last_became_active":"2023-01-24T03:30:20.984540+0000","last_became_peered":"2023-01-24T03:30:20.984540+0000","last_unstale":"2023-01-24T03:30:24.994605+0000","last_undegraded":"2023-01-24T03:30:24.994605+0000","last_fullsized":"2023-01-24T03:30:24.994605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:16:39.993359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.995936+0000","last_change":"2023-01-24T03:30:20.986567+0000","last_active":"2023-01-24T03:30:24.995936+0000","last_peered":"2023-01-24T03:30:24.995936+0000","last_clean":"2023-01-24T03:30:24.995936+0000","last_became_active":"2023-01-24T03:30:20.986224+0000","last_became_peered":"2023-01-24T03:30:20.986224+0000","last_unstale":"2023-01-24T03:30:24.995936+0000","last_undegraded":"2023-01-24T03:30:24.995936+0000","last_fullsized":"2023-01-24T03:30:24.995936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:09:14.692354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005087+0000","last_change":"2023-01-24T03:30:20.988843+0000","last_active":"2023-01-24T03:30:25.005087+0000","last_peered":"2023-01-24T03:30:25.005087+0000","last_clean":"2023-01-24T03:30:25.005087+0000","last_became_active":"2023-01-24T03:30:20.988486+0000","last_became_peered":"2023-01-24T03:30:20.988486+0000","last_unstale":"2023-01-24T03:30:25.005087+0000","last_undegraded":"2023-01-24T03:30:25.005087+0000","last_fullsized":"2023-01-24T03:30:25.005087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:18:20.538844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996110+0000","last_change":"2023-01-24T03:30:21.000738+0000","last_active":"2023-01-24T03:30:24.996110+0000","last_peered":"2023-01-24T03:30:24.996110+0000","last_clean":"2023-01-24T03:30:24.996110+0000","last_became_active":"2023-01-24T03:30:21.000458+0000","last_became_peered":"2023-01-24T03:30:21.000458+0000","last_unstale":"2023-01-24T03:30:24.996110+0000","last_undegraded":"2023-01-24T03:30:24.996110+0000","last_fullsized":"2023-01-24T03:30:24.996110+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:21:42.876426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992383+0000","last_change":"2023-01-24T03:30:20.985062+0000","last_active":"2023-01-24T03:30:24.992383+0000","last_peered":"2023-01-24T03:30:24.992383+0000","last_clean":"2023-01-24T03:30:24.992383+0000","last_became_active":"2023-01-24T03:30:20.984759+0000","last_became_peered":"2023-01-24T03:30:20.984759+0000","last_unstale":"2023-01-24T03:30:24.992383+0000","last_undegraded":"2023-01-24T03:30:24.992383+0000","last_fullsized":"2023-01-24T03:30:24.992383+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:38:34.615748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992439+0000","last_change":"2023-01-24T03:30:20.989080+0000","last_active":"2023-01-24T03:30:24.992439+0000","last_peered":"2023-01-24T03:30:24.992439+0000","last_clean":"2023-01-24T03:30:24.992439+0000","last_became_active":"2023-01-24T03:30:20.988823+0000","last_became_peered":"2023-01-24T03:30:20.988823+0000","last_unstale":"2023-01-24T03:30:24.992439+0000","last_undegraded":"2023-01-24T03:30:24.992439+0000","last_fullsized":"2023-01-24T03:30:24.992439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:22:21.807370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993008+0000","last_change":"2023-01-24T03:30:20.984222+0000","last_active":"2023-01-24T03:30:24.993008+0000","last_peered":"2023-01-24T03:30:24.993008+0000","last_clean":"2023-01-24T03:30:24.993008+0000","last_became_active":"2023-01-24T03:30:20.984002+0000","last_became_peered":"2023-01-24T03:30:20.984002+0000","last_unstale":"2023-01-24T03:30:24.993008+0000","last_undegraded":"2023-01-24T03:30:24.993008+0000","last_fullsized":"2023-01-24T03:30:24.993008+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:54:24.005486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.997598+0000","last_change":"2023-01-24T03:30:20.988197+0000","last_active":"2023-01-24T03:30:24.997598+0000","last_peered":"2023-01-24T03:30:24.997598+0000","last_clean":"2023-01-24T03:30:24.997598+0000","last_became_active":"2023-01-24T03:30:20.987941+0000","last_became_peered":"2023-01-24T03:30:20.987941+0000","last_unstale":"2023-01-24T03:30:24.997598+0000","last_undegraded":"2023-01-24T03:30:24.997598+0000","last_fullsized":"2023-01-24T03:30:24.997598+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T15:08:29.659964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992957+0000","last_change":"2023-01-24T03:30:20.988701+0000","last_active":"2023-01-24T03:30:24.992957+0000","last_peered":"2023-01-24T03:30:24.992957+0000","last_clean":"2023-01-24T03:30:24.992957+0000","last_became_active":"2023-01-24T03:30:20.988476+0000","last_became_peered":"2023-01-24T03:30:20.988476+0000","last_unstale":"2023-01-24T03:30:24.992957+0000","last_undegraded":"2023-01-24T03:30:24.992957+0000","last_fullsized":"2023-01-24T03:30:24.992957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:33:29.463194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996546+0000","last_change":"2023-01-24T03:30:21.001125+0000","last_active":"2023-01-24T03:30:24.996546+0000","last_peered":"2023-01-24T03:30:24.996546+0000","last_clean":"2023-01-24T03:30:24.996546+0000","last_became_active":"2023-01-24T03:30:21.000873+0000","last_became_peered":"2023-01-24T03:30:21.000873+0000","last_unstale":"2023-01-24T03:30:24.996546+0000","last_undegraded":"2023-01-24T03:30:24.996546+0000","last_fullsized":"2023-01-24T03:30:24.996546+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:22:16.136875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993432+0000","last_change":"2023-01-24T03:30:20.992639+0000","last_active":"2023-01-24T03:30:24.993432+0000","last_peered":"2023-01-24T03:30:24.993432+0000","last_clean":"2023-01-24T03:30:24.993432+0000","last_became_active":"2023-01-24T03:30:20.992386+0000","last_became_peered":"2023-01-24T03:30:20.992386+0000","last_unstale":"2023-01-24T03:30:24.993432+0000","last_undegraded":"2023-01-24T03:30:24.993432+0000","last_fullsized":"2023-01-24T03:30:24.993432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:29:17.912198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992608+0000","last_change":"2023-01-24T03:30:20.994066+0000","last_active":"2023-01-24T03:30:24.992608+0000","last_peered":"2023-01-24T03:30:24.992608+0000","last_clean":"2023-01-24T03:30:24.992608+0000","last_became_active":"2023-01-24T03:30:20.993820+0000","last_became_peered":"2023-01-24T03:30:20.993820+0000","last_unstale":"2023-01-24T03:30:24.992608+0000","last_undegraded":"2023-01-24T03:30:24.992608+0000","last_fullsized":"2023-01-24T03:30:24.992608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:31:23.874058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996188+0000","last_change":"2023-01-24T03:30:21.001652+0000","last_active":"2023-01-24T03:30:24.996188+0000","last_peered":"2023-01-24T03:30:24.996188+0000","last_clean":"2023-01-24T03:30:24.996188+0000","last_became_active":"2023-01-24T03:30:21.001400+0000","last_became_peered":"2023-01-24T03:30:21.001400+0000","last_unstale":"2023-01-24T03:30:24.996188+0000","last_undegraded":"2023-01-24T03:30:24.996188+0000","last_fullsized":"2023-01-24T03:30:24.996188+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:55:53.249816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002243+0000","last_change":"2023-01-24T03:30:20.998304+0000","last_active":"2023-01-24T03:30:25.002243+0000","last_peered":"2023-01-24T03:30:25.002243+0000","last_clean":"2023-01-24T03:30:25.002243+0000","last_became_active":"2023-01-24T03:30:20.996288+0000","last_became_peered":"2023-01-24T03:30:20.996288+0000","last_unstale":"2023-01-24T03:30:25.002243+0000","last_undegraded":"2023-01-24T03:30:25.002243+0000","last_fullsized":"2023-01-24T03:30:25.002243+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:11:08.120197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002638+0000","last_change":"2023-01-24T03:30:20.996489+0000","last_active":"2023-01-24T03:30:25.002638+0000","last_peered":"2023-01-24T03:30:25.002638+0000","last_clean":"2023-01-24T03:30:25.002638+0000","last_became_active":"2023-01-24T03:30:20.994371+0000","last_became_peered":"2023-01-24T03:30:20.994371+0000","last_unstale":"2023-01-24T03:30:25.002638+0000","last_undegraded":"2023-01-24T03:30:25.002638+0000","last_fullsized":"2023-01-24T03:30:25.002638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:52:21.369928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005744+0000","last_change":"2023-01-24T03:30:20.997753+0000","last_active":"2023-01-24T03:30:25.005744+0000","last_peered":"2023-01-24T03:30:25.005744+0000","last_clean":"2023-01-24T03:30:25.005744+0000","last_became_active":"2023-01-24T03:30:20.997475+0000","last_became_peered":"2023-01-24T03:30:20.997475+0000","last_unstale":"2023-01-24T03:30:25.005744+0000","last_undegraded":"2023-01-24T03:30:25.005744+0000","last_fullsized":"2023-01-24T03:30:25.005744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:43:16.199843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.013267+0000","last_change":"2023-01-24T03:30:25.004944+0000","last_active":"2023-01-24T03:30:25.013267+0000","last_peered":"2023-01-24T03:30:25.013267+0000","last_clean":"2023-01-24T03:30:25.013267+0000","last_became_active":"2023-01-24T03:30:25.004590+0000","last_became_peered":"2023-01-24T03:30:25.004590+0000","last_unstale":"2023-01-24T03:30:25.013267+0000","last_undegraded":"2023-01-24T03:30:25.013267+0000","last_fullsized":"2023-01-24T03:30:25.013267+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:04:11.054225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2,0],"acting":[4,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"24'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.380450+0000","last_change":"2023-01-24T03:30:20.607211+0000","last_active":"2023-01-24T03:30:25.380450+0000","last_peered":"2023-01-24T03:30:25.380450+0000","last_clean":"2023-01-24T03:30:25.380450+0000","last_became_active":"2023-01-24T03:30:20.606855+0000","last_became_peered":"2023-01-24T03:30:20.606855+0000","last_unstale":"2023-01-24T03:30:25.380450+0000","last_undegraded":"2023-01-24T03:30:25.380450+0000","last_fullsized":"2023-01-24T03:30:25.380450+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:42:29.300089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991276+0000","last_change":"2023-01-24T03:30:20.993128+0000","last_active":"2023-01-24T03:30:24.991276+0000","last_peered":"2023-01-24T03:30:24.991276+0000","last_clean":"2023-01-24T03:30:24.991276+0000","last_became_active":"2023-01-24T03:30:20.992967+0000","last_became_peered":"2023-01-24T03:30:20.992967+0000","last_unstale":"2023-01-24T03:30:24.991276+0000","last_undegraded":"2023-01-24T03:30:24.991276+0000","last_fullsized":"2023-01-24T03:30:24.991276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:09:59.170218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.008814+0000","last_change":"2023-01-24T03:30:25.006569+0000","last_active":"2023-01-24T03:30:25.008814+0000","last_peered":"2023-01-24T03:30:25.008814+0000","last_clean":"2023-01-24T03:30:25.008814+0000","last_became_active":"2023-01-24T03:30:25.006199+0000","last_became_peered":"2023-01-24T03:30:25.006199+0000","last_unstale":"2023-01-24T03:30:25.008814+0000","last_undegraded":"2023-01-24T03:30:25.008814+0000","last_fullsized":"2023-01-24T03:30:25.008814+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:31:46.839485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,7],"acting":[6,4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.001971+0000","last_change":"2023-01-24T03:30:20.612738+0000","last_active":"2023-01-24T03:30:25.001971+0000","last_peered":"2023-01-24T03:30:25.001971+0000","last_clean":"2023-01-24T03:30:25.001971+0000","last_became_active":"2023-01-24T03:30:20.612375+0000","last_became_peered":"2023-01-24T03:30:20.612375+0000","last_unstale":"2023-01-24T03:30:25.001971+0000","last_undegraded":"2023-01-24T03:30:25.001971+0000","last_fullsized":"2023-01-24T03:30:25.001971+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:17:43.251945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993741+0000","last_change":"2023-01-24T03:30:20.996368+0000","last_active":"2023-01-24T03:30:24.993741+0000","last_peered":"2023-01-24T03:30:24.993741+0000","last_clean":"2023-01-24T03:30:24.993741+0000","last_became_active":"2023-01-24T03:30:20.996051+0000","last_became_peered":"2023-01-24T03:30:20.996051+0000","last_unstale":"2023-01-24T03:30:24.993741+0000","last_undegraded":"2023-01-24T03:30:24.993741+0000","last_fullsized":"2023-01-24T03:30:24.993741+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:53:28.193862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.014921+0000","last_change":"2023-01-24T03:30:25.008528+0000","last_active":"2023-01-24T03:30:25.014921+0000","last_peered":"2023-01-24T03:30:25.014921+0000","last_clean":"2023-01-24T03:30:25.014921+0000","last_became_active":"2023-01-24T03:30:25.008146+0000","last_became_peered":"2023-01-24T03:30:25.008146+0000","last_unstale":"2023-01-24T03:30:25.014921+0000","last_undegraded":"2023-01-24T03:30:25.014921+0000","last_fullsized":"2023-01-24T03:30:25.014921+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:17:11.457359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1,2],"acting":[0,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991341+0000","last_change":"2023-01-24T03:30:20.608932+0000","last_active":"2023-01-24T03:30:24.991341+0000","last_peered":"2023-01-24T03:30:24.991341+0000","last_clean":"2023-01-24T03:30:24.991341+0000","last_became_active":"2023-01-24T03:30:20.608135+0000","last_became_peered":"2023-01-24T03:30:20.608135+0000","last_unstale":"2023-01-24T03:30:24.991341+0000","last_undegraded":"2023-01-24T03:30:24.991341+0000","last_fullsized":"2023-01-24T03:30:24.991341+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:38:30.012279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990753+0000","last_change":"2023-01-24T03:30:20.986490+0000","last_active":"2023-01-24T03:30:24.990753+0000","last_peered":"2023-01-24T03:30:24.990753+0000","last_clean":"2023-01-24T03:30:24.990753+0000","last_became_active":"2023-01-24T03:30:20.986245+0000","last_became_peered":"2023-01-24T03:30:20.986245+0000","last_unstale":"2023-01-24T03:30:24.990753+0000","last_undegraded":"2023-01-24T03:30:24.990753+0000","last_fullsized":"2023-01-24T03:30:24.990753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:38:30.801032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.012186+0000","last_change":"2023-01-24T03:30:25.008221+0000","last_active":"2023-01-24T03:30:25.012186+0000","last_peered":"2023-01-24T03:30:25.012186+0000","last_clean":"2023-01-24T03:30:25.012186+0000","last_became_active":"2023-01-24T03:30:25.007855+0000","last_became_peered":"2023-01-24T03:30:25.007855+0000","last_unstale":"2023-01-24T03:30:25.012186+0000","last_undegraded":"2023-01-24T03:30:25.012186+0000","last_fullsized":"2023-01-24T03:30:25.012186+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:45:15.317067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,1],"acting":[6,4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992445+0000","last_change":"2023-01-24T03:30:20.607385+0000","last_active":"2023-01-24T03:30:24.992445+0000","last_peered":"2023-01-24T03:30:24.992445+0000","last_clean":"2023-01-24T03:30:24.992445+0000","last_became_active":"2023-01-24T03:30:20.607123+0000","last_became_peered":"2023-01-24T03:30:20.607123+0000","last_unstale":"2023-01-24T03:30:24.992445+0000","last_undegraded":"2023-01-24T03:30:24.992445+0000","last_fullsized":"2023-01-24T03:30:24.992445+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T15:24:00.474758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991258+0000","last_change":"2023-01-24T03:30:20.994124+0000","last_active":"2023-01-24T03:30:24.991258+0000","last_peered":"2023-01-24T03:30:24.991258+0000","last_clean":"2023-01-24T03:30:24.991258+0000","last_became_active":"2023-01-24T03:30:20.993876+0000","last_became_peered":"2023-01-24T03:30:20.993876+0000","last_unstale":"2023-01-24T03:30:24.991258+0000","last_undegraded":"2023-01-24T03:30:24.991258+0000","last_fullsized":"2023-01-24T03:30:24.991258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:07:26.252741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.011535+0000","last_change":"2023-01-24T03:30:25.006312+0000","last_active":"2023-01-24T03:30:25.011535+0000","last_peered":"2023-01-24T03:30:25.011535+0000","last_clean":"2023-01-24T03:30:25.011535+0000","last_became_active":"2023-01-24T03:30:25.005946+0000","last_became_peered":"2023-01-24T03:30:25.005946+0000","last_unstale":"2023-01-24T03:30:25.011535+0000","last_undegraded":"2023-01-24T03:30:25.011535+0000","last_fullsized":"2023-01-24T03:30:25.011535+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:35:02.760190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6,0],"acting":[7,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.006326+0000","last_change":"2023-01-24T03:30:20.610825+0000","last_active":"2023-01-24T03:30:25.006326+0000","last_peered":"2023-01-24T03:30:25.006326+0000","last_clean":"2023-01-24T03:30:25.006326+0000","last_became_active":"2023-01-24T03:30:20.610540+0000","last_became_peered":"2023-01-24T03:30:20.610540+0000","last_unstale":"2023-01-24T03:30:25.006326+0000","last_undegraded":"2023-01-24T03:30:25.006326+0000","last_fullsized":"2023-01-24T03:30:25.006326+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:23:15.279283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990818+0000","last_change":"2023-01-24T03:30:20.994605+0000","last_active":"2023-01-24T03:30:24.990818+0000","last_peered":"2023-01-24T03:30:24.990818+0000","last_clean":"2023-01-24T03:30:24.990818+0000","last_became_active":"2023-01-24T03:30:20.994356+0000","last_became_peered":"2023-01-24T03:30:20.994356+0000","last_unstale":"2023-01-24T03:30:24.990818+0000","last_undegraded":"2023-01-24T03:30:24.990818+0000","last_fullsized":"2023-01-24T03:30:24.990818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:06:08.797922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.018595+0000","last_change":"2023-01-24T03:30:25.012301+0000","last_active":"2023-01-24T03:30:25.018595+0000","last_peered":"2023-01-24T03:30:25.018595+0000","last_clean":"2023-01-24T03:30:25.018595+0000","last_became_active":"2023-01-24T03:30:25.011891+0000","last_became_peered":"2023-01-24T03:30:25.011891+0000","last_unstale":"2023-01-24T03:30:25.018595+0000","last_undegraded":"2023-01-24T03:30:25.018595+0000","last_fullsized":"2023-01-24T03:30:25.018595+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:15:46.012675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,5],"acting":[3,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"24'3","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.986137+0000","last_change":"2023-01-24T03:30:20.609828+0000","last_active":"2023-01-24T03:30:25.986137+0000","last_peered":"2023-01-24T03:30:25.986137+0000","last_clean":"2023-01-24T03:30:25.986137+0000","last_became_active":"2023-01-24T03:30:20.609492+0000","last_became_peered":"2023-01-24T03:30:20.609492+0000","last_unstale":"2023-01-24T03:30:25.986137+0000","last_undegraded":"2023-01-24T03:30:25.986137+0000","last_fullsized":"2023-01-24T03:30:25.986137+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:22:53.430711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002553+0000","last_change":"2023-01-24T03:30:20.999747+0000","last_active":"2023-01-24T03:30:25.002553+0000","last_peered":"2023-01-24T03:30:25.002553+0000","last_clean":"2023-01-24T03:30:25.002553+0000","last_became_active":"2023-01-24T03:30:20.999003+0000","last_became_peered":"2023-01-24T03:30:20.999003+0000","last_unstale":"2023-01-24T03:30:25.002553+0000","last_undegraded":"2023-01-24T03:30:25.002553+0000","last_fullsized":"2023-01-24T03:30:25.002553+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:19:22.205251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016573+0000","last_change":"2023-01-24T03:30:25.010659+0000","last_active":"2023-01-24T03:30:25.016573+0000","last_peered":"2023-01-24T03:30:25.016573+0000","last_clean":"2023-01-24T03:30:25.016573+0000","last_became_active":"2023-01-24T03:30:25.010237+0000","last_became_peered":"2023-01-24T03:30:25.010237+0000","last_unstale":"2023-01-24T03:30:25.016573+0000","last_undegraded":"2023-01-24T03:30:25.016573+0000","last_fullsized":"2023-01-24T03:30:25.016573+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:01:53.501070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3,7],"acting":[1,5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"24'11","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:48.616383+0000","last_change":"2023-01-24T03:30:20.610682+0000","last_active":"2023-01-24T03:30:48.616383+0000","last_peered":"2023-01-24T03:30:48.616383+0000","last_clean":"2023-01-24T03:30:48.616383+0000","last_became_active":"2023-01-24T03:30:20.610463+0000","last_became_peered":"2023-01-24T03:30:20.610463+0000","last_unstale":"2023-01-24T03:30:48.616383+0000","last_undegraded":"2023-01-24T03:30:48.616383+0000","last_fullsized":"2023-01-24T03:30:48.616383+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:25:00.447596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992845+0000","last_change":"2023-01-24T03:30:20.987326+0000","last_active":"2023-01-24T03:30:24.992845+0000","last_peered":"2023-01-24T03:30:24.992845+0000","last_clean":"2023-01-24T03:30:24.992845+0000","last_became_active":"2023-01-24T03:30:20.987042+0000","last_became_peered":"2023-01-24T03:30:20.987042+0000","last_unstale":"2023-01-24T03:30:24.992845+0000","last_undegraded":"2023-01-24T03:30:24.992845+0000","last_fullsized":"2023-01-24T03:30:24.992845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:57:43.257624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.020022+0000","last_change":"2023-01-24T03:30:25.012004+0000","last_active":"2023-01-24T03:30:25.020022+0000","last_peered":"2023-01-24T03:30:25.020022+0000","last_clean":"2023-01-24T03:30:25.020022+0000","last_became_active":"2023-01-24T03:30:25.007410+0000","last_became_peered":"2023-01-24T03:30:25.007410+0000","last_unstale":"2023-01-24T03:30:25.020022+0000","last_undegraded":"2023-01-24T03:30:25.020022+0000","last_fullsized":"2023-01-24T03:30:25.020022+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:34:21.681011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3,0],"acting":[5,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"24'2","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985201+0000","last_change":"2023-01-24T03:30:20.613026+0000","last_active":"2023-01-24T03:30:25.985201+0000","last_peered":"2023-01-24T03:30:25.985201+0000","last_clean":"2023-01-24T03:30:25.985201+0000","last_became_active":"2023-01-24T03:30:20.612605+0000","last_became_peered":"2023-01-24T03:30:20.612605+0000","last_unstale":"2023-01-24T03:30:25.985201+0000","last_undegraded":"2023-01-24T03:30:25.985201+0000","last_fullsized":"2023-01-24T03:30:25.985201+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:45:08.016740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991113+0000","last_change":"2023-01-24T03:30:20.991929+0000","last_active":"2023-01-24T03:30:24.991113+0000","last_peered":"2023-01-24T03:30:24.991113+0000","last_clean":"2023-01-24T03:30:24.991113+0000","last_became_active":"2023-01-24T03:30:20.991693+0000","last_became_peered":"2023-01-24T03:30:20.991693+0000","last_unstale":"2023-01-24T03:30:24.991113+0000","last_undegraded":"2023-01-24T03:30:24.991113+0000","last_fullsized":"2023-01-24T03:30:24.991113+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:55:51.637568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.018472+0000","last_change":"2023-01-24T03:30:25.010572+0000","last_active":"2023-01-24T03:30:25.018472+0000","last_peered":"2023-01-24T03:30:25.018472+0000","last_clean":"2023-01-24T03:30:25.018472+0000","last_became_active":"2023-01-24T03:30:25.010193+0000","last_became_peered":"2023-01-24T03:30:25.010193+0000","last_unstale":"2023-01-24T03:30:25.018472+0000","last_undegraded":"2023-01-24T03:30:25.018472+0000","last_fullsized":"2023-01-24T03:30:25.018472+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:25:39.676179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,1],"acting":[3,7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993579+0000","last_change":"2023-01-24T03:30:20.612047+0000","last_active":"2023-01-24T03:30:24.993579+0000","last_peered":"2023-01-24T03:30:24.993579+0000","last_clean":"2023-01-24T03:30:24.993579+0000","last_became_active":"2023-01-24T03:30:20.611798+0000","last_became_peered":"2023-01-24T03:30:20.611798+0000","last_unstale":"2023-01-24T03:30:24.993579+0000","last_undegraded":"2023-01-24T03:30:24.993579+0000","last_fullsized":"2023-01-24T03:30:24.993579+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:01:41.731689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005194+0000","last_change":"2023-01-24T03:30:20.996432+0000","last_active":"2023-01-24T03:30:25.005194+0000","last_peered":"2023-01-24T03:30:25.005194+0000","last_clean":"2023-01-24T03:30:25.005194+0000","last_became_active":"2023-01-24T03:30:20.996173+0000","last_became_peered":"2023-01-24T03:30:20.996173+0000","last_unstale":"2023-01-24T03:30:25.005194+0000","last_undegraded":"2023-01-24T03:30:25.005194+0000","last_fullsized":"2023-01-24T03:30:25.005194+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:13:31.596587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.018507+0000","last_change":"2023-01-24T03:30:25.008479+0000","last_active":"2023-01-24T03:30:25.018507+0000","last_peered":"2023-01-24T03:30:25.018507+0000","last_clean":"2023-01-24T03:30:25.018507+0000","last_became_active":"2023-01-24T03:30:25.007858+0000","last_became_peered":"2023-01-24T03:30:25.007858+0000","last_unstale":"2023-01-24T03:30:25.018507+0000","last_undegraded":"2023-01-24T03:30:25.018507+0000","last_fullsized":"2023-01-24T03:30:25.018507+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:10:38.425874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,0],"acting":[3,1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996134+0000","last_change":"2023-01-24T03:30:20.612848+0000","last_active":"2023-01-24T03:30:24.996134+0000","last_peered":"2023-01-24T03:30:24.996134+0000","last_clean":"2023-01-24T03:30:24.996134+0000","last_became_active":"2023-01-24T03:30:20.612600+0000","last_became_peered":"2023-01-24T03:30:20.612600+0000","last_unstale":"2023-01-24T03:30:24.996134+0000","last_undegraded":"2023-01-24T03:30:24.996134+0000","last_fullsized":"2023-01-24T03:30:24.996134+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:46:45.634472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993883+0000","last_change":"2023-01-24T03:30:20.991133+0000","last_active":"2023-01-24T03:30:24.993883+0000","last_peered":"2023-01-24T03:30:24.993883+0000","last_clean":"2023-01-24T03:30:24.993883+0000","last_became_active":"2023-01-24T03:30:20.990856+0000","last_became_peered":"2023-01-24T03:30:20.990856+0000","last_unstale":"2023-01-24T03:30:24.993883+0000","last_undegraded":"2023-01-24T03:30:24.993883+0000","last_fullsized":"2023-01-24T03:30:24.993883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:27:32.163224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.022452+0000","last_change":"2023-01-24T03:30:25.017038+0000","last_active":"2023-01-24T03:30:25.022452+0000","last_peered":"2023-01-24T03:30:25.022452+0000","last_clean":"2023-01-24T03:30:25.022452+0000","last_became_active":"2023-01-24T03:30:25.016684+0000","last_became_peered":"2023-01-24T03:30:25.016684+0000","last_unstale":"2023-01-24T03:30:25.022452+0000","last_undegraded":"2023-01-24T03:30:25.022452+0000","last_fullsized":"2023-01-24T03:30:25.022452+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:57:30.028834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1,6],"acting":[5,3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"24'8","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.988793+0000","last_change":"2023-01-24T03:30:20.604617+0000","last_active":"2023-01-24T03:30:25.988793+0000","last_peered":"2023-01-24T03:30:25.988793+0000","last_clean":"2023-01-24T03:30:25.988793+0000","last_became_active":"2023-01-24T03:30:20.604445+0000","last_became_peered":"2023-01-24T03:30:20.604445+0000","last_unstale":"2023-01-24T03:30:25.988793+0000","last_undegraded":"2023-01-24T03:30:25.988793+0000","last_fullsized":"2023-01-24T03:30:25.988793+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:06:51.460895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993017+0000","last_change":"2023-01-24T03:30:20.995872+0000","last_active":"2023-01-24T03:30:24.993017+0000","last_peered":"2023-01-24T03:30:24.993017+0000","last_clean":"2023-01-24T03:30:24.993017+0000","last_became_active":"2023-01-24T03:30:20.995599+0000","last_became_peered":"2023-01-24T03:30:20.995599+0000","last_unstale":"2023-01-24T03:30:24.993017+0000","last_undegraded":"2023-01-24T03:30:24.993017+0000","last_fullsized":"2023-01-24T03:30:24.993017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:53:54.356899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.020130+0000","last_change":"2023-01-24T03:30:25.016101+0000","last_active":"2023-01-24T03:30:25.020130+0000","last_peered":"2023-01-24T03:30:25.020130+0000","last_clean":"2023-01-24T03:30:25.020130+0000","last_became_active":"2023-01-24T03:30:25.015705+0000","last_became_peered":"2023-01-24T03:30:25.015705+0000","last_unstale":"2023-01-24T03:30:25.020130+0000","last_undegraded":"2023-01-24T03:30:25.020130+0000","last_fullsized":"2023-01-24T03:30:25.020130+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:54:37.798191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3,4],"acting":[7,6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"24'1","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.446876+0000","last_change":"2023-01-24T03:30:20.610632+0000","last_active":"2023-01-24T03:30:25.446876+0000","last_peered":"2023-01-24T03:30:25.446876+0000","last_clean":"2023-01-24T03:30:25.446876+0000","last_became_active":"2023-01-24T03:30:20.610367+0000","last_became_peered":"2023-01-24T03:30:20.610367+0000","last_unstale":"2023-01-24T03:30:25.446876+0000","last_undegraded":"2023-01-24T03:30:25.446876+0000","last_fullsized":"2023-01-24T03:30:25.446876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:26:05.938178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.004623+0000","last_change":"2023-01-24T03:30:20.996493+0000","last_active":"2023-01-24T03:30:25.004623+0000","last_peered":"2023-01-24T03:30:25.004623+0000","last_clean":"2023-01-24T03:30:25.004623+0000","last_became_active":"2023-01-24T03:30:20.996229+0000","last_became_peered":"2023-01-24T03:30:20.996229+0000","last_unstale":"2023-01-24T03:30:25.004623+0000","last_undegraded":"2023-01-24T03:30:25.004623+0000","last_fullsized":"2023-01-24T03:30:25.004623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:23:53.642522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016195+0000","last_change":"2023-01-24T03:30:25.009215+0000","last_active":"2023-01-24T03:30:25.016195+0000","last_peered":"2023-01-24T03:30:25.016195+0000","last_clean":"2023-01-24T03:30:25.016195+0000","last_became_active":"2023-01-24T03:30:25.008213+0000","last_became_peered":"2023-01-24T03:30:25.008213+0000","last_unstale":"2023-01-24T03:30:25.016195+0000","last_undegraded":"2023-01-24T03:30:25.016195+0000","last_fullsized":"2023-01-24T03:30:25.016195+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:48:37.517172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,5],"acting":[2,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"24'2","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985223+0000","last_change":"2023-01-24T03:30:20.612158+0000","last_active":"2023-01-24T03:30:25.985223+0000","last_peered":"2023-01-24T03:30:25.985223+0000","last_clean":"2023-01-24T03:30:25.985223+0000","last_became_active":"2023-01-24T03:30:20.611931+0000","last_became_peered":"2023-01-24T03:30:20.611931+0000","last_unstale":"2023-01-24T03:30:25.985223+0000","last_undegraded":"2023-01-24T03:30:25.985223+0000","last_fullsized":"2023-01-24T03:30:25.985223+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:30:37.532390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992500+0000","last_change":"2023-01-24T03:30:20.991582+0000","last_active":"2023-01-24T03:30:24.992500+0000","last_peered":"2023-01-24T03:30:24.992500+0000","last_clean":"2023-01-24T03:30:24.992500+0000","last_became_active":"2023-01-24T03:30:20.991335+0000","last_became_peered":"2023-01-24T03:30:20.991335+0000","last_unstale":"2023-01-24T03:30:24.992500+0000","last_undegraded":"2023-01-24T03:30:24.992500+0000","last_fullsized":"2023-01-24T03:30:24.992500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:45:20.786712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016224+0000","last_change":"2023-01-24T03:30:25.008058+0000","last_active":"2023-01-24T03:30:25.016224+0000","last_peered":"2023-01-24T03:30:25.016224+0000","last_clean":"2023-01-24T03:30:25.016224+0000","last_became_active":"2023-01-24T03:30:25.007711+0000","last_became_peered":"2023-01-24T03:30:25.007711+0000","last_unstale":"2023-01-24T03:30:25.016224+0000","last_undegraded":"2023-01-24T03:30:25.016224+0000","last_fullsized":"2023-01-24T03:30:25.016224+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:26:56.270674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5,6],"acting":[2,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"24'1","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.447163+0000","last_change":"2023-01-24T03:30:20.610495+0000","last_active":"2023-01-24T03:30:25.447163+0000","last_peered":"2023-01-24T03:30:25.447163+0000","last_clean":"2023-01-24T03:30:25.447163+0000","last_became_active":"2023-01-24T03:30:20.610035+0000","last_became_peered":"2023-01-24T03:30:20.610035+0000","last_unstale":"2023-01-24T03:30:25.447163+0000","last_undegraded":"2023-01-24T03:30:25.447163+0000","last_fullsized":"2023-01-24T03:30:25.447163+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:25:30.944275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.989631+0000","last_change":"2023-01-24T03:30:20.994362+0000","last_active":"2023-01-24T03:30:24.989631+0000","last_peered":"2023-01-24T03:30:24.989631+0000","last_clean":"2023-01-24T03:30:24.989631+0000","last_became_active":"2023-01-24T03:30:20.994101+0000","last_became_peered":"2023-01-24T03:30:20.994101+0000","last_unstale":"2023-01-24T03:30:24.989631+0000","last_undegraded":"2023-01-24T03:30:24.989631+0000","last_fullsized":"2023-01-24T03:30:24.989631+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:55:12.837646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.015999+0000","last_change":"2023-01-24T03:30:25.006995+0000","last_active":"2023-01-24T03:30:25.015999+0000","last_peered":"2023-01-24T03:30:25.015999+0000","last_clean":"2023-01-24T03:30:25.015999+0000","last_became_active":"2023-01-24T03:30:25.006473+0000","last_became_peered":"2023-01-24T03:30:25.006473+0000","last_unstale":"2023-01-24T03:30:25.015999+0000","last_undegraded":"2023-01-24T03:30:25.015999+0000","last_fullsized":"2023-01-24T03:30:25.015999+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:48:16.320162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3,6],"acting":[2,4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005342+0000","last_change":"2023-01-24T03:30:20.606526+0000","last_active":"2023-01-24T03:30:25.005342+0000","last_peered":"2023-01-24T03:30:25.005342+0000","last_clean":"2023-01-24T03:30:25.005342+0000","last_became_active":"2023-01-24T03:30:20.606301+0000","last_became_peered":"2023-01-24T03:30:20.606301+0000","last_unstale":"2023-01-24T03:30:25.005342+0000","last_undegraded":"2023-01-24T03:30:25.005342+0000","last_fullsized":"2023-01-24T03:30:25.005342+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:43:28.702475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992428+0000","last_change":"2023-01-24T03:30:20.993379+0000","last_active":"2023-01-24T03:30:24.992428+0000","last_peered":"2023-01-24T03:30:24.992428+0000","last_clean":"2023-01-24T03:30:24.992428+0000","last_became_active":"2023-01-24T03:30:20.993151+0000","last_became_peered":"2023-01-24T03:30:20.993151+0000","last_unstale":"2023-01-24T03:30:24.992428+0000","last_undegraded":"2023-01-24T03:30:24.992428+0000","last_fullsized":"2023-01-24T03:30:24.992428+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:00:45.845082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.020912+0000","last_change":"2023-01-24T03:30:25.013701+0000","last_active":"2023-01-24T03:30:25.020912+0000","last_peered":"2023-01-24T03:30:25.020912+0000","last_clean":"2023-01-24T03:30:25.020912+0000","last_became_active":"2023-01-24T03:30:25.013290+0000","last_became_peered":"2023-01-24T03:30:25.013290+0000","last_unstale":"2023-01-24T03:30:25.020912+0000","last_undegraded":"2023-01-24T03:30:25.020912+0000","last_fullsized":"2023-01-24T03:30:25.020912+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:32:17.340583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5,6],"acting":[2,7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"24'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985860+0000","last_change":"2023-01-24T03:30:20.609656+0000","last_active":"2023-01-24T03:30:25.985860+0000","last_peered":"2023-01-24T03:30:25.985860+0000","last_clean":"2023-01-24T03:30:25.985860+0000","last_became_active":"2023-01-24T03:30:20.609374+0000","last_became_peered":"2023-01-24T03:30:20.609374+0000","last_unstale":"2023-01-24T03:30:25.985860+0000","last_undegraded":"2023-01-24T03:30:25.985860+0000","last_fullsized":"2023-01-24T03:30:25.985860+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:58:12.653746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992726+0000","last_change":"2023-01-24T03:30:20.986089+0000","last_active":"2023-01-24T03:30:24.992726+0000","last_peered":"2023-01-24T03:30:24.992726+0000","last_clean":"2023-01-24T03:30:24.992726+0000","last_became_active":"2023-01-24T03:30:20.985857+0000","last_became_peered":"2023-01-24T03:30:20.985857+0000","last_unstale":"2023-01-24T03:30:24.992726+0000","last_undegraded":"2023-01-24T03:30:24.992726+0000","last_fullsized":"2023-01-24T03:30:24.992726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:00:57.338328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016603+0000","last_change":"2023-01-24T03:30:25.009962+0000","last_active":"2023-01-24T03:30:25.016603+0000","last_peered":"2023-01-24T03:30:25.016603+0000","last_clean":"2023-01-24T03:30:25.016603+0000","last_became_active":"2023-01-24T03:30:25.009623+0000","last_became_peered":"2023-01-24T03:30:25.009623+0000","last_unstale":"2023-01-24T03:30:25.016603+0000","last_undegraded":"2023-01-24T03:30:25.016603+0000","last_fullsized":"2023-01-24T03:30:25.016603+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:24:42.958531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0,6],"acting":[1,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.989699+0000","last_change":"2023-01-24T03:30:20.608114+0000","last_active":"2023-01-24T03:30:24.989699+0000","last_peered":"2023-01-24T03:30:24.989699+0000","last_clean":"2023-01-24T03:30:24.989699+0000","last_became_active":"2023-01-24T03:30:20.607816+0000","last_became_peered":"2023-01-24T03:30:20.607816+0000","last_unstale":"2023-01-24T03:30:24.989699+0000","last_undegraded":"2023-01-24T03:30:24.989699+0000","last_fullsized":"2023-01-24T03:30:24.989699+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:40:00.784496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992908+0000","last_change":"2023-01-24T03:30:20.982589+0000","last_active":"2023-01-24T03:30:24.992908+0000","last_peered":"2023-01-24T03:30:24.992908+0000","last_clean":"2023-01-24T03:30:24.992908+0000","last_became_active":"2023-01-24T03:30:20.979779+0000","last_became_peered":"2023-01-24T03:30:20.979779+0000","last_unstale":"2023-01-24T03:30:24.992908+0000","last_undegraded":"2023-01-24T03:30:24.992908+0000","last_fullsized":"2023-01-24T03:30:24.992908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:01:18.491363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.019938+0000","last_change":"2023-01-24T03:30:25.013327+0000","last_active":"2023-01-24T03:30:25.019938+0000","last_peered":"2023-01-24T03:30:25.019938+0000","last_clean":"2023-01-24T03:30:25.019938+0000","last_became_active":"2023-01-24T03:30:25.011303+0000","last_became_peered":"2023-01-24T03:30:25.011303+0000","last_unstale":"2023-01-24T03:30:25.019938+0000","last_undegraded":"2023-01-24T03:30:25.019938+0000","last_fullsized":"2023-01-24T03:30:25.019938+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:14:43.169287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6,2],"acting":[5,4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992390+0000","last_change":"2023-01-24T03:30:20.602369+0000","last_active":"2023-01-24T03:30:24.992390+0000","last_peered":"2023-01-24T03:30:24.992390+0000","last_clean":"2023-01-24T03:30:24.992390+0000","last_became_active":"2023-01-24T03:30:20.601996+0000","last_became_peered":"2023-01-24T03:30:20.601996+0000","last_unstale":"2023-01-24T03:30:24.992390+0000","last_undegraded":"2023-01-24T03:30:24.992390+0000","last_fullsized":"2023-01-24T03:30:24.992390+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:33:55.328080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993109+0000","last_change":"2023-01-24T03:30:20.988738+0000","last_active":"2023-01-24T03:30:24.993109+0000","last_peered":"2023-01-24T03:30:24.993109+0000","last_clean":"2023-01-24T03:30:24.993109+0000","last_became_active":"2023-01-24T03:30:20.987190+0000","last_became_peered":"2023-01-24T03:30:20.987190+0000","last_unstale":"2023-01-24T03:30:24.993109+0000","last_undegraded":"2023-01-24T03:30:24.993109+0000","last_fullsized":"2023-01-24T03:30:24.993109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:32:05.689674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.013303+0000","last_change":"2023-01-24T03:30:25.008401+0000","last_active":"2023-01-24T03:30:25.013303+0000","last_peered":"2023-01-24T03:30:25.013303+0000","last_clean":"2023-01-24T03:30:25.013303+0000","last_became_active":"2023-01-24T03:30:25.007973+0000","last_became_peered":"2023-01-24T03:30:25.007973+0000","last_unstale":"2023-01-24T03:30:25.013303+0000","last_undegraded":"2023-01-24T03:30:25.013303+0000","last_fullsized":"2023-01-24T03:30:25.013303+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:36:22.339295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,2],"acting":[4,5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991180+0000","last_change":"2023-01-24T03:30:20.608145+0000","last_active":"2023-01-24T03:30:24.991180+0000","last_peered":"2023-01-24T03:30:24.991180+0000","last_clean":"2023-01-24T03:30:24.991180+0000","last_became_active":"2023-01-24T03:30:20.607165+0000","last_became_peered":"2023-01-24T03:30:20.607165+0000","last_unstale":"2023-01-24T03:30:24.991180+0000","last_undegraded":"2023-01-24T03:30:24.991180+0000","last_fullsized":"2023-01-24T03:30:24.991180+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:58:56.490482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002416+0000","last_change":"2023-01-24T03:30:20.998777+0000","last_active":"2023-01-24T03:30:25.002416+0000","last_peered":"2023-01-24T03:30:25.002416+0000","last_clean":"2023-01-24T03:30:25.002416+0000","last_became_active":"2023-01-24T03:30:20.998519+0000","last_became_peered":"2023-01-24T03:30:20.998519+0000","last_unstale":"2023-01-24T03:30:25.002416+0000","last_undegraded":"2023-01-24T03:30:25.002416+0000","last_fullsized":"2023-01-24T03:30:25.002416+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:55:55.320556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.012455+0000","last_change":"2023-01-24T03:30:25.008154+0000","last_active":"2023-01-24T03:30:25.012455+0000","last_peered":"2023-01-24T03:30:25.012455+0000","last_clean":"2023-01-24T03:30:25.012455+0000","last_became_active":"2023-01-24T03:30:25.007797+0000","last_became_peered":"2023-01-24T03:30:25.007797+0000","last_unstale":"2023-01-24T03:30:25.012455+0000","last_undegraded":"2023-01-24T03:30:25.012455+0000","last_fullsized":"2023-01-24T03:30:25.012455+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:29:05.153733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7,0],"acting":[4,3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"24'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985835+0000","last_change":"2023-01-24T03:30:20.610690+0000","last_active":"2023-01-24T03:30:25.985835+0000","last_peered":"2023-01-24T03:30:25.985835+0000","last_clean":"2023-01-24T03:30:25.985835+0000","last_became_active":"2023-01-24T03:30:20.610409+0000","last_became_peered":"2023-01-24T03:30:20.610409+0000","last_unstale":"2023-01-24T03:30:25.985835+0000","last_undegraded":"2023-01-24T03:30:25.985835+0000","last_fullsized":"2023-01-24T03:30:25.985835+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:43:37.740181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993033+0000","last_change":"2023-01-24T03:30:16.085885+0000","last_active":"2023-01-24T03:30:24.993033+0000","last_peered":"2023-01-24T03:30:24.993033+0000","last_clean":"2023-01-24T03:30:24.993033+0000","last_became_active":"2023-01-24T03:30:16.085525+0000","last_became_peered":"2023-01-24T03:30:16.085525+0000","last_unstale":"2023-01-24T03:30:24.993033+0000","last_undegraded":"2023-01-24T03:30:24.993033+0000","last_fullsized":"2023-01-24T03:30:24.993033+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:57:36.499644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991099+0000","last_change":"2023-01-24T03:30:20.996370+0000","last_active":"2023-01-24T03:30:24.991099+0000","last_peered":"2023-01-24T03:30:24.991099+0000","last_clean":"2023-01-24T03:30:24.991099+0000","last_became_active":"2023-01-24T03:30:20.996122+0000","last_became_peered":"2023-01-24T03:30:20.996122+0000","last_unstale":"2023-01-24T03:30:24.991099+0000","last_undegraded":"2023-01-24T03:30:24.991099+0000","last_fullsized":"2023-01-24T03:30:24.991099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:13:06.913474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.014170+0000","last_change":"2023-01-24T03:30:25.007262+0000","last_active":"2023-01-24T03:30:25.014170+0000","last_peered":"2023-01-24T03:30:25.014170+0000","last_clean":"2023-01-24T03:30:25.014170+0000","last_became_active":"2023-01-24T03:30:25.006862+0000","last_became_peered":"2023-01-24T03:30:25.006862+0000","last_unstale":"2023-01-24T03:30:25.014170+0000","last_undegraded":"2023-01-24T03:30:25.014170+0000","last_fullsized":"2023-01-24T03:30:25.014170+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:42:05.268862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,0],"acting":[3,1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991165+0000","last_change":"2023-01-24T03:30:16.057240+0000","last_active":"2023-01-24T03:30:24.991165+0000","last_peered":"2023-01-24T03:30:24.991165+0000","last_clean":"2023-01-24T03:30:24.991165+0000","last_became_active":"2023-01-24T03:30:16.056941+0000","last_became_peered":"2023-01-24T03:30:16.056941+0000","last_unstale":"2023-01-24T03:30:24.991165+0000","last_undegraded":"2023-01-24T03:30:24.991165+0000","last_fullsized":"2023-01-24T03:30:24.991165+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:22:26.168527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"24'1","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.447279+0000","last_change":"2023-01-24T03:30:20.606130+0000","last_active":"2023-01-24T03:30:25.447279+0000","last_peered":"2023-01-24T03:30:25.447279+0000","last_clean":"2023-01-24T03:30:25.447279+0000","last_became_active":"2023-01-24T03:30:20.605899+0000","last_became_peered":"2023-01-24T03:30:20.605899+0000","last_unstale":"2023-01-24T03:30:25.447279+0000","last_undegraded":"2023-01-24T03:30:25.447279+0000","last_fullsized":"2023-01-24T03:30:25.447279+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:30:47.956569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992411+0000","last_change":"2023-01-24T03:30:20.988906+0000","last_active":"2023-01-24T03:30:24.992411+0000","last_peered":"2023-01-24T03:30:24.992411+0000","last_clean":"2023-01-24T03:30:24.992411+0000","last_became_active":"2023-01-24T03:30:20.988688+0000","last_became_peered":"2023-01-24T03:30:20.988688+0000","last_unstale":"2023-01-24T03:30:24.992411+0000","last_undegraded":"2023-01-24T03:30:24.992411+0000","last_fullsized":"2023-01-24T03:30:24.992411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:25:43.365167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.019966+0000","last_change":"2023-01-24T03:30:25.013834+0000","last_active":"2023-01-24T03:30:25.019966+0000","last_peered":"2023-01-24T03:30:25.019966+0000","last_clean":"2023-01-24T03:30:25.019966+0000","last_became_active":"2023-01-24T03:30:25.011774+0000","last_became_peered":"2023-01-24T03:30:25.011774+0000","last_unstale":"2023-01-24T03:30:25.019966+0000","last_undegraded":"2023-01-24T03:30:25.019966+0000","last_fullsized":"2023-01-24T03:30:25.019966+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:02:50.158049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0,4],"acting":[5,1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002878+0000","last_change":"2023-01-24T03:30:16.051822+0000","last_active":"2023-01-24T03:30:25.002878+0000","last_peered":"2023-01-24T03:30:25.002878+0000","last_clean":"2023-01-24T03:30:25.002878+0000","last_became_active":"2023-01-24T03:30:16.051525+0000","last_became_peered":"2023-01-24T03:30:16.051525+0000","last_unstale":"2023-01-24T03:30:25.002878+0000","last_undegraded":"2023-01-24T03:30:25.002878+0000","last_fullsized":"2023-01-24T03:30:25.002878+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:37:30.802424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"16'117","reported_seq":166,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996317+0000","last_change":"2023-01-24T03:30:16.054907+0000","last_active":"2023-01-24T03:30:24.996317+0000","last_peered":"2023-01-24T03:30:24.996317+0000","last_clean":"2023-01-24T03:30:24.996317+0000","last_became_active":"2023-01-24T03:30:16.054071+0000","last_became_peered":"2023-01-24T03:30:16.054071+0000","last_unstale":"2023-01-24T03:30:24.996317+0000","last_undegraded":"2023-01-24T03:30:24.996317+0000","last_fullsized":"2023-01-24T03:30:24.996317+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:31:26.790766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992331+0000","last_change":"2023-01-24T03:30:20.607914+0000","last_active":"2023-01-24T03:30:24.992331+0000","last_peered":"2023-01-24T03:30:24.992331+0000","last_clean":"2023-01-24T03:30:24.992331+0000","last_became_active":"2023-01-24T03:30:20.607663+0000","last_became_peered":"2023-01-24T03:30:20.607663+0000","last_unstale":"2023-01-24T03:30:24.992331+0000","last_undegraded":"2023-01-24T03:30:24.992331+0000","last_fullsized":"2023-01-24T03:30:24.992331+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:32:24.271747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993825+0000","last_change":"2023-01-24T03:30:20.998105+0000","last_active":"2023-01-24T03:30:24.993825+0000","last_peered":"2023-01-24T03:30:24.993825+0000","last_clean":"2023-01-24T03:30:24.993825+0000","last_became_active":"2023-01-24T03:30:20.997871+0000","last_became_peered":"2023-01-24T03:30:20.997871+0000","last_unstale":"2023-01-24T03:30:24.993825+0000","last_undegraded":"2023-01-24T03:30:24.993825+0000","last_fullsized":"2023-01-24T03:30:24.993825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:47:42.740428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016190+0000","last_change":"2023-01-24T03:30:25.008079+0000","last_active":"2023-01-24T03:30:25.016190+0000","last_peered":"2023-01-24T03:30:25.016190+0000","last_clean":"2023-01-24T03:30:25.016190+0000","last_became_active":"2023-01-24T03:30:25.007596+0000","last_became_peered":"2023-01-24T03:30:25.007596+0000","last_unstale":"2023-01-24T03:30:25.016190+0000","last_undegraded":"2023-01-24T03:30:25.016190+0000","last_fullsized":"2023-01-24T03:30:25.016190+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:30:57.179885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1,7],"acting":[2,5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996872+0000","last_change":"2023-01-24T03:30:16.272445+0000","last_active":"2023-01-24T03:30:24.996872+0000","last_peered":"2023-01-24T03:30:24.996872+0000","last_clean":"2023-01-24T03:30:24.996872+0000","last_became_active":"2023-01-24T03:30:16.272184+0000","last_became_peered":"2023-01-24T03:30:16.272184+0000","last_unstale":"2023-01-24T03:30:24.996872+0000","last_undegraded":"2023-01-24T03:30:24.996872+0000","last_fullsized":"2023-01-24T03:30:24.996872+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:17:05.555068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002328+0000","last_change":"2023-01-24T03:30:20.612832+0000","last_active":"2023-01-24T03:30:25.002328+0000","last_peered":"2023-01-24T03:30:25.002328+0000","last_clean":"2023-01-24T03:30:25.002328+0000","last_became_active":"2023-01-24T03:30:20.612569+0000","last_became_peered":"2023-01-24T03:30:20.612569+0000","last_unstale":"2023-01-24T03:30:25.002328+0000","last_undegraded":"2023-01-24T03:30:25.002328+0000","last_fullsized":"2023-01-24T03:30:25.002328+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:37:40.534015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002330+0000","last_change":"2023-01-24T03:30:20.998171+0000","last_active":"2023-01-24T03:30:25.002330+0000","last_peered":"2023-01-24T03:30:25.002330+0000","last_clean":"2023-01-24T03:30:25.002330+0000","last_became_active":"2023-01-24T03:30:20.994887+0000","last_became_peered":"2023-01-24T03:30:20.994887+0000","last_unstale":"2023-01-24T03:30:25.002330+0000","last_undegraded":"2023-01-24T03:30:25.002330+0000","last_fullsized":"2023-01-24T03:30:25.002330+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:57:40.909930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.020081+0000","last_change":"2023-01-24T03:30:25.014948+0000","last_active":"2023-01-24T03:30:25.020081+0000","last_peered":"2023-01-24T03:30:25.020081+0000","last_clean":"2023-01-24T03:30:25.020081+0000","last_became_active":"2023-01-24T03:30:25.014572+0000","last_became_peered":"2023-01-24T03:30:25.014572+0000","last_unstale":"2023-01-24T03:30:25.020081+0000","last_undegraded":"2023-01-24T03:30:25.020081+0000","last_fullsized":"2023-01-24T03:30:25.020081+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:59:13.108074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5,3],"acting":[7,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996112+0000","last_change":"2023-01-24T03:30:16.272574+0000","last_active":"2023-01-24T03:30:24.996112+0000","last_peered":"2023-01-24T03:30:24.996112+0000","last_clean":"2023-01-24T03:30:24.996112+0000","last_became_active":"2023-01-24T03:30:16.272252+0000","last_became_peered":"2023-01-24T03:30:16.272252+0000","last_unstale":"2023-01-24T03:30:24.996112+0000","last_undegraded":"2023-01-24T03:30:24.996112+0000","last_fullsized":"2023-01-24T03:30:24.996112+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:12:08.393585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991181+0000","last_change":"2023-01-24T03:30:20.606878+0000","last_active":"2023-01-24T03:30:24.991181+0000","last_peered":"2023-01-24T03:30:24.991181+0000","last_clean":"2023-01-24T03:30:24.991181+0000","last_became_active":"2023-01-24T03:30:20.606442+0000","last_became_peered":"2023-01-24T03:30:20.606442+0000","last_unstale":"2023-01-24T03:30:24.991181+0000","last_undegraded":"2023-01-24T03:30:24.991181+0000","last_fullsized":"2023-01-24T03:30:24.991181+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:26:16.916080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990646+0000","last_change":"2023-01-24T03:30:20.990840+0000","last_active":"2023-01-24T03:30:24.990646+0000","last_peered":"2023-01-24T03:30:24.990646+0000","last_clean":"2023-01-24T03:30:24.990646+0000","last_became_active":"2023-01-24T03:30:20.990615+0000","last_became_peered":"2023-01-24T03:30:20.990615+0000","last_unstale":"2023-01-24T03:30:24.990646+0000","last_undegraded":"2023-01-24T03:30:24.990646+0000","last_fullsized":"2023-01-24T03:30:24.990646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T15:06:37.670127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.013106+0000","last_change":"2023-01-24T03:30:25.006375+0000","last_active":"2023-01-24T03:30:25.013106+0000","last_peered":"2023-01-24T03:30:25.013106+0000","last_clean":"2023-01-24T03:30:25.013106+0000","last_became_active":"2023-01-24T03:30:25.006027+0000","last_became_peered":"2023-01-24T03:30:25.006027+0000","last_unstale":"2023-01-24T03:30:25.013106+0000","last_undegraded":"2023-01-24T03:30:25.013106+0000","last_fullsized":"2023-01-24T03:30:25.013106+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:51:29.481545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,6],"acting":[0,1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005695+0000","last_change":"2023-01-24T03:30:16.272475+0000","last_active":"2023-01-24T03:30:25.005695+0000","last_peered":"2023-01-24T03:30:25.005695+0000","last_clean":"2023-01-24T03:30:25.005695+0000","last_became_active":"2023-01-24T03:30:16.272140+0000","last_became_peered":"2023-01-24T03:30:16.272140+0000","last_unstale":"2023-01-24T03:30:25.005695+0000","last_undegraded":"2023-01-24T03:30:25.005695+0000","last_fullsized":"2023-01-24T03:30:25.005695+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:48:16.300876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992670+0000","last_change":"2023-01-24T03:30:20.609391+0000","last_active":"2023-01-24T03:30:24.992670+0000","last_peered":"2023-01-24T03:30:24.992670+0000","last_clean":"2023-01-24T03:30:24.992670+0000","last_became_active":"2023-01-24T03:30:20.609124+0000","last_became_peered":"2023-01-24T03:30:20.609124+0000","last_unstale":"2023-01-24T03:30:24.992670+0000","last_undegraded":"2023-01-24T03:30:24.992670+0000","last_fullsized":"2023-01-24T03:30:24.992670+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:36:49.616649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.001859+0000","last_change":"2023-01-24T03:30:20.996985+0000","last_active":"2023-01-24T03:30:25.001859+0000","last_peered":"2023-01-24T03:30:25.001859+0000","last_clean":"2023-01-24T03:30:25.001859+0000","last_became_active":"2023-01-24T03:30:20.996612+0000","last_became_peered":"2023-01-24T03:30:20.996612+0000","last_unstale":"2023-01-24T03:30:25.001859+0000","last_undegraded":"2023-01-24T03:30:25.001859+0000","last_fullsized":"2023-01-24T03:30:25.001859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:10:50.378902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.012218+0000","last_change":"2023-01-24T03:30:25.008155+0000","last_active":"2023-01-24T03:30:25.012218+0000","last_peered":"2023-01-24T03:30:25.012218+0000","last_clean":"2023-01-24T03:30:25.012218+0000","last_became_active":"2023-01-24T03:30:25.007674+0000","last_became_peered":"2023-01-24T03:30:25.007674+0000","last_unstale":"2023-01-24T03:30:25.012218+0000","last_undegraded":"2023-01-24T03:30:25.012218+0000","last_fullsized":"2023-01-24T03:30:25.012218+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:55:46.448808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,2],"acting":[6,0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990827+0000","last_change":"2023-01-24T03:30:16.272372+0000","last_active":"2023-01-24T03:30:24.990827+0000","last_peered":"2023-01-24T03:30:24.990827+0000","last_clean":"2023-01-24T03:30:24.990827+0000","last_became_active":"2023-01-24T03:30:16.272104+0000","last_became_peered":"2023-01-24T03:30:16.272104+0000","last_unstale":"2023-01-24T03:30:24.990827+0000","last_undegraded":"2023-01-24T03:30:24.990827+0000","last_fullsized":"2023-01-24T03:30:24.990827+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:15:10.865596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002114+0000","last_change":"2023-01-24T03:30:20.612284+0000","last_active":"2023-01-24T03:30:25.002114+0000","last_peered":"2023-01-24T03:30:25.002114+0000","last_clean":"2023-01-24T03:30:25.002114+0000","last_became_active":"2023-01-24T03:30:20.612029+0000","last_became_peered":"2023-01-24T03:30:20.612029+0000","last_unstale":"2023-01-24T03:30:25.002114+0000","last_undegraded":"2023-01-24T03:30:25.002114+0000","last_fullsized":"2023-01-24T03:30:25.002114+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:18:58.755974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992643+0000","last_change":"2023-01-24T03:30:20.989834+0000","last_active":"2023-01-24T03:30:24.992643+0000","last_peered":"2023-01-24T03:30:24.992643+0000","last_clean":"2023-01-24T03:30:24.992643+0000","last_became_active":"2023-01-24T03:30:20.989645+0000","last_became_peered":"2023-01-24T03:30:20.989645+0000","last_unstale":"2023-01-24T03:30:24.992643+0000","last_undegraded":"2023-01-24T03:30:24.992643+0000","last_fullsized":"2023-01-24T03:30:24.992643+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:42:06.477043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016259+0000","last_change":"2023-01-24T03:30:25.011952+0000","last_active":"2023-01-24T03:30:25.016259+0000","last_peered":"2023-01-24T03:30:25.016259+0000","last_clean":"2023-01-24T03:30:25.016259+0000","last_became_active":"2023-01-24T03:30:25.011612+0000","last_became_peered":"2023-01-24T03:30:25.011612+0000","last_unstale":"2023-01-24T03:30:25.016259+0000","last_undegraded":"2023-01-24T03:30:25.016259+0000","last_fullsized":"2023-01-24T03:30:25.016259+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:51:52.027285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6,5],"acting":[0,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990404+0000","last_change":"2023-01-24T03:30:16.048979+0000","last_active":"2023-01-24T03:30:24.990404+0000","last_peered":"2023-01-24T03:30:24.990404+0000","last_clean":"2023-01-24T03:30:24.990404+0000","last_became_active":"2023-01-24T03:30:16.048641+0000","last_became_peered":"2023-01-24T03:30:16.048641+0000","last_unstale":"2023-01-24T03:30:24.990404+0000","last_undegraded":"2023-01-24T03:30:24.990404+0000","last_fullsized":"2023-01-24T03:30:24.990404+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T15:11:26.211024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005087+0000","last_change":"2023-01-24T03:30:20.606906+0000","last_active":"2023-01-24T03:30:25.005087+0000","last_peered":"2023-01-24T03:30:25.005087+0000","last_clean":"2023-01-24T03:30:25.005087+0000","last_became_active":"2023-01-24T03:30:20.606640+0000","last_became_peered":"2023-01-24T03:30:20.606640+0000","last_unstale":"2023-01-24T03:30:25.005087+0000","last_undegraded":"2023-01-24T03:30:25.005087+0000","last_fullsized":"2023-01-24T03:30:25.005087+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:21:09.008653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002357+0000","last_change":"2023-01-24T03:30:20.999886+0000","last_active":"2023-01-24T03:30:25.002357+0000","last_peered":"2023-01-24T03:30:25.002357+0000","last_clean":"2023-01-24T03:30:25.002357+0000","last_became_active":"2023-01-24T03:30:20.999394+0000","last_became_peered":"2023-01-24T03:30:20.999394+0000","last_unstale":"2023-01-24T03:30:25.002357+0000","last_undegraded":"2023-01-24T03:30:25.002357+0000","last_fullsized":"2023-01-24T03:30:25.002357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:30:22.395611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016356+0000","last_change":"2023-01-24T03:30:25.006190+0000","last_active":"2023-01-24T03:30:25.016356+0000","last_peered":"2023-01-24T03:30:25.016356+0000","last_clean":"2023-01-24T03:30:25.016356+0000","last_became_active":"2023-01-24T03:30:25.005825+0000","last_became_peered":"2023-01-24T03:30:25.005825+0000","last_unstale":"2023-01-24T03:30:25.016356+0000","last_undegraded":"2023-01-24T03:30:25.016356+0000","last_fullsized":"2023-01-24T03:30:25.016356+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:51:50.888938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,0],"acting":[1,3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"24'1","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.446961+0000","last_change":"2023-01-24T03:30:20.611079+0000","last_active":"2023-01-24T03:30:25.446961+0000","last_peered":"2023-01-24T03:30:25.446961+0000","last_clean":"2023-01-24T03:30:25.446961+0000","last_became_active":"2023-01-24T03:30:20.610841+0000","last_became_peered":"2023-01-24T03:30:20.610841+0000","last_unstale":"2023-01-24T03:30:25.446961+0000","last_undegraded":"2023-01-24T03:30:25.446961+0000","last_fullsized":"2023-01-24T03:30:25.446961+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:34:24.922661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.994693+0000","last_change":"2023-01-24T03:30:20.991418+0000","last_active":"2023-01-24T03:30:24.994693+0000","last_peered":"2023-01-24T03:30:24.994693+0000","last_clean":"2023-01-24T03:30:24.994693+0000","last_became_active":"2023-01-24T03:30:20.991167+0000","last_became_peered":"2023-01-24T03:30:20.991167+0000","last_unstale":"2023-01-24T03:30:24.994693+0000","last_undegraded":"2023-01-24T03:30:24.994693+0000","last_fullsized":"2023-01-24T03:30:24.994693+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:51:52.736691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.013519+0000","last_change":"2023-01-24T03:30:25.010394+0000","last_active":"2023-01-24T03:30:25.013519+0000","last_peered":"2023-01-24T03:30:25.013519+0000","last_clean":"2023-01-24T03:30:25.013519+0000","last_became_active":"2023-01-24T03:30:25.010044+0000","last_became_peered":"2023-01-24T03:30:25.010044+0000","last_unstale":"2023-01-24T03:30:25.013519+0000","last_undegraded":"2023-01-24T03:30:25.013519+0000","last_fullsized":"2023-01-24T03:30:25.013519+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:08:55.250184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1,6],"acting":[7,4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"24'2","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.984877+0000","last_change":"2023-01-24T03:30:20.607538+0000","last_active":"2023-01-24T03:30:25.984877+0000","last_peered":"2023-01-24T03:30:25.984877+0000","last_clean":"2023-01-24T03:30:25.984877+0000","last_became_active":"2023-01-24T03:30:20.607285+0000","last_became_peered":"2023-01-24T03:30:20.607285+0000","last_unstale":"2023-01-24T03:30:25.984877+0000","last_undegraded":"2023-01-24T03:30:25.984877+0000","last_fullsized":"2023-01-24T03:30:25.984877+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:42:21.836603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992857+0000","last_change":"2023-01-24T03:30:20.986791+0000","last_active":"2023-01-24T03:30:24.992857+0000","last_peered":"2023-01-24T03:30:24.992857+0000","last_clean":"2023-01-24T03:30:24.992857+0000","last_became_active":"2023-01-24T03:30:20.986516+0000","last_became_peered":"2023-01-24T03:30:20.986516+0000","last_unstale":"2023-01-24T03:30:24.992857+0000","last_undegraded":"2023-01-24T03:30:24.992857+0000","last_fullsized":"2023-01-24T03:30:24.992857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:53:19.191166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.015383+0000","last_change":"2023-01-24T03:30:25.010751+0000","last_active":"2023-01-24T03:30:25.015383+0000","last_peered":"2023-01-24T03:30:25.015383+0000","last_clean":"2023-01-24T03:30:25.015383+0000","last_became_active":"2023-01-24T03:30:25.010332+0000","last_became_peered":"2023-01-24T03:30:25.010332+0000","last_unstale":"2023-01-24T03:30:25.015383+0000","last_undegraded":"2023-01-24T03:30:25.015383+0000","last_fullsized":"2023-01-24T03:30:25.015383+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:48:30.190712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7,4],"acting":[5,0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"24'2","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985166+0000","last_change":"2023-01-24T03:30:20.608509+0000","last_active":"2023-01-24T03:30:25.985166+0000","last_peered":"2023-01-24T03:30:25.985166+0000","last_clean":"2023-01-24T03:30:25.985166+0000","last_became_active":"2023-01-24T03:30:20.608313+0000","last_became_peered":"2023-01-24T03:30:20.608313+0000","last_unstale":"2023-01-24T03:30:25.985166+0000","last_undegraded":"2023-01-24T03:30:25.985166+0000","last_fullsized":"2023-01-24T03:30:25.985166+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:31:52.253588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992448+0000","last_change":"2023-01-24T03:30:20.992334+0000","last_active":"2023-01-24T03:30:24.992448+0000","last_peered":"2023-01-24T03:30:24.992448+0000","last_clean":"2023-01-24T03:30:24.992448+0000","last_became_active":"2023-01-24T03:30:20.992027+0000","last_became_peered":"2023-01-24T03:30:20.992027+0000","last_unstale":"2023-01-24T03:30:24.992448+0000","last_undegraded":"2023-01-24T03:30:24.992448+0000","last_fullsized":"2023-01-24T03:30:24.992448+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:57:56.985507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.012177+0000","last_change":"2023-01-24T03:30:25.004167+0000","last_active":"2023-01-24T03:30:25.012177+0000","last_peered":"2023-01-24T03:30:25.012177+0000","last_clean":"2023-01-24T03:30:25.012177+0000","last_became_active":"2023-01-24T03:30:25.003752+0000","last_became_peered":"2023-01-24T03:30:25.003752+0000","last_unstale":"2023-01-24T03:30:25.012177+0000","last_undegraded":"2023-01-24T03:30:25.012177+0000","last_fullsized":"2023-01-24T03:30:25.012177+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:19:16.285529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"24'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.448123+0000","last_change":"2023-01-24T03:30:20.603542+0000","last_active":"2023-01-24T03:30:25.448123+0000","last_peered":"2023-01-24T03:30:25.448123+0000","last_clean":"2023-01-24T03:30:25.448123+0000","last_became_active":"2023-01-24T03:30:20.603287+0000","last_became_peered":"2023-01-24T03:30:20.603287+0000","last_unstale":"2023-01-24T03:30:25.448123+0000","last_undegraded":"2023-01-24T03:30:25.448123+0000","last_fullsized":"2023-01-24T03:30:25.448123+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:03:56.865763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991098+0000","last_change":"2023-01-24T03:30:20.989519+0000","last_active":"2023-01-24T03:30:24.991098+0000","last_peered":"2023-01-24T03:30:24.991098+0000","last_clean":"2023-01-24T03:30:24.991098+0000","last_became_active":"2023-01-24T03:30:20.989286+0000","last_became_peered":"2023-01-24T03:30:20.989286+0000","last_unstale":"2023-01-24T03:30:24.991098+0000","last_undegraded":"2023-01-24T03:30:24.991098+0000","last_fullsized":"2023-01-24T03:30:24.991098+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:17:30.829327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016619+0000","last_change":"2023-01-24T03:30:25.010262+0000","last_active":"2023-01-24T03:30:25.016619+0000","last_peered":"2023-01-24T03:30:25.016619+0000","last_clean":"2023-01-24T03:30:25.016619+0000","last_became_active":"2023-01-24T03:30:25.009930+0000","last_became_peered":"2023-01-24T03:30:25.009930+0000","last_unstale":"2023-01-24T03:30:25.016619+0000","last_undegraded":"2023-01-24T03:30:25.016619+0000","last_fullsized":"2023-01-24T03:30:25.016619+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:36:17.807046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7,2],"acting":[1,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"24'4","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985911+0000","last_change":"2023-01-24T03:30:20.607334+0000","last_active":"2023-01-24T03:30:25.985911+0000","last_peered":"2023-01-24T03:30:25.985911+0000","last_clean":"2023-01-24T03:30:25.985911+0000","last_became_active":"2023-01-24T03:30:20.607079+0000","last_became_peered":"2023-01-24T03:30:20.607079+0000","last_unstale":"2023-01-24T03:30:25.985911+0000","last_undegraded":"2023-01-24T03:30:25.985911+0000","last_fullsized":"2023-01-24T03:30:25.985911+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:05:18.886532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992554+0000","last_change":"2023-01-24T03:30:20.987567+0000","last_active":"2023-01-24T03:30:24.992554+0000","last_peered":"2023-01-24T03:30:24.992554+0000","last_clean":"2023-01-24T03:30:24.992554+0000","last_became_active":"2023-01-24T03:30:20.987058+0000","last_became_peered":"2023-01-24T03:30:20.987058+0000","last_unstale":"2023-01-24T03:30:24.992554+0000","last_undegraded":"2023-01-24T03:30:24.992554+0000","last_fullsized":"2023-01-24T03:30:24.992554+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:58:30.391733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.989538+0000","last_change":"2023-01-24T03:30:20.986393+0000","last_active":"2023-01-24T03:30:24.989538+0000","last_peered":"2023-01-24T03:30:24.989538+0000","last_clean":"2023-01-24T03:30:24.989538+0000","last_became_active":"2023-01-24T03:30:20.986089+0000","last_became_peered":"2023-01-24T03:30:20.986089+0000","last_unstale":"2023-01-24T03:30:24.989538+0000","last_undegraded":"2023-01-24T03:30:24.989538+0000","last_fullsized":"2023-01-24T03:30:24.989538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:44:21.992291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990902+0000","last_change":"2023-01-24T03:30:20.991468+0000","last_active":"2023-01-24T03:30:24.990902+0000","last_peered":"2023-01-24T03:30:24.990902+0000","last_clean":"2023-01-24T03:30:24.990902+0000","last_became_active":"2023-01-24T03:30:20.991227+0000","last_became_peered":"2023-01-24T03:30:20.991227+0000","last_unstale":"2023-01-24T03:30:24.990902+0000","last_undegraded":"2023-01-24T03:30:24.990902+0000","last_fullsized":"2023-01-24T03:30:24.990902+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:27:25.806340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992602+0000","last_change":"2023-01-24T03:30:20.989824+0000","last_active":"2023-01-24T03:30:24.992602+0000","last_peered":"2023-01-24T03:30:24.992602+0000","last_clean":"2023-01-24T03:30:24.992602+0000","last_became_active":"2023-01-24T03:30:20.989577+0000","last_became_peered":"2023-01-24T03:30:20.989577+0000","last_unstale":"2023-01-24T03:30:24.992602+0000","last_undegraded":"2023-01-24T03:30:24.992602+0000","last_fullsized":"2023-01-24T03:30:24.992602+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:48:17.948459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992452+0000","last_change":"2023-01-24T03:30:20.996266+0000","last_active":"2023-01-24T03:30:24.992452+0000","last_peered":"2023-01-24T03:30:24.992452+0000","last_clean":"2023-01-24T03:30:24.992452+0000","last_became_active":"2023-01-24T03:30:20.995610+0000","last_became_peered":"2023-01-24T03:30:20.995610+0000","last_unstale":"2023-01-24T03:30:24.992452+0000","last_undegraded":"2023-01-24T03:30:24.992452+0000","last_fullsized":"2023-01-24T03:30:24.992452+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:24:43.022530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.001943+0000","last_change":"2023-01-24T03:30:20.997081+0000","last_active":"2023-01-24T03:30:25.001943+0000","last_peered":"2023-01-24T03:30:25.001943+0000","last_clean":"2023-01-24T03:30:25.001943+0000","last_became_active":"2023-01-24T03:30:20.996837+0000","last_became_peered":"2023-01-24T03:30:20.996837+0000","last_unstale":"2023-01-24T03:30:25.001943+0000","last_undegraded":"2023-01-24T03:30:25.001943+0000","last_fullsized":"2023-01-24T03:30:25.001943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:42:06.134817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":4746,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64574,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9180,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":8832,"kb_avail":94362660,"statfs":{"total":96636764160,"available":96627363840,"internally_reserved":0,"allocated":348160,"data_stored":698731,"data_compressed":32287,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":9043968},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]}]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8576,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94363264,"statfs":{"total":96636764160,"available":96627982336,"internally_reserved":0,"allocated":253952,"data_stored":104434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9152,"kb_used_data":312,"kb_used_omap":0,"kb_used_meta":8832,"kb_avail":94362688,"statfs":{"total":96636764160,"available":96627392512,"internally_reserved":0,"allocated":319488,"data_stored":694802,"data_compressed":32287,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":9043968},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574859,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8640,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94363200,"statfs":{"total":96636764160,"available":96627916800,"internally_reserved":0,"allocated":253952,"data_stored":104382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]}]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8512,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94363328,"statfs":{"total":96636764160,"available":96628047872,"internally_reserved":0,"allocated":253952,"data_stored":104434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.254}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8580,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94363260,"statfs":{"total":96636764160,"available":96627978240,"internally_reserved":0,"allocated":258048,"data_stored":104524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574858,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8660,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94363180,"statfs":{"total":96636764160,"available":96627896320,"internally_reserved":0,"allocated":274432,"data_stored":108466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]}]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8580,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94363260,"statfs":{"total":96636764160,"available":96627978240,"internally_reserved":0,"allocated":258048,"data_stored":103705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.311}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32287,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32287,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-01-24T03:30:53.231 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-24T03:30:53.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.375+0000 7fa6f08a6700 1 Processor -- start 2023-01-24T03:30:53.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.375+0000 7fa6f08a6700 1 -- start start 2023-01-24T03:30:53.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.375+0000 7fa6f08a6700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa6ec0b0aa0 0x7fa6ec155da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:53.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.375+0000 7fa6f08a6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6ec0ca3e0 0x7fa6ec0ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:53.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.375+0000 7fa6f08a6700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6ec0b0f40 con 0x7fa6ec0c7e40 2023-01-24T03:30:53.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.375+0000 7fa6f08a6700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6ec0b1170 con 0x7fa6ec0b0aa0 2023-01-24T03:30:53.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.375+0000 7fa6f08a6700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ec0b1450 con 0x7fa6ec0ca3e0 2023-01-24T03:30:53.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.376+0000 7fa6e1d9b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa6ec0b0aa0 0x7fa6ec155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:53.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.376+0000 7fa6ead9d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6ec0ca3e0 0x7fa6ec0ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:53.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.376+0000 7fa6e1d9b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa6ec0b0aa0 0x7fa6ec155da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.112:43718/0 (socket says 172.21.15.112:43718) 2023-01-24T03:30:53.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.376+0000 7fa6e1d9b700 1 -- 172.21.15.112:0/1045848049 learned_addr learned my addr 172.21.15.112:0/1045848049 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:53.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.376+0000 7fa6e1d9b700 1 -- 172.21.15.112:0/1045848049 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6ec0ca3e0 msgr2=0x7fa6ec0ca7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:53.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.376+0000 7fa6e1d9b700 1 --2- 172.21.15.112:0/1045848049 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6ec0ca3e0 0x7fa6ec0ca7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:53.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.376+0000 7fa6e1d9b700 1 -- 172.21.15.112:0/1045848049 >> v1:172.21.15.142:6789/0 conn(0x7fa6ec0c7e40 legacy=0x7fa6ec0ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:53.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.376+0000 7fa6e1d9b700 1 -- 172.21.15.112:0/1045848049 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ec0b1730 con 0x7fa6ec0b0aa0 2023-01-24T03:30:53.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.376+0000 7fa6e1d9b700 1 --2- 172.21.15.112:0/1045848049 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6ec0b0aa0 0x7fa6ec155da0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4006fd0 tx=0x7fa6d40090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=165a36dda0278d1b server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:53.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.376+0000 7fa6e9d9b700 1 -- 172.21.15.112:0/1045848049 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d4011630 con 0x7fa6ec0b0aa0 2023-01-24T03:30:53.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.376+0000 7fa6e9d9b700 1 -- 172.21.15.112:0/1045848049 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6d40117d0 con 0x7fa6ec0b0aa0 2023-01-24T03:30:53.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.376+0000 7fa6e9d9b700 1 -- 172.21.15.112:0/1045848049 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d4011b90 con 0x7fa6ec0b0aa0 2023-01-24T03:30:53.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.377+0000 7fa6f08a6700 1 -- 172.21.15.112:0/1045848049 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6ec0b0aa0 msgr2=0x7fa6ec155da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:53.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.377+0000 7fa6f08a6700 1 --2- 172.21.15.112:0/1045848049 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6ec0b0aa0 0x7fa6ec155da0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4006fd0 tx=0x7fa6d40090f0 comp rx=0 tx=0).stop 2023-01-24T03:30:53.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.377+0000 7fa6f08a6700 1 -- 172.21.15.112:0/1045848049 shutdown_connections 2023-01-24T03:30:53.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.377+0000 7fa6f08a6700 1 --2- 172.21.15.112:0/1045848049 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6ec0ca3e0 0x7fa6ec0ca7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:53.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.377+0000 7fa6f08a6700 1 --2- 172.21.15.112:0/1045848049 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6ec0b0aa0 0x7fa6ec155da0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:53.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.377+0000 7fa6f08a6700 1 -- 172.21.15.112:0/1045848049 >> 172.21.15.112:0/1045848049 conn(0x7fa6ec05e2c0 msgr2=0x7fa6ec0cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:53.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.377+0000 7fa6f08a6700 1 -- 172.21.15.112:0/1045848049 shutdown_connections 2023-01-24T03:30:53.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.377+0000 7fa6f08a6700 1 -- 172.21.15.112:0/1045848049 wait complete. 2023-01-24T03:30:53.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.378+0000 7fa6f08a6700 1 Processor -- start 2023-01-24T03:30:53.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.378+0000 7fa6f08a6700 1 -- start start 2023-01-24T03:30:53.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.379+0000 7fa6f08a6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6ec0b0aa0 0x7fa6ec165eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:53.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.379+0000 7fa6f08a6700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6ec0c7e40 0x7fa6ec151f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:53.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.379+0000 7fa6f08a6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa6ec0ca3e0 0x7fa6ec0c2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:53.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.379+0000 7fa6f08a6700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ec0c1610 con 0x7fa6ec0ca3e0 2023-01-24T03:30:53.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.379+0000 7fa6f08a6700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ec0c1880 con 0x7fa6ec0c7e40 2023-01-24T03:30:53.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.379+0000 7fa6f08a6700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ec0c1b60 con 0x7fa6ec0b0aa0 2023-01-24T03:30:53.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.379+0000 7fa6ead9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa6ec0ca3e0 0x7fa6ec0c2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:53.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.379+0000 7fa6ead9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa6ec0ca3e0 0x7fa6ec0c2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51232/0 (socket says 172.21.15.112:51232) 2023-01-24T03:30:53.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.379+0000 7fa6ead9d700 1 -- 172.21.15.112:0/1069575640 learned_addr learned my addr 172.21.15.112:0/1069575640 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:53.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.379+0000 7fa6e1d9b700 1 --2- 172.21.15.112:0/1069575640 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6ec0c7e40 0x7fa6ec151f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:53.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.379+0000 7fa6ea59c700 1 --2- 172.21.15.112:0/1069575640 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6ec0b0aa0 0x7fa6ec165eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:53.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.379+0000 7fa6ead9d700 1 -- 172.21.15.112:0/1069575640 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6ec0b0aa0 msgr2=0x7fa6ec165eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:53.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.379+0000 7fa6ead9d700 1 --2- 172.21.15.112:0/1069575640 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6ec0b0aa0 0x7fa6ec165eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:53.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.379+0000 7fa6ead9d700 1 -- 172.21.15.112:0/1069575640 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6ec0c7e40 msgr2=0x7fa6ec151f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:53.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.379+0000 7fa6ead9d700 1 --2- 172.21.15.112:0/1069575640 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6ec0c7e40 0x7fa6ec151f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:53.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.379+0000 7fa6ead9d700 1 -- 172.21.15.112:0/1069575640 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d4009370 con 0x7fa6ec0ca3e0 2023-01-24T03:30:53.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.379+0000 7fa6e1d9b700 1 --2- 172.21.15.112:0/1069575640 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6ec0c7e40 0x7fa6ec151f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:30:53.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.379+0000 7fa6ead9d700 1 --2- 172.21.15.112:0/1069575640 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa6ec0ca3e0 0x7fa6ec0c2e90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa6e4009e90 tx=0x7fa6e400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:53.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.379+0000 7fa6e2ffd700 1 -- 172.21.15.112:0/1069575640 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e400cc20 con 0x7fa6ec0ca3e0 2023-01-24T03:30:53.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.379+0000 7fa6e2ffd700 1 -- 172.21.15.112:0/1069575640 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6e400f040 con 0x7fa6ec0ca3e0 2023-01-24T03:30:53.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.380+0000 7fa6e2ffd700 1 -- 172.21.15.112:0/1069575640 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e4014480 con 0x7fa6ec0ca3e0 2023-01-24T03:30:53.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.380+0000 7fa6f08a6700 1 -- 172.21.15.112:0/1069575640 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6ec0c1e40 con 0x7fa6ec0ca3e0 2023-01-24T03:30:53.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.380+0000 7fa6f08a6700 1 -- 172.21.15.112:0/1069575640 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6ec0c2400 con 0x7fa6ec0ca3e0 2023-01-24T03:30:53.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.381+0000 7fa6f08a6700 1 -- 172.21.15.112:0/1069575640 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6ec0c2ce0 con 0x7fa6ec0ca3e0 2023-01-24T03:30:53.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.381+0000 7fa6e2ffd700 1 -- 172.21.15.112:0/1069575640 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fa6e4014830 con 0x7fa6ec0ca3e0 2023-01-24T03:30:53.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.381+0000 7fa6e2ffd700 1 --2- 172.21.15.112:0/1069575640 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa6d00611c0 0x7fa6d0063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:53.393 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.381+0000 7fa6e2ffd700 1 -- 172.21.15.112:0/1069575640 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fa6e407e550 con 0x7fa6ec0ca3e0 2023-01-24T03:30:53.393 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.381+0000 7fa6ea59c700 1 --2- 172.21.15.112:0/1069575640 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa6d00611c0 0x7fa6d0063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:53.393 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.382+0000 7fa6ea59c700 1 --2- 172.21.15.112:0/1069575640 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa6d00611c0 0x7fa6d0063670 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc003640 tx=0x7fa6dc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:53.393 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.385+0000 7fa6e2ffd700 1 -- 172.21.15.112:0/1069575640 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fa6e404e390 con 0x7fa6ec0ca3e0 2023-01-24T03:30:53.562 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.560+0000 7fa6f08a6700 1 -- 172.21.15.112:0/1069575640 --> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa6ec04bb80 con 0x7fa6d00611c0 2023-01-24T03:30:53.571 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.569+0000 7fa6e2ffd700 1 -- 172.21.15.112:0/1069575640 <== mgr.4104 v2:172.21.15.112:6832/84276 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+365110 (secure 0 0 0) 0x7fa6ec04bb80 con 0x7fa6d00611c0 2023-01-24T03:30:53.575 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:30:53.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.573+0000 7fa6f08a6700 1 -- 172.21.15.112:0/1069575640 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa6d00611c0 msgr2=0x7fa6d0063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:53.579 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.573+0000 7fa6f08a6700 1 --2- 172.21.15.112:0/1069575640 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa6d00611c0 0x7fa6d0063670 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc003640 tx=0x7fa6dc009040 comp rx=0 tx=0).stop 2023-01-24T03:30:53.579 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.573+0000 7fa6f08a6700 1 -- 172.21.15.112:0/1069575640 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa6ec0ca3e0 msgr2=0x7fa6ec0c2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:53.579 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.573+0000 7fa6f08a6700 1 --2- 172.21.15.112:0/1069575640 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa6ec0ca3e0 0x7fa6ec0c2e90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa6e4009e90 tx=0x7fa6e400b040 comp rx=0 tx=0).stop 2023-01-24T03:30:53.579 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.573+0000 7fa6f08a6700 1 -- 172.21.15.112:0/1069575640 shutdown_connections 2023-01-24T03:30:53.580 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.573+0000 7fa6f08a6700 1 --2- 172.21.15.112:0/1069575640 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa6d00611c0 0x7fa6d0063670 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:53.580 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.573+0000 7fa6f08a6700 1 --2- 172.21.15.112:0/1069575640 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa6ec0ca3e0 0x7fa6ec0c2e90 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:53.580 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.573+0000 7fa6f08a6700 1 --2- 172.21.15.112:0/1069575640 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6ec0b0aa0 0x7fa6ec165eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:53.580 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.573+0000 7fa6f08a6700 1 --2- 172.21.15.112:0/1069575640 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6ec0c7e40 0x7fa6ec151f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:53.581 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.573+0000 7fa6f08a6700 1 -- 172.21.15.112:0/1069575640 >> 172.21.15.112:0/1069575640 conn(0x7fa6ec05e2c0 msgr2=0x7fa6ec0680c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:53.581 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.573+0000 7fa6f08a6700 1 -- 172.21.15.112:0/1069575640 shutdown_connections 2023-01-24T03:30:53.581 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.573+0000 7fa6f08a6700 1 -- 172.21.15.112:0/1069575640 wait complete. 2023-01-24T03:30:53.581 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2023-01-24T03:30:53.587 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2023-01-24T03:30:53.009421+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595114,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":209,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":69880,"kb_used_data":2168,"kb_used_omap":0,"kb_used_meta":67648,"kb_avail":754904840,"statfs":{"total":773094113280,"available":773022556160,"internally_reserved":0,"allocated":2220032,"data_stored":2023478,"data_compressed":64574,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":69271552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":263,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002903"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993289+0000","last_change":"2023-01-24T03:30:20.986308+0000","last_active":"2023-01-24T03:30:24.993289+0000","last_peered":"2023-01-24T03:30:24.993289+0000","last_clean":"2023-01-24T03:30:24.993289+0000","last_became_active":"2023-01-24T03:30:20.986025+0000","last_became_peered":"2023-01-24T03:30:20.986025+0000","last_unstale":"2023-01-24T03:30:24.993289+0000","last_undegraded":"2023-01-24T03:30:24.993289+0000","last_fullsized":"2023-01-24T03:30:24.993289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:47:58.616698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.006202+0000","last_change":"2023-01-24T03:30:20.994478+0000","last_active":"2023-01-24T03:30:25.006202+0000","last_peered":"2023-01-24T03:30:25.006202+0000","last_clean":"2023-01-24T03:30:25.006202+0000","last_became_active":"2023-01-24T03:30:20.994207+0000","last_became_peered":"2023-01-24T03:30:20.994207+0000","last_unstale":"2023-01-24T03:30:25.006202+0000","last_undegraded":"2023-01-24T03:30:25.006202+0000","last_fullsized":"2023-01-24T03:30:25.006202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:02:53.230231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993177+0000","last_change":"2023-01-24T03:30:20.989651+0000","last_active":"2023-01-24T03:30:24.993177+0000","last_peered":"2023-01-24T03:30:24.993177+0000","last_clean":"2023-01-24T03:30:24.993177+0000","last_became_active":"2023-01-24T03:30:20.987585+0000","last_became_peered":"2023-01-24T03:30:20.987585+0000","last_unstale":"2023-01-24T03:30:24.993177+0000","last_undegraded":"2023-01-24T03:30:24.993177+0000","last_fullsized":"2023-01-24T03:30:24.993177+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:36:02.177544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990511+0000","last_change":"2023-01-24T03:30:20.993601+0000","last_active":"2023-01-24T03:30:24.990511+0000","last_peered":"2023-01-24T03:30:24.990511+0000","last_clean":"2023-01-24T03:30:24.990511+0000","last_became_active":"2023-01-24T03:30:20.993418+0000","last_became_peered":"2023-01-24T03:30:20.993418+0000","last_unstale":"2023-01-24T03:30:24.990511+0000","last_undegraded":"2023-01-24T03:30:24.990511+0000","last_fullsized":"2023-01-24T03:30:24.990511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:47:50.260497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996430+0000","last_change":"2023-01-24T03:30:20.991605+0000","last_active":"2023-01-24T03:30:24.996430+0000","last_peered":"2023-01-24T03:30:24.996430+0000","last_clean":"2023-01-24T03:30:24.996430+0000","last_became_active":"2023-01-24T03:30:20.991361+0000","last_became_peered":"2023-01-24T03:30:20.991361+0000","last_unstale":"2023-01-24T03:30:24.996430+0000","last_undegraded":"2023-01-24T03:30:24.996430+0000","last_fullsized":"2023-01-24T03:30:24.996430+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:31:25.418430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993288+0000","last_change":"2023-01-24T03:30:20.985058+0000","last_active":"2023-01-24T03:30:24.993288+0000","last_peered":"2023-01-24T03:30:24.993288+0000","last_clean":"2023-01-24T03:30:24.993288+0000","last_became_active":"2023-01-24T03:30:20.984754+0000","last_became_peered":"2023-01-24T03:30:20.984754+0000","last_unstale":"2023-01-24T03:30:24.993288+0000","last_undegraded":"2023-01-24T03:30:24.993288+0000","last_fullsized":"2023-01-24T03:30:24.993288+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:12:03.225109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996279+0000","last_change":"2023-01-24T03:30:21.000890+0000","last_active":"2023-01-24T03:30:24.996279+0000","last_peered":"2023-01-24T03:30:24.996279+0000","last_clean":"2023-01-24T03:30:24.996279+0000","last_became_active":"2023-01-24T03:30:21.000438+0000","last_became_peered":"2023-01-24T03:30:21.000438+0000","last_unstale":"2023-01-24T03:30:24.996279+0000","last_undegraded":"2023-01-24T03:30:24.996279+0000","last_fullsized":"2023-01-24T03:30:24.996279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:17:12.999664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.004717+0000","last_change":"2023-01-24T03:30:20.986753+0000","last_active":"2023-01-24T03:30:25.004717+0000","last_peered":"2023-01-24T03:30:25.004717+0000","last_clean":"2023-01-24T03:30:25.004717+0000","last_became_active":"2023-01-24T03:30:20.986524+0000","last_became_peered":"2023-01-24T03:30:20.986524+0000","last_unstale":"2023-01-24T03:30:25.004717+0000","last_undegraded":"2023-01-24T03:30:25.004717+0000","last_fullsized":"2023-01-24T03:30:25.004717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:24:55.699379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992350+0000","last_change":"2023-01-24T03:30:20.991136+0000","last_active":"2023-01-24T03:30:24.992350+0000","last_peered":"2023-01-24T03:30:24.992350+0000","last_clean":"2023-01-24T03:30:24.992350+0000","last_became_active":"2023-01-24T03:30:20.990925+0000","last_became_peered":"2023-01-24T03:30:20.990925+0000","last_unstale":"2023-01-24T03:30:24.992350+0000","last_undegraded":"2023-01-24T03:30:24.992350+0000","last_fullsized":"2023-01-24T03:30:24.992350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:45:37.485199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.006401+0000","last_change":"2023-01-24T03:30:20.995730+0000","last_active":"2023-01-24T03:30:25.006401+0000","last_peered":"2023-01-24T03:30:25.006401+0000","last_clean":"2023-01-24T03:30:25.006401+0000","last_became_active":"2023-01-24T03:30:20.994880+0000","last_became_peered":"2023-01-24T03:30:20.994880+0000","last_unstale":"2023-01-24T03:30:25.006401+0000","last_undegraded":"2023-01-24T03:30:25.006401+0000","last_fullsized":"2023-01-24T03:30:25.006401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:07:21.546386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993144+0000","last_change":"2023-01-24T03:30:20.993328+0000","last_active":"2023-01-24T03:30:24.993144+0000","last_peered":"2023-01-24T03:30:24.993144+0000","last_clean":"2023-01-24T03:30:24.993144+0000","last_became_active":"2023-01-24T03:30:20.992730+0000","last_became_peered":"2023-01-24T03:30:20.992730+0000","last_unstale":"2023-01-24T03:30:24.993144+0000","last_undegraded":"2023-01-24T03:30:24.993144+0000","last_fullsized":"2023-01-24T03:30:24.993144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:43:39.048550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.994605+0000","last_change":"2023-01-24T03:30:20.984993+0000","last_active":"2023-01-24T03:30:24.994605+0000","last_peered":"2023-01-24T03:30:24.994605+0000","last_clean":"2023-01-24T03:30:24.994605+0000","last_became_active":"2023-01-24T03:30:20.984540+0000","last_became_peered":"2023-01-24T03:30:20.984540+0000","last_unstale":"2023-01-24T03:30:24.994605+0000","last_undegraded":"2023-01-24T03:30:24.994605+0000","last_fullsized":"2023-01-24T03:30:24.994605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:16:39.993359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.995936+0000","last_change":"2023-01-24T03:30:20.986567+0000","last_active":"2023-01-24T03:30:24.995936+0000","last_peered":"2023-01-24T03:30:24.995936+0000","last_clean":"2023-01-24T03:30:24.995936+0000","last_became_active":"2023-01-24T03:30:20.986224+0000","last_became_peered":"2023-01-24T03:30:20.986224+0000","last_unstale":"2023-01-24T03:30:24.995936+0000","last_undegraded":"2023-01-24T03:30:24.995936+0000","last_fullsized":"2023-01-24T03:30:24.995936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:09:14.692354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005087+0000","last_change":"2023-01-24T03:30:20.988843+0000","last_active":"2023-01-24T03:30:25.005087+0000","last_peered":"2023-01-24T03:30:25.005087+0000","last_clean":"2023-01-24T03:30:25.005087+0000","last_became_active":"2023-01-24T03:30:20.988486+0000","last_became_peered":"2023-01-24T03:30:20.988486+0000","last_unstale":"2023-01-24T03:30:25.005087+0000","last_undegraded":"2023-01-24T03:30:25.005087+0000","last_fullsized":"2023-01-24T03:30:25.005087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:18:20.538844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996110+0000","last_change":"2023-01-24T03:30:21.000738+0000","last_active":"2023-01-24T03:30:24.996110+0000","last_peered":"2023-01-24T03:30:24.996110+0000","last_clean":"2023-01-24T03:30:24.996110+0000","last_became_active":"2023-01-24T03:30:21.000458+0000","last_became_peered":"2023-01-24T03:30:21.000458+0000","last_unstale":"2023-01-24T03:30:24.996110+0000","last_undegraded":"2023-01-24T03:30:24.996110+0000","last_fullsized":"2023-01-24T03:30:24.996110+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:21:42.876426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992383+0000","last_change":"2023-01-24T03:30:20.985062+0000","last_active":"2023-01-24T03:30:24.992383+0000","last_peered":"2023-01-24T03:30:24.992383+0000","last_clean":"2023-01-24T03:30:24.992383+0000","last_became_active":"2023-01-24T03:30:20.984759+0000","last_became_peered":"2023-01-24T03:30:20.984759+0000","last_unstale":"2023-01-24T03:30:24.992383+0000","last_undegraded":"2023-01-24T03:30:24.992383+0000","last_fullsized":"2023-01-24T03:30:24.992383+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:38:34.615748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992439+0000","last_change":"2023-01-24T03:30:20.989080+0000","last_active":"2023-01-24T03:30:24.992439+0000","last_peered":"2023-01-24T03:30:24.992439+0000","last_clean":"2023-01-24T03:30:24.992439+0000","last_became_active":"2023-01-24T03:30:20.988823+0000","last_became_peered":"2023-01-24T03:30:20.988823+0000","last_unstale":"2023-01-24T03:30:24.992439+0000","last_undegraded":"2023-01-24T03:30:24.992439+0000","last_fullsized":"2023-01-24T03:30:24.992439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:22:21.807370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993008+0000","last_change":"2023-01-24T03:30:20.984222+0000","last_active":"2023-01-24T03:30:24.993008+0000","last_peered":"2023-01-24T03:30:24.993008+0000","last_clean":"2023-01-24T03:30:24.993008+0000","last_became_active":"2023-01-24T03:30:20.984002+0000","last_became_peered":"2023-01-24T03:30:20.984002+0000","last_unstale":"2023-01-24T03:30:24.993008+0000","last_undegraded":"2023-01-24T03:30:24.993008+0000","last_fullsized":"2023-01-24T03:30:24.993008+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:54:24.005486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.997598+0000","last_change":"2023-01-24T03:30:20.988197+0000","last_active":"2023-01-24T03:30:24.997598+0000","last_peered":"2023-01-24T03:30:24.997598+0000","last_clean":"2023-01-24T03:30:24.997598+0000","last_became_active":"2023-01-24T03:30:20.987941+0000","last_became_peered":"2023-01-24T03:30:20.987941+0000","last_unstale":"2023-01-24T03:30:24.997598+0000","last_undegraded":"2023-01-24T03:30:24.997598+0000","last_fullsized":"2023-01-24T03:30:24.997598+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T15:08:29.659964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992957+0000","last_change":"2023-01-24T03:30:20.988701+0000","last_active":"2023-01-24T03:30:24.992957+0000","last_peered":"2023-01-24T03:30:24.992957+0000","last_clean":"2023-01-24T03:30:24.992957+0000","last_became_active":"2023-01-24T03:30:20.988476+0000","last_became_peered":"2023-01-24T03:30:20.988476+0000","last_unstale":"2023-01-24T03:30:24.992957+0000","last_undegraded":"2023-01-24T03:30:24.992957+0000","last_fullsized":"2023-01-24T03:30:24.992957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:33:29.463194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996546+0000","last_change":"2023-01-24T03:30:21.001125+0000","last_active":"2023-01-24T03:30:24.996546+0000","last_peered":"2023-01-24T03:30:24.996546+0000","last_clean":"2023-01-24T03:30:24.996546+0000","last_became_active":"2023-01-24T03:30:21.000873+0000","last_became_peered":"2023-01-24T03:30:21.000873+0000","last_unstale":"2023-01-24T03:30:24.996546+0000","last_undegraded":"2023-01-24T03:30:24.996546+0000","last_fullsized":"2023-01-24T03:30:24.996546+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:22:16.136875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993432+0000","last_change":"2023-01-24T03:30:20.992639+0000","last_active":"2023-01-24T03:30:24.993432+0000","last_peered":"2023-01-24T03:30:24.993432+0000","last_clean":"2023-01-24T03:30:24.993432+0000","last_became_active":"2023-01-24T03:30:20.992386+0000","last_became_peered":"2023-01-24T03:30:20.992386+0000","last_unstale":"2023-01-24T03:30:24.993432+0000","last_undegraded":"2023-01-24T03:30:24.993432+0000","last_fullsized":"2023-01-24T03:30:24.993432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:29:17.912198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992608+0000","last_change":"2023-01-24T03:30:20.994066+0000","last_active":"2023-01-24T03:30:24.992608+0000","last_peered":"2023-01-24T03:30:24.992608+0000","last_clean":"2023-01-24T03:30:24.992608+0000","last_became_active":"2023-01-24T03:30:20.993820+0000","last_became_peered":"2023-01-24T03:30:20.993820+0000","last_unstale":"2023-01-24T03:30:24.992608+0000","last_undegraded":"2023-01-24T03:30:24.992608+0000","last_fullsized":"2023-01-24T03:30:24.992608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:31:23.874058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996188+0000","last_change":"2023-01-24T03:30:21.001652+0000","last_active":"2023-01-24T03:30:24.996188+0000","last_peered":"2023-01-24T03:30:24.996188+0000","last_clean":"2023-01-24T03:30:24.996188+0000","last_became_active":"2023-01-24T03:30:21.001400+0000","last_became_peered":"2023-01-24T03:30:21.001400+0000","last_unstale":"2023-01-24T03:30:24.996188+0000","last_undegraded":"2023-01-24T03:30:24.996188+0000","last_fullsized":"2023-01-24T03:30:24.996188+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:55:53.249816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002243+0000","last_change":"2023-01-24T03:30:20.998304+0000","last_active":"2023-01-24T03:30:25.002243+0000","last_peered":"2023-01-24T03:30:25.002243+0000","last_clean":"2023-01-24T03:30:25.002243+0000","last_became_active":"2023-01-24T03:30:20.996288+0000","last_became_peered":"2023-01-24T03:30:20.996288+0000","last_unstale":"2023-01-24T03:30:25.002243+0000","last_undegraded":"2023-01-24T03:30:25.002243+0000","last_fullsized":"2023-01-24T03:30:25.002243+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:11:08.120197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002638+0000","last_change":"2023-01-24T03:30:20.996489+0000","last_active":"2023-01-24T03:30:25.002638+0000","last_peered":"2023-01-24T03:30:25.002638+0000","last_clean":"2023-01-24T03:30:25.002638+0000","last_became_active":"2023-01-24T03:30:20.994371+0000","last_became_peered":"2023-01-24T03:30:20.994371+0000","last_unstale":"2023-01-24T03:30:25.002638+0000","last_undegraded":"2023-01-24T03:30:25.002638+0000","last_fullsized":"2023-01-24T03:30:25.002638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:52:21.369928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005744+0000","last_change":"2023-01-24T03:30:20.997753+0000","last_active":"2023-01-24T03:30:25.005744+0000","last_peered":"2023-01-24T03:30:25.005744+0000","last_clean":"2023-01-24T03:30:25.005744+0000","last_became_active":"2023-01-24T03:30:20.997475+0000","last_became_peered":"2023-01-24T03:30:20.997475+0000","last_unstale":"2023-01-24T03:30:25.005744+0000","last_undegraded":"2023-01-24T03:30:25.005744+0000","last_fullsized":"2023-01-24T03:30:25.005744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:43:16.199843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.013267+0000","last_change":"2023-01-24T03:30:25.004944+0000","last_active":"2023-01-24T03:30:25.013267+0000","last_peered":"2023-01-24T03:30:25.013267+0000","last_clean":"2023-01-24T03:30:25.013267+0000","last_became_active":"2023-01-24T03:30:25.004590+0000","last_became_peered":"2023-01-24T03:30:25.004590+0000","last_unstale":"2023-01-24T03:30:25.013267+0000","last_undegraded":"2023-01-24T03:30:25.013267+0000","last_fullsized":"2023-01-24T03:30:25.013267+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:04:11.054225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2,0],"acting":[4,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"24'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.380450+0000","last_change":"2023-01-24T03:30:20.607211+0000","last_active":"2023-01-24T03:30:25.380450+0000","last_peered":"2023-01-24T03:30:25.380450+0000","last_clean":"2023-01-24T03:30:25.380450+0000","last_became_active":"2023-01-24T03:30:20.606855+0000","last_became_peered":"2023-01-24T03:30:20.606855+0000","last_unstale":"2023-01-24T03:30:25.380450+0000","last_undegraded":"2023-01-24T03:30:25.380450+0000","last_fullsized":"2023-01-24T03:30:25.380450+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:42:29.300089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991276+0000","last_change":"2023-01-24T03:30:20.993128+0000","last_active":"2023-01-24T03:30:24.991276+0000","last_peered":"2023-01-24T03:30:24.991276+0000","last_clean":"2023-01-24T03:30:24.991276+0000","last_became_active":"2023-01-24T03:30:20.992967+0000","last_became_peered":"2023-01-24T03:30:20.992967+0000","last_unstale":"2023-01-24T03:30:24.991276+0000","last_undegraded":"2023-01-24T03:30:24.991276+0000","last_fullsized":"2023-01-24T03:30:24.991276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:09:59.170218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.008814+0000","last_change":"2023-01-24T03:30:25.006569+0000","last_active":"2023-01-24T03:30:25.008814+0000","last_peered":"2023-01-24T03:30:25.008814+0000","last_clean":"2023-01-24T03:30:25.008814+0000","last_became_active":"2023-01-24T03:30:25.006199+0000","last_became_peered":"2023-01-24T03:30:25.006199+0000","last_unstale":"2023-01-24T03:30:25.008814+0000","last_undegraded":"2023-01-24T03:30:25.008814+0000","last_fullsized":"2023-01-24T03:30:25.008814+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:31:46.839485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,7],"acting":[6,4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.001971+0000","last_change":"2023-01-24T03:30:20.612738+0000","last_active":"2023-01-24T03:30:25.001971+0000","last_peered":"2023-01-24T03:30:25.001971+0000","last_clean":"2023-01-24T03:30:25.001971+0000","last_became_active":"2023-01-24T03:30:20.612375+0000","last_became_peered":"2023-01-24T03:30:20.612375+0000","last_unstale":"2023-01-24T03:30:25.001971+0000","last_undegraded":"2023-01-24T03:30:25.001971+0000","last_fullsized":"2023-01-24T03:30:25.001971+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:17:43.251945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993741+0000","last_change":"2023-01-24T03:30:20.996368+0000","last_active":"2023-01-24T03:30:24.993741+0000","last_peered":"2023-01-24T03:30:24.993741+0000","last_clean":"2023-01-24T03:30:24.993741+0000","last_became_active":"2023-01-24T03:30:20.996051+0000","last_became_peered":"2023-01-24T03:30:20.996051+0000","last_unstale":"2023-01-24T03:30:24.993741+0000","last_undegraded":"2023-01-24T03:30:24.993741+0000","last_fullsized":"2023-01-24T03:30:24.993741+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:53:28.193862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.014921+0000","last_change":"2023-01-24T03:30:25.008528+0000","last_active":"2023-01-24T03:30:25.014921+0000","last_peered":"2023-01-24T03:30:25.014921+0000","last_clean":"2023-01-24T03:30:25.014921+0000","last_became_active":"2023-01-24T03:30:25.008146+0000","last_became_peered":"2023-01-24T03:30:25.008146+0000","last_unstale":"2023-01-24T03:30:25.014921+0000","last_undegraded":"2023-01-24T03:30:25.014921+0000","last_fullsized":"2023-01-24T03:30:25.014921+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:17:11.457359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1,2],"acting":[0,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991341+0000","last_change":"2023-01-24T03:30:20.608932+0000","last_active":"2023-01-24T03:30:24.991341+0000","last_peered":"2023-01-24T03:30:24.991341+0000","last_clean":"2023-01-24T03:30:24.991341+0000","last_became_active":"2023-01-24T03:30:20.608135+0000","last_became_peered":"2023-01-24T03:30:20.608135+0000","last_unstale":"2023-01-24T03:30:24.991341+0000","last_undegraded":"2023-01-24T03:30:24.991341+0000","last_fullsized":"2023-01-24T03:30:24.991341+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:38:30.012279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990753+0000","last_change":"2023-01-24T03:30:20.986490+0000","last_active":"2023-01-24T03:30:24.990753+0000","last_peered":"2023-01-24T03:30:24.990753+0000","last_clean":"2023-01-24T03:30:24.990753+0000","last_became_active":"2023-01-24T03:30:20.986245+0000","last_became_peered":"2023-01-24T03:30:20.986245+0000","last_unstale":"2023-01-24T03:30:24.990753+0000","last_undegraded":"2023-01-24T03:30:24.990753+0000","last_fullsized":"2023-01-24T03:30:24.990753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:38:30.801032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.012186+0000","last_change":"2023-01-24T03:30:25.008221+0000","last_active":"2023-01-24T03:30:25.012186+0000","last_peered":"2023-01-24T03:30:25.012186+0000","last_clean":"2023-01-24T03:30:25.012186+0000","last_became_active":"2023-01-24T03:30:25.007855+0000","last_became_peered":"2023-01-24T03:30:25.007855+0000","last_unstale":"2023-01-24T03:30:25.012186+0000","last_undegraded":"2023-01-24T03:30:25.012186+0000","last_fullsized":"2023-01-24T03:30:25.012186+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:45:15.317067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,1],"acting":[6,4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992445+0000","last_change":"2023-01-24T03:30:20.607385+0000","last_active":"2023-01-24T03:30:24.992445+0000","last_peered":"2023-01-24T03:30:24.992445+0000","last_clean":"2023-01-24T03:30:24.992445+0000","last_became_active":"2023-01-24T03:30:20.607123+0000","last_became_peered":"2023-01-24T03:30:20.607123+0000","last_unstale":"2023-01-24T03:30:24.992445+0000","last_undegraded":"2023-01-24T03:30:24.992445+0000","last_fullsized":"2023-01-24T03:30:24.992445+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T15:24:00.474758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991258+0000","last_change":"2023-01-24T03:30:20.994124+0000","last_active":"2023-01-24T03:30:24.991258+0000","last_peered":"2023-01-24T03:30:24.991258+0000","last_clean":"2023-01-24T03:30:24.991258+0000","last_became_active":"2023-01-24T03:30:20.993876+0000","last_became_peered":"2023-01-24T03:30:20.993876+0000","last_unstale":"2023-01-24T03:30:24.991258+0000","last_undegraded":"2023-01-24T03:30:24.991258+0000","last_fullsized":"2023-01-24T03:30:24.991258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:07:26.252741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.011535+0000","last_change":"2023-01-24T03:30:25.006312+0000","last_active":"2023-01-24T03:30:25.011535+0000","last_peered":"2023-01-24T03:30:25.011535+0000","last_clean":"2023-01-24T03:30:25.011535+0000","last_became_active":"2023-01-24T03:30:25.005946+0000","last_became_peered":"2023-01-24T03:30:25.005946+0000","last_unstale":"2023-01-24T03:30:25.011535+0000","last_undegraded":"2023-01-24T03:30:25.011535+0000","last_fullsized":"2023-01-24T03:30:25.011535+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:35:02.760190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6,0],"acting":[7,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.006326+0000","last_change":"2023-01-24T03:30:20.610825+0000","last_active":"2023-01-24T03:30:25.006326+0000","last_peered":"2023-01-24T03:30:25.006326+0000","last_clean":"2023-01-24T03:30:25.006326+0000","last_became_active":"2023-01-24T03:30:20.610540+0000","last_became_peered":"2023-01-24T03:30:20.610540+0000","last_unstale":"2023-01-24T03:30:25.006326+0000","last_undegraded":"2023-01-24T03:30:25.006326+0000","last_fullsized":"2023-01-24T03:30:25.006326+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:23:15.279283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990818+0000","last_change":"2023-01-24T03:30:20.994605+0000","last_active":"2023-01-24T03:30:24.990818+0000","last_peered":"2023-01-24T03:30:24.990818+0000","last_clean":"2023-01-24T03:30:24.990818+0000","last_became_active":"2023-01-24T03:30:20.994356+0000","last_became_peered":"2023-01-24T03:30:20.994356+0000","last_unstale":"2023-01-24T03:30:24.990818+0000","last_undegraded":"2023-01-24T03:30:24.990818+0000","last_fullsized":"2023-01-24T03:30:24.990818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:06:08.797922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.018595+0000","last_change":"2023-01-24T03:30:25.012301+0000","last_active":"2023-01-24T03:30:25.018595+0000","last_peered":"2023-01-24T03:30:25.018595+0000","last_clean":"2023-01-24T03:30:25.018595+0000","last_became_active":"2023-01-24T03:30:25.011891+0000","last_became_peered":"2023-01-24T03:30:25.011891+0000","last_unstale":"2023-01-24T03:30:25.018595+0000","last_undegraded":"2023-01-24T03:30:25.018595+0000","last_fullsized":"2023-01-24T03:30:25.018595+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:15:46.012675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,5],"acting":[3,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"24'3","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.986137+0000","last_change":"2023-01-24T03:30:20.609828+0000","last_active":"2023-01-24T03:30:25.986137+0000","last_peered":"2023-01-24T03:30:25.986137+0000","last_clean":"2023-01-24T03:30:25.986137+0000","last_became_active":"2023-01-24T03:30:20.609492+0000","last_became_peered":"2023-01-24T03:30:20.609492+0000","last_unstale":"2023-01-24T03:30:25.986137+0000","last_undegraded":"2023-01-24T03:30:25.986137+0000","last_fullsized":"2023-01-24T03:30:25.986137+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:22:53.430711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002553+0000","last_change":"2023-01-24T03:30:20.999747+0000","last_active":"2023-01-24T03:30:25.002553+0000","last_peered":"2023-01-24T03:30:25.002553+0000","last_clean":"2023-01-24T03:30:25.002553+0000","last_became_active":"2023-01-24T03:30:20.999003+0000","last_became_peered":"2023-01-24T03:30:20.999003+0000","last_unstale":"2023-01-24T03:30:25.002553+0000","last_undegraded":"2023-01-24T03:30:25.002553+0000","last_fullsized":"2023-01-24T03:30:25.002553+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:19:22.205251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016573+0000","last_change":"2023-01-24T03:30:25.010659+0000","last_active":"2023-01-24T03:30:25.016573+0000","last_peered":"2023-01-24T03:30:25.016573+0000","last_clean":"2023-01-24T03:30:25.016573+0000","last_became_active":"2023-01-24T03:30:25.010237+0000","last_became_peered":"2023-01-24T03:30:25.010237+0000","last_unstale":"2023-01-24T03:30:25.016573+0000","last_undegraded":"2023-01-24T03:30:25.016573+0000","last_fullsized":"2023-01-24T03:30:25.016573+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:01:53.501070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3,7],"acting":[1,5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"24'11","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:48.616383+0000","last_change":"2023-01-24T03:30:20.610682+0000","last_active":"2023-01-24T03:30:48.616383+0000","last_peered":"2023-01-24T03:30:48.616383+0000","last_clean":"2023-01-24T03:30:48.616383+0000","last_became_active":"2023-01-24T03:30:20.610463+0000","last_became_peered":"2023-01-24T03:30:20.610463+0000","last_unstale":"2023-01-24T03:30:48.616383+0000","last_undegraded":"2023-01-24T03:30:48.616383+0000","last_fullsized":"2023-01-24T03:30:48.616383+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:25:00.447596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992845+0000","last_change":"2023-01-24T03:30:20.987326+0000","last_active":"2023-01-24T03:30:24.992845+0000","last_peered":"2023-01-24T03:30:24.992845+0000","last_clean":"2023-01-24T03:30:24.992845+0000","last_became_active":"2023-01-24T03:30:20.987042+0000","last_became_peered":"2023-01-24T03:30:20.987042+0000","last_unstale":"2023-01-24T03:30:24.992845+0000","last_undegraded":"2023-01-24T03:30:24.992845+0000","last_fullsized":"2023-01-24T03:30:24.992845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:57:43.257624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.020022+0000","last_change":"2023-01-24T03:30:25.012004+0000","last_active":"2023-01-24T03:30:25.020022+0000","last_peered":"2023-01-24T03:30:25.020022+0000","last_clean":"2023-01-24T03:30:25.020022+0000","last_became_active":"2023-01-24T03:30:25.007410+0000","last_became_peered":"2023-01-24T03:30:25.007410+0000","last_unstale":"2023-01-24T03:30:25.020022+0000","last_undegraded":"2023-01-24T03:30:25.020022+0000","last_fullsized":"2023-01-24T03:30:25.020022+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:34:21.681011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3,0],"acting":[5,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"24'2","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985201+0000","last_change":"2023-01-24T03:30:20.613026+0000","last_active":"2023-01-24T03:30:25.985201+0000","last_peered":"2023-01-24T03:30:25.985201+0000","last_clean":"2023-01-24T03:30:25.985201+0000","last_became_active":"2023-01-24T03:30:20.612605+0000","last_became_peered":"2023-01-24T03:30:20.612605+0000","last_unstale":"2023-01-24T03:30:25.985201+0000","last_undegraded":"2023-01-24T03:30:25.985201+0000","last_fullsized":"2023-01-24T03:30:25.985201+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:45:08.016740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991113+0000","last_change":"2023-01-24T03:30:20.991929+0000","last_active":"2023-01-24T03:30:24.991113+0000","last_peered":"2023-01-24T03:30:24.991113+0000","last_clean":"2023-01-24T03:30:24.991113+0000","last_became_active":"2023-01-24T03:30:20.991693+0000","last_became_peered":"2023-01-24T03:30:20.991693+0000","last_unstale":"2023-01-24T03:30:24.991113+0000","last_undegraded":"2023-01-24T03:30:24.991113+0000","last_fullsized":"2023-01-24T03:30:24.991113+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:55:51.637568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.018472+0000","last_change":"2023-01-24T03:30:25.010572+0000","last_active":"2023-01-24T03:30:25.018472+0000","last_peered":"2023-01-24T03:30:25.018472+0000","last_clean":"2023-01-24T03:30:25.018472+0000","last_became_active":"2023-01-24T03:30:25.010193+0000","last_became_peered":"2023-01-24T03:30:25.010193+0000","last_unstale":"2023-01-24T03:30:25.018472+0000","last_undegraded":"2023-01-24T03:30:25.018472+0000","last_fullsized":"2023-01-24T03:30:25.018472+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:25:39.676179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,1],"acting":[3,7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993579+0000","last_change":"2023-01-24T03:30:20.612047+0000","last_active":"2023-01-24T03:30:24.993579+0000","last_peered":"2023-01-24T03:30:24.993579+0000","last_clean":"2023-01-24T03:30:24.993579+0000","last_became_active":"2023-01-24T03:30:20.611798+0000","last_became_peered":"2023-01-24T03:30:20.611798+0000","last_unstale":"2023-01-24T03:30:24.993579+0000","last_undegraded":"2023-01-24T03:30:24.993579+0000","last_fullsized":"2023-01-24T03:30:24.993579+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:01:41.731689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005194+0000","last_change":"2023-01-24T03:30:20.996432+0000","last_active":"2023-01-24T03:30:25.005194+0000","last_peered":"2023-01-24T03:30:25.005194+0000","last_clean":"2023-01-24T03:30:25.005194+0000","last_became_active":"2023-01-24T03:30:20.996173+0000","last_became_peered":"2023-01-24T03:30:20.996173+0000","last_unstale":"2023-01-24T03:30:25.005194+0000","last_undegraded":"2023-01-24T03:30:25.005194+0000","last_fullsized":"2023-01-24T03:30:25.005194+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:13:31.596587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.018507+0000","last_change":"2023-01-24T03:30:25.008479+0000","last_active":"2023-01-24T03:30:25.018507+0000","last_peered":"2023-01-24T03:30:25.018507+0000","last_clean":"2023-01-24T03:30:25.018507+0000","last_became_active":"2023-01-24T03:30:25.007858+0000","last_became_peered":"2023-01-24T03:30:25.007858+0000","last_unstale":"2023-01-24T03:30:25.018507+0000","last_undegraded":"2023-01-24T03:30:25.018507+0000","last_fullsized":"2023-01-24T03:30:25.018507+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:10:38.425874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,0],"acting":[3,1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996134+0000","last_change":"2023-01-24T03:30:20.612848+0000","last_active":"2023-01-24T03:30:24.996134+0000","last_peered":"2023-01-24T03:30:24.996134+0000","last_clean":"2023-01-24T03:30:24.996134+0000","last_became_active":"2023-01-24T03:30:20.612600+0000","last_became_peered":"2023-01-24T03:30:20.612600+0000","last_unstale":"2023-01-24T03:30:24.996134+0000","last_undegraded":"2023-01-24T03:30:24.996134+0000","last_fullsized":"2023-01-24T03:30:24.996134+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:46:45.634472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993883+0000","last_change":"2023-01-24T03:30:20.991133+0000","last_active":"2023-01-24T03:30:24.993883+0000","last_peered":"2023-01-24T03:30:24.993883+0000","last_clean":"2023-01-24T03:30:24.993883+0000","last_became_active":"2023-01-24T03:30:20.990856+0000","last_became_peered":"2023-01-24T03:30:20.990856+0000","last_unstale":"2023-01-24T03:30:24.993883+0000","last_undegraded":"2023-01-24T03:30:24.993883+0000","last_fullsized":"2023-01-24T03:30:24.993883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:27:32.163224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.022452+0000","last_change":"2023-01-24T03:30:25.017038+0000","last_active":"2023-01-24T03:30:25.022452+0000","last_peered":"2023-01-24T03:30:25.022452+0000","last_clean":"2023-01-24T03:30:25.022452+0000","last_became_active":"2023-01-24T03:30:25.016684+0000","last_became_peered":"2023-01-24T03:30:25.016684+0000","last_unstale":"2023-01-24T03:30:25.022452+0000","last_undegraded":"2023-01-24T03:30:25.022452+0000","last_fullsized":"2023-01-24T03:30:25.022452+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:57:30.028834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1,6],"acting":[5,3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"24'8","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.988793+0000","last_change":"2023-01-24T03:30:20.604617+0000","last_active":"2023-01-24T03:30:25.988793+0000","last_peered":"2023-01-24T03:30:25.988793+0000","last_clean":"2023-01-24T03:30:25.988793+0000","last_became_active":"2023-01-24T03:30:20.604445+0000","last_became_peered":"2023-01-24T03:30:20.604445+0000","last_unstale":"2023-01-24T03:30:25.988793+0000","last_undegraded":"2023-01-24T03:30:25.988793+0000","last_fullsized":"2023-01-24T03:30:25.988793+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:06:51.460895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993017+0000","last_change":"2023-01-24T03:30:20.995872+0000","last_active":"2023-01-24T03:30:24.993017+0000","last_peered":"2023-01-24T03:30:24.993017+0000","last_clean":"2023-01-24T03:30:24.993017+0000","last_became_active":"2023-01-24T03:30:20.995599+0000","last_became_peered":"2023-01-24T03:30:20.995599+0000","last_unstale":"2023-01-24T03:30:24.993017+0000","last_undegraded":"2023-01-24T03:30:24.993017+0000","last_fullsized":"2023-01-24T03:30:24.993017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:53:54.356899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.020130+0000","last_change":"2023-01-24T03:30:25.016101+0000","last_active":"2023-01-24T03:30:25.020130+0000","last_peered":"2023-01-24T03:30:25.020130+0000","last_clean":"2023-01-24T03:30:25.020130+0000","last_became_active":"2023-01-24T03:30:25.015705+0000","last_became_peered":"2023-01-24T03:30:25.015705+0000","last_unstale":"2023-01-24T03:30:25.020130+0000","last_undegraded":"2023-01-24T03:30:25.020130+0000","last_fullsized":"2023-01-24T03:30:25.020130+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:54:37.798191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3,4],"acting":[7,6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"24'1","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.446876+0000","last_change":"2023-01-24T03:30:20.610632+0000","last_active":"2023-01-24T03:30:25.446876+0000","last_peered":"2023-01-24T03:30:25.446876+0000","last_clean":"2023-01-24T03:30:25.446876+0000","last_became_active":"2023-01-24T03:30:20.610367+0000","last_became_peered":"2023-01-24T03:30:20.610367+0000","last_unstale":"2023-01-24T03:30:25.446876+0000","last_undegraded":"2023-01-24T03:30:25.446876+0000","last_fullsized":"2023-01-24T03:30:25.446876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:26:05.938178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.004623+0000","last_change":"2023-01-24T03:30:20.996493+0000","last_active":"2023-01-24T03:30:25.004623+0000","last_peered":"2023-01-24T03:30:25.004623+0000","last_clean":"2023-01-24T03:30:25.004623+0000","last_became_active":"2023-01-24T03:30:20.996229+0000","last_became_peered":"2023-01-24T03:30:20.996229+0000","last_unstale":"2023-01-24T03:30:25.004623+0000","last_undegraded":"2023-01-24T03:30:25.004623+0000","last_fullsized":"2023-01-24T03:30:25.004623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:23:53.642522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016195+0000","last_change":"2023-01-24T03:30:25.009215+0000","last_active":"2023-01-24T03:30:25.016195+0000","last_peered":"2023-01-24T03:30:25.016195+0000","last_clean":"2023-01-24T03:30:25.016195+0000","last_became_active":"2023-01-24T03:30:25.008213+0000","last_became_peered":"2023-01-24T03:30:25.008213+0000","last_unstale":"2023-01-24T03:30:25.016195+0000","last_undegraded":"2023-01-24T03:30:25.016195+0000","last_fullsized":"2023-01-24T03:30:25.016195+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:48:37.517172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,5],"acting":[2,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"24'2","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985223+0000","last_change":"2023-01-24T03:30:20.612158+0000","last_active":"2023-01-24T03:30:25.985223+0000","last_peered":"2023-01-24T03:30:25.985223+0000","last_clean":"2023-01-24T03:30:25.985223+0000","last_became_active":"2023-01-24T03:30:20.611931+0000","last_became_peered":"2023-01-24T03:30:20.611931+0000","last_unstale":"2023-01-24T03:30:25.985223+0000","last_undegraded":"2023-01-24T03:30:25.985223+0000","last_fullsized":"2023-01-24T03:30:25.985223+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:30:37.532390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992500+0000","last_change":"2023-01-24T03:30:20.991582+0000","last_active":"2023-01-24T03:30:24.992500+0000","last_peered":"2023-01-24T03:30:24.992500+0000","last_clean":"2023-01-24T03:30:24.992500+0000","last_became_active":"2023-01-24T03:30:20.991335+0000","last_became_peered":"2023-01-24T03:30:20.991335+0000","last_unstale":"2023-01-24T03:30:24.992500+0000","last_undegraded":"2023-01-24T03:30:24.992500+0000","last_fullsized":"2023-01-24T03:30:24.992500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:45:20.786712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016224+0000","last_change":"2023-01-24T03:30:25.008058+0000","last_active":"2023-01-24T03:30:25.016224+0000","last_peered":"2023-01-24T03:30:25.016224+0000","last_clean":"2023-01-24T03:30:25.016224+0000","last_became_active":"2023-01-24T03:30:25.007711+0000","last_became_peered":"2023-01-24T03:30:25.007711+0000","last_unstale":"2023-01-24T03:30:25.016224+0000","last_undegraded":"2023-01-24T03:30:25.016224+0000","last_fullsized":"2023-01-24T03:30:25.016224+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:26:56.270674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5,6],"acting":[2,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"24'1","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.447163+0000","last_change":"2023-01-24T03:30:20.610495+0000","last_active":"2023-01-24T03:30:25.447163+0000","last_peered":"2023-01-24T03:30:25.447163+0000","last_clean":"2023-01-24T03:30:25.447163+0000","last_became_active":"2023-01-24T03:30:20.610035+0000","last_became_peered":"2023-01-24T03:30:20.610035+0000","last_unstale":"2023-01-24T03:30:25.447163+0000","last_undegraded":"2023-01-24T03:30:25.447163+0000","last_fullsized":"2023-01-24T03:30:25.447163+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:25:30.944275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.989631+0000","last_change":"2023-01-24T03:30:20.994362+0000","last_active":"2023-01-24T03:30:24.989631+0000","last_peered":"2023-01-24T03:30:24.989631+0000","last_clean":"2023-01-24T03:30:24.989631+0000","last_became_active":"2023-01-24T03:30:20.994101+0000","last_became_peered":"2023-01-24T03:30:20.994101+0000","last_unstale":"2023-01-24T03:30:24.989631+0000","last_undegraded":"2023-01-24T03:30:24.989631+0000","last_fullsized":"2023-01-24T03:30:24.989631+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:55:12.837646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.015999+0000","last_change":"2023-01-24T03:30:25.006995+0000","last_active":"2023-01-24T03:30:25.015999+0000","last_peered":"2023-01-24T03:30:25.015999+0000","last_clean":"2023-01-24T03:30:25.015999+0000","last_became_active":"2023-01-24T03:30:25.006473+0000","last_became_peered":"2023-01-24T03:30:25.006473+0000","last_unstale":"2023-01-24T03:30:25.015999+0000","last_undegraded":"2023-01-24T03:30:25.015999+0000","last_fullsized":"2023-01-24T03:30:25.015999+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:48:16.320162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3,6],"acting":[2,4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005342+0000","last_change":"2023-01-24T03:30:20.606526+0000","last_active":"2023-01-24T03:30:25.005342+0000","last_peered":"2023-01-24T03:30:25.005342+0000","last_clean":"2023-01-24T03:30:25.005342+0000","last_became_active":"2023-01-24T03:30:20.606301+0000","last_became_peered":"2023-01-24T03:30:20.606301+0000","last_unstale":"2023-01-24T03:30:25.005342+0000","last_undegraded":"2023-01-24T03:30:25.005342+0000","last_fullsized":"2023-01-24T03:30:25.005342+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:43:28.702475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992428+0000","last_change":"2023-01-24T03:30:20.993379+0000","last_active":"2023-01-24T03:30:24.992428+0000","last_peered":"2023-01-24T03:30:24.992428+0000","last_clean":"2023-01-24T03:30:24.992428+0000","last_became_active":"2023-01-24T03:30:20.993151+0000","last_became_peered":"2023-01-24T03:30:20.993151+0000","last_unstale":"2023-01-24T03:30:24.992428+0000","last_undegraded":"2023-01-24T03:30:24.992428+0000","last_fullsized":"2023-01-24T03:30:24.992428+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:00:45.845082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.020912+0000","last_change":"2023-01-24T03:30:25.013701+0000","last_active":"2023-01-24T03:30:25.020912+0000","last_peered":"2023-01-24T03:30:25.020912+0000","last_clean":"2023-01-24T03:30:25.020912+0000","last_became_active":"2023-01-24T03:30:25.013290+0000","last_became_peered":"2023-01-24T03:30:25.013290+0000","last_unstale":"2023-01-24T03:30:25.020912+0000","last_undegraded":"2023-01-24T03:30:25.020912+0000","last_fullsized":"2023-01-24T03:30:25.020912+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:32:17.340583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5,6],"acting":[2,7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"24'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985860+0000","last_change":"2023-01-24T03:30:20.609656+0000","last_active":"2023-01-24T03:30:25.985860+0000","last_peered":"2023-01-24T03:30:25.985860+0000","last_clean":"2023-01-24T03:30:25.985860+0000","last_became_active":"2023-01-24T03:30:20.609374+0000","last_became_peered":"2023-01-24T03:30:20.609374+0000","last_unstale":"2023-01-24T03:30:25.985860+0000","last_undegraded":"2023-01-24T03:30:25.985860+0000","last_fullsized":"2023-01-24T03:30:25.985860+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:58:12.653746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992726+0000","last_change":"2023-01-24T03:30:20.986089+0000","last_active":"2023-01-24T03:30:24.992726+0000","last_peered":"2023-01-24T03:30:24.992726+0000","last_clean":"2023-01-24T03:30:24.992726+0000","last_became_active":"2023-01-24T03:30:20.985857+0000","last_became_peered":"2023-01-24T03:30:20.985857+0000","last_unstale":"2023-01-24T03:30:24.992726+0000","last_undegraded":"2023-01-24T03:30:24.992726+0000","last_fullsized":"2023-01-24T03:30:24.992726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:00:57.338328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016603+0000","last_change":"2023-01-24T03:30:25.009962+0000","last_active":"2023-01-24T03:30:25.016603+0000","last_peered":"2023-01-24T03:30:25.016603+0000","last_clean":"2023-01-24T03:30:25.016603+0000","last_became_active":"2023-01-24T03:30:25.009623+0000","last_became_peered":"2023-01-24T03:30:25.009623+0000","last_unstale":"2023-01-24T03:30:25.016603+0000","last_undegraded":"2023-01-24T03:30:25.016603+0000","last_fullsized":"2023-01-24T03:30:25.016603+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:24:42.958531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0,6],"acting":[1,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.989699+0000","last_change":"2023-01-24T03:30:20.608114+0000","last_active":"2023-01-24T03:30:24.989699+0000","last_peered":"2023-01-24T03:30:24.989699+0000","last_clean":"2023-01-24T03:30:24.989699+0000","last_became_active":"2023-01-24T03:30:20.607816+0000","last_became_peered":"2023-01-24T03:30:20.607816+0000","last_unstale":"2023-01-24T03:30:24.989699+0000","last_undegraded":"2023-01-24T03:30:24.989699+0000","last_fullsized":"2023-01-24T03:30:24.989699+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:40:00.784496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992908+0000","last_change":"2023-01-24T03:30:20.982589+0000","last_active":"2023-01-24T03:30:24.992908+0000","last_peered":"2023-01-24T03:30:24.992908+0000","last_clean":"2023-01-24T03:30:24.992908+0000","last_became_active":"2023-01-24T03:30:20.979779+0000","last_became_peered":"2023-01-24T03:30:20.979779+0000","last_unstale":"2023-01-24T03:30:24.992908+0000","last_undegraded":"2023-01-24T03:30:24.992908+0000","last_fullsized":"2023-01-24T03:30:24.992908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:01:18.491363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.019938+0000","last_change":"2023-01-24T03:30:25.013327+0000","last_active":"2023-01-24T03:30:25.019938+0000","last_peered":"2023-01-24T03:30:25.019938+0000","last_clean":"2023-01-24T03:30:25.019938+0000","last_became_active":"2023-01-24T03:30:25.011303+0000","last_became_peered":"2023-01-24T03:30:25.011303+0000","last_unstale":"2023-01-24T03:30:25.019938+0000","last_undegraded":"2023-01-24T03:30:25.019938+0000","last_fullsized":"2023-01-24T03:30:25.019938+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:14:43.169287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6,2],"acting":[5,4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992390+0000","last_change":"2023-01-24T03:30:20.602369+0000","last_active":"2023-01-24T03:30:24.992390+0000","last_peered":"2023-01-24T03:30:24.992390+0000","last_clean":"2023-01-24T03:30:24.992390+0000","last_became_active":"2023-01-24T03:30:20.601996+0000","last_became_peered":"2023-01-24T03:30:20.601996+0000","last_unstale":"2023-01-24T03:30:24.992390+0000","last_undegraded":"2023-01-24T03:30:24.992390+0000","last_fullsized":"2023-01-24T03:30:24.992390+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:33:55.328080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993109+0000","last_change":"2023-01-24T03:30:20.988738+0000","last_active":"2023-01-24T03:30:24.993109+0000","last_peered":"2023-01-24T03:30:24.993109+0000","last_clean":"2023-01-24T03:30:24.993109+0000","last_became_active":"2023-01-24T03:30:20.987190+0000","last_became_peered":"2023-01-24T03:30:20.987190+0000","last_unstale":"2023-01-24T03:30:24.993109+0000","last_undegraded":"2023-01-24T03:30:24.993109+0000","last_fullsized":"2023-01-24T03:30:24.993109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:32:05.689674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.013303+0000","last_change":"2023-01-24T03:30:25.008401+0000","last_active":"2023-01-24T03:30:25.013303+0000","last_peered":"2023-01-24T03:30:25.013303+0000","last_clean":"2023-01-24T03:30:25.013303+0000","last_became_active":"2023-01-24T03:30:25.007973+0000","last_became_peered":"2023-01-24T03:30:25.007973+0000","last_unstale":"2023-01-24T03:30:25.013303+0000","last_undegraded":"2023-01-24T03:30:25.013303+0000","last_fullsized":"2023-01-24T03:30:25.013303+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:36:22.339295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,2],"acting":[4,5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991180+0000","last_change":"2023-01-24T03:30:20.608145+0000","last_active":"2023-01-24T03:30:24.991180+0000","last_peered":"2023-01-24T03:30:24.991180+0000","last_clean":"2023-01-24T03:30:24.991180+0000","last_became_active":"2023-01-24T03:30:20.607165+0000","last_became_peered":"2023-01-24T03:30:20.607165+0000","last_unstale":"2023-01-24T03:30:24.991180+0000","last_undegraded":"2023-01-24T03:30:24.991180+0000","last_fullsized":"2023-01-24T03:30:24.991180+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:58:56.490482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002416+0000","last_change":"2023-01-24T03:30:20.998777+0000","last_active":"2023-01-24T03:30:25.002416+0000","last_peered":"2023-01-24T03:30:25.002416+0000","last_clean":"2023-01-24T03:30:25.002416+0000","last_became_active":"2023-01-24T03:30:20.998519+0000","last_became_peered":"2023-01-24T03:30:20.998519+0000","last_unstale":"2023-01-24T03:30:25.002416+0000","last_undegraded":"2023-01-24T03:30:25.002416+0000","last_fullsized":"2023-01-24T03:30:25.002416+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:55:55.320556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.012455+0000","last_change":"2023-01-24T03:30:25.008154+0000","last_active":"2023-01-24T03:30:25.012455+0000","last_peered":"2023-01-24T03:30:25.012455+0000","last_clean":"2023-01-24T03:30:25.012455+0000","last_became_active":"2023-01-24T03:30:25.007797+0000","last_became_peered":"2023-01-24T03:30:25.007797+0000","last_unstale":"2023-01-24T03:30:25.012455+0000","last_undegraded":"2023-01-24T03:30:25.012455+0000","last_fullsized":"2023-01-24T03:30:25.012455+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:29:05.153733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7,0],"acting":[4,3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"24'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985835+0000","last_change":"2023-01-24T03:30:20.610690+0000","last_active":"2023-01-24T03:30:25.985835+0000","last_peered":"2023-01-24T03:30:25.985835+0000","last_clean":"2023-01-24T03:30:25.985835+0000","last_became_active":"2023-01-24T03:30:20.610409+0000","last_became_peered":"2023-01-24T03:30:20.610409+0000","last_unstale":"2023-01-24T03:30:25.985835+0000","last_undegraded":"2023-01-24T03:30:25.985835+0000","last_fullsized":"2023-01-24T03:30:25.985835+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:43:37.740181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993033+0000","last_change":"2023-01-24T03:30:16.085885+0000","last_active":"2023-01-24T03:30:24.993033+0000","last_peered":"2023-01-24T03:30:24.993033+0000","last_clean":"2023-01-24T03:30:24.993033+0000","last_became_active":"2023-01-24T03:30:16.085525+0000","last_became_peered":"2023-01-24T03:30:16.085525+0000","last_unstale":"2023-01-24T03:30:24.993033+0000","last_undegraded":"2023-01-24T03:30:24.993033+0000","last_fullsized":"2023-01-24T03:30:24.993033+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:57:36.499644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991099+0000","last_change":"2023-01-24T03:30:20.996370+0000","last_active":"2023-01-24T03:30:24.991099+0000","last_peered":"2023-01-24T03:30:24.991099+0000","last_clean":"2023-01-24T03:30:24.991099+0000","last_became_active":"2023-01-24T03:30:20.996122+0000","last_became_peered":"2023-01-24T03:30:20.996122+0000","last_unstale":"2023-01-24T03:30:24.991099+0000","last_undegraded":"2023-01-24T03:30:24.991099+0000","last_fullsized":"2023-01-24T03:30:24.991099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:13:06.913474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.014170+0000","last_change":"2023-01-24T03:30:25.007262+0000","last_active":"2023-01-24T03:30:25.014170+0000","last_peered":"2023-01-24T03:30:25.014170+0000","last_clean":"2023-01-24T03:30:25.014170+0000","last_became_active":"2023-01-24T03:30:25.006862+0000","last_became_peered":"2023-01-24T03:30:25.006862+0000","last_unstale":"2023-01-24T03:30:25.014170+0000","last_undegraded":"2023-01-24T03:30:25.014170+0000","last_fullsized":"2023-01-24T03:30:25.014170+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:42:05.268862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,0],"acting":[3,1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991165+0000","last_change":"2023-01-24T03:30:16.057240+0000","last_active":"2023-01-24T03:30:24.991165+0000","last_peered":"2023-01-24T03:30:24.991165+0000","last_clean":"2023-01-24T03:30:24.991165+0000","last_became_active":"2023-01-24T03:30:16.056941+0000","last_became_peered":"2023-01-24T03:30:16.056941+0000","last_unstale":"2023-01-24T03:30:24.991165+0000","last_undegraded":"2023-01-24T03:30:24.991165+0000","last_fullsized":"2023-01-24T03:30:24.991165+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:22:26.168527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"24'1","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.447279+0000","last_change":"2023-01-24T03:30:20.606130+0000","last_active":"2023-01-24T03:30:25.447279+0000","last_peered":"2023-01-24T03:30:25.447279+0000","last_clean":"2023-01-24T03:30:25.447279+0000","last_became_active":"2023-01-24T03:30:20.605899+0000","last_became_peered":"2023-01-24T03:30:20.605899+0000","last_unstale":"2023-01-24T03:30:25.447279+0000","last_undegraded":"2023-01-24T03:30:25.447279+0000","last_fullsized":"2023-01-24T03:30:25.447279+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:30:47.956569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992411+0000","last_change":"2023-01-24T03:30:20.988906+0000","last_active":"2023-01-24T03:30:24.992411+0000","last_peered":"2023-01-24T03:30:24.992411+0000","last_clean":"2023-01-24T03:30:24.992411+0000","last_became_active":"2023-01-24T03:30:20.988688+0000","last_became_peered":"2023-01-24T03:30:20.988688+0000","last_unstale":"2023-01-24T03:30:24.992411+0000","last_undegraded":"2023-01-24T03:30:24.992411+0000","last_fullsized":"2023-01-24T03:30:24.992411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:25:43.365167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.019966+0000","last_change":"2023-01-24T03:30:25.013834+0000","last_active":"2023-01-24T03:30:25.019966+0000","last_peered":"2023-01-24T03:30:25.019966+0000","last_clean":"2023-01-24T03:30:25.019966+0000","last_became_active":"2023-01-24T03:30:25.011774+0000","last_became_peered":"2023-01-24T03:30:25.011774+0000","last_unstale":"2023-01-24T03:30:25.019966+0000","last_undegraded":"2023-01-24T03:30:25.019966+0000","last_fullsized":"2023-01-24T03:30:25.019966+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:02:50.158049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0,4],"acting":[5,1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002878+0000","last_change":"2023-01-24T03:30:16.051822+0000","last_active":"2023-01-24T03:30:25.002878+0000","last_peered":"2023-01-24T03:30:25.002878+0000","last_clean":"2023-01-24T03:30:25.002878+0000","last_became_active":"2023-01-24T03:30:16.051525+0000","last_became_peered":"2023-01-24T03:30:16.051525+0000","last_unstale":"2023-01-24T03:30:25.002878+0000","last_undegraded":"2023-01-24T03:30:25.002878+0000","last_fullsized":"2023-01-24T03:30:25.002878+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:37:30.802424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"16'117","reported_seq":166,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996317+0000","last_change":"2023-01-24T03:30:16.054907+0000","last_active":"2023-01-24T03:30:24.996317+0000","last_peered":"2023-01-24T03:30:24.996317+0000","last_clean":"2023-01-24T03:30:24.996317+0000","last_became_active":"2023-01-24T03:30:16.054071+0000","last_became_peered":"2023-01-24T03:30:16.054071+0000","last_unstale":"2023-01-24T03:30:24.996317+0000","last_undegraded":"2023-01-24T03:30:24.996317+0000","last_fullsized":"2023-01-24T03:30:24.996317+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:31:26.790766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992331+0000","last_change":"2023-01-24T03:30:20.607914+0000","last_active":"2023-01-24T03:30:24.992331+0000","last_peered":"2023-01-24T03:30:24.992331+0000","last_clean":"2023-01-24T03:30:24.992331+0000","last_became_active":"2023-01-24T03:30:20.607663+0000","last_became_peered":"2023-01-24T03:30:20.607663+0000","last_unstale":"2023-01-24T03:30:24.992331+0000","last_undegraded":"2023-01-24T03:30:24.992331+0000","last_fullsized":"2023-01-24T03:30:24.992331+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:32:24.271747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993825+0000","last_change":"2023-01-24T03:30:20.998105+0000","last_active":"2023-01-24T03:30:24.993825+0000","last_peered":"2023-01-24T03:30:24.993825+0000","last_clean":"2023-01-24T03:30:24.993825+0000","last_became_active":"2023-01-24T03:30:20.997871+0000","last_became_peered":"2023-01-24T03:30:20.997871+0000","last_unstale":"2023-01-24T03:30:24.993825+0000","last_undegraded":"2023-01-24T03:30:24.993825+0000","last_fullsized":"2023-01-24T03:30:24.993825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:47:42.740428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016190+0000","last_change":"2023-01-24T03:30:25.008079+0000","last_active":"2023-01-24T03:30:25.016190+0000","last_peered":"2023-01-24T03:30:25.016190+0000","last_clean":"2023-01-24T03:30:25.016190+0000","last_became_active":"2023-01-24T03:30:25.007596+0000","last_became_peered":"2023-01-24T03:30:25.007596+0000","last_unstale":"2023-01-24T03:30:25.016190+0000","last_undegraded":"2023-01-24T03:30:25.016190+0000","last_fullsized":"2023-01-24T03:30:25.016190+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:30:57.179885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1,7],"acting":[2,5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996872+0000","last_change":"2023-01-24T03:30:16.272445+0000","last_active":"2023-01-24T03:30:24.996872+0000","last_peered":"2023-01-24T03:30:24.996872+0000","last_clean":"2023-01-24T03:30:24.996872+0000","last_became_active":"2023-01-24T03:30:16.272184+0000","last_became_peered":"2023-01-24T03:30:16.272184+0000","last_unstale":"2023-01-24T03:30:24.996872+0000","last_undegraded":"2023-01-24T03:30:24.996872+0000","last_fullsized":"2023-01-24T03:30:24.996872+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:17:05.555068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002328+0000","last_change":"2023-01-24T03:30:20.612832+0000","last_active":"2023-01-24T03:30:25.002328+0000","last_peered":"2023-01-24T03:30:25.002328+0000","last_clean":"2023-01-24T03:30:25.002328+0000","last_became_active":"2023-01-24T03:30:20.612569+0000","last_became_peered":"2023-01-24T03:30:20.612569+0000","last_unstale":"2023-01-24T03:30:25.002328+0000","last_undegraded":"2023-01-24T03:30:25.002328+0000","last_fullsized":"2023-01-24T03:30:25.002328+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:37:40.534015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002330+0000","last_change":"2023-01-24T03:30:20.998171+0000","last_active":"2023-01-24T03:30:25.002330+0000","last_peered":"2023-01-24T03:30:25.002330+0000","last_clean":"2023-01-24T03:30:25.002330+0000","last_became_active":"2023-01-24T03:30:20.994887+0000","last_became_peered":"2023-01-24T03:30:20.994887+0000","last_unstale":"2023-01-24T03:30:25.002330+0000","last_undegraded":"2023-01-24T03:30:25.002330+0000","last_fullsized":"2023-01-24T03:30:25.002330+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:57:40.909930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.020081+0000","last_change":"2023-01-24T03:30:25.014948+0000","last_active":"2023-01-24T03:30:25.020081+0000","last_peered":"2023-01-24T03:30:25.020081+0000","last_clean":"2023-01-24T03:30:25.020081+0000","last_became_active":"2023-01-24T03:30:25.014572+0000","last_became_peered":"2023-01-24T03:30:25.014572+0000","last_unstale":"2023-01-24T03:30:25.020081+0000","last_undegraded":"2023-01-24T03:30:25.020081+0000","last_fullsized":"2023-01-24T03:30:25.020081+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:59:13.108074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5,3],"acting":[7,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996112+0000","last_change":"2023-01-24T03:30:16.272574+0000","last_active":"2023-01-24T03:30:24.996112+0000","last_peered":"2023-01-24T03:30:24.996112+0000","last_clean":"2023-01-24T03:30:24.996112+0000","last_became_active":"2023-01-24T03:30:16.272252+0000","last_became_peered":"2023-01-24T03:30:16.272252+0000","last_unstale":"2023-01-24T03:30:24.996112+0000","last_undegraded":"2023-01-24T03:30:24.996112+0000","last_fullsized":"2023-01-24T03:30:24.996112+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:12:08.393585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991181+0000","last_change":"2023-01-24T03:30:20.606878+0000","last_active":"2023-01-24T03:30:24.991181+0000","last_peered":"2023-01-24T03:30:24.991181+0000","last_clean":"2023-01-24T03:30:24.991181+0000","last_became_active":"2023-01-24T03:30:20.606442+0000","last_became_peered":"2023-01-24T03:30:20.606442+0000","last_unstale":"2023-01-24T03:30:24.991181+0000","last_undegraded":"2023-01-24T03:30:24.991181+0000","last_fullsized":"2023-01-24T03:30:24.991181+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:26:16.916080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990646+0000","last_change":"2023-01-24T03:30:20.990840+0000","last_active":"2023-01-24T03:30:24.990646+0000","last_peered":"2023-01-24T03:30:24.990646+0000","last_clean":"2023-01-24T03:30:24.990646+0000","last_became_active":"2023-01-24T03:30:20.990615+0000","last_became_peered":"2023-01-24T03:30:20.990615+0000","last_unstale":"2023-01-24T03:30:24.990646+0000","last_undegraded":"2023-01-24T03:30:24.990646+0000","last_fullsized":"2023-01-24T03:30:24.990646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T15:06:37.670127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.013106+0000","last_change":"2023-01-24T03:30:25.006375+0000","last_active":"2023-01-24T03:30:25.013106+0000","last_peered":"2023-01-24T03:30:25.013106+0000","last_clean":"2023-01-24T03:30:25.013106+0000","last_became_active":"2023-01-24T03:30:25.006027+0000","last_became_peered":"2023-01-24T03:30:25.006027+0000","last_unstale":"2023-01-24T03:30:25.013106+0000","last_undegraded":"2023-01-24T03:30:25.013106+0000","last_fullsized":"2023-01-24T03:30:25.013106+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:51:29.481545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,6],"acting":[0,1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005695+0000","last_change":"2023-01-24T03:30:16.272475+0000","last_active":"2023-01-24T03:30:25.005695+0000","last_peered":"2023-01-24T03:30:25.005695+0000","last_clean":"2023-01-24T03:30:25.005695+0000","last_became_active":"2023-01-24T03:30:16.272140+0000","last_became_peered":"2023-01-24T03:30:16.272140+0000","last_unstale":"2023-01-24T03:30:25.005695+0000","last_undegraded":"2023-01-24T03:30:25.005695+0000","last_fullsized":"2023-01-24T03:30:25.005695+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:48:16.300876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992670+0000","last_change":"2023-01-24T03:30:20.609391+0000","last_active":"2023-01-24T03:30:24.992670+0000","last_peered":"2023-01-24T03:30:24.992670+0000","last_clean":"2023-01-24T03:30:24.992670+0000","last_became_active":"2023-01-24T03:30:20.609124+0000","last_became_peered":"2023-01-24T03:30:20.609124+0000","last_unstale":"2023-01-24T03:30:24.992670+0000","last_undegraded":"2023-01-24T03:30:24.992670+0000","last_fullsized":"2023-01-24T03:30:24.992670+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:36:49.616649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.001859+0000","last_change":"2023-01-24T03:30:20.996985+0000","last_active":"2023-01-24T03:30:25.001859+0000","last_peered":"2023-01-24T03:30:25.001859+0000","last_clean":"2023-01-24T03:30:25.001859+0000","last_became_active":"2023-01-24T03:30:20.996612+0000","last_became_peered":"2023-01-24T03:30:20.996612+0000","last_unstale":"2023-01-24T03:30:25.001859+0000","last_undegraded":"2023-01-24T03:30:25.001859+0000","last_fullsized":"2023-01-24T03:30:25.001859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:10:50.378902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.012218+0000","last_change":"2023-01-24T03:30:25.008155+0000","last_active":"2023-01-24T03:30:25.012218+0000","last_peered":"2023-01-24T03:30:25.012218+0000","last_clean":"2023-01-24T03:30:25.012218+0000","last_became_active":"2023-01-24T03:30:25.007674+0000","last_became_peered":"2023-01-24T03:30:25.007674+0000","last_unstale":"2023-01-24T03:30:25.012218+0000","last_undegraded":"2023-01-24T03:30:25.012218+0000","last_fullsized":"2023-01-24T03:30:25.012218+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:55:46.448808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,2],"acting":[6,0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990827+0000","last_change":"2023-01-24T03:30:16.272372+0000","last_active":"2023-01-24T03:30:24.990827+0000","last_peered":"2023-01-24T03:30:24.990827+0000","last_clean":"2023-01-24T03:30:24.990827+0000","last_became_active":"2023-01-24T03:30:16.272104+0000","last_became_peered":"2023-01-24T03:30:16.272104+0000","last_unstale":"2023-01-24T03:30:24.990827+0000","last_undegraded":"2023-01-24T03:30:24.990827+0000","last_fullsized":"2023-01-24T03:30:24.990827+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:15:10.865596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002114+0000","last_change":"2023-01-24T03:30:20.612284+0000","last_active":"2023-01-24T03:30:25.002114+0000","last_peered":"2023-01-24T03:30:25.002114+0000","last_clean":"2023-01-24T03:30:25.002114+0000","last_became_active":"2023-01-24T03:30:20.612029+0000","last_became_peered":"2023-01-24T03:30:20.612029+0000","last_unstale":"2023-01-24T03:30:25.002114+0000","last_undegraded":"2023-01-24T03:30:25.002114+0000","last_fullsized":"2023-01-24T03:30:25.002114+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:18:58.755974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992643+0000","last_change":"2023-01-24T03:30:20.989834+0000","last_active":"2023-01-24T03:30:24.992643+0000","last_peered":"2023-01-24T03:30:24.992643+0000","last_clean":"2023-01-24T03:30:24.992643+0000","last_became_active":"2023-01-24T03:30:20.989645+0000","last_became_peered":"2023-01-24T03:30:20.989645+0000","last_unstale":"2023-01-24T03:30:24.992643+0000","last_undegraded":"2023-01-24T03:30:24.992643+0000","last_fullsized":"2023-01-24T03:30:24.992643+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:42:06.477043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016259+0000","last_change":"2023-01-24T03:30:25.011952+0000","last_active":"2023-01-24T03:30:25.016259+0000","last_peered":"2023-01-24T03:30:25.016259+0000","last_clean":"2023-01-24T03:30:25.016259+0000","last_became_active":"2023-01-24T03:30:25.011612+0000","last_became_peered":"2023-01-24T03:30:25.011612+0000","last_unstale":"2023-01-24T03:30:25.016259+0000","last_undegraded":"2023-01-24T03:30:25.016259+0000","last_fullsized":"2023-01-24T03:30:25.016259+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:51:52.027285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6,5],"acting":[0,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990404+0000","last_change":"2023-01-24T03:30:16.048979+0000","last_active":"2023-01-24T03:30:24.990404+0000","last_peered":"2023-01-24T03:30:24.990404+0000","last_clean":"2023-01-24T03:30:24.990404+0000","last_became_active":"2023-01-24T03:30:16.048641+0000","last_became_peered":"2023-01-24T03:30:16.048641+0000","last_unstale":"2023-01-24T03:30:24.990404+0000","last_undegraded":"2023-01-24T03:30:24.990404+0000","last_fullsized":"2023-01-24T03:30:24.990404+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T15:11:26.211024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005087+0000","last_change":"2023-01-24T03:30:20.606906+0000","last_active":"2023-01-24T03:30:25.005087+0000","last_peered":"2023-01-24T03:30:25.005087+0000","last_clean":"2023-01-24T03:30:25.005087+0000","last_became_active":"2023-01-24T03:30:20.606640+0000","last_became_peered":"2023-01-24T03:30:20.606640+0000","last_unstale":"2023-01-24T03:30:25.005087+0000","last_undegraded":"2023-01-24T03:30:25.005087+0000","last_fullsized":"2023-01-24T03:30:25.005087+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:21:09.008653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002357+0000","last_change":"2023-01-24T03:30:20.999886+0000","last_active":"2023-01-24T03:30:25.002357+0000","last_peered":"2023-01-24T03:30:25.002357+0000","last_clean":"2023-01-24T03:30:25.002357+0000","last_became_active":"2023-01-24T03:30:20.999394+0000","last_became_peered":"2023-01-24T03:30:20.999394+0000","last_unstale":"2023-01-24T03:30:25.002357+0000","last_undegraded":"2023-01-24T03:30:25.002357+0000","last_fullsized":"2023-01-24T03:30:25.002357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:30:22.395611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016356+0000","last_change":"2023-01-24T03:30:25.006190+0000","last_active":"2023-01-24T03:30:25.016356+0000","last_peered":"2023-01-24T03:30:25.016356+0000","last_clean":"2023-01-24T03:30:25.016356+0000","last_became_active":"2023-01-24T03:30:25.005825+0000","last_became_peered":"2023-01-24T03:30:25.005825+0000","last_unstale":"2023-01-24T03:30:25.016356+0000","last_undegraded":"2023-01-24T03:30:25.016356+0000","last_fullsized":"2023-01-24T03:30:25.016356+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:51:50.888938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,0],"acting":[1,3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"24'1","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.446961+0000","last_change":"2023-01-24T03:30:20.611079+0000","last_active":"2023-01-24T03:30:25.446961+0000","last_peered":"2023-01-24T03:30:25.446961+0000","last_clean":"2023-01-24T03:30:25.446961+0000","last_became_active":"2023-01-24T03:30:20.610841+0000","last_became_peered":"2023-01-24T03:30:20.610841+0000","last_unstale":"2023-01-24T03:30:25.446961+0000","last_undegraded":"2023-01-24T03:30:25.446961+0000","last_fullsized":"2023-01-24T03:30:25.446961+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:34:24.922661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.994693+0000","last_change":"2023-01-24T03:30:20.991418+0000","last_active":"2023-01-24T03:30:24.994693+0000","last_peered":"2023-01-24T03:30:24.994693+0000","last_clean":"2023-01-24T03:30:24.994693+0000","last_became_active":"2023-01-24T03:30:20.991167+0000","last_became_peered":"2023-01-24T03:30:20.991167+0000","last_unstale":"2023-01-24T03:30:24.994693+0000","last_undegraded":"2023-01-24T03:30:24.994693+0000","last_fullsized":"2023-01-24T03:30:24.994693+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:51:52.736691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.013519+0000","last_change":"2023-01-24T03:30:25.010394+0000","last_active":"2023-01-24T03:30:25.013519+0000","last_peered":"2023-01-24T03:30:25.013519+0000","last_clean":"2023-01-24T03:30:25.013519+0000","last_became_active":"2023-01-24T03:30:25.010044+0000","last_became_peered":"2023-01-24T03:30:25.010044+0000","last_unstale":"2023-01-24T03:30:25.013519+0000","last_undegraded":"2023-01-24T03:30:25.013519+0000","last_fullsized":"2023-01-24T03:30:25.013519+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:08:55.250184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1,6],"acting":[7,4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"24'2","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.984877+0000","last_change":"2023-01-24T03:30:20.607538+0000","last_active":"2023-01-24T03:30:25.984877+0000","last_peered":"2023-01-24T03:30:25.984877+0000","last_clean":"2023-01-24T03:30:25.984877+0000","last_became_active":"2023-01-24T03:30:20.607285+0000","last_became_peered":"2023-01-24T03:30:20.607285+0000","last_unstale":"2023-01-24T03:30:25.984877+0000","last_undegraded":"2023-01-24T03:30:25.984877+0000","last_fullsized":"2023-01-24T03:30:25.984877+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:42:21.836603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992857+0000","last_change":"2023-01-24T03:30:20.986791+0000","last_active":"2023-01-24T03:30:24.992857+0000","last_peered":"2023-01-24T03:30:24.992857+0000","last_clean":"2023-01-24T03:30:24.992857+0000","last_became_active":"2023-01-24T03:30:20.986516+0000","last_became_peered":"2023-01-24T03:30:20.986516+0000","last_unstale":"2023-01-24T03:30:24.992857+0000","last_undegraded":"2023-01-24T03:30:24.992857+0000","last_fullsized":"2023-01-24T03:30:24.992857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:53:19.191166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.015383+0000","last_change":"2023-01-24T03:30:25.010751+0000","last_active":"2023-01-24T03:30:25.015383+0000","last_peered":"2023-01-24T03:30:25.015383+0000","last_clean":"2023-01-24T03:30:25.015383+0000","last_became_active":"2023-01-24T03:30:25.010332+0000","last_became_peered":"2023-01-24T03:30:25.010332+0000","last_unstale":"2023-01-24T03:30:25.015383+0000","last_undegraded":"2023-01-24T03:30:25.015383+0000","last_fullsized":"2023-01-24T03:30:25.015383+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:48:30.190712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7,4],"acting":[5,0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"24'2","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985166+0000","last_change":"2023-01-24T03:30:20.608509+0000","last_active":"2023-01-24T03:30:25.985166+0000","last_peered":"2023-01-24T03:30:25.985166+0000","last_clean":"2023-01-24T03:30:25.985166+0000","last_became_active":"2023-01-24T03:30:20.608313+0000","last_became_peered":"2023-01-24T03:30:20.608313+0000","last_unstale":"2023-01-24T03:30:25.985166+0000","last_undegraded":"2023-01-24T03:30:25.985166+0000","last_fullsized":"2023-01-24T03:30:25.985166+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:31:52.253588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992448+0000","last_change":"2023-01-24T03:30:20.992334+0000","last_active":"2023-01-24T03:30:24.992448+0000","last_peered":"2023-01-24T03:30:24.992448+0000","last_clean":"2023-01-24T03:30:24.992448+0000","last_became_active":"2023-01-24T03:30:20.992027+0000","last_became_peered":"2023-01-24T03:30:20.992027+0000","last_unstale":"2023-01-24T03:30:24.992448+0000","last_undegraded":"2023-01-24T03:30:24.992448+0000","last_fullsized":"2023-01-24T03:30:24.992448+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:57:56.985507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.012177+0000","last_change":"2023-01-24T03:30:25.004167+0000","last_active":"2023-01-24T03:30:25.012177+0000","last_peered":"2023-01-24T03:30:25.012177+0000","last_clean":"2023-01-24T03:30:25.012177+0000","last_became_active":"2023-01-24T03:30:25.003752+0000","last_became_peered":"2023-01-24T03:30:25.003752+0000","last_unstale":"2023-01-24T03:30:25.012177+0000","last_undegraded":"2023-01-24T03:30:25.012177+0000","last_fullsized":"2023-01-24T03:30:25.012177+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:19:16.285529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"24'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.448123+0000","last_change":"2023-01-24T03:30:20.603542+0000","last_active":"2023-01-24T03:30:25.448123+0000","last_peered":"2023-01-24T03:30:25.448123+0000","last_clean":"2023-01-24T03:30:25.448123+0000","last_became_active":"2023-01-24T03:30:20.603287+0000","last_became_peered":"2023-01-24T03:30:20.603287+0000","last_unstale":"2023-01-24T03:30:25.448123+0000","last_undegraded":"2023-01-24T03:30:25.448123+0000","last_fullsized":"2023-01-24T03:30:25.448123+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:03:56.865763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991098+0000","last_change":"2023-01-24T03:30:20.989519+0000","last_active":"2023-01-24T03:30:24.991098+0000","last_peered":"2023-01-24T03:30:24.991098+0000","last_clean":"2023-01-24T03:30:24.991098+0000","last_became_active":"2023-01-24T03:30:20.989286+0000","last_became_peered":"2023-01-24T03:30:20.989286+0000","last_unstale":"2023-01-24T03:30:24.991098+0000","last_undegraded":"2023-01-24T03:30:24.991098+0000","last_fullsized":"2023-01-24T03:30:24.991098+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:17:30.829327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016619+0000","last_change":"2023-01-24T03:30:25.010262+0000","last_active":"2023-01-24T03:30:25.016619+0000","last_peered":"2023-01-24T03:30:25.016619+0000","last_clean":"2023-01-24T03:30:25.016619+0000","last_became_active":"2023-01-24T03:30:25.009930+0000","last_became_peered":"2023-01-24T03:30:25.009930+0000","last_unstale":"2023-01-24T03:30:25.016619+0000","last_undegraded":"2023-01-24T03:30:25.016619+0000","last_fullsized":"2023-01-24T03:30:25.016619+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:36:17.807046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7,2],"acting":[1,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"24'4","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985911+0000","last_change":"2023-01-24T03:30:20.607334+0000","last_active":"2023-01-24T03:30:25.985911+0000","last_peered":"2023-01-24T03:30:25.985911+0000","last_clean":"2023-01-24T03:30:25.985911+0000","last_became_active":"2023-01-24T03:30:20.607079+0000","last_became_peered":"2023-01-24T03:30:20.607079+0000","last_unstale":"2023-01-24T03:30:25.985911+0000","last_undegraded":"2023-01-24T03:30:25.985911+0000","last_fullsized":"2023-01-24T03:30:25.985911+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:05:18.886532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992554+0000","last_change":"2023-01-24T03:30:20.987567+0000","last_active":"2023-01-24T03:30:24.992554+0000","last_peered":"2023-01-24T03:30:24.992554+0000","last_clean":"2023-01-24T03:30:24.992554+0000","last_became_active":"2023-01-24T03:30:20.987058+0000","last_became_peered":"2023-01-24T03:30:20.987058+0000","last_unstale":"2023-01-24T03:30:24.992554+0000","last_undegraded":"2023-01-24T03:30:24.992554+0000","last_fullsized":"2023-01-24T03:30:24.992554+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:58:30.391733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.989538+0000","last_change":"2023-01-24T03:30:20.986393+0000","last_active":"2023-01-24T03:30:24.989538+0000","last_peered":"2023-01-24T03:30:24.989538+0000","last_clean":"2023-01-24T03:30:24.989538+0000","last_became_active":"2023-01-24T03:30:20.986089+0000","last_became_peered":"2023-01-24T03:30:20.986089+0000","last_unstale":"2023-01-24T03:30:24.989538+0000","last_undegraded":"2023-01-24T03:30:24.989538+0000","last_fullsized":"2023-01-24T03:30:24.989538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:44:21.992291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990902+0000","last_change":"2023-01-24T03:30:20.991468+0000","last_active":"2023-01-24T03:30:24.990902+0000","last_peered":"2023-01-24T03:30:24.990902+0000","last_clean":"2023-01-24T03:30:24.990902+0000","last_became_active":"2023-01-24T03:30:20.991227+0000","last_became_peered":"2023-01-24T03:30:20.991227+0000","last_unstale":"2023-01-24T03:30:24.990902+0000","last_undegraded":"2023-01-24T03:30:24.990902+0000","last_fullsized":"2023-01-24T03:30:24.990902+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:27:25.806340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992602+0000","last_change":"2023-01-24T03:30:20.989824+0000","last_active":"2023-01-24T03:30:24.992602+0000","last_peered":"2023-01-24T03:30:24.992602+0000","last_clean":"2023-01-24T03:30:24.992602+0000","last_became_active":"2023-01-24T03:30:20.989577+0000","last_became_peered":"2023-01-24T03:30:20.989577+0000","last_unstale":"2023-01-24T03:30:24.992602+0000","last_undegraded":"2023-01-24T03:30:24.992602+0000","last_fullsized":"2023-01-24T03:30:24.992602+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:48:17.948459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992452+0000","last_change":"2023-01-24T03:30:20.996266+0000","last_active":"2023-01-24T03:30:24.992452+0000","last_peered":"2023-01-24T03:30:24.992452+0000","last_clean":"2023-01-24T03:30:24.992452+0000","last_became_active":"2023-01-24T03:30:20.995610+0000","last_became_peered":"2023-01-24T03:30:20.995610+0000","last_unstale":"2023-01-24T03:30:24.992452+0000","last_undegraded":"2023-01-24T03:30:24.992452+0000","last_fullsized":"2023-01-24T03:30:24.992452+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:24:43.022530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.001943+0000","last_change":"2023-01-24T03:30:20.997081+0000","last_active":"2023-01-24T03:30:25.001943+0000","last_peered":"2023-01-24T03:30:25.001943+0000","last_clean":"2023-01-24T03:30:25.001943+0000","last_became_active":"2023-01-24T03:30:20.996837+0000","last_became_peered":"2023-01-24T03:30:20.996837+0000","last_unstale":"2023-01-24T03:30:25.001943+0000","last_undegraded":"2023-01-24T03:30:25.001943+0000","last_fullsized":"2023-01-24T03:30:25.001943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:42:06.134817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":4746,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64574,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9180,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":8832,"kb_avail":94362660,"statfs":{"total":96636764160,"available":96627363840,"internally_reserved":0,"allocated":348160,"data_stored":698731,"data_compressed":32287,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":9043968},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]}]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8576,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94363264,"statfs":{"total":96636764160,"available":96627982336,"internally_reserved":0,"allocated":253952,"data_stored":104434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9152,"kb_used_data":312,"kb_used_omap":0,"kb_used_meta":8832,"kb_avail":94362688,"statfs":{"total":96636764160,"available":96627392512,"internally_reserved":0,"allocated":319488,"data_stored":694802,"data_compressed":32287,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":9043968},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574859,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8640,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94363200,"statfs":{"total":96636764160,"available":96627916800,"internally_reserved":0,"allocated":253952,"data_stored":104382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]}]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8512,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94363328,"statfs":{"total":96636764160,"available":96628047872,"internally_reserved":0,"allocated":253952,"data_stored":104434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.254}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8580,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94363260,"statfs":{"total":96636764160,"available":96627978240,"internally_reserved":0,"allocated":258048,"data_stored":104524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574858,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8660,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94363180,"statfs":{"total":96636764160,"available":96627896320,"internally_reserved":0,"allocated":274432,"data_stored":108466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]}]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8580,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94363260,"statfs":{"total":96636764160,"available":96627978240,"internally_reserved":0,"allocated":258048,"data_stored":103705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.311}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32287,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32287,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-01-24T03:30:53.595 INFO:tasks.mds_thrash.ceph_manager:clean! 2023-01-24T03:30:53.595 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-24T03:30:53.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.738+0000 7fecc8287700 1 Processor -- start 2023-01-24T03:30:53.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.738+0000 7fecc8287700 1 -- start start 2023-01-24T03:30:53.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.738+0000 7fecc8287700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fecc00c7e40 0x7fecc00b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:53.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.738+0000 7fecc8287700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fecc0155da0 0x7fecc00ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:53.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.738+0000 7fecc8287700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecc0156240 con 0x7fecc00aef20 2023-01-24T03:30:53.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.738+0000 7fecc8287700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecc00af320 con 0x7fecc0155da0 2023-01-24T03:30:53.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.738+0000 7fecc8287700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecc00b1170 con 0x7fecc00c7e40 2023-01-24T03:30:53.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.739+0000 7fecc5822700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fecc0155da0 0x7fecc00ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:53.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.739+0000 7fecc6824700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fecc00aef20 0x7fecc00ca3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:60456/0 (socket says 172.21.15.112:60456) 2023-01-24T03:30:53.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.739+0000 7fecc6824700 1 -- 172.21.15.112:0/2126061045 learned_addr learned my addr 172.21.15.112:0/2126061045 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:53.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.739+0000 7fecc6023700 1 --2- 172.21.15.112:0/2126061045 >> v2:172.21.15.142:3300/0 conn(0x7fecc00c7e40 0x7fecc00b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:53.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.739+0000 7fecc5822700 1 -- 172.21.15.112:0/2126061045 >> v2:172.21.15.142:3300/0 conn(0x7fecc00c7e40 msgr2=0x7fecc00b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:53.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.739+0000 7fecc5822700 1 --2- 172.21.15.112:0/2126061045 >> v2:172.21.15.142:3300/0 conn(0x7fecc00c7e40 0x7fecc00b0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:53.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.739+0000 7fecc5822700 1 -- 172.21.15.112:0/2126061045 >> v1:172.21.15.112:6789/0 conn(0x7fecc00aef20 legacy=0x7fecc00ca3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:53.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.739+0000 7fecc5021700 1 -- 172.21.15.112:0/2126061045 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1625039906 0 0) 0x7fecc0156240 con 0x7fecc00aef20 2023-01-24T03:30:53.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.739+0000 7fecc5822700 1 -- 172.21.15.112:0/2126061045 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc00b1450 con 0x7fecc0155da0 2023-01-24T03:30:53.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.739+0000 7fecc6023700 1 --2- 172.21.15.112:0/2126061045 >> v2:172.21.15.142:3300/0 conn(0x7fecc00c7e40 0x7fecc00b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-01-24T03:30:53.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.739+0000 7fecc5822700 1 --2- 172.21.15.112:0/2126061045 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fecc0155da0 0x7fecc00ae9e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fecc0089cf0 tx=0x7fecb4009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23143d0bff1770c0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:53.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.739+0000 7fecc5021700 1 -- 172.21.15.112:0/2126061045 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb4011630 con 0x7fecc0155da0 2023-01-24T03:30:53.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.739+0000 7fecc5021700 1 -- 172.21.15.112:0/2126061045 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecb4011790 con 0x7fecc0155da0 2023-01-24T03:30:53.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.739+0000 7fecc5021700 1 -- 172.21.15.112:0/2126061045 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb4011b50 con 0x7fecc0155da0 2023-01-24T03:30:53.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.740+0000 7fecc8287700 1 -- 172.21.15.112:0/2126061045 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fecc0155da0 msgr2=0x7fecc00ae9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:53.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.740+0000 7fecc8287700 1 --2- 172.21.15.112:0/2126061045 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fecc0155da0 0x7fecc00ae9e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fecc0089cf0 tx=0x7fecb4009060 comp rx=0 tx=0).stop 2023-01-24T03:30:53.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.740+0000 7fecc8287700 1 -- 172.21.15.112:0/2126061045 shutdown_connections 2023-01-24T03:30:53.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.740+0000 7fecc8287700 1 --2- 172.21.15.112:0/2126061045 >> v2:172.21.15.142:3300/0 conn(0x7fecc00c7e40 0x7fecc00b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:53.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.740+0000 7fecc8287700 1 --2- 172.21.15.112:0/2126061045 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fecc0155da0 0x7fecc00ae9e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:53.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.740+0000 7fecc8287700 1 -- 172.21.15.112:0/2126061045 >> 172.21.15.112:0/2126061045 conn(0x7fecc005e2c0 msgr2=0x7fecc00cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:53.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.740+0000 7fecc8287700 1 -- 172.21.15.112:0/2126061045 shutdown_connections 2023-01-24T03:30:53.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.740+0000 7fecc8287700 1 -- 172.21.15.112:0/2126061045 wait complete. 2023-01-24T03:30:53.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.741+0000 7fecc8287700 1 Processor -- start 2023-01-24T03:30:53.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.741+0000 7fecc8287700 1 -- start start 2023-01-24T03:30:53.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.741+0000 7fecc8287700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fecc00aef20 0x7fecc0168ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:53.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.741+0000 7fecc8287700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fecc00c7e40 0x7fecc00c4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:53.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.741+0000 7fecc8287700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fecc0155da0 0x7fecc014bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:53.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.741+0000 7fecc8287700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc0168530 con 0x7fecc0155da0 2023-01-24T03:30:53.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.741+0000 7fecc8287700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc00c2ad0 con 0x7fecc00aef20 2023-01-24T03:30:53.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.741+0000 7fecc8287700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecc00c2db0 con 0x7fecc00c7e40 2023-01-24T03:30:53.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.742+0000 7fecc6824700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fecc0155da0 0x7fecc014bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:53.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.742+0000 7fecc6824700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fecc0155da0 0x7fecc014bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51244/0 (socket says 172.21.15.112:51244) 2023-01-24T03:30:53.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.742+0000 7fecc6824700 1 -- 172.21.15.112:0/1388816494 learned_addr learned my addr 172.21.15.112:0/1388816494 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:53.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.742+0000 7fecc6023700 1 --2- 172.21.15.112:0/1388816494 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fecc00aef20 0x7fecc0168ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:53.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.742+0000 7fecc5822700 1 --2- 172.21.15.112:0/1388816494 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fecc00c7e40 0x7fecc00c4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:53.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.742+0000 7fecc6824700 1 -- 172.21.15.112:0/1388816494 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fecc00c7e40 msgr2=0x7fecc00c4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:53.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.742+0000 7fecc6824700 1 --2- 172.21.15.112:0/1388816494 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fecc00c7e40 0x7fecc00c4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:53.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.742+0000 7fecc6824700 1 -- 172.21.15.112:0/1388816494 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fecc00aef20 msgr2=0x7fecc0168ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:53.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.742+0000 7fecc6824700 1 --2- 172.21.15.112:0/1388816494 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fecc00aef20 0x7fecc0168ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:53.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.742+0000 7fecc6824700 1 -- 172.21.15.112:0/1388816494 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecb40092a0 con 0x7fecc0155da0 2023-01-24T03:30:53.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.742+0000 7fecc6824700 1 --2- 172.21.15.112:0/1388816494 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fecc0155da0 0x7fecc014bca0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fecb800c460 tx=0x7fecb8007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:53.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.742+0000 7fecbe7fc700 1 -- 172.21.15.112:0/1388816494 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb80072e0 con 0x7fecc0155da0 2023-01-24T03:30:53.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.742+0000 7fecbe7fc700 1 -- 172.21.15.112:0/1388816494 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecb8007460 con 0x7fecc0155da0 2023-01-24T03:30:53.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.742+0000 7fecc8287700 1 -- 172.21.15.112:0/1388816494 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecc00c3090 con 0x7fecc0155da0 2023-01-24T03:30:53.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.742+0000 7fecbe7fc700 1 -- 172.21.15.112:0/1388816494 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb80059b0 con 0x7fecc0155da0 2023-01-24T03:30:53.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.742+0000 7fecc8287700 1 -- 172.21.15.112:0/1388816494 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecc0168780 con 0x7fecc0155da0 2023-01-24T03:30:53.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.743+0000 7fecc8287700 1 -- 172.21.15.112:0/1388816494 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecc01689a0 con 0x7fecc0155da0 2023-01-24T03:30:53.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.743+0000 7fecbe7fc700 1 -- 172.21.15.112:0/1388816494 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fecb8019030 con 0x7fecc0155da0 2023-01-24T03:30:53.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.744+0000 7fecbe7fc700 1 --2- 172.21.15.112:0/1388816494 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7feca00611c0 0x7feca0063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:53.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.744+0000 7fecbe7fc700 1 -- 172.21.15.112:0/1388816494 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fecb807ca50 con 0x7fecc0155da0 2023-01-24T03:30:53.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.744+0000 7fecc6023700 1 --2- 172.21.15.112:0/1388816494 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7feca00611c0 0x7feca0063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:53.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.744+0000 7fecc6023700 1 --2- 172.21.15.112:0/1388816494 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7feca00611c0 0x7feca0063670 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fecb000e4b0 tx=0x7fecb0007670 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:53.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.748+0000 7fecbe7fc700 1 -- 172.21.15.112:0/1388816494 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fecb804c890 con 0x7fecc0155da0 2023-01-24T03:30:53.924 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.922+0000 7fecc8287700 1 -- 172.21.15.112:0/1388816494 --> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fecc004bb80 con 0x7feca00611c0 2023-01-24T03:30:53.933 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.931+0000 7fecbe7fc700 1 -- 172.21.15.112:0/1388816494 <== mgr.4104 v2:172.21.15.112:6832/84276 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+365110 (secure 0 0 0) 0x7fecc004bb80 con 0x7feca00611c0 2023-01-24T03:30:53.937 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:30:53.940 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.935+0000 7fecc8287700 1 -- 172.21.15.112:0/1388816494 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7feca00611c0 msgr2=0x7feca0063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:53.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.935+0000 7fecc8287700 1 --2- 172.21.15.112:0/1388816494 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7feca00611c0 0x7feca0063670 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fecb000e4b0 tx=0x7fecb0007670 comp rx=0 tx=0).stop 2023-01-24T03:30:53.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.935+0000 7fecc8287700 1 -- 172.21.15.112:0/1388816494 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fecc0155da0 msgr2=0x7fecc014bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:53.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.935+0000 7fecc8287700 1 --2- 172.21.15.112:0/1388816494 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fecc0155da0 0x7fecc014bca0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fecb800c460 tx=0x7fecb8007e10 comp rx=0 tx=0).stop 2023-01-24T03:30:53.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.935+0000 7fecc8287700 1 -- 172.21.15.112:0/1388816494 shutdown_connections 2023-01-24T03:30:53.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.935+0000 7fecc8287700 1 --2- 172.21.15.112:0/1388816494 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7feca00611c0 0x7feca0063670 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:53.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.935+0000 7fecc8287700 1 --2- 172.21.15.112:0/1388816494 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fecc0155da0 0x7fecc014bca0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:53.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.935+0000 7fecc8287700 1 --2- 172.21.15.112:0/1388816494 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fecc00aef20 0x7fecc0168ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:53.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.935+0000 7fecc8287700 1 --2- 172.21.15.112:0/1388816494 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fecc00c7e40 0x7fecc00c4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:53.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.935+0000 7fecc8287700 1 -- 172.21.15.112:0/1388816494 >> 172.21.15.112:0/1388816494 conn(0x7fecc005e2c0 msgr2=0x7fecc0157f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:53.943 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.935+0000 7fecc8287700 1 -- 172.21.15.112:0/1388816494 shutdown_connections 2023-01-24T03:30:53.943 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:53.935+0000 7fecc8287700 1 -- 172.21.15.112:0/1388816494 wait complete. 2023-01-24T03:30:53.943 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2023-01-24T03:30:53.949 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2023-01-24T03:30:53.009421+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595114,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":209,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":338,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":338,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":69880,"kb_used_data":2168,"kb_used_omap":0,"kb_used_meta":67648,"kb_avail":754904840,"statfs":{"total":773094113280,"available":773022556160,"internally_reserved":0,"allocated":2220032,"data_stored":2023478,"data_compressed":64574,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":69271552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":263,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002903"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993289+0000","last_change":"2023-01-24T03:30:20.986308+0000","last_active":"2023-01-24T03:30:24.993289+0000","last_peered":"2023-01-24T03:30:24.993289+0000","last_clean":"2023-01-24T03:30:24.993289+0000","last_became_active":"2023-01-24T03:30:20.986025+0000","last_became_peered":"2023-01-24T03:30:20.986025+0000","last_unstale":"2023-01-24T03:30:24.993289+0000","last_undegraded":"2023-01-24T03:30:24.993289+0000","last_fullsized":"2023-01-24T03:30:24.993289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:47:58.616698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.006202+0000","last_change":"2023-01-24T03:30:20.994478+0000","last_active":"2023-01-24T03:30:25.006202+0000","last_peered":"2023-01-24T03:30:25.006202+0000","last_clean":"2023-01-24T03:30:25.006202+0000","last_became_active":"2023-01-24T03:30:20.994207+0000","last_became_peered":"2023-01-24T03:30:20.994207+0000","last_unstale":"2023-01-24T03:30:25.006202+0000","last_undegraded":"2023-01-24T03:30:25.006202+0000","last_fullsized":"2023-01-24T03:30:25.006202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:02:53.230231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993177+0000","last_change":"2023-01-24T03:30:20.989651+0000","last_active":"2023-01-24T03:30:24.993177+0000","last_peered":"2023-01-24T03:30:24.993177+0000","last_clean":"2023-01-24T03:30:24.993177+0000","last_became_active":"2023-01-24T03:30:20.987585+0000","last_became_peered":"2023-01-24T03:30:20.987585+0000","last_unstale":"2023-01-24T03:30:24.993177+0000","last_undegraded":"2023-01-24T03:30:24.993177+0000","last_fullsized":"2023-01-24T03:30:24.993177+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:36:02.177544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990511+0000","last_change":"2023-01-24T03:30:20.993601+0000","last_active":"2023-01-24T03:30:24.990511+0000","last_peered":"2023-01-24T03:30:24.990511+0000","last_clean":"2023-01-24T03:30:24.990511+0000","last_became_active":"2023-01-24T03:30:20.993418+0000","last_became_peered":"2023-01-24T03:30:20.993418+0000","last_unstale":"2023-01-24T03:30:24.990511+0000","last_undegraded":"2023-01-24T03:30:24.990511+0000","last_fullsized":"2023-01-24T03:30:24.990511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:47:50.260497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996430+0000","last_change":"2023-01-24T03:30:20.991605+0000","last_active":"2023-01-24T03:30:24.996430+0000","last_peered":"2023-01-24T03:30:24.996430+0000","last_clean":"2023-01-24T03:30:24.996430+0000","last_became_active":"2023-01-24T03:30:20.991361+0000","last_became_peered":"2023-01-24T03:30:20.991361+0000","last_unstale":"2023-01-24T03:30:24.996430+0000","last_undegraded":"2023-01-24T03:30:24.996430+0000","last_fullsized":"2023-01-24T03:30:24.996430+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:31:25.418430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993288+0000","last_change":"2023-01-24T03:30:20.985058+0000","last_active":"2023-01-24T03:30:24.993288+0000","last_peered":"2023-01-24T03:30:24.993288+0000","last_clean":"2023-01-24T03:30:24.993288+0000","last_became_active":"2023-01-24T03:30:20.984754+0000","last_became_peered":"2023-01-24T03:30:20.984754+0000","last_unstale":"2023-01-24T03:30:24.993288+0000","last_undegraded":"2023-01-24T03:30:24.993288+0000","last_fullsized":"2023-01-24T03:30:24.993288+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:12:03.225109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996279+0000","last_change":"2023-01-24T03:30:21.000890+0000","last_active":"2023-01-24T03:30:24.996279+0000","last_peered":"2023-01-24T03:30:24.996279+0000","last_clean":"2023-01-24T03:30:24.996279+0000","last_became_active":"2023-01-24T03:30:21.000438+0000","last_became_peered":"2023-01-24T03:30:21.000438+0000","last_unstale":"2023-01-24T03:30:24.996279+0000","last_undegraded":"2023-01-24T03:30:24.996279+0000","last_fullsized":"2023-01-24T03:30:24.996279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:17:12.999664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.004717+0000","last_change":"2023-01-24T03:30:20.986753+0000","last_active":"2023-01-24T03:30:25.004717+0000","last_peered":"2023-01-24T03:30:25.004717+0000","last_clean":"2023-01-24T03:30:25.004717+0000","last_became_active":"2023-01-24T03:30:20.986524+0000","last_became_peered":"2023-01-24T03:30:20.986524+0000","last_unstale":"2023-01-24T03:30:25.004717+0000","last_undegraded":"2023-01-24T03:30:25.004717+0000","last_fullsized":"2023-01-24T03:30:25.004717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:24:55.699379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992350+0000","last_change":"2023-01-24T03:30:20.991136+0000","last_active":"2023-01-24T03:30:24.992350+0000","last_peered":"2023-01-24T03:30:24.992350+0000","last_clean":"2023-01-24T03:30:24.992350+0000","last_became_active":"2023-01-24T03:30:20.990925+0000","last_became_peered":"2023-01-24T03:30:20.990925+0000","last_unstale":"2023-01-24T03:30:24.992350+0000","last_undegraded":"2023-01-24T03:30:24.992350+0000","last_fullsized":"2023-01-24T03:30:24.992350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:45:37.485199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.006401+0000","last_change":"2023-01-24T03:30:20.995730+0000","last_active":"2023-01-24T03:30:25.006401+0000","last_peered":"2023-01-24T03:30:25.006401+0000","last_clean":"2023-01-24T03:30:25.006401+0000","last_became_active":"2023-01-24T03:30:20.994880+0000","last_became_peered":"2023-01-24T03:30:20.994880+0000","last_unstale":"2023-01-24T03:30:25.006401+0000","last_undegraded":"2023-01-24T03:30:25.006401+0000","last_fullsized":"2023-01-24T03:30:25.006401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:07:21.546386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993144+0000","last_change":"2023-01-24T03:30:20.993328+0000","last_active":"2023-01-24T03:30:24.993144+0000","last_peered":"2023-01-24T03:30:24.993144+0000","last_clean":"2023-01-24T03:30:24.993144+0000","last_became_active":"2023-01-24T03:30:20.992730+0000","last_became_peered":"2023-01-24T03:30:20.992730+0000","last_unstale":"2023-01-24T03:30:24.993144+0000","last_undegraded":"2023-01-24T03:30:24.993144+0000","last_fullsized":"2023-01-24T03:30:24.993144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:43:39.048550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.994605+0000","last_change":"2023-01-24T03:30:20.984993+0000","last_active":"2023-01-24T03:30:24.994605+0000","last_peered":"2023-01-24T03:30:24.994605+0000","last_clean":"2023-01-24T03:30:24.994605+0000","last_became_active":"2023-01-24T03:30:20.984540+0000","last_became_peered":"2023-01-24T03:30:20.984540+0000","last_unstale":"2023-01-24T03:30:24.994605+0000","last_undegraded":"2023-01-24T03:30:24.994605+0000","last_fullsized":"2023-01-24T03:30:24.994605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:16:39.993359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.995936+0000","last_change":"2023-01-24T03:30:20.986567+0000","last_active":"2023-01-24T03:30:24.995936+0000","last_peered":"2023-01-24T03:30:24.995936+0000","last_clean":"2023-01-24T03:30:24.995936+0000","last_became_active":"2023-01-24T03:30:20.986224+0000","last_became_peered":"2023-01-24T03:30:20.986224+0000","last_unstale":"2023-01-24T03:30:24.995936+0000","last_undegraded":"2023-01-24T03:30:24.995936+0000","last_fullsized":"2023-01-24T03:30:24.995936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:09:14.692354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005087+0000","last_change":"2023-01-24T03:30:20.988843+0000","last_active":"2023-01-24T03:30:25.005087+0000","last_peered":"2023-01-24T03:30:25.005087+0000","last_clean":"2023-01-24T03:30:25.005087+0000","last_became_active":"2023-01-24T03:30:20.988486+0000","last_became_peered":"2023-01-24T03:30:20.988486+0000","last_unstale":"2023-01-24T03:30:25.005087+0000","last_undegraded":"2023-01-24T03:30:25.005087+0000","last_fullsized":"2023-01-24T03:30:25.005087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:18:20.538844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996110+0000","last_change":"2023-01-24T03:30:21.000738+0000","last_active":"2023-01-24T03:30:24.996110+0000","last_peered":"2023-01-24T03:30:24.996110+0000","last_clean":"2023-01-24T03:30:24.996110+0000","last_became_active":"2023-01-24T03:30:21.000458+0000","last_became_peered":"2023-01-24T03:30:21.000458+0000","last_unstale":"2023-01-24T03:30:24.996110+0000","last_undegraded":"2023-01-24T03:30:24.996110+0000","last_fullsized":"2023-01-24T03:30:24.996110+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:21:42.876426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992383+0000","last_change":"2023-01-24T03:30:20.985062+0000","last_active":"2023-01-24T03:30:24.992383+0000","last_peered":"2023-01-24T03:30:24.992383+0000","last_clean":"2023-01-24T03:30:24.992383+0000","last_became_active":"2023-01-24T03:30:20.984759+0000","last_became_peered":"2023-01-24T03:30:20.984759+0000","last_unstale":"2023-01-24T03:30:24.992383+0000","last_undegraded":"2023-01-24T03:30:24.992383+0000","last_fullsized":"2023-01-24T03:30:24.992383+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:38:34.615748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992439+0000","last_change":"2023-01-24T03:30:20.989080+0000","last_active":"2023-01-24T03:30:24.992439+0000","last_peered":"2023-01-24T03:30:24.992439+0000","last_clean":"2023-01-24T03:30:24.992439+0000","last_became_active":"2023-01-24T03:30:20.988823+0000","last_became_peered":"2023-01-24T03:30:20.988823+0000","last_unstale":"2023-01-24T03:30:24.992439+0000","last_undegraded":"2023-01-24T03:30:24.992439+0000","last_fullsized":"2023-01-24T03:30:24.992439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:22:21.807370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993008+0000","last_change":"2023-01-24T03:30:20.984222+0000","last_active":"2023-01-24T03:30:24.993008+0000","last_peered":"2023-01-24T03:30:24.993008+0000","last_clean":"2023-01-24T03:30:24.993008+0000","last_became_active":"2023-01-24T03:30:20.984002+0000","last_became_peered":"2023-01-24T03:30:20.984002+0000","last_unstale":"2023-01-24T03:30:24.993008+0000","last_undegraded":"2023-01-24T03:30:24.993008+0000","last_fullsized":"2023-01-24T03:30:24.993008+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:54:24.005486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.997598+0000","last_change":"2023-01-24T03:30:20.988197+0000","last_active":"2023-01-24T03:30:24.997598+0000","last_peered":"2023-01-24T03:30:24.997598+0000","last_clean":"2023-01-24T03:30:24.997598+0000","last_became_active":"2023-01-24T03:30:20.987941+0000","last_became_peered":"2023-01-24T03:30:20.987941+0000","last_unstale":"2023-01-24T03:30:24.997598+0000","last_undegraded":"2023-01-24T03:30:24.997598+0000","last_fullsized":"2023-01-24T03:30:24.997598+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T15:08:29.659964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992957+0000","last_change":"2023-01-24T03:30:20.988701+0000","last_active":"2023-01-24T03:30:24.992957+0000","last_peered":"2023-01-24T03:30:24.992957+0000","last_clean":"2023-01-24T03:30:24.992957+0000","last_became_active":"2023-01-24T03:30:20.988476+0000","last_became_peered":"2023-01-24T03:30:20.988476+0000","last_unstale":"2023-01-24T03:30:24.992957+0000","last_undegraded":"2023-01-24T03:30:24.992957+0000","last_fullsized":"2023-01-24T03:30:24.992957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:33:29.463194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996546+0000","last_change":"2023-01-24T03:30:21.001125+0000","last_active":"2023-01-24T03:30:24.996546+0000","last_peered":"2023-01-24T03:30:24.996546+0000","last_clean":"2023-01-24T03:30:24.996546+0000","last_became_active":"2023-01-24T03:30:21.000873+0000","last_became_peered":"2023-01-24T03:30:21.000873+0000","last_unstale":"2023-01-24T03:30:24.996546+0000","last_undegraded":"2023-01-24T03:30:24.996546+0000","last_fullsized":"2023-01-24T03:30:24.996546+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:22:16.136875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993432+0000","last_change":"2023-01-24T03:30:20.992639+0000","last_active":"2023-01-24T03:30:24.993432+0000","last_peered":"2023-01-24T03:30:24.993432+0000","last_clean":"2023-01-24T03:30:24.993432+0000","last_became_active":"2023-01-24T03:30:20.992386+0000","last_became_peered":"2023-01-24T03:30:20.992386+0000","last_unstale":"2023-01-24T03:30:24.993432+0000","last_undegraded":"2023-01-24T03:30:24.993432+0000","last_fullsized":"2023-01-24T03:30:24.993432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:29:17.912198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992608+0000","last_change":"2023-01-24T03:30:20.994066+0000","last_active":"2023-01-24T03:30:24.992608+0000","last_peered":"2023-01-24T03:30:24.992608+0000","last_clean":"2023-01-24T03:30:24.992608+0000","last_became_active":"2023-01-24T03:30:20.993820+0000","last_became_peered":"2023-01-24T03:30:20.993820+0000","last_unstale":"2023-01-24T03:30:24.992608+0000","last_undegraded":"2023-01-24T03:30:24.992608+0000","last_fullsized":"2023-01-24T03:30:24.992608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:31:23.874058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996188+0000","last_change":"2023-01-24T03:30:21.001652+0000","last_active":"2023-01-24T03:30:24.996188+0000","last_peered":"2023-01-24T03:30:24.996188+0000","last_clean":"2023-01-24T03:30:24.996188+0000","last_became_active":"2023-01-24T03:30:21.001400+0000","last_became_peered":"2023-01-24T03:30:21.001400+0000","last_unstale":"2023-01-24T03:30:24.996188+0000","last_undegraded":"2023-01-24T03:30:24.996188+0000","last_fullsized":"2023-01-24T03:30:24.996188+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:55:53.249816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002243+0000","last_change":"2023-01-24T03:30:20.998304+0000","last_active":"2023-01-24T03:30:25.002243+0000","last_peered":"2023-01-24T03:30:25.002243+0000","last_clean":"2023-01-24T03:30:25.002243+0000","last_became_active":"2023-01-24T03:30:20.996288+0000","last_became_peered":"2023-01-24T03:30:20.996288+0000","last_unstale":"2023-01-24T03:30:25.002243+0000","last_undegraded":"2023-01-24T03:30:25.002243+0000","last_fullsized":"2023-01-24T03:30:25.002243+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:11:08.120197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002638+0000","last_change":"2023-01-24T03:30:20.996489+0000","last_active":"2023-01-24T03:30:25.002638+0000","last_peered":"2023-01-24T03:30:25.002638+0000","last_clean":"2023-01-24T03:30:25.002638+0000","last_became_active":"2023-01-24T03:30:20.994371+0000","last_became_peered":"2023-01-24T03:30:20.994371+0000","last_unstale":"2023-01-24T03:30:25.002638+0000","last_undegraded":"2023-01-24T03:30:25.002638+0000","last_fullsized":"2023-01-24T03:30:25.002638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:52:21.369928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005744+0000","last_change":"2023-01-24T03:30:20.997753+0000","last_active":"2023-01-24T03:30:25.005744+0000","last_peered":"2023-01-24T03:30:25.005744+0000","last_clean":"2023-01-24T03:30:25.005744+0000","last_became_active":"2023-01-24T03:30:20.997475+0000","last_became_peered":"2023-01-24T03:30:20.997475+0000","last_unstale":"2023-01-24T03:30:25.005744+0000","last_undegraded":"2023-01-24T03:30:25.005744+0000","last_fullsized":"2023-01-24T03:30:25.005744+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:43:16.199843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.013267+0000","last_change":"2023-01-24T03:30:25.004944+0000","last_active":"2023-01-24T03:30:25.013267+0000","last_peered":"2023-01-24T03:30:25.013267+0000","last_clean":"2023-01-24T03:30:25.013267+0000","last_became_active":"2023-01-24T03:30:25.004590+0000","last_became_peered":"2023-01-24T03:30:25.004590+0000","last_unstale":"2023-01-24T03:30:25.013267+0000","last_undegraded":"2023-01-24T03:30:25.013267+0000","last_fullsized":"2023-01-24T03:30:25.013267+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:04:11.054225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2,0],"acting":[4,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"24'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.380450+0000","last_change":"2023-01-24T03:30:20.607211+0000","last_active":"2023-01-24T03:30:25.380450+0000","last_peered":"2023-01-24T03:30:25.380450+0000","last_clean":"2023-01-24T03:30:25.380450+0000","last_became_active":"2023-01-24T03:30:20.606855+0000","last_became_peered":"2023-01-24T03:30:20.606855+0000","last_unstale":"2023-01-24T03:30:25.380450+0000","last_undegraded":"2023-01-24T03:30:25.380450+0000","last_fullsized":"2023-01-24T03:30:25.380450+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:42:29.300089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991276+0000","last_change":"2023-01-24T03:30:20.993128+0000","last_active":"2023-01-24T03:30:24.991276+0000","last_peered":"2023-01-24T03:30:24.991276+0000","last_clean":"2023-01-24T03:30:24.991276+0000","last_became_active":"2023-01-24T03:30:20.992967+0000","last_became_peered":"2023-01-24T03:30:20.992967+0000","last_unstale":"2023-01-24T03:30:24.991276+0000","last_undegraded":"2023-01-24T03:30:24.991276+0000","last_fullsized":"2023-01-24T03:30:24.991276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:09:59.170218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.008814+0000","last_change":"2023-01-24T03:30:25.006569+0000","last_active":"2023-01-24T03:30:25.008814+0000","last_peered":"2023-01-24T03:30:25.008814+0000","last_clean":"2023-01-24T03:30:25.008814+0000","last_became_active":"2023-01-24T03:30:25.006199+0000","last_became_peered":"2023-01-24T03:30:25.006199+0000","last_unstale":"2023-01-24T03:30:25.008814+0000","last_undegraded":"2023-01-24T03:30:25.008814+0000","last_fullsized":"2023-01-24T03:30:25.008814+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:31:46.839485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3,7],"acting":[6,4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.001971+0000","last_change":"2023-01-24T03:30:20.612738+0000","last_active":"2023-01-24T03:30:25.001971+0000","last_peered":"2023-01-24T03:30:25.001971+0000","last_clean":"2023-01-24T03:30:25.001971+0000","last_became_active":"2023-01-24T03:30:20.612375+0000","last_became_peered":"2023-01-24T03:30:20.612375+0000","last_unstale":"2023-01-24T03:30:25.001971+0000","last_undegraded":"2023-01-24T03:30:25.001971+0000","last_fullsized":"2023-01-24T03:30:25.001971+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:17:43.251945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993741+0000","last_change":"2023-01-24T03:30:20.996368+0000","last_active":"2023-01-24T03:30:24.993741+0000","last_peered":"2023-01-24T03:30:24.993741+0000","last_clean":"2023-01-24T03:30:24.993741+0000","last_became_active":"2023-01-24T03:30:20.996051+0000","last_became_peered":"2023-01-24T03:30:20.996051+0000","last_unstale":"2023-01-24T03:30:24.993741+0000","last_undegraded":"2023-01-24T03:30:24.993741+0000","last_fullsized":"2023-01-24T03:30:24.993741+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:53:28.193862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.014921+0000","last_change":"2023-01-24T03:30:25.008528+0000","last_active":"2023-01-24T03:30:25.014921+0000","last_peered":"2023-01-24T03:30:25.014921+0000","last_clean":"2023-01-24T03:30:25.014921+0000","last_became_active":"2023-01-24T03:30:25.008146+0000","last_became_peered":"2023-01-24T03:30:25.008146+0000","last_unstale":"2023-01-24T03:30:25.014921+0000","last_undegraded":"2023-01-24T03:30:25.014921+0000","last_fullsized":"2023-01-24T03:30:25.014921+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:17:11.457359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1,2],"acting":[0,7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991341+0000","last_change":"2023-01-24T03:30:20.608932+0000","last_active":"2023-01-24T03:30:24.991341+0000","last_peered":"2023-01-24T03:30:24.991341+0000","last_clean":"2023-01-24T03:30:24.991341+0000","last_became_active":"2023-01-24T03:30:20.608135+0000","last_became_peered":"2023-01-24T03:30:20.608135+0000","last_unstale":"2023-01-24T03:30:24.991341+0000","last_undegraded":"2023-01-24T03:30:24.991341+0000","last_fullsized":"2023-01-24T03:30:24.991341+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:38:30.012279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990753+0000","last_change":"2023-01-24T03:30:20.986490+0000","last_active":"2023-01-24T03:30:24.990753+0000","last_peered":"2023-01-24T03:30:24.990753+0000","last_clean":"2023-01-24T03:30:24.990753+0000","last_became_active":"2023-01-24T03:30:20.986245+0000","last_became_peered":"2023-01-24T03:30:20.986245+0000","last_unstale":"2023-01-24T03:30:24.990753+0000","last_undegraded":"2023-01-24T03:30:24.990753+0000","last_fullsized":"2023-01-24T03:30:24.990753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:38:30.801032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.012186+0000","last_change":"2023-01-24T03:30:25.008221+0000","last_active":"2023-01-24T03:30:25.012186+0000","last_peered":"2023-01-24T03:30:25.012186+0000","last_clean":"2023-01-24T03:30:25.012186+0000","last_became_active":"2023-01-24T03:30:25.007855+0000","last_became_peered":"2023-01-24T03:30:25.007855+0000","last_unstale":"2023-01-24T03:30:25.012186+0000","last_undegraded":"2023-01-24T03:30:25.012186+0000","last_fullsized":"2023-01-24T03:30:25.012186+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:45:15.317067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7,1],"acting":[6,4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992445+0000","last_change":"2023-01-24T03:30:20.607385+0000","last_active":"2023-01-24T03:30:24.992445+0000","last_peered":"2023-01-24T03:30:24.992445+0000","last_clean":"2023-01-24T03:30:24.992445+0000","last_became_active":"2023-01-24T03:30:20.607123+0000","last_became_peered":"2023-01-24T03:30:20.607123+0000","last_unstale":"2023-01-24T03:30:24.992445+0000","last_undegraded":"2023-01-24T03:30:24.992445+0000","last_fullsized":"2023-01-24T03:30:24.992445+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T15:24:00.474758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991258+0000","last_change":"2023-01-24T03:30:20.994124+0000","last_active":"2023-01-24T03:30:24.991258+0000","last_peered":"2023-01-24T03:30:24.991258+0000","last_clean":"2023-01-24T03:30:24.991258+0000","last_became_active":"2023-01-24T03:30:20.993876+0000","last_became_peered":"2023-01-24T03:30:20.993876+0000","last_unstale":"2023-01-24T03:30:24.991258+0000","last_undegraded":"2023-01-24T03:30:24.991258+0000","last_fullsized":"2023-01-24T03:30:24.991258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:07:26.252741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.011535+0000","last_change":"2023-01-24T03:30:25.006312+0000","last_active":"2023-01-24T03:30:25.011535+0000","last_peered":"2023-01-24T03:30:25.011535+0000","last_clean":"2023-01-24T03:30:25.011535+0000","last_became_active":"2023-01-24T03:30:25.005946+0000","last_became_peered":"2023-01-24T03:30:25.005946+0000","last_unstale":"2023-01-24T03:30:25.011535+0000","last_undegraded":"2023-01-24T03:30:25.011535+0000","last_fullsized":"2023-01-24T03:30:25.011535+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:35:02.760190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6,0],"acting":[7,4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.006326+0000","last_change":"2023-01-24T03:30:20.610825+0000","last_active":"2023-01-24T03:30:25.006326+0000","last_peered":"2023-01-24T03:30:25.006326+0000","last_clean":"2023-01-24T03:30:25.006326+0000","last_became_active":"2023-01-24T03:30:20.610540+0000","last_became_peered":"2023-01-24T03:30:20.610540+0000","last_unstale":"2023-01-24T03:30:25.006326+0000","last_undegraded":"2023-01-24T03:30:25.006326+0000","last_fullsized":"2023-01-24T03:30:25.006326+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:23:15.279283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990818+0000","last_change":"2023-01-24T03:30:20.994605+0000","last_active":"2023-01-24T03:30:24.990818+0000","last_peered":"2023-01-24T03:30:24.990818+0000","last_clean":"2023-01-24T03:30:24.990818+0000","last_became_active":"2023-01-24T03:30:20.994356+0000","last_became_peered":"2023-01-24T03:30:20.994356+0000","last_unstale":"2023-01-24T03:30:24.990818+0000","last_undegraded":"2023-01-24T03:30:24.990818+0000","last_fullsized":"2023-01-24T03:30:24.990818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:06:08.797922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.018595+0000","last_change":"2023-01-24T03:30:25.012301+0000","last_active":"2023-01-24T03:30:25.018595+0000","last_peered":"2023-01-24T03:30:25.018595+0000","last_clean":"2023-01-24T03:30:25.018595+0000","last_became_active":"2023-01-24T03:30:25.011891+0000","last_became_peered":"2023-01-24T03:30:25.011891+0000","last_unstale":"2023-01-24T03:30:25.018595+0000","last_undegraded":"2023-01-24T03:30:25.018595+0000","last_fullsized":"2023-01-24T03:30:25.018595+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:15:46.012675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,5],"acting":[3,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"24'3","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.986137+0000","last_change":"2023-01-24T03:30:20.609828+0000","last_active":"2023-01-24T03:30:25.986137+0000","last_peered":"2023-01-24T03:30:25.986137+0000","last_clean":"2023-01-24T03:30:25.986137+0000","last_became_active":"2023-01-24T03:30:20.609492+0000","last_became_peered":"2023-01-24T03:30:20.609492+0000","last_unstale":"2023-01-24T03:30:25.986137+0000","last_undegraded":"2023-01-24T03:30:25.986137+0000","last_fullsized":"2023-01-24T03:30:25.986137+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:22:53.430711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002553+0000","last_change":"2023-01-24T03:30:20.999747+0000","last_active":"2023-01-24T03:30:25.002553+0000","last_peered":"2023-01-24T03:30:25.002553+0000","last_clean":"2023-01-24T03:30:25.002553+0000","last_became_active":"2023-01-24T03:30:20.999003+0000","last_became_peered":"2023-01-24T03:30:20.999003+0000","last_unstale":"2023-01-24T03:30:25.002553+0000","last_undegraded":"2023-01-24T03:30:25.002553+0000","last_fullsized":"2023-01-24T03:30:25.002553+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:19:22.205251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016573+0000","last_change":"2023-01-24T03:30:25.010659+0000","last_active":"2023-01-24T03:30:25.016573+0000","last_peered":"2023-01-24T03:30:25.016573+0000","last_clean":"2023-01-24T03:30:25.016573+0000","last_became_active":"2023-01-24T03:30:25.010237+0000","last_became_peered":"2023-01-24T03:30:25.010237+0000","last_unstale":"2023-01-24T03:30:25.016573+0000","last_undegraded":"2023-01-24T03:30:25.016573+0000","last_fullsized":"2023-01-24T03:30:25.016573+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:01:53.501070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3,7],"acting":[1,5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"24'11","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:48.616383+0000","last_change":"2023-01-24T03:30:20.610682+0000","last_active":"2023-01-24T03:30:48.616383+0000","last_peered":"2023-01-24T03:30:48.616383+0000","last_clean":"2023-01-24T03:30:48.616383+0000","last_became_active":"2023-01-24T03:30:20.610463+0000","last_became_peered":"2023-01-24T03:30:20.610463+0000","last_unstale":"2023-01-24T03:30:48.616383+0000","last_undegraded":"2023-01-24T03:30:48.616383+0000","last_fullsized":"2023-01-24T03:30:48.616383+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:25:00.447596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992845+0000","last_change":"2023-01-24T03:30:20.987326+0000","last_active":"2023-01-24T03:30:24.992845+0000","last_peered":"2023-01-24T03:30:24.992845+0000","last_clean":"2023-01-24T03:30:24.992845+0000","last_became_active":"2023-01-24T03:30:20.987042+0000","last_became_peered":"2023-01-24T03:30:20.987042+0000","last_unstale":"2023-01-24T03:30:24.992845+0000","last_undegraded":"2023-01-24T03:30:24.992845+0000","last_fullsized":"2023-01-24T03:30:24.992845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:57:43.257624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.020022+0000","last_change":"2023-01-24T03:30:25.012004+0000","last_active":"2023-01-24T03:30:25.020022+0000","last_peered":"2023-01-24T03:30:25.020022+0000","last_clean":"2023-01-24T03:30:25.020022+0000","last_became_active":"2023-01-24T03:30:25.007410+0000","last_became_peered":"2023-01-24T03:30:25.007410+0000","last_unstale":"2023-01-24T03:30:25.020022+0000","last_undegraded":"2023-01-24T03:30:25.020022+0000","last_fullsized":"2023-01-24T03:30:25.020022+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:34:21.681011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3,0],"acting":[5,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"24'2","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985201+0000","last_change":"2023-01-24T03:30:20.613026+0000","last_active":"2023-01-24T03:30:25.985201+0000","last_peered":"2023-01-24T03:30:25.985201+0000","last_clean":"2023-01-24T03:30:25.985201+0000","last_became_active":"2023-01-24T03:30:20.612605+0000","last_became_peered":"2023-01-24T03:30:20.612605+0000","last_unstale":"2023-01-24T03:30:25.985201+0000","last_undegraded":"2023-01-24T03:30:25.985201+0000","last_fullsized":"2023-01-24T03:30:25.985201+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:45:08.016740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991113+0000","last_change":"2023-01-24T03:30:20.991929+0000","last_active":"2023-01-24T03:30:24.991113+0000","last_peered":"2023-01-24T03:30:24.991113+0000","last_clean":"2023-01-24T03:30:24.991113+0000","last_became_active":"2023-01-24T03:30:20.991693+0000","last_became_peered":"2023-01-24T03:30:20.991693+0000","last_unstale":"2023-01-24T03:30:24.991113+0000","last_undegraded":"2023-01-24T03:30:24.991113+0000","last_fullsized":"2023-01-24T03:30:24.991113+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:55:51.637568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.018472+0000","last_change":"2023-01-24T03:30:25.010572+0000","last_active":"2023-01-24T03:30:25.018472+0000","last_peered":"2023-01-24T03:30:25.018472+0000","last_clean":"2023-01-24T03:30:25.018472+0000","last_became_active":"2023-01-24T03:30:25.010193+0000","last_became_peered":"2023-01-24T03:30:25.010193+0000","last_unstale":"2023-01-24T03:30:25.018472+0000","last_undegraded":"2023-01-24T03:30:25.018472+0000","last_fullsized":"2023-01-24T03:30:25.018472+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:25:39.676179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2,1],"acting":[3,7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993579+0000","last_change":"2023-01-24T03:30:20.612047+0000","last_active":"2023-01-24T03:30:24.993579+0000","last_peered":"2023-01-24T03:30:24.993579+0000","last_clean":"2023-01-24T03:30:24.993579+0000","last_became_active":"2023-01-24T03:30:20.611798+0000","last_became_peered":"2023-01-24T03:30:20.611798+0000","last_unstale":"2023-01-24T03:30:24.993579+0000","last_undegraded":"2023-01-24T03:30:24.993579+0000","last_fullsized":"2023-01-24T03:30:24.993579+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:01:41.731689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005194+0000","last_change":"2023-01-24T03:30:20.996432+0000","last_active":"2023-01-24T03:30:25.005194+0000","last_peered":"2023-01-24T03:30:25.005194+0000","last_clean":"2023-01-24T03:30:25.005194+0000","last_became_active":"2023-01-24T03:30:20.996173+0000","last_became_peered":"2023-01-24T03:30:20.996173+0000","last_unstale":"2023-01-24T03:30:25.005194+0000","last_undegraded":"2023-01-24T03:30:25.005194+0000","last_fullsized":"2023-01-24T03:30:25.005194+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:13:31.596587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.018507+0000","last_change":"2023-01-24T03:30:25.008479+0000","last_active":"2023-01-24T03:30:25.018507+0000","last_peered":"2023-01-24T03:30:25.018507+0000","last_clean":"2023-01-24T03:30:25.018507+0000","last_became_active":"2023-01-24T03:30:25.007858+0000","last_became_peered":"2023-01-24T03:30:25.007858+0000","last_unstale":"2023-01-24T03:30:25.018507+0000","last_undegraded":"2023-01-24T03:30:25.018507+0000","last_fullsized":"2023-01-24T03:30:25.018507+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:10:38.425874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,0],"acting":[3,1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996134+0000","last_change":"2023-01-24T03:30:20.612848+0000","last_active":"2023-01-24T03:30:24.996134+0000","last_peered":"2023-01-24T03:30:24.996134+0000","last_clean":"2023-01-24T03:30:24.996134+0000","last_became_active":"2023-01-24T03:30:20.612600+0000","last_became_peered":"2023-01-24T03:30:20.612600+0000","last_unstale":"2023-01-24T03:30:24.996134+0000","last_undegraded":"2023-01-24T03:30:24.996134+0000","last_fullsized":"2023-01-24T03:30:24.996134+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:46:45.634472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993883+0000","last_change":"2023-01-24T03:30:20.991133+0000","last_active":"2023-01-24T03:30:24.993883+0000","last_peered":"2023-01-24T03:30:24.993883+0000","last_clean":"2023-01-24T03:30:24.993883+0000","last_became_active":"2023-01-24T03:30:20.990856+0000","last_became_peered":"2023-01-24T03:30:20.990856+0000","last_unstale":"2023-01-24T03:30:24.993883+0000","last_undegraded":"2023-01-24T03:30:24.993883+0000","last_fullsized":"2023-01-24T03:30:24.993883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:27:32.163224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.022452+0000","last_change":"2023-01-24T03:30:25.017038+0000","last_active":"2023-01-24T03:30:25.022452+0000","last_peered":"2023-01-24T03:30:25.022452+0000","last_clean":"2023-01-24T03:30:25.022452+0000","last_became_active":"2023-01-24T03:30:25.016684+0000","last_became_peered":"2023-01-24T03:30:25.016684+0000","last_unstale":"2023-01-24T03:30:25.022452+0000","last_undegraded":"2023-01-24T03:30:25.022452+0000","last_fullsized":"2023-01-24T03:30:25.022452+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:57:30.028834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1,6],"acting":[5,3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"24'8","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.988793+0000","last_change":"2023-01-24T03:30:20.604617+0000","last_active":"2023-01-24T03:30:25.988793+0000","last_peered":"2023-01-24T03:30:25.988793+0000","last_clean":"2023-01-24T03:30:25.988793+0000","last_became_active":"2023-01-24T03:30:20.604445+0000","last_became_peered":"2023-01-24T03:30:20.604445+0000","last_unstale":"2023-01-24T03:30:25.988793+0000","last_undegraded":"2023-01-24T03:30:25.988793+0000","last_fullsized":"2023-01-24T03:30:25.988793+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:06:51.460895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993017+0000","last_change":"2023-01-24T03:30:20.995872+0000","last_active":"2023-01-24T03:30:24.993017+0000","last_peered":"2023-01-24T03:30:24.993017+0000","last_clean":"2023-01-24T03:30:24.993017+0000","last_became_active":"2023-01-24T03:30:20.995599+0000","last_became_peered":"2023-01-24T03:30:20.995599+0000","last_unstale":"2023-01-24T03:30:24.993017+0000","last_undegraded":"2023-01-24T03:30:24.993017+0000","last_fullsized":"2023-01-24T03:30:24.993017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:53:54.356899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.020130+0000","last_change":"2023-01-24T03:30:25.016101+0000","last_active":"2023-01-24T03:30:25.020130+0000","last_peered":"2023-01-24T03:30:25.020130+0000","last_clean":"2023-01-24T03:30:25.020130+0000","last_became_active":"2023-01-24T03:30:25.015705+0000","last_became_peered":"2023-01-24T03:30:25.015705+0000","last_unstale":"2023-01-24T03:30:25.020130+0000","last_undegraded":"2023-01-24T03:30:25.020130+0000","last_fullsized":"2023-01-24T03:30:25.020130+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:54:37.798191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3,4],"acting":[7,6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"24'1","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.446876+0000","last_change":"2023-01-24T03:30:20.610632+0000","last_active":"2023-01-24T03:30:25.446876+0000","last_peered":"2023-01-24T03:30:25.446876+0000","last_clean":"2023-01-24T03:30:25.446876+0000","last_became_active":"2023-01-24T03:30:20.610367+0000","last_became_peered":"2023-01-24T03:30:20.610367+0000","last_unstale":"2023-01-24T03:30:25.446876+0000","last_undegraded":"2023-01-24T03:30:25.446876+0000","last_fullsized":"2023-01-24T03:30:25.446876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:26:05.938178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.004623+0000","last_change":"2023-01-24T03:30:20.996493+0000","last_active":"2023-01-24T03:30:25.004623+0000","last_peered":"2023-01-24T03:30:25.004623+0000","last_clean":"2023-01-24T03:30:25.004623+0000","last_became_active":"2023-01-24T03:30:20.996229+0000","last_became_peered":"2023-01-24T03:30:20.996229+0000","last_unstale":"2023-01-24T03:30:25.004623+0000","last_undegraded":"2023-01-24T03:30:25.004623+0000","last_fullsized":"2023-01-24T03:30:25.004623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:23:53.642522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016195+0000","last_change":"2023-01-24T03:30:25.009215+0000","last_active":"2023-01-24T03:30:25.016195+0000","last_peered":"2023-01-24T03:30:25.016195+0000","last_clean":"2023-01-24T03:30:25.016195+0000","last_became_active":"2023-01-24T03:30:25.008213+0000","last_became_peered":"2023-01-24T03:30:25.008213+0000","last_unstale":"2023-01-24T03:30:25.016195+0000","last_undegraded":"2023-01-24T03:30:25.016195+0000","last_fullsized":"2023-01-24T03:30:25.016195+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:48:37.517172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1,5],"acting":[2,0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"24'2","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985223+0000","last_change":"2023-01-24T03:30:20.612158+0000","last_active":"2023-01-24T03:30:25.985223+0000","last_peered":"2023-01-24T03:30:25.985223+0000","last_clean":"2023-01-24T03:30:25.985223+0000","last_became_active":"2023-01-24T03:30:20.611931+0000","last_became_peered":"2023-01-24T03:30:20.611931+0000","last_unstale":"2023-01-24T03:30:25.985223+0000","last_undegraded":"2023-01-24T03:30:25.985223+0000","last_fullsized":"2023-01-24T03:30:25.985223+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:30:37.532390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992500+0000","last_change":"2023-01-24T03:30:20.991582+0000","last_active":"2023-01-24T03:30:24.992500+0000","last_peered":"2023-01-24T03:30:24.992500+0000","last_clean":"2023-01-24T03:30:24.992500+0000","last_became_active":"2023-01-24T03:30:20.991335+0000","last_became_peered":"2023-01-24T03:30:20.991335+0000","last_unstale":"2023-01-24T03:30:24.992500+0000","last_undegraded":"2023-01-24T03:30:24.992500+0000","last_fullsized":"2023-01-24T03:30:24.992500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:45:20.786712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016224+0000","last_change":"2023-01-24T03:30:25.008058+0000","last_active":"2023-01-24T03:30:25.016224+0000","last_peered":"2023-01-24T03:30:25.016224+0000","last_clean":"2023-01-24T03:30:25.016224+0000","last_became_active":"2023-01-24T03:30:25.007711+0000","last_became_peered":"2023-01-24T03:30:25.007711+0000","last_unstale":"2023-01-24T03:30:25.016224+0000","last_undegraded":"2023-01-24T03:30:25.016224+0000","last_fullsized":"2023-01-24T03:30:25.016224+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:26:56.270674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5,6],"acting":[2,0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"24'1","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.447163+0000","last_change":"2023-01-24T03:30:20.610495+0000","last_active":"2023-01-24T03:30:25.447163+0000","last_peered":"2023-01-24T03:30:25.447163+0000","last_clean":"2023-01-24T03:30:25.447163+0000","last_became_active":"2023-01-24T03:30:20.610035+0000","last_became_peered":"2023-01-24T03:30:20.610035+0000","last_unstale":"2023-01-24T03:30:25.447163+0000","last_undegraded":"2023-01-24T03:30:25.447163+0000","last_fullsized":"2023-01-24T03:30:25.447163+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:25:30.944275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.989631+0000","last_change":"2023-01-24T03:30:20.994362+0000","last_active":"2023-01-24T03:30:24.989631+0000","last_peered":"2023-01-24T03:30:24.989631+0000","last_clean":"2023-01-24T03:30:24.989631+0000","last_became_active":"2023-01-24T03:30:20.994101+0000","last_became_peered":"2023-01-24T03:30:20.994101+0000","last_unstale":"2023-01-24T03:30:24.989631+0000","last_undegraded":"2023-01-24T03:30:24.989631+0000","last_fullsized":"2023-01-24T03:30:24.989631+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:55:12.837646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.015999+0000","last_change":"2023-01-24T03:30:25.006995+0000","last_active":"2023-01-24T03:30:25.015999+0000","last_peered":"2023-01-24T03:30:25.015999+0000","last_clean":"2023-01-24T03:30:25.015999+0000","last_became_active":"2023-01-24T03:30:25.006473+0000","last_became_peered":"2023-01-24T03:30:25.006473+0000","last_unstale":"2023-01-24T03:30:25.015999+0000","last_undegraded":"2023-01-24T03:30:25.015999+0000","last_fullsized":"2023-01-24T03:30:25.015999+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:48:16.320162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3,6],"acting":[2,4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005342+0000","last_change":"2023-01-24T03:30:20.606526+0000","last_active":"2023-01-24T03:30:25.005342+0000","last_peered":"2023-01-24T03:30:25.005342+0000","last_clean":"2023-01-24T03:30:25.005342+0000","last_became_active":"2023-01-24T03:30:20.606301+0000","last_became_peered":"2023-01-24T03:30:20.606301+0000","last_unstale":"2023-01-24T03:30:25.005342+0000","last_undegraded":"2023-01-24T03:30:25.005342+0000","last_fullsized":"2023-01-24T03:30:25.005342+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:43:28.702475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992428+0000","last_change":"2023-01-24T03:30:20.993379+0000","last_active":"2023-01-24T03:30:24.992428+0000","last_peered":"2023-01-24T03:30:24.992428+0000","last_clean":"2023-01-24T03:30:24.992428+0000","last_became_active":"2023-01-24T03:30:20.993151+0000","last_became_peered":"2023-01-24T03:30:20.993151+0000","last_unstale":"2023-01-24T03:30:24.992428+0000","last_undegraded":"2023-01-24T03:30:24.992428+0000","last_fullsized":"2023-01-24T03:30:24.992428+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:00:45.845082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.020912+0000","last_change":"2023-01-24T03:30:25.013701+0000","last_active":"2023-01-24T03:30:25.020912+0000","last_peered":"2023-01-24T03:30:25.020912+0000","last_clean":"2023-01-24T03:30:25.020912+0000","last_became_active":"2023-01-24T03:30:25.013290+0000","last_became_peered":"2023-01-24T03:30:25.013290+0000","last_unstale":"2023-01-24T03:30:25.020912+0000","last_undegraded":"2023-01-24T03:30:25.020912+0000","last_fullsized":"2023-01-24T03:30:25.020912+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:32:17.340583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5,6],"acting":[2,7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"24'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985860+0000","last_change":"2023-01-24T03:30:20.609656+0000","last_active":"2023-01-24T03:30:25.985860+0000","last_peered":"2023-01-24T03:30:25.985860+0000","last_clean":"2023-01-24T03:30:25.985860+0000","last_became_active":"2023-01-24T03:30:20.609374+0000","last_became_peered":"2023-01-24T03:30:20.609374+0000","last_unstale":"2023-01-24T03:30:25.985860+0000","last_undegraded":"2023-01-24T03:30:25.985860+0000","last_fullsized":"2023-01-24T03:30:25.985860+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:58:12.653746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992726+0000","last_change":"2023-01-24T03:30:20.986089+0000","last_active":"2023-01-24T03:30:24.992726+0000","last_peered":"2023-01-24T03:30:24.992726+0000","last_clean":"2023-01-24T03:30:24.992726+0000","last_became_active":"2023-01-24T03:30:20.985857+0000","last_became_peered":"2023-01-24T03:30:20.985857+0000","last_unstale":"2023-01-24T03:30:24.992726+0000","last_undegraded":"2023-01-24T03:30:24.992726+0000","last_fullsized":"2023-01-24T03:30:24.992726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:00:57.338328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016603+0000","last_change":"2023-01-24T03:30:25.009962+0000","last_active":"2023-01-24T03:30:25.016603+0000","last_peered":"2023-01-24T03:30:25.016603+0000","last_clean":"2023-01-24T03:30:25.016603+0000","last_became_active":"2023-01-24T03:30:25.009623+0000","last_became_peered":"2023-01-24T03:30:25.009623+0000","last_unstale":"2023-01-24T03:30:25.016603+0000","last_undegraded":"2023-01-24T03:30:25.016603+0000","last_fullsized":"2023-01-24T03:30:25.016603+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:24:42.958531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0,6],"acting":[1,4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.989699+0000","last_change":"2023-01-24T03:30:20.608114+0000","last_active":"2023-01-24T03:30:24.989699+0000","last_peered":"2023-01-24T03:30:24.989699+0000","last_clean":"2023-01-24T03:30:24.989699+0000","last_became_active":"2023-01-24T03:30:20.607816+0000","last_became_peered":"2023-01-24T03:30:20.607816+0000","last_unstale":"2023-01-24T03:30:24.989699+0000","last_undegraded":"2023-01-24T03:30:24.989699+0000","last_fullsized":"2023-01-24T03:30:24.989699+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:40:00.784496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992908+0000","last_change":"2023-01-24T03:30:20.982589+0000","last_active":"2023-01-24T03:30:24.992908+0000","last_peered":"2023-01-24T03:30:24.992908+0000","last_clean":"2023-01-24T03:30:24.992908+0000","last_became_active":"2023-01-24T03:30:20.979779+0000","last_became_peered":"2023-01-24T03:30:20.979779+0000","last_unstale":"2023-01-24T03:30:24.992908+0000","last_undegraded":"2023-01-24T03:30:24.992908+0000","last_fullsized":"2023-01-24T03:30:24.992908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:01:18.491363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.019938+0000","last_change":"2023-01-24T03:30:25.013327+0000","last_active":"2023-01-24T03:30:25.019938+0000","last_peered":"2023-01-24T03:30:25.019938+0000","last_clean":"2023-01-24T03:30:25.019938+0000","last_became_active":"2023-01-24T03:30:25.011303+0000","last_became_peered":"2023-01-24T03:30:25.011303+0000","last_unstale":"2023-01-24T03:30:25.019938+0000","last_undegraded":"2023-01-24T03:30:25.019938+0000","last_fullsized":"2023-01-24T03:30:25.019938+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:14:43.169287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6,2],"acting":[5,4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992390+0000","last_change":"2023-01-24T03:30:20.602369+0000","last_active":"2023-01-24T03:30:24.992390+0000","last_peered":"2023-01-24T03:30:24.992390+0000","last_clean":"2023-01-24T03:30:24.992390+0000","last_became_active":"2023-01-24T03:30:20.601996+0000","last_became_peered":"2023-01-24T03:30:20.601996+0000","last_unstale":"2023-01-24T03:30:24.992390+0000","last_undegraded":"2023-01-24T03:30:24.992390+0000","last_fullsized":"2023-01-24T03:30:24.992390+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:33:55.328080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993109+0000","last_change":"2023-01-24T03:30:20.988738+0000","last_active":"2023-01-24T03:30:24.993109+0000","last_peered":"2023-01-24T03:30:24.993109+0000","last_clean":"2023-01-24T03:30:24.993109+0000","last_became_active":"2023-01-24T03:30:20.987190+0000","last_became_peered":"2023-01-24T03:30:20.987190+0000","last_unstale":"2023-01-24T03:30:24.993109+0000","last_undegraded":"2023-01-24T03:30:24.993109+0000","last_fullsized":"2023-01-24T03:30:24.993109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:32:05.689674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.013303+0000","last_change":"2023-01-24T03:30:25.008401+0000","last_active":"2023-01-24T03:30:25.013303+0000","last_peered":"2023-01-24T03:30:25.013303+0000","last_clean":"2023-01-24T03:30:25.013303+0000","last_became_active":"2023-01-24T03:30:25.007973+0000","last_became_peered":"2023-01-24T03:30:25.007973+0000","last_unstale":"2023-01-24T03:30:25.013303+0000","last_undegraded":"2023-01-24T03:30:25.013303+0000","last_fullsized":"2023-01-24T03:30:25.013303+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:36:22.339295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,2],"acting":[4,5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991180+0000","last_change":"2023-01-24T03:30:20.608145+0000","last_active":"2023-01-24T03:30:24.991180+0000","last_peered":"2023-01-24T03:30:24.991180+0000","last_clean":"2023-01-24T03:30:24.991180+0000","last_became_active":"2023-01-24T03:30:20.607165+0000","last_became_peered":"2023-01-24T03:30:20.607165+0000","last_unstale":"2023-01-24T03:30:24.991180+0000","last_undegraded":"2023-01-24T03:30:24.991180+0000","last_fullsized":"2023-01-24T03:30:24.991180+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:58:56.490482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002416+0000","last_change":"2023-01-24T03:30:20.998777+0000","last_active":"2023-01-24T03:30:25.002416+0000","last_peered":"2023-01-24T03:30:25.002416+0000","last_clean":"2023-01-24T03:30:25.002416+0000","last_became_active":"2023-01-24T03:30:20.998519+0000","last_became_peered":"2023-01-24T03:30:20.998519+0000","last_unstale":"2023-01-24T03:30:25.002416+0000","last_undegraded":"2023-01-24T03:30:25.002416+0000","last_fullsized":"2023-01-24T03:30:25.002416+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:55:55.320556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.012455+0000","last_change":"2023-01-24T03:30:25.008154+0000","last_active":"2023-01-24T03:30:25.012455+0000","last_peered":"2023-01-24T03:30:25.012455+0000","last_clean":"2023-01-24T03:30:25.012455+0000","last_became_active":"2023-01-24T03:30:25.007797+0000","last_became_peered":"2023-01-24T03:30:25.007797+0000","last_unstale":"2023-01-24T03:30:25.012455+0000","last_undegraded":"2023-01-24T03:30:25.012455+0000","last_fullsized":"2023-01-24T03:30:25.012455+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:29:05.153733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7,0],"acting":[4,3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"24'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985835+0000","last_change":"2023-01-24T03:30:20.610690+0000","last_active":"2023-01-24T03:30:25.985835+0000","last_peered":"2023-01-24T03:30:25.985835+0000","last_clean":"2023-01-24T03:30:25.985835+0000","last_became_active":"2023-01-24T03:30:20.610409+0000","last_became_peered":"2023-01-24T03:30:20.610409+0000","last_unstale":"2023-01-24T03:30:25.985835+0000","last_undegraded":"2023-01-24T03:30:25.985835+0000","last_fullsized":"2023-01-24T03:30:25.985835+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:43:37.740181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993033+0000","last_change":"2023-01-24T03:30:16.085885+0000","last_active":"2023-01-24T03:30:24.993033+0000","last_peered":"2023-01-24T03:30:24.993033+0000","last_clean":"2023-01-24T03:30:24.993033+0000","last_became_active":"2023-01-24T03:30:16.085525+0000","last_became_peered":"2023-01-24T03:30:16.085525+0000","last_unstale":"2023-01-24T03:30:24.993033+0000","last_undegraded":"2023-01-24T03:30:24.993033+0000","last_fullsized":"2023-01-24T03:30:24.993033+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:57:36.499644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991099+0000","last_change":"2023-01-24T03:30:20.996370+0000","last_active":"2023-01-24T03:30:24.991099+0000","last_peered":"2023-01-24T03:30:24.991099+0000","last_clean":"2023-01-24T03:30:24.991099+0000","last_became_active":"2023-01-24T03:30:20.996122+0000","last_became_peered":"2023-01-24T03:30:20.996122+0000","last_unstale":"2023-01-24T03:30:24.991099+0000","last_undegraded":"2023-01-24T03:30:24.991099+0000","last_fullsized":"2023-01-24T03:30:24.991099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:13:06.913474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.014170+0000","last_change":"2023-01-24T03:30:25.007262+0000","last_active":"2023-01-24T03:30:25.014170+0000","last_peered":"2023-01-24T03:30:25.014170+0000","last_clean":"2023-01-24T03:30:25.014170+0000","last_became_active":"2023-01-24T03:30:25.006862+0000","last_became_peered":"2023-01-24T03:30:25.006862+0000","last_unstale":"2023-01-24T03:30:25.014170+0000","last_undegraded":"2023-01-24T03:30:25.014170+0000","last_fullsized":"2023-01-24T03:30:25.014170+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:42:05.268862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,0],"acting":[3,1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991165+0000","last_change":"2023-01-24T03:30:16.057240+0000","last_active":"2023-01-24T03:30:24.991165+0000","last_peered":"2023-01-24T03:30:24.991165+0000","last_clean":"2023-01-24T03:30:24.991165+0000","last_became_active":"2023-01-24T03:30:16.056941+0000","last_became_peered":"2023-01-24T03:30:16.056941+0000","last_unstale":"2023-01-24T03:30:24.991165+0000","last_undegraded":"2023-01-24T03:30:24.991165+0000","last_fullsized":"2023-01-24T03:30:24.991165+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:22:26.168527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"24'1","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.447279+0000","last_change":"2023-01-24T03:30:20.606130+0000","last_active":"2023-01-24T03:30:25.447279+0000","last_peered":"2023-01-24T03:30:25.447279+0000","last_clean":"2023-01-24T03:30:25.447279+0000","last_became_active":"2023-01-24T03:30:20.605899+0000","last_became_peered":"2023-01-24T03:30:20.605899+0000","last_unstale":"2023-01-24T03:30:25.447279+0000","last_undegraded":"2023-01-24T03:30:25.447279+0000","last_fullsized":"2023-01-24T03:30:25.447279+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:30:47.956569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992411+0000","last_change":"2023-01-24T03:30:20.988906+0000","last_active":"2023-01-24T03:30:24.992411+0000","last_peered":"2023-01-24T03:30:24.992411+0000","last_clean":"2023-01-24T03:30:24.992411+0000","last_became_active":"2023-01-24T03:30:20.988688+0000","last_became_peered":"2023-01-24T03:30:20.988688+0000","last_unstale":"2023-01-24T03:30:24.992411+0000","last_undegraded":"2023-01-24T03:30:24.992411+0000","last_fullsized":"2023-01-24T03:30:24.992411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:25:43.365167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.019966+0000","last_change":"2023-01-24T03:30:25.013834+0000","last_active":"2023-01-24T03:30:25.019966+0000","last_peered":"2023-01-24T03:30:25.019966+0000","last_clean":"2023-01-24T03:30:25.019966+0000","last_became_active":"2023-01-24T03:30:25.011774+0000","last_became_peered":"2023-01-24T03:30:25.011774+0000","last_unstale":"2023-01-24T03:30:25.019966+0000","last_undegraded":"2023-01-24T03:30:25.019966+0000","last_fullsized":"2023-01-24T03:30:25.019966+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:02:50.158049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0,4],"acting":[5,1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002878+0000","last_change":"2023-01-24T03:30:16.051822+0000","last_active":"2023-01-24T03:30:25.002878+0000","last_peered":"2023-01-24T03:30:25.002878+0000","last_clean":"2023-01-24T03:30:25.002878+0000","last_became_active":"2023-01-24T03:30:16.051525+0000","last_became_peered":"2023-01-24T03:30:16.051525+0000","last_unstale":"2023-01-24T03:30:25.002878+0000","last_undegraded":"2023-01-24T03:30:25.002878+0000","last_fullsized":"2023-01-24T03:30:25.002878+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:37:30.802424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"16'117","reported_seq":166,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996317+0000","last_change":"2023-01-24T03:30:16.054907+0000","last_active":"2023-01-24T03:30:24.996317+0000","last_peered":"2023-01-24T03:30:24.996317+0000","last_clean":"2023-01-24T03:30:24.996317+0000","last_became_active":"2023-01-24T03:30:16.054071+0000","last_became_peered":"2023-01-24T03:30:16.054071+0000","last_unstale":"2023-01-24T03:30:24.996317+0000","last_undegraded":"2023-01-24T03:30:24.996317+0000","last_fullsized":"2023-01-24T03:30:24.996317+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:31:26.790766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992331+0000","last_change":"2023-01-24T03:30:20.607914+0000","last_active":"2023-01-24T03:30:24.992331+0000","last_peered":"2023-01-24T03:30:24.992331+0000","last_clean":"2023-01-24T03:30:24.992331+0000","last_became_active":"2023-01-24T03:30:20.607663+0000","last_became_peered":"2023-01-24T03:30:20.607663+0000","last_unstale":"2023-01-24T03:30:24.992331+0000","last_undegraded":"2023-01-24T03:30:24.992331+0000","last_fullsized":"2023-01-24T03:30:24.992331+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:32:24.271747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.993825+0000","last_change":"2023-01-24T03:30:20.998105+0000","last_active":"2023-01-24T03:30:24.993825+0000","last_peered":"2023-01-24T03:30:24.993825+0000","last_clean":"2023-01-24T03:30:24.993825+0000","last_became_active":"2023-01-24T03:30:20.997871+0000","last_became_peered":"2023-01-24T03:30:20.997871+0000","last_unstale":"2023-01-24T03:30:24.993825+0000","last_undegraded":"2023-01-24T03:30:24.993825+0000","last_fullsized":"2023-01-24T03:30:24.993825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:47:42.740428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016190+0000","last_change":"2023-01-24T03:30:25.008079+0000","last_active":"2023-01-24T03:30:25.016190+0000","last_peered":"2023-01-24T03:30:25.016190+0000","last_clean":"2023-01-24T03:30:25.016190+0000","last_became_active":"2023-01-24T03:30:25.007596+0000","last_became_peered":"2023-01-24T03:30:25.007596+0000","last_unstale":"2023-01-24T03:30:25.016190+0000","last_undegraded":"2023-01-24T03:30:25.016190+0000","last_fullsized":"2023-01-24T03:30:25.016190+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:30:57.179885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1,7],"acting":[2,5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996872+0000","last_change":"2023-01-24T03:30:16.272445+0000","last_active":"2023-01-24T03:30:24.996872+0000","last_peered":"2023-01-24T03:30:24.996872+0000","last_clean":"2023-01-24T03:30:24.996872+0000","last_became_active":"2023-01-24T03:30:16.272184+0000","last_became_peered":"2023-01-24T03:30:16.272184+0000","last_unstale":"2023-01-24T03:30:24.996872+0000","last_undegraded":"2023-01-24T03:30:24.996872+0000","last_fullsized":"2023-01-24T03:30:24.996872+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:17:05.555068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002328+0000","last_change":"2023-01-24T03:30:20.612832+0000","last_active":"2023-01-24T03:30:25.002328+0000","last_peered":"2023-01-24T03:30:25.002328+0000","last_clean":"2023-01-24T03:30:25.002328+0000","last_became_active":"2023-01-24T03:30:20.612569+0000","last_became_peered":"2023-01-24T03:30:20.612569+0000","last_unstale":"2023-01-24T03:30:25.002328+0000","last_undegraded":"2023-01-24T03:30:25.002328+0000","last_fullsized":"2023-01-24T03:30:25.002328+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:37:40.534015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002330+0000","last_change":"2023-01-24T03:30:20.998171+0000","last_active":"2023-01-24T03:30:25.002330+0000","last_peered":"2023-01-24T03:30:25.002330+0000","last_clean":"2023-01-24T03:30:25.002330+0000","last_became_active":"2023-01-24T03:30:20.994887+0000","last_became_peered":"2023-01-24T03:30:20.994887+0000","last_unstale":"2023-01-24T03:30:25.002330+0000","last_undegraded":"2023-01-24T03:30:25.002330+0000","last_fullsized":"2023-01-24T03:30:25.002330+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:57:40.909930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.020081+0000","last_change":"2023-01-24T03:30:25.014948+0000","last_active":"2023-01-24T03:30:25.020081+0000","last_peered":"2023-01-24T03:30:25.020081+0000","last_clean":"2023-01-24T03:30:25.020081+0000","last_became_active":"2023-01-24T03:30:25.014572+0000","last_became_peered":"2023-01-24T03:30:25.014572+0000","last_unstale":"2023-01-24T03:30:25.020081+0000","last_undegraded":"2023-01-24T03:30:25.020081+0000","last_fullsized":"2023-01-24T03:30:25.020081+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:59:13.108074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5,3],"acting":[7,2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.996112+0000","last_change":"2023-01-24T03:30:16.272574+0000","last_active":"2023-01-24T03:30:24.996112+0000","last_peered":"2023-01-24T03:30:24.996112+0000","last_clean":"2023-01-24T03:30:24.996112+0000","last_became_active":"2023-01-24T03:30:16.272252+0000","last_became_peered":"2023-01-24T03:30:16.272252+0000","last_unstale":"2023-01-24T03:30:24.996112+0000","last_undegraded":"2023-01-24T03:30:24.996112+0000","last_fullsized":"2023-01-24T03:30:24.996112+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T09:12:08.393585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991181+0000","last_change":"2023-01-24T03:30:20.606878+0000","last_active":"2023-01-24T03:30:24.991181+0000","last_peered":"2023-01-24T03:30:24.991181+0000","last_clean":"2023-01-24T03:30:24.991181+0000","last_became_active":"2023-01-24T03:30:20.606442+0000","last_became_peered":"2023-01-24T03:30:20.606442+0000","last_unstale":"2023-01-24T03:30:24.991181+0000","last_undegraded":"2023-01-24T03:30:24.991181+0000","last_fullsized":"2023-01-24T03:30:24.991181+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:26:16.916080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990646+0000","last_change":"2023-01-24T03:30:20.990840+0000","last_active":"2023-01-24T03:30:24.990646+0000","last_peered":"2023-01-24T03:30:24.990646+0000","last_clean":"2023-01-24T03:30:24.990646+0000","last_became_active":"2023-01-24T03:30:20.990615+0000","last_became_peered":"2023-01-24T03:30:20.990615+0000","last_unstale":"2023-01-24T03:30:24.990646+0000","last_undegraded":"2023-01-24T03:30:24.990646+0000","last_fullsized":"2023-01-24T03:30:24.990646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T15:06:37.670127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.013106+0000","last_change":"2023-01-24T03:30:25.006375+0000","last_active":"2023-01-24T03:30:25.013106+0000","last_peered":"2023-01-24T03:30:25.013106+0000","last_clean":"2023-01-24T03:30:25.013106+0000","last_became_active":"2023-01-24T03:30:25.006027+0000","last_became_peered":"2023-01-24T03:30:25.006027+0000","last_unstale":"2023-01-24T03:30:25.013106+0000","last_undegraded":"2023-01-24T03:30:25.013106+0000","last_fullsized":"2023-01-24T03:30:25.013106+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:51:29.481545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,6],"acting":[0,1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005695+0000","last_change":"2023-01-24T03:30:16.272475+0000","last_active":"2023-01-24T03:30:25.005695+0000","last_peered":"2023-01-24T03:30:25.005695+0000","last_clean":"2023-01-24T03:30:25.005695+0000","last_became_active":"2023-01-24T03:30:16.272140+0000","last_became_peered":"2023-01-24T03:30:16.272140+0000","last_unstale":"2023-01-24T03:30:25.005695+0000","last_undegraded":"2023-01-24T03:30:25.005695+0000","last_fullsized":"2023-01-24T03:30:25.005695+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T04:48:16.300876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992670+0000","last_change":"2023-01-24T03:30:20.609391+0000","last_active":"2023-01-24T03:30:24.992670+0000","last_peered":"2023-01-24T03:30:24.992670+0000","last_clean":"2023-01-24T03:30:24.992670+0000","last_became_active":"2023-01-24T03:30:20.609124+0000","last_became_peered":"2023-01-24T03:30:20.609124+0000","last_unstale":"2023-01-24T03:30:24.992670+0000","last_undegraded":"2023-01-24T03:30:24.992670+0000","last_fullsized":"2023-01-24T03:30:24.992670+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:36:49.616649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.001859+0000","last_change":"2023-01-24T03:30:20.996985+0000","last_active":"2023-01-24T03:30:25.001859+0000","last_peered":"2023-01-24T03:30:25.001859+0000","last_clean":"2023-01-24T03:30:25.001859+0000","last_became_active":"2023-01-24T03:30:20.996612+0000","last_became_peered":"2023-01-24T03:30:20.996612+0000","last_unstale":"2023-01-24T03:30:25.001859+0000","last_undegraded":"2023-01-24T03:30:25.001859+0000","last_fullsized":"2023-01-24T03:30:25.001859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:10:50.378902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.012218+0000","last_change":"2023-01-24T03:30:25.008155+0000","last_active":"2023-01-24T03:30:25.012218+0000","last_peered":"2023-01-24T03:30:25.012218+0000","last_clean":"2023-01-24T03:30:25.012218+0000","last_became_active":"2023-01-24T03:30:25.007674+0000","last_became_peered":"2023-01-24T03:30:25.007674+0000","last_unstale":"2023-01-24T03:30:25.012218+0000","last_undegraded":"2023-01-24T03:30:25.012218+0000","last_fullsized":"2023-01-24T03:30:25.012218+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:55:46.448808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5,2],"acting":[6,0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990827+0000","last_change":"2023-01-24T03:30:16.272372+0000","last_active":"2023-01-24T03:30:24.990827+0000","last_peered":"2023-01-24T03:30:24.990827+0000","last_clean":"2023-01-24T03:30:24.990827+0000","last_became_active":"2023-01-24T03:30:16.272104+0000","last_became_peered":"2023-01-24T03:30:16.272104+0000","last_unstale":"2023-01-24T03:30:24.990827+0000","last_undegraded":"2023-01-24T03:30:24.990827+0000","last_fullsized":"2023-01-24T03:30:24.990827+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:15:10.865596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002114+0000","last_change":"2023-01-24T03:30:20.612284+0000","last_active":"2023-01-24T03:30:25.002114+0000","last_peered":"2023-01-24T03:30:25.002114+0000","last_clean":"2023-01-24T03:30:25.002114+0000","last_became_active":"2023-01-24T03:30:20.612029+0000","last_became_peered":"2023-01-24T03:30:20.612029+0000","last_unstale":"2023-01-24T03:30:25.002114+0000","last_undegraded":"2023-01-24T03:30:25.002114+0000","last_fullsized":"2023-01-24T03:30:25.002114+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:18:58.755974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992643+0000","last_change":"2023-01-24T03:30:20.989834+0000","last_active":"2023-01-24T03:30:24.992643+0000","last_peered":"2023-01-24T03:30:24.992643+0000","last_clean":"2023-01-24T03:30:24.992643+0000","last_became_active":"2023-01-24T03:30:20.989645+0000","last_became_peered":"2023-01-24T03:30:20.989645+0000","last_unstale":"2023-01-24T03:30:24.992643+0000","last_undegraded":"2023-01-24T03:30:24.992643+0000","last_fullsized":"2023-01-24T03:30:24.992643+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:42:06.477043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016259+0000","last_change":"2023-01-24T03:30:25.011952+0000","last_active":"2023-01-24T03:30:25.016259+0000","last_peered":"2023-01-24T03:30:25.016259+0000","last_clean":"2023-01-24T03:30:25.016259+0000","last_became_active":"2023-01-24T03:30:25.011612+0000","last_became_peered":"2023-01-24T03:30:25.011612+0000","last_unstale":"2023-01-24T03:30:25.016259+0000","last_undegraded":"2023-01-24T03:30:25.016259+0000","last_fullsized":"2023-01-24T03:30:25.016259+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:51:52.027285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6,5],"acting":[0,1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990404+0000","last_change":"2023-01-24T03:30:16.048979+0000","last_active":"2023-01-24T03:30:24.990404+0000","last_peered":"2023-01-24T03:30:24.990404+0000","last_clean":"2023-01-24T03:30:24.990404+0000","last_became_active":"2023-01-24T03:30:16.048641+0000","last_became_peered":"2023-01-24T03:30:16.048641+0000","last_unstale":"2023-01-24T03:30:24.990404+0000","last_undegraded":"2023-01-24T03:30:24.990404+0000","last_fullsized":"2023-01-24T03:30:24.990404+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:15.032884+0000","last_clean_scrub_stamp":"2023-01-24T03:30:15.032884+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T15:11:26.211024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.005087+0000","last_change":"2023-01-24T03:30:20.606906+0000","last_active":"2023-01-24T03:30:25.005087+0000","last_peered":"2023-01-24T03:30:25.005087+0000","last_clean":"2023-01-24T03:30:25.005087+0000","last_became_active":"2023-01-24T03:30:20.606640+0000","last_became_peered":"2023-01-24T03:30:20.606640+0000","last_unstale":"2023-01-24T03:30:25.005087+0000","last_undegraded":"2023-01-24T03:30:25.005087+0000","last_fullsized":"2023-01-24T03:30:25.005087+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:21:09.008653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.002357+0000","last_change":"2023-01-24T03:30:20.999886+0000","last_active":"2023-01-24T03:30:25.002357+0000","last_peered":"2023-01-24T03:30:25.002357+0000","last_clean":"2023-01-24T03:30:25.002357+0000","last_became_active":"2023-01-24T03:30:20.999394+0000","last_became_peered":"2023-01-24T03:30:20.999394+0000","last_unstale":"2023-01-24T03:30:25.002357+0000","last_undegraded":"2023-01-24T03:30:25.002357+0000","last_fullsized":"2023-01-24T03:30:25.002357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:30:22.395611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016356+0000","last_change":"2023-01-24T03:30:25.006190+0000","last_active":"2023-01-24T03:30:25.016356+0000","last_peered":"2023-01-24T03:30:25.016356+0000","last_clean":"2023-01-24T03:30:25.016356+0000","last_became_active":"2023-01-24T03:30:25.005825+0000","last_became_peered":"2023-01-24T03:30:25.005825+0000","last_unstale":"2023-01-24T03:30:25.016356+0000","last_undegraded":"2023-01-24T03:30:25.016356+0000","last_fullsized":"2023-01-24T03:30:25.016356+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T08:51:50.888938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,0],"acting":[1,3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"24'1","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.446961+0000","last_change":"2023-01-24T03:30:20.611079+0000","last_active":"2023-01-24T03:30:25.446961+0000","last_peered":"2023-01-24T03:30:25.446961+0000","last_clean":"2023-01-24T03:30:25.446961+0000","last_became_active":"2023-01-24T03:30:20.610841+0000","last_became_peered":"2023-01-24T03:30:20.610841+0000","last_unstale":"2023-01-24T03:30:25.446961+0000","last_undegraded":"2023-01-24T03:30:25.446961+0000","last_fullsized":"2023-01-24T03:30:25.446961+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:34:24.922661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.994693+0000","last_change":"2023-01-24T03:30:20.991418+0000","last_active":"2023-01-24T03:30:24.994693+0000","last_peered":"2023-01-24T03:30:24.994693+0000","last_clean":"2023-01-24T03:30:24.994693+0000","last_became_active":"2023-01-24T03:30:20.991167+0000","last_became_peered":"2023-01-24T03:30:20.991167+0000","last_unstale":"2023-01-24T03:30:24.994693+0000","last_undegraded":"2023-01-24T03:30:24.994693+0000","last_fullsized":"2023-01-24T03:30:24.994693+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:51:52.736691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.013519+0000","last_change":"2023-01-24T03:30:25.010394+0000","last_active":"2023-01-24T03:30:25.013519+0000","last_peered":"2023-01-24T03:30:25.013519+0000","last_clean":"2023-01-24T03:30:25.013519+0000","last_became_active":"2023-01-24T03:30:25.010044+0000","last_became_peered":"2023-01-24T03:30:25.010044+0000","last_unstale":"2023-01-24T03:30:25.013519+0000","last_undegraded":"2023-01-24T03:30:25.013519+0000","last_fullsized":"2023-01-24T03:30:25.013519+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:08:55.250184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1,6],"acting":[7,4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"24'2","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.984877+0000","last_change":"2023-01-24T03:30:20.607538+0000","last_active":"2023-01-24T03:30:25.984877+0000","last_peered":"2023-01-24T03:30:25.984877+0000","last_clean":"2023-01-24T03:30:25.984877+0000","last_became_active":"2023-01-24T03:30:20.607285+0000","last_became_peered":"2023-01-24T03:30:20.607285+0000","last_unstale":"2023-01-24T03:30:25.984877+0000","last_undegraded":"2023-01-24T03:30:25.984877+0000","last_fullsized":"2023-01-24T03:30:25.984877+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:42:21.836603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992857+0000","last_change":"2023-01-24T03:30:20.986791+0000","last_active":"2023-01-24T03:30:24.992857+0000","last_peered":"2023-01-24T03:30:24.992857+0000","last_clean":"2023-01-24T03:30:24.992857+0000","last_became_active":"2023-01-24T03:30:20.986516+0000","last_became_peered":"2023-01-24T03:30:20.986516+0000","last_unstale":"2023-01-24T03:30:24.992857+0000","last_undegraded":"2023-01-24T03:30:24.992857+0000","last_fullsized":"2023-01-24T03:30:24.992857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:53:19.191166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.015383+0000","last_change":"2023-01-24T03:30:25.010751+0000","last_active":"2023-01-24T03:30:25.015383+0000","last_peered":"2023-01-24T03:30:25.015383+0000","last_clean":"2023-01-24T03:30:25.015383+0000","last_became_active":"2023-01-24T03:30:25.010332+0000","last_became_peered":"2023-01-24T03:30:25.010332+0000","last_unstale":"2023-01-24T03:30:25.015383+0000","last_undegraded":"2023-01-24T03:30:25.015383+0000","last_fullsized":"2023-01-24T03:30:25.015383+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T13:48:30.190712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7,4],"acting":[5,0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"24'2","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985166+0000","last_change":"2023-01-24T03:30:20.608509+0000","last_active":"2023-01-24T03:30:25.985166+0000","last_peered":"2023-01-24T03:30:25.985166+0000","last_clean":"2023-01-24T03:30:25.985166+0000","last_became_active":"2023-01-24T03:30:20.608313+0000","last_became_peered":"2023-01-24T03:30:20.608313+0000","last_unstale":"2023-01-24T03:30:25.985166+0000","last_undegraded":"2023-01-24T03:30:25.985166+0000","last_fullsized":"2023-01-24T03:30:25.985166+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T05:31:52.253588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992448+0000","last_change":"2023-01-24T03:30:20.992334+0000","last_active":"2023-01-24T03:30:24.992448+0000","last_peered":"2023-01-24T03:30:24.992448+0000","last_clean":"2023-01-24T03:30:24.992448+0000","last_became_active":"2023-01-24T03:30:20.992027+0000","last_became_peered":"2023-01-24T03:30:20.992027+0000","last_unstale":"2023-01-24T03:30:24.992448+0000","last_undegraded":"2023-01-24T03:30:24.992448+0000","last_fullsized":"2023-01-24T03:30:24.992448+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T12:57:56.985507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.012177+0000","last_change":"2023-01-24T03:30:25.004167+0000","last_active":"2023-01-24T03:30:25.012177+0000","last_peered":"2023-01-24T03:30:25.012177+0000","last_clean":"2023-01-24T03:30:25.012177+0000","last_became_active":"2023-01-24T03:30:25.003752+0000","last_became_peered":"2023-01-24T03:30:25.003752+0000","last_unstale":"2023-01-24T03:30:25.012177+0000","last_undegraded":"2023-01-24T03:30:25.012177+0000","last_fullsized":"2023-01-24T03:30:25.012177+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:19:16.285529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,2],"acting":[4,6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"24'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.448123+0000","last_change":"2023-01-24T03:30:20.603542+0000","last_active":"2023-01-24T03:30:25.448123+0000","last_peered":"2023-01-24T03:30:25.448123+0000","last_clean":"2023-01-24T03:30:25.448123+0000","last_became_active":"2023-01-24T03:30:20.603287+0000","last_became_peered":"2023-01-24T03:30:20.603287+0000","last_unstale":"2023-01-24T03:30:25.448123+0000","last_undegraded":"2023-01-24T03:30:25.448123+0000","last_fullsized":"2023-01-24T03:30:25.448123+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T11:03:56.865763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.991098+0000","last_change":"2023-01-24T03:30:20.989519+0000","last_active":"2023-01-24T03:30:24.991098+0000","last_peered":"2023-01-24T03:30:24.991098+0000","last_clean":"2023-01-24T03:30:24.991098+0000","last_became_active":"2023-01-24T03:30:20.989286+0000","last_became_peered":"2023-01-24T03:30:20.989286+0000","last_unstale":"2023-01-24T03:30:24.991098+0000","last_undegraded":"2023-01-24T03:30:24.991098+0000","last_fullsized":"2023-01-24T03:30:24.991098+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:17:30.829327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.016619+0000","last_change":"2023-01-24T03:30:25.010262+0000","last_active":"2023-01-24T03:30:25.016619+0000","last_peered":"2023-01-24T03:30:25.016619+0000","last_clean":"2023-01-24T03:30:25.016619+0000","last_became_active":"2023-01-24T03:30:25.009930+0000","last_became_peered":"2023-01-24T03:30:25.009930+0000","last_unstale":"2023-01-24T03:30:25.016619+0000","last_undegraded":"2023-01-24T03:30:25.016619+0000","last_fullsized":"2023-01-24T03:30:25.016619+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:23.979279+0000","last_clean_scrub_stamp":"2023-01-24T03:30:23.979279+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:36:17.807046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7,2],"acting":[1,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"24'4","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.985911+0000","last_change":"2023-01-24T03:30:20.607334+0000","last_active":"2023-01-24T03:30:25.985911+0000","last_peered":"2023-01-24T03:30:25.985911+0000","last_clean":"2023-01-24T03:30:25.985911+0000","last_became_active":"2023-01-24T03:30:20.607079+0000","last_became_peered":"2023-01-24T03:30:20.607079+0000","last_unstale":"2023-01-24T03:30:25.985911+0000","last_undegraded":"2023-01-24T03:30:25.985911+0000","last_fullsized":"2023-01-24T03:30:25.985911+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:19.578496+0000","last_clean_scrub_stamp":"2023-01-24T03:30:19.578496+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T10:05:18.886532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992554+0000","last_change":"2023-01-24T03:30:20.987567+0000","last_active":"2023-01-24T03:30:24.992554+0000","last_peered":"2023-01-24T03:30:24.992554+0000","last_clean":"2023-01-24T03:30:24.992554+0000","last_became_active":"2023-01-24T03:30:20.987058+0000","last_became_peered":"2023-01-24T03:30:20.987058+0000","last_unstale":"2023-01-24T03:30:24.992554+0000","last_undegraded":"2023-01-24T03:30:24.992554+0000","last_fullsized":"2023-01-24T03:30:24.992554+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:58:30.391733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.989538+0000","last_change":"2023-01-24T03:30:20.986393+0000","last_active":"2023-01-24T03:30:24.989538+0000","last_peered":"2023-01-24T03:30:24.989538+0000","last_clean":"2023-01-24T03:30:24.989538+0000","last_became_active":"2023-01-24T03:30:20.986089+0000","last_became_peered":"2023-01-24T03:30:20.986089+0000","last_unstale":"2023-01-24T03:30:24.989538+0000","last_undegraded":"2023-01-24T03:30:24.989538+0000","last_fullsized":"2023-01-24T03:30:24.989538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T06:44:21.992291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.990902+0000","last_change":"2023-01-24T03:30:20.991468+0000","last_active":"2023-01-24T03:30:24.990902+0000","last_peered":"2023-01-24T03:30:24.990902+0000","last_clean":"2023-01-24T03:30:24.990902+0000","last_became_active":"2023-01-24T03:30:20.991227+0000","last_became_peered":"2023-01-24T03:30:20.991227+0000","last_unstale":"2023-01-24T03:30:24.990902+0000","last_undegraded":"2023-01-24T03:30:24.990902+0000","last_fullsized":"2023-01-24T03:30:24.990902+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T07:27:25.806340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992602+0000","last_change":"2023-01-24T03:30:20.989824+0000","last_active":"2023-01-24T03:30:24.992602+0000","last_peered":"2023-01-24T03:30:24.992602+0000","last_clean":"2023-01-24T03:30:24.992602+0000","last_became_active":"2023-01-24T03:30:20.989577+0000","last_became_peered":"2023-01-24T03:30:20.989577+0000","last_unstale":"2023-01-24T03:30:24.992602+0000","last_undegraded":"2023-01-24T03:30:24.992602+0000","last_fullsized":"2023-01-24T03:30:24.992602+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T03:48:17.948459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:24.992452+0000","last_change":"2023-01-24T03:30:20.996266+0000","last_active":"2023-01-24T03:30:24.992452+0000","last_peered":"2023-01-24T03:30:24.992452+0000","last_clean":"2023-01-24T03:30:24.992452+0000","last_became_active":"2023-01-24T03:30:20.995610+0000","last_became_peered":"2023-01-24T03:30:20.995610+0000","last_unstale":"2023-01-24T03:30:24.992452+0000","last_undegraded":"2023-01-24T03:30:24.992452+0000","last_fullsized":"2023-01-24T03:30:24.992452+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:24:43.022530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-01-24T03:30:25.001943+0000","last_change":"2023-01-24T03:30:20.997081+0000","last_active":"2023-01-24T03:30:25.001943+0000","last_peered":"2023-01-24T03:30:25.001943+0000","last_clean":"2023-01-24T03:30:25.001943+0000","last_became_active":"2023-01-24T03:30:20.996837+0000","last_became_peered":"2023-01-24T03:30:20.996837+0000","last_unstale":"2023-01-24T03:30:25.001943+0000","last_undegraded":"2023-01-24T03:30:25.001943+0000","last_fullsized":"2023-01-24T03:30:25.001943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T03:30:20.587093+0000","last_clean_scrub_stamp":"2023-01-24T03:30:20.587093+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T14:42:06.134817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":4746,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64574,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9180,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":8832,"kb_avail":94362660,"statfs":{"total":96636764160,"available":96627363840,"internally_reserved":0,"allocated":348160,"data_stored":698731,"data_compressed":32287,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":9043968},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]}]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8576,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94363264,"statfs":{"total":96636764160,"available":96627982336,"internally_reserved":0,"allocated":253952,"data_stored":104434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":9152,"kb_used_data":312,"kb_used_omap":0,"kb_used_meta":8832,"kb_avail":94362688,"statfs":{"total":96636764160,"available":96627392512,"internally_reserved":0,"allocated":319488,"data_stored":694802,"data_compressed":32287,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":9043968},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574859,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8640,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94363200,"statfs":{"total":96636764160,"available":96627916800,"internally_reserved":0,"allocated":253952,"data_stored":104382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]}]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8512,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":8256,"kb_avail":94363328,"statfs":{"total":96636764160,"available":96628047872,"internally_reserved":0,"allocated":253952,"data_stored":104434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8454144},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.254}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8580,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94363260,"statfs":{"total":96636764160,"available":96627978240,"internally_reserved":0,"allocated":258048,"data_stored":104524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574858,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8660,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":8384,"kb_avail":94363180,"statfs":{"total":96636764160,"available":96627896320,"internally_reserved":0,"allocated":274432,"data_stored":108466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8585216},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]}]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8580,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":8320,"kb_avail":94363260,"statfs":{"total":96636764160,"available":96627978240,"internally_reserved":0,"allocated":258048,"data_stored":103705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8519680},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.311}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32287,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32287,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-01-24T03:30:53.956 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T03:30:54.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.099+0000 7f3108f79700 1 Processor -- start 2023-01-24T03:30:54.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.099+0000 7f3108f79700 1 -- start start 2023-01-24T03:30:54.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.100+0000 7f3108f79700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f310406d320 0x7f3104155db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:54.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.100+0000 7f3108f79700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3104067f20 0x7f310414efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:54.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.100+0000 7f3108f79700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31040c54d0 con 0x7f310406cf50 2023-01-24T03:30:54.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.100+0000 7f3108f79700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31040c49c0 con 0x7f310406d320 2023-01-24T03:30:54.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.100+0000 7f3108f79700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31041562f0 con 0x7f3104067f20 2023-01-24T03:30:54.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.100+0000 7f310259c700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f310406cf50 0x7f31040c5960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:60468/0 (socket says 172.21.15.112:60468) 2023-01-24T03:30:54.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.100+0000 7f310259c700 1 -- 172.21.15.112:0/507372898 learned_addr learned my addr 172.21.15.112:0/507372898 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:54.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.101+0000 7f3102d9d700 1 --2- 172.21.15.112:0/507372898 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3104067f20 0x7f310414efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:54.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.101+0000 7f3101d9b700 1 --2- 172.21.15.112:0/507372898 >> v2:172.21.15.142:3300/0 conn(0x7f310406d320 0x7f3104155db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:54.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.101+0000 7f310159a700 1 -- 172.21.15.112:0/507372898 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 202008687 0 0) 0x7f31040c54d0 con 0x7f310406cf50 2023-01-24T03:30:54.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.101+0000 7f310159a700 1 -- 172.21.15.112:0/507372898 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30e40033f0 con 0x7f310406cf50 2023-01-24T03:30:54.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.101+0000 7f310159a700 1 -- 172.21.15.112:0/507372898 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1836855465 0 0) 0x7f30e40033f0 con 0x7f310406cf50 2023-01-24T03:30:54.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.101+0000 7f310159a700 1 -- 172.21.15.112:0/507372898 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3104067f20 msgr2=0x7f310414efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:54.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.101+0000 7f310159a700 1 --2- 172.21.15.112:0/507372898 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3104067f20 0x7f310414efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:54.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.101+0000 7f310159a700 1 -- 172.21.15.112:0/507372898 >> v2:172.21.15.142:3300/0 conn(0x7f310406d320 msgr2=0x7f3104155db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:54.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.101+0000 7f310159a700 1 --2- 172.21.15.112:0/507372898 >> v2:172.21.15.142:3300/0 conn(0x7f310406d320 0x7f3104155db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:54.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.101+0000 7f310159a700 1 -- 172.21.15.112:0/507372898 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31040c6070 con 0x7f310406cf50 2023-01-24T03:30:54.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.101+0000 7f310159a700 1 -- 172.21.15.112:0/507372898 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7f30f40017f0 con 0x7f310406cf50 2023-01-24T03:30:54.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.101+0000 7f310159a700 1 -- 172.21.15.112:0/507372898 >> v1:172.21.15.112:6789/0 conn(0x7f310406cf50 legacy=0x7f31040c5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:54.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.101+0000 7f310159a700 1 --2- 172.21.15.112:0/507372898 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3104067f20 0x7f30e40038e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:54.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.101+0000 7f310159a700 1 --2- 172.21.15.112:0/507372898 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f30e4003e60 0x7f30e4004690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:54.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.101+0000 7f310159a700 1 --2- 172.21.15.112:0/507372898 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f30e4004bd0 0x7f30e4005040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:54.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.101+0000 7f310159a700 1 -- 172.21.15.112:0/507372898 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31040c49c0 con 0x7f30e4004bd0 2023-01-24T03:30:54.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.101+0000 7f310159a700 1 -- 172.21.15.112:0/507372898 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31041562f0 con 0x7f30e4003e60 2023-01-24T03:30:54.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.101+0000 7f310159a700 1 -- 172.21.15.112:0/507372898 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30e400b910 con 0x7f3104067f20 2023-01-24T03:30:54.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.101+0000 7f3102d9d700 1 --2- 172.21.15.112:0/507372898 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f30e4004bd0 0x7f30e4005040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:54.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.101+0000 7f310259c700 1 --2- 172.21.15.112:0/507372898 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3104067f20 0x7f30e40038e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:54.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.101+0000 7f3101d9b700 1 --2- 172.21.15.112:0/507372898 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f30e4003e60 0x7f30e4004690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:54.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.101+0000 7f3102d9d700 1 -- 172.21.15.112:0/507372898 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3104067f20 msgr2=0x7f30e40038e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:54.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.101+0000 7f3102d9d700 1 --2- 172.21.15.112:0/507372898 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3104067f20 0x7f30e40038e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:54.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.101+0000 7f3102d9d700 1 -- 172.21.15.112:0/507372898 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f30e4003e60 msgr2=0x7f30e4004690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:54.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.101+0000 7f3102d9d700 1 --2- 172.21.15.112:0/507372898 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f30e4003e60 0x7f30e4004690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:54.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.101+0000 7f3102d9d700 1 -- 172.21.15.112:0/507372898 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e40033f0 con 0x7f30e4004bd0 2023-01-24T03:30:54.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.101+0000 7f3102d9d700 1 --2- 172.21.15.112:0/507372898 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f30e4004bd0 0x7f30e4005040 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f30fc009a80 tx=0x7f30fc009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59e0e59d90835ad1 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:54.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.102+0000 7f310159a700 1 -- 172.21.15.112:0/507372898 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30fc009430 con 0x7f30e4004bd0 2023-01-24T03:30:54.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.102+0000 7f310159a700 1 -- 172.21.15.112:0/507372898 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30fc00e5e0 con 0x7f30e4004bd0 2023-01-24T03:30:54.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.102+0000 7f310159a700 1 -- 172.21.15.112:0/507372898 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30fc00e9a0 con 0x7f30e4004bd0 2023-01-24T03:30:54.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.102+0000 7f3108f79700 1 -- 172.21.15.112:0/507372898 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f30e4004bd0 msgr2=0x7f30e4005040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:54.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.102+0000 7f3108f79700 1 --2- 172.21.15.112:0/507372898 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f30e4004bd0 0x7f30e4005040 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f30fc009a80 tx=0x7f30fc009060 comp rx=0 tx=0).stop 2023-01-24T03:30:54.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.102+0000 7f3102d9d700 1 -- 172.21.15.112:0/507372898 reap_dead start 2023-01-24T03:30:54.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.102+0000 7f3108f79700 1 -- 172.21.15.112:0/507372898 shutdown_connections 2023-01-24T03:30:54.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.102+0000 7f3108f79700 1 -- 172.21.15.112:0/507372898 >> 172.21.15.112:0/507372898 conn(0x7f31040c9820 msgr2=0x7f31040c9c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:54.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.102+0000 7f3108f79700 1 -- 172.21.15.112:0/507372898 shutdown_connections 2023-01-24T03:30:54.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.102+0000 7f3108f79700 1 -- 172.21.15.112:0/507372898 wait complete. 2023-01-24T03:30:54.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.103+0000 7f3108f79700 1 Processor -- start 2023-01-24T03:30:54.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.103+0000 7f3108f79700 1 -- start start 2023-01-24T03:30:54.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.104+0000 7f3108f79700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f30e4004bd0 0x7f310414af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:54.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.104+0000 7f3108f79700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f310414fe10 0x7f31041577f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:54.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.104+0000 7f3108f79700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3104157d30 0x7f31041514e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:54.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.104+0000 7f3108f79700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31041568c0 con 0x7f310414fe10 2023-01-24T03:30:54.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.104+0000 7f3108f79700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310414b4a0 con 0x7f3104157d30 2023-01-24T03:30:54.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.104+0000 7f3108f79700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3104159610 con 0x7f30e4004bd0 2023-01-24T03:30:54.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.104+0000 7f3101d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f310414fe10 0x7f31041577f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:54.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.104+0000 7f3101d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f310414fe10 0x7f31041577f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51262/0 (socket says 172.21.15.112:51262) 2023-01-24T03:30:54.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.104+0000 7f3101d9b700 1 -- 172.21.15.112:0/2447555498 learned_addr learned my addr 172.21.15.112:0/2447555498 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:54.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.104+0000 7f310259c700 1 --2- 172.21.15.112:0/2447555498 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f30e4004bd0 0x7f310414af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:54.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.104+0000 7f3102d9d700 1 --2- 172.21.15.112:0/2447555498 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3104157d30 0x7f31041514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:54.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.104+0000 7f3101d9b700 1 -- 172.21.15.112:0/2447555498 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f30e4004bd0 msgr2=0x7f310414af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:54.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.104+0000 7f3101d9b700 1 --2- 172.21.15.112:0/2447555498 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f30e4004bd0 0x7f310414af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:54.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.104+0000 7f3101d9b700 1 -- 172.21.15.112:0/2447555498 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3104157d30 msgr2=0x7f31041514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:54.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.104+0000 7f3101d9b700 1 --2- 172.21.15.112:0/2447555498 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3104157d30 0x7f31041514e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:54.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.104+0000 7f3101d9b700 1 -- 172.21.15.112:0/2447555498 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31041581a0 con 0x7f310414fe10 2023-01-24T03:30:54.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.104+0000 7f3101d9b700 1 --2- 172.21.15.112:0/2447555498 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f310414fe10 0x7f31041577f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f30ec0119f0 tx=0x7f30ec009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:54.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.105+0000 7f30fa7fc700 1 -- 172.21.15.112:0/2447555498 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ec00f6e0 con 0x7f310414fe10 2023-01-24T03:30:54.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.105+0000 7f30fa7fc700 1 -- 172.21.15.112:0/2447555498 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30ec00f840 con 0x7f310414fe10 2023-01-24T03:30:54.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.105+0000 7f3108f79700 1 -- 172.21.15.112:0/2447555498 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31041565e0 con 0x7f310414fe10 2023-01-24T03:30:54.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.105+0000 7f30fa7fc700 1 -- 172.21.15.112:0/2447555498 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ec00fbe0 con 0x7f310414fe10 2023-01-24T03:30:54.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.105+0000 7f3108f79700 1 -- 172.21.15.112:0/2447555498 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3104150f40 con 0x7f310414fe10 2023-01-24T03:30:54.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.106+0000 7f3108f79700 1 -- 172.21.15.112:0/2447555498 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31040c3f40 con 0x7f310414fe10 2023-01-24T03:30:54.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.106+0000 7f30fa7fc700 1 -- 172.21.15.112:0/2447555498 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f30ec024070 con 0x7f310414fe10 2023-01-24T03:30:54.124 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.106+0000 7f30fa7fc700 1 --2- 172.21.15.112:0/2447555498 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f30e80611c0 0x7f30e8063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:54.124 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.106+0000 7f30fa7fc700 1 -- 172.21.15.112:0/2447555498 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f30ec018030 con 0x7f310414fe10 2023-01-24T03:30:54.124 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.106+0000 7f310259c700 1 --2- 172.21.15.112:0/2447555498 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f30e80611c0 0x7f30e8063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:54.125 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.107+0000 7f310259c700 1 --2- 172.21.15.112:0/2447555498 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f30e80611c0 0x7f30e8063670 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f30f4001460 tx=0x7f310406a030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:54.125 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.110+0000 7f30fa7fc700 1 -- 172.21.15.112:0/2447555498 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f30ec04dca0 con 0x7f310414fe10 2023-01-24T03:30:54.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.288+0000 7f3108f79700 1 -- 172.21.15.112:0/2447555498 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f310404bb80 con 0x7f310414fe10 2023-01-24T03:30:54.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.289+0000 7f30fa7fc700 1 -- 172.21.15.112:0/2447555498 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18755 (secure 0 0 0) 0x7f30ec022020 con 0x7f310414fe10 2023-01-24T03:30:54.292 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:30:54.292 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":24,"fsid":"054b9ac9-b7b0-40e5-b932-6c8a955c5a72","created":"2023-01-24T03:30:03.798533+0000","modified":"2023-01-24T03:30:25.373340+0000","last_up_change":"2023-01-24T03:30:13.945316+0000","last_in_change":"2023-01-24T03:30:06.198433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-01-24T03:30:14.931166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-01-24T03:30:15.023730+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-01-24T03:30:19.060597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-01-24T03:30:19.943686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-01-24T03:30:22.999036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"fe6ad231-a39a-42eb-9a3a-49d008dd35e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6809","nonce":84512},{"type":"v1","addr":"172.21.15.112:6811","nonce":84512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":84512},{"type":"v1","addr":"172.21.15.112:6817","nonce":84512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6825","nonce":84512},{"type":"v1","addr":"172.21.15.112:6828","nonce":84512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6819","nonce":84512},{"type":"v1","addr":"172.21.15.112:6822","nonce":84512}]},"public_addr":"172.21.15.112:6811/84512","cluster_addr":"172.21.15.112:6817/84512","heartbeat_back_addr":"172.21.15.112:6828/84512","heartbeat_front_addr":"172.21.15.112:6822/84512","state":["exists","up"]},{"osd":1,"uuid":"8526a1d2-d540-4021-9766-c4bbb30317ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":84513},{"type":"v1","addr":"172.21.15.112:6812","nonce":84513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6815","nonce":84513},{"type":"v1","addr":"172.21.15.112:6818","nonce":84513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6827","nonce":84513},{"type":"v1","addr":"172.21.15.112:6830","nonce":84513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6821","nonce":84513},{"type":"v1","addr":"172.21.15.112:6824","nonce":84513}]},"public_addr":"172.21.15.112:6812/84513","cluster_addr":"172.21.15.112:6818/84513","heartbeat_back_addr":"172.21.15.112:6830/84513","heartbeat_front_addr":"172.21.15.112:6824/84513","state":["exists","up"]},{"osd":2,"uuid":"041ef8e4-4858-432d-9476-cbdb3a0dc47c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":84511},{"type":"v1","addr":"172.21.15.112:6813","nonce":84511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":84511},{"type":"v1","addr":"172.21.15.112:6820","nonce":84511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6829","nonce":84511},{"type":"v1","addr":"172.21.15.112:6831","nonce":84511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6823","nonce":84511},{"type":"v1","addr":"172.21.15.112:6826","nonce":84511}]},"public_addr":"172.21.15.112:6813/84511","cluster_addr":"172.21.15.112:6820/84511","heartbeat_back_addr":"172.21.15.112:6831/84511","heartbeat_front_addr":"172.21.15.112:6826/84511","state":["exists","up"]},{"osd":3,"uuid":"987587cf-1357-4a70-ad85-67b189c67729","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":84510},{"type":"v1","addr":"172.21.15.112:6801","nonce":84510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":84510},{"type":"v1","addr":"172.21.15.112:6803","nonce":84510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":84510},{"type":"v1","addr":"172.21.15.112:6807","nonce":84510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":84510},{"type":"v1","addr":"172.21.15.112:6805","nonce":84510}]},"public_addr":"172.21.15.112:6801/84510","cluster_addr":"172.21.15.112:6803/84510","heartbeat_back_addr":"172.21.15.112:6807/84510","heartbeat_front_addr":"172.21.15.112:6805/84510","state":["exists","up"]},{"osd":4,"uuid":"8503512b-43db-40b8-8b2b-c86cf2e54e17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":84614},{"type":"v1","addr":"172.21.15.142:6801","nonce":84614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":84614},{"type":"v1","addr":"172.21.15.142:6803","nonce":84614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":84614},{"type":"v1","addr":"172.21.15.142:6807","nonce":84614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":84614},{"type":"v1","addr":"172.21.15.142:6805","nonce":84614}]},"public_addr":"172.21.15.142:6801/84614","cluster_addr":"172.21.15.142:6803/84614","heartbeat_back_addr":"172.21.15.142:6807/84614","heartbeat_front_addr":"172.21.15.142:6805/84614","state":["exists","up"]},{"osd":5,"uuid":"d3bc0030-8b08-4645-9ec2-1f8a4e29ef36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":84638},{"type":"v1","addr":"172.21.15.142:6817","nonce":84638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":84638},{"type":"v1","addr":"172.21.15.142:6819","nonce":84638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":84638},{"type":"v1","addr":"172.21.15.142:6823","nonce":84638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":84638},{"type":"v1","addr":"172.21.15.142:6821","nonce":84638}]},"public_addr":"172.21.15.142:6817/84638","cluster_addr":"172.21.15.142:6819/84638","heartbeat_back_addr":"172.21.15.142:6823/84638","heartbeat_front_addr":"172.21.15.142:6821/84638","state":["exists","up"]},{"osd":6,"uuid":"c7649e3b-832d-4e6f-8866-b15e5d814476","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":84645},{"type":"v1","addr":"172.21.15.142:6825","nonce":84645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":84645},{"type":"v1","addr":"172.21.15.142:6827","nonce":84645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":84645},{"type":"v1","addr":"172.21.15.142:6831","nonce":84645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":84645},{"type":"v1","addr":"172.21.15.142:6829","nonce":84645}]},"public_addr":"172.21.15.142:6825/84645","cluster_addr":"172.21.15.142:6827/84645","heartbeat_back_addr":"172.21.15.142:6831/84645","heartbeat_front_addr":"172.21.15.142:6829/84645","state":["exists","up"]},{"osd":7,"uuid":"2ea5b3bf-884e-4b20-b572-1443efdc7477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":84615},{"type":"v1","addr":"172.21.15.142:6809","nonce":84615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":84615},{"type":"v1","addr":"172.21.15.142:6811","nonce":84615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":84615},{"type":"v1","addr":"172.21.15.142:6815","nonce":84615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":84615},{"type":"v1","addr":"172.21.15.142:6813","nonce":84615}]},"public_addr":"172.21.15.142:6809/84615","cluster_addr":"172.21.15.142:6811/84615","heartbeat_back_addr":"172.21.15.142:6815/84615","heartbeat_front_addr":"172.21.15.142:6813/84615","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.370027+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.335683+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.350667+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.350794+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.391119+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.422933+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.400439+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.354018+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T03:30:54.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.291+0000 7f3108f79700 1 -- 172.21.15.112:0/2447555498 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f30e80611c0 msgr2=0x7f30e8063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:54.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.291+0000 7f3108f79700 1 --2- 172.21.15.112:0/2447555498 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f30e80611c0 0x7f30e8063670 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f30f4001460 tx=0x7f310406a030 comp rx=0 tx=0).stop 2023-01-24T03:30:54.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.291+0000 7f3108f79700 1 -- 172.21.15.112:0/2447555498 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f310414fe10 msgr2=0x7f31041577f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:54.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.291+0000 7f3108f79700 1 --2- 172.21.15.112:0/2447555498 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f310414fe10 0x7f31041577f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f30ec0119f0 tx=0x7f30ec009060 comp rx=0 tx=0).stop 2023-01-24T03:30:54.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.292+0000 7f3108f79700 1 -- 172.21.15.112:0/2447555498 shutdown_connections 2023-01-24T03:30:54.295 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.292+0000 7f3108f79700 1 --2- 172.21.15.112:0/2447555498 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f30e80611c0 0x7f30e8063670 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:54.295 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.292+0000 7f3108f79700 1 --2- 172.21.15.112:0/2447555498 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3104157d30 0x7f31041514e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:54.295 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.292+0000 7f3108f79700 1 --2- 172.21.15.112:0/2447555498 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f30e4004bd0 0x7f310414af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:54.295 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.292+0000 7f3108f79700 1 --2- 172.21.15.112:0/2447555498 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f310414fe10 0x7f31041577f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:54.295 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.292+0000 7f3108f79700 1 -- 172.21.15.112:0/2447555498 >> 172.21.15.112:0/2447555498 conn(0x7f31040c9820 msgr2=0x7f31040c5960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:54.296 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.292+0000 7f3108f79700 1 -- 172.21.15.112:0/2447555498 shutdown_connections 2023-01-24T03:30:54.296 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.292+0000 7f3108f79700 1 -- 172.21.15.112:0/2447555498 wait complete. 2023-01-24T03:30:54.305 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-01-24T03:30:14.931166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-01-24T03:30:15.023730+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-01-24T03:30:19.060597+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-01-24T03:30:19.943686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 5, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-01-24T03:30:22.999036+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-01-24T03:30:54.306 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-01-24T03:30:54.420 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.418+0000 7f76102dd700 1 Processor -- start 2023-01-24T03:30:54.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.418+0000 7f76102dd700 1 -- start start 2023-01-24T03:30:54.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.419+0000 7f76102dd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f76080694c0 0x7f7608069890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:54.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.419+0000 7f76102dd700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7608155ab0 con 0x7f7608068d20 2023-01-24T03:30:54.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.419+0000 7f76102dd700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76080c70e0 con 0x7f760805e420 2023-01-24T03:30:54.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.419+0000 7f76102dd700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7608155550 con 0x7f76080694c0 2023-01-24T03:30:54.428 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.419+0000 7f760e87a700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f7608068d20 0x7f76080ae9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:60490/0 (socket says 172.21.15.112:60490) 2023-01-24T03:30:54.428 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.419+0000 7f760e87a700 1 -- 172.21.15.112:0/1545707653 learned_addr learned my addr 172.21.15.112:0/1545707653 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:54.428 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.420+0000 7f760e079700 1 --2- 172.21.15.112:0/1545707653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f76080694c0 0x7f7608069890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:54.428 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.420+0000 7f760d077700 1 -- 172.21.15.112:0/1545707653 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 178337063 0 0) 0x7f7608155ab0 con 0x7f7608068d20 2023-01-24T03:30:54.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.420+0000 7f760d077700 1 -- 172.21.15.112:0/1545707653 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75f40033f0 con 0x7f7608068d20 2023-01-24T03:30:54.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.420+0000 7f760d077700 1 -- 172.21.15.112:0/1545707653 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2762182710 0 0) 0x7f75f40033f0 con 0x7f7608068d20 2023-01-24T03:30:54.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.420+0000 7f760d077700 1 -- 172.21.15.112:0/1545707653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f76080694c0 msgr2=0x7f7608069890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:54.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.420+0000 7f760d077700 1 --2- 172.21.15.112:0/1545707653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f76080694c0 0x7f7608069890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:54.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.420+0000 7f760d077700 1 -- 172.21.15.112:0/1545707653 >> v1:172.21.15.142:6789/0 conn(0x7f760805e420 legacy=0x7f76080c68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:54.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.420+0000 7f760d077700 1 -- 172.21.15.112:0/1545707653 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76080c73c0 con 0x7f7608068d20 2023-01-24T03:30:54.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.420+0000 7f760d077700 1 -- 172.21.15.112:0/1545707653 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7f7600001470 con 0x7f7608068d20 2023-01-24T03:30:54.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.420+0000 7f760d077700 1 -- 172.21.15.112:0/1545707653 >> v1:172.21.15.112:6789/0 conn(0x7f7608068d20 legacy=0x7f76080ae9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:54.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.420+0000 7f760d077700 1 --2- 172.21.15.112:0/1545707653 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f75f4003910 0x7f75f4003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:54.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.420+0000 7f760d077700 1 --2- 172.21.15.112:0/1545707653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f76080694c0 0x7f75f4004690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:54.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.420+0000 7f760d077700 1 --2- 172.21.15.112:0/1545707653 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75f4004bd0 0x7f75f4005040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:54.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.420+0000 7f760d077700 1 -- 172.21.15.112:0/1545707653 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f400b990 con 0x7f75f4004bd0 2023-01-24T03:30:54.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.420+0000 7f760d077700 1 -- 172.21.15.112:0/1545707653 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f400bad0 con 0x7f75f4003910 2023-01-24T03:30:54.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.420+0000 7f760d077700 1 -- 172.21.15.112:0/1545707653 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75f400bc90 con 0x7f76080694c0 2023-01-24T03:30:54.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.420+0000 7f760e87a700 1 --2- 172.21.15.112:0/1545707653 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75f4004bd0 0x7f75f4005040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:54.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.420+0000 7f760e079700 1 --2- 172.21.15.112:0/1545707653 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f75f4003910 0x7f75f4003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:54.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.420+0000 7f760d878700 1 --2- 172.21.15.112:0/1545707653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f76080694c0 0x7f75f4004690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:54.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.420+0000 7f760e87a700 1 -- 172.21.15.112:0/1545707653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f76080694c0 msgr2=0x7f75f4004690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:54.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.420+0000 7f760e87a700 1 --2- 172.21.15.112:0/1545707653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f76080694c0 0x7f75f4004690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:54.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.420+0000 7f760e87a700 1 -- 172.21.15.112:0/1545707653 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f75f4003910 msgr2=0x7f75f4003ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:54.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.420+0000 7f760e87a700 1 --2- 172.21.15.112:0/1545707653 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f75f4003910 0x7f75f4003ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:54.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.420+0000 7f760e87a700 1 -- 172.21.15.112:0/1545707653 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f40033f0 con 0x7f75f4004bd0 2023-01-24T03:30:54.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.420+0000 7f760e87a700 1 --2- 172.21.15.112:0/1545707653 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75f4004bd0 0x7f75f4005040 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f760000c190 tx=0x7f7600001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab6bf4a34b66dd24 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:54.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.421+0000 7f760d077700 1 -- 172.21.15.112:0/1545707653 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f760000cac0 con 0x7f75f4004bd0 2023-01-24T03:30:54.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.421+0000 7f760d077700 1 -- 172.21.15.112:0/1545707653 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f760000cc20 con 0x7f75f4004bd0 2023-01-24T03:30:54.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.421+0000 7f760d077700 1 -- 172.21.15.112:0/1545707653 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76000070d0 con 0x7f75f4004bd0 2023-01-24T03:30:54.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.421+0000 7f76102dd700 1 -- 172.21.15.112:0/1545707653 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75f4004bd0 msgr2=0x7f75f4005040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:54.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.421+0000 7f76102dd700 1 --2- 172.21.15.112:0/1545707653 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75f4004bd0 0x7f75f4005040 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f760000c190 tx=0x7f7600001e70 comp rx=0 tx=0).stop 2023-01-24T03:30:54.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.421+0000 7f760e87a700 1 -- 172.21.15.112:0/1545707653 reap_dead start 2023-01-24T03:30:54.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.421+0000 7f76102dd700 1 -- 172.21.15.112:0/1545707653 shutdown_connections 2023-01-24T03:30:54.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.421+0000 7f76102dd700 1 -- 172.21.15.112:0/1545707653 >> 172.21.15.112:0/1545707653 conn(0x7f760805d830 msgr2=0x7f760805dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:54.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.421+0000 7f76102dd700 1 -- 172.21.15.112:0/1545707653 shutdown_connections 2023-01-24T03:30:54.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.421+0000 7f76102dd700 1 -- 172.21.15.112:0/1545707653 wait complete. 2023-01-24T03:30:54.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.422+0000 7f76102dd700 1 Processor -- start 2023-01-24T03:30:54.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.422+0000 7f76102dd700 1 -- start start 2023-01-24T03:30:54.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.423+0000 7f76102dd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75f4004bd0 0x7f76080c1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:54.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.423+0000 7f76102dd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f76080c2ae0 0x7f760814d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:54.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.423+0000 7f76102dd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f760814dbf0 0x7f760814ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:54.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.423+0000 7f76102dd700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76080c1130 con 0x7f75f4004bd0 2023-01-24T03:30:54.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.423+0000 7f76102dd700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76080c0260 con 0x7f76080c2ae0 2023-01-24T03:30:54.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.423+0000 7f76102dd700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f760814e060 con 0x7f760814dbf0 2023-01-24T03:30:54.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.423+0000 7f760e079700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75f4004bd0 0x7f76080c1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:54.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.423+0000 7f760e079700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75f4004bd0 0x7f76080c1ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51280/0 (socket says 172.21.15.112:51280) 2023-01-24T03:30:54.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.423+0000 7f760e079700 1 -- 172.21.15.112:0/1831234628 learned_addr learned my addr 172.21.15.112:0/1831234628 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:54.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.423+0000 7f760d878700 1 --2- 172.21.15.112:0/1831234628 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f76080c2ae0 0x7f760814d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:54.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.423+0000 7f760e87a700 1 --2- 172.21.15.112:0/1831234628 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f760814dbf0 0x7f760814ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:54.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.423+0000 7f760e079700 1 -- 172.21.15.112:0/1831234628 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f760814dbf0 msgr2=0x7f760814ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:54.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.423+0000 7f760e079700 1 --2- 172.21.15.112:0/1831234628 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f760814dbf0 0x7f760814ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:54.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.423+0000 7f760e079700 1 -- 172.21.15.112:0/1831234628 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f76080c2ae0 msgr2=0x7f760814d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:54.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.423+0000 7f760e079700 1 --2- 172.21.15.112:0/1831234628 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f76080c2ae0 0x7f760814d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:54.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.423+0000 7f760e079700 1 -- 172.21.15.112:0/1831234628 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76080c04b0 con 0x7f75f4004bd0 2023-01-24T03:30:54.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.423+0000 7f760e87a700 1 --2- 172.21.15.112:0/1831234628 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f760814dbf0 0x7f760814ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:30:54.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.423+0000 7f760e079700 1 --2- 172.21.15.112:0/1831234628 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75f4004bd0 0x7f76080c1ec0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f75f8011f00 tx=0x7f75f800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:54.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.424+0000 7f76067fc700 1 -- 172.21.15.112:0/1831234628 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f800e040 con 0x7f75f4004bd0 2023-01-24T03:30:54.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.424+0000 7f76067fc700 1 -- 172.21.15.112:0/1831234628 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75f800f040 con 0x7f75f4004bd0 2023-01-24T03:30:54.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.424+0000 7f76067fc700 1 -- 172.21.15.112:0/1831234628 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f800fec0 con 0x7f75f4004bd0 2023-01-24T03:30:54.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.424+0000 7f76102dd700 1 -- 172.21.15.112:0/1831234628 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76080c0790 con 0x7f75f4004bd0 2023-01-24T03:30:54.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.424+0000 7f76102dd700 1 -- 172.21.15.112:0/1831234628 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76080c0d50 con 0x7f75f4004bd0 2023-01-24T03:30:54.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.424+0000 7f76102dd700 1 -- 172.21.15.112:0/1831234628 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f760805e230 con 0x7f75f4004bd0 2023-01-24T03:30:54.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.425+0000 7f76067fc700 1 -- 172.21.15.112:0/1831234628 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f75f8015640 con 0x7f75f4004bd0 2023-01-24T03:30:54.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.425+0000 7f76067fc700 1 --2- 172.21.15.112:0/1831234628 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f75e80611f0 0x7f75e80636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:54.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.425+0000 7f76067fc700 1 -- 172.21.15.112:0/1831234628 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f75f807e410 con 0x7f75f4004bd0 2023-01-24T03:30:54.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.425+0000 7f760d878700 1 --2- 172.21.15.112:0/1831234628 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f75e80611f0 0x7f75e80636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:54.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.425+0000 7f760d878700 1 --2- 172.21.15.112:0/1831234628 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f75e80611f0 0x7f75e80636a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f75fc007fd0 tx=0x7f75fc007bb0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:54.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.429+0000 7f76067fc700 1 -- 172.21.15.112:0/1831234628 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f75f804e250 con 0x7f75f4004bd0 2023-01-24T03:30:54.608 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.607+0000 7f76102dd700 1 -- 172.21.15.112:0/1831234628 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f760806f3a0 con 0x7f75f4004bd0 2023-01-24T03:30:54.609 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.607+0000 7f76067fc700 1 -- 172.21.15.112:0/1831234628 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v24) v1 ==== 92+0+10 (secure 0 0 0) 0x7f75f8022020 con 0x7f75f4004bd0 2023-01-24T03:30:54.609 INFO:teuthology.orchestra.run.smithi112.stdout:pg_num: 8 2023-01-24T03:30:54.612 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.609+0000 7f76102dd700 1 -- 172.21.15.112:0/1831234628 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f75e80611f0 msgr2=0x7f75e80636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:54.612 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.609+0000 7f76102dd700 1 --2- 172.21.15.112:0/1831234628 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f75e80611f0 0x7f75e80636a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f75fc007fd0 tx=0x7f75fc007bb0 comp rx=0 tx=0).stop 2023-01-24T03:30:54.612 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.609+0000 7f76102dd700 1 -- 172.21.15.112:0/1831234628 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75f4004bd0 msgr2=0x7f76080c1ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:54.612 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.609+0000 7f76102dd700 1 --2- 172.21.15.112:0/1831234628 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75f4004bd0 0x7f76080c1ec0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f75f8011f00 tx=0x7f75f800b040 comp rx=0 tx=0).stop 2023-01-24T03:30:54.612 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.609+0000 7f76102dd700 1 -- 172.21.15.112:0/1831234628 shutdown_connections 2023-01-24T03:30:54.613 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.609+0000 7f76102dd700 1 --2- 172.21.15.112:0/1831234628 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f75e80611f0 0x7f75e80636a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:54.613 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.609+0000 7f76102dd700 1 --2- 172.21.15.112:0/1831234628 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f75f4004bd0 0x7f76080c1ec0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:54.613 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.609+0000 7f76102dd700 1 --2- 172.21.15.112:0/1831234628 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f760814dbf0 0x7f760814ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:54.613 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.609+0000 7f76102dd700 1 --2- 172.21.15.112:0/1831234628 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f76080c2ae0 0x7f760814d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:54.613 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.609+0000 7f76102dd700 1 -- 172.21.15.112:0/1831234628 >> 172.21.15.112:0/1831234628 conn(0x7f760805d830 msgr2=0x7f76080caec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:54.614 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.609+0000 7f76102dd700 1 -- 172.21.15.112:0/1831234628 shutdown_connections 2023-01-24T03:30:54.614 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.609+0000 7f76102dd700 1 -- 172.21.15.112:0/1831234628 wait complete. 2023-01-24T03:30:54.623 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-01-24T03:30:54.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.735+0000 7f7b1c19e700 1 Processor -- start 2023-01-24T03:30:54.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.735+0000 7f7b1c19e700 1 -- start start 2023-01-24T03:30:54.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.736+0000 7f7b1c19e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7b140694c0 0x7f7b14069890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:54.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.736+0000 7f7b1c19e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7b14068d20 0x7f7b14155f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:54.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.736+0000 7f7b1c19e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b14155ab0 con 0x7f7b1405e420 2023-01-24T03:30:54.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.736+0000 7f7b1c19e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b14155550 con 0x7f7b14068d20 2023-01-24T03:30:54.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.736+0000 7f7b1c19e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b14156480 con 0x7f7b140694c0 2023-01-24T03:30:54.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.736+0000 7f7b1a73b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7b14068d20 0x7f7b14155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:54.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.736+0000 7f7b19f3a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7b140694c0 0x7f7b14069890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:54.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.736+0000 7f7b1a73b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7b14068d20 0x7f7b14155f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.112:43784/0 (socket says 172.21.15.112:43784) 2023-01-24T03:30:54.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.736+0000 7f7b1a73b700 1 -- 172.21.15.112:0/973568488 learned_addr learned my addr 172.21.15.112:0/973568488 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:54.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.736+0000 7f7b19739700 1 --1- 172.21.15.112:0/973568488 >> v1:172.21.15.142:6789/0 conn(0x7f7b1405e420 0x7f7b140c68d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.112:46070/0 (socket says 172.21.15.112:46070) 2023-01-24T03:30:54.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.737+0000 7f7b1a73b700 1 -- 172.21.15.112:0/973568488 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7b140694c0 msgr2=0x7f7b14069890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:54.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.737+0000 7f7b1a73b700 1 --2- 172.21.15.112:0/973568488 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7b140694c0 0x7f7b14069890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:54.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.737+0000 7f7b1a73b700 1 -- 172.21.15.112:0/973568488 >> v1:172.21.15.142:6789/0 conn(0x7f7b1405e420 legacy=0x7f7b140c68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:54.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.737+0000 7f7b1a73b700 1 -- 172.21.15.112:0/973568488 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b140c70e0 con 0x7f7b14068d20 2023-01-24T03:30:54.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.737+0000 7f7b1a73b700 1 --2- 172.21.15.112:0/973568488 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7b14068d20 0x7f7b14155f40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c006fd0 tx=0x7f7b0c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13699178a1f50be5 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:54.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.737+0000 7f7b18f38700 1 -- 172.21.15.112:0/973568488 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c011630 con 0x7f7b14068d20 2023-01-24T03:30:54.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.737+0000 7f7b18f38700 1 -- 172.21.15.112:0/973568488 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b0c0117d0 con 0x7f7b14068d20 2023-01-24T03:30:54.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.737+0000 7f7b18f38700 1 -- 172.21.15.112:0/973568488 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c011b90 con 0x7f7b14068d20 2023-01-24T03:30:54.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.737+0000 7f7b1c19e700 1 -- 172.21.15.112:0/973568488 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7b14068d20 msgr2=0x7f7b14155f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:54.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.737+0000 7f7b1c19e700 1 --2- 172.21.15.112:0/973568488 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7b14068d20 0x7f7b14155f40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c006fd0 tx=0x7f7b0c0090f0 comp rx=0 tx=0).stop 2023-01-24T03:30:54.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.738+0000 7f7b1c19e700 1 -- 172.21.15.112:0/973568488 shutdown_connections 2023-01-24T03:30:54.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.738+0000 7f7b1c19e700 1 --2- 172.21.15.112:0/973568488 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7b14068d20 0x7f7b14155f40 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:54.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.738+0000 7f7b1c19e700 1 --2- 172.21.15.112:0/973568488 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7b140694c0 0x7f7b14069890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:54.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.738+0000 7f7b1c19e700 1 -- 172.21.15.112:0/973568488 >> 172.21.15.112:0/973568488 conn(0x7f7b1405d830 msgr2=0x7f7b1405dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:54.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.738+0000 7f7b1c19e700 1 -- 172.21.15.112:0/973568488 shutdown_connections 2023-01-24T03:30:54.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.738+0000 7f7b1c19e700 1 -- 172.21.15.112:0/973568488 wait complete. 2023-01-24T03:30:54.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.739+0000 7f7b1c19e700 1 Processor -- start 2023-01-24T03:30:54.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.739+0000 7f7b1c19e700 1 -- start start 2023-01-24T03:30:54.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.739+0000 7f7b1c19e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7b1405e420 0x7f7b140c8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:54.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.739+0000 7f7b1c19e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7b14068d20 0x7f7b14150350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:54.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.739+0000 7f7b1c19e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b140694c0 0x7f7b14150890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:54.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.739+0000 7f7b1c19e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b141522b0 con 0x7f7b140694c0 2023-01-24T03:30:54.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.739+0000 7f7b1c19e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b14152df0 con 0x7f7b1405e420 2023-01-24T03:30:54.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.739+0000 7f7b1c19e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b14158760 con 0x7f7b14068d20 2023-01-24T03:30:54.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.739+0000 7f7b1a73b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b140694c0 0x7f7b14150890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:54.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.739+0000 7f7b1a73b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b140694c0 0x7f7b14150890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51296/0 (socket says 172.21.15.112:51296) 2023-01-24T03:30:54.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.739+0000 7f7b1a73b700 1 -- 172.21.15.112:0/3217184171 learned_addr learned my addr 172.21.15.112:0/3217184171 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:54.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.740+0000 7f7b19f3a700 1 --2- 172.21.15.112:0/3217184171 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7b1405e420 0x7f7b140c8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:54.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.740+0000 7f7b19739700 1 --2- 172.21.15.112:0/3217184171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7b14068d20 0x7f7b14150350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:54.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.740+0000 7f7b1a73b700 1 -- 172.21.15.112:0/3217184171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7b14068d20 msgr2=0x7f7b14150350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:54.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.740+0000 7f7b1a73b700 1 --2- 172.21.15.112:0/3217184171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7b14068d20 0x7f7b14150350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:54.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.740+0000 7f7b1a73b700 1 -- 172.21.15.112:0/3217184171 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7b1405e420 msgr2=0x7f7b140c8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:54.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.740+0000 7f7b1a73b700 1 --2- 172.21.15.112:0/3217184171 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7b1405e420 0x7f7b140c8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:54.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.740+0000 7f7b1a73b700 1 -- 172.21.15.112:0/3217184171 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b0c009370 con 0x7f7b140694c0 2023-01-24T03:30:54.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.740+0000 7f7b19739700 1 --2- 172.21.15.112:0/3217184171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7b14068d20 0x7f7b14150350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-01-24T03:30:54.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.740+0000 7f7b1a73b700 1 --2- 172.21.15.112:0/3217184171 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b140694c0 0x7f7b14150890 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c009130 tx=0x7f7b0c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:54.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.740+0000 7f7b11ffb700 1 -- 172.21.15.112:0/3217184171 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c027480 con 0x7f7b140694c0 2023-01-24T03:30:54.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.740+0000 7f7b11ffb700 1 -- 172.21.15.112:0/3217184171 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b0c020070 con 0x7f7b140694c0 2023-01-24T03:30:54.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.740+0000 7f7b1c19e700 1 -- 172.21.15.112:0/3217184171 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b141589d0 con 0x7f7b140694c0 2023-01-24T03:30:54.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.740+0000 7f7b11ffb700 1 -- 172.21.15.112:0/3217184171 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c027870 con 0x7f7b140694c0 2023-01-24T03:30:54.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.740+0000 7f7b1c19e700 1 -- 172.21.15.112:0/3217184171 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b141527a0 con 0x7f7b140694c0 2023-01-24T03:30:54.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.741+0000 7f7b1c19e700 1 -- 172.21.15.112:0/3217184171 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b1406f3a0 con 0x7f7b140694c0 2023-01-24T03:30:54.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.741+0000 7f7b11ffb700 1 -- 172.21.15.112:0/3217184171 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f7b0c017020 con 0x7f7b140694c0 2023-01-24T03:30:54.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.742+0000 7f7b11ffb700 1 --2- 172.21.15.112:0/3217184171 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7af40611c0 0x7f7af4063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:54.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.742+0000 7f7b11ffb700 1 -- 172.21.15.112:0/3217184171 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f7b0c022070 con 0x7f7b140694c0 2023-01-24T03:30:54.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.742+0000 7f7b19f3a700 1 --2- 172.21.15.112:0/3217184171 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7af40611c0 0x7f7af4063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:54.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.742+0000 7f7b19f3a700 1 --2- 172.21.15.112:0/3217184171 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7af40611c0 0x7f7af4063670 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7b04007600 tx=0x7f7b0400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:54.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.746+0000 7f7b11ffb700 1 -- 172.21.15.112:0/3217184171 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f7b0c04f620 con 0x7f7b140694c0 2023-01-24T03:30:54.924 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.922+0000 7f7b1c19e700 1 -- 172.21.15.112:0/3217184171 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7b140c2880 con 0x7f7b140694c0 2023-01-24T03:30:54.924 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.922+0000 7f7b11ffb700 1 -- 172.21.15.112:0/3217184171 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7b0c052c40 con 0x7f7b140694c0 2023-01-24T03:30:54.925 INFO:teuthology.orchestra.run.smithi112.stdout:pg_num: 1 2023-01-24T03:30:54.926 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.924+0000 7f7b1c19e700 1 -- 172.21.15.112:0/3217184171 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7af40611c0 msgr2=0x7f7af4063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:54.927 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.924+0000 7f7b1c19e700 1 --2- 172.21.15.112:0/3217184171 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7af40611c0 0x7f7af4063670 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7b04007600 tx=0x7f7b0400b040 comp rx=0 tx=0).stop 2023-01-24T03:30:54.927 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.924+0000 7f7b1c19e700 1 -- 172.21.15.112:0/3217184171 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b140694c0 msgr2=0x7f7b14150890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:54.927 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.924+0000 7f7b1c19e700 1 --2- 172.21.15.112:0/3217184171 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b140694c0 0x7f7b14150890 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c009130 tx=0x7f7b0c016000 comp rx=0 tx=0).stop 2023-01-24T03:30:54.927 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.924+0000 7f7b1c19e700 1 -- 172.21.15.112:0/3217184171 shutdown_connections 2023-01-24T03:30:54.928 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.924+0000 7f7b1c19e700 1 --2- 172.21.15.112:0/3217184171 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7af40611c0 0x7f7af4063670 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:54.928 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.924+0000 7f7b1c19e700 1 --2- 172.21.15.112:0/3217184171 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b140694c0 0x7f7b14150890 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:54.928 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.924+0000 7f7b1c19e700 1 --2- 172.21.15.112:0/3217184171 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7b1405e420 0x7f7b140c8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:54.928 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.924+0000 7f7b1c19e700 1 --2- 172.21.15.112:0/3217184171 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7b14068d20 0x7f7b14150350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:54.928 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.924+0000 7f7b1c19e700 1 -- 172.21.15.112:0/3217184171 >> 172.21.15.112:0/3217184171 conn(0x7f7b1405d830 msgr2=0x7f7b14153df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:54.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.924+0000 7f7b1c19e700 1 -- 172.21.15.112:0/3217184171 shutdown_connections 2023-01-24T03:30:54.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:54.924+0000 7f7b1c19e700 1 -- 172.21.15.112:0/3217184171 wait complete. 2023-01-24T03:30:54.938 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-01-24T03:30:55.053 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.051+0000 7fac1551f700 1 Processor -- start 2023-01-24T03:30:55.053 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.051+0000 7fac1551f700 1 -- start start 2023-01-24T03:30:55.055 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.051+0000 7fac1551f700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fac100c8e20 0x7fac10069960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:55.055 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.051+0000 7fac1551f700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fac101569e0 0x7fac100ae9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:55.056 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.051+0000 7fac1551f700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac101591f0 con 0x7fac1005e340 2023-01-24T03:30:55.056 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.052+0000 7fac1551f700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac10159a10 con 0x7fac101569e0 2023-01-24T03:30:55.056 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.052+0000 7fac1551f700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac10156e10 con 0x7fac100c8e20 2023-01-24T03:30:55.056 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.052+0000 7fac0e7fc700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fac1005e340 0x7fac10158ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:60522/0 (socket says 172.21.15.112:60522) 2023-01-24T03:30:55.057 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.052+0000 7fac0e7fc700 1 -- 172.21.15.112:0/835829895 learned_addr learned my addr 172.21.15.112:0/835829895 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:55.057 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.052+0000 7fac0f7fe700 1 --2- 172.21.15.112:0/835829895 >> v2:172.21.15.112:3300/0 conn(0x7fac101569e0 0x7fac100ae9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:55.057 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.052+0000 7fac0effd700 1 --2- 172.21.15.112:0/835829895 >> v2:172.21.15.142:3300/0 conn(0x7fac100c8e20 0x7fac10069960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:55.057 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.052+0000 7fac0dffb700 1 -- 172.21.15.112:0/835829895 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1800838344 0 0) 0x7fac101591f0 con 0x7fac1005e340 2023-01-24T03:30:55.057 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.052+0000 7fac0dffb700 1 -- 172.21.15.112:0/835829895 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabf00033f0 con 0x7fac1005e340 2023-01-24T03:30:55.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.052+0000 7fac0f7fe700 1 -- 172.21.15.112:0/835829895 >> v2:172.21.15.142:3300/0 conn(0x7fac100c8e20 msgr2=0x7fac10069960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:55.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.052+0000 7fac0f7fe700 1 --2- 172.21.15.112:0/835829895 >> v2:172.21.15.142:3300/0 conn(0x7fac100c8e20 0x7fac10069960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:55.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.052+0000 7fac0f7fe700 1 -- 172.21.15.112:0/835829895 >> v1:172.21.15.112:6789/0 conn(0x7fac1005e340 legacy=0x7fac10158ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:55.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.052+0000 7fac0f7fe700 1 -- 172.21.15.112:0/835829895 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac10159450 con 0x7fac101569e0 2023-01-24T03:30:55.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.052+0000 7fac0f7fe700 1 --2- 172.21.15.112:0/835829895 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fac101569e0 0x7fac100ae9f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fac10089390 tx=0x7fac00009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=983a3c844dcfb37e server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:55.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.052+0000 7fac0dffb700 1 -- 172.21.15.112:0/835829895 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac000105b0 con 0x7fac101569e0 2023-01-24T03:30:55.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.052+0000 7fac0dffb700 1 -- 172.21.15.112:0/835829895 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac00010710 con 0x7fac101569e0 2023-01-24T03:30:55.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.052+0000 7fac0dffb700 1 -- 172.21.15.112:0/835829895 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac00010ad0 con 0x7fac101569e0 2023-01-24T03:30:55.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.053+0000 7fac1551f700 1 -- 172.21.15.112:0/835829895 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fac101569e0 msgr2=0x7fac100ae9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:55.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.053+0000 7fac1551f700 1 --2- 172.21.15.112:0/835829895 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fac101569e0 0x7fac100ae9f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fac10089390 tx=0x7fac00009060 comp rx=0 tx=0).stop 2023-01-24T03:30:55.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.053+0000 7fac1551f700 1 -- 172.21.15.112:0/835829895 shutdown_connections 2023-01-24T03:30:55.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.053+0000 7fac1551f700 1 --2- 172.21.15.112:0/835829895 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fac101569e0 0x7fac100ae9f0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:55.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.053+0000 7fac1551f700 1 --2- 172.21.15.112:0/835829895 >> v2:172.21.15.142:3300/0 conn(0x7fac100c8e20 0x7fac10069960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:55.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.053+0000 7fac1551f700 1 -- 172.21.15.112:0/835829895 >> 172.21.15.112:0/835829895 conn(0x7fac1005d830 msgr2=0x7fac1005dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:55.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.053+0000 7fac1551f700 1 -- 172.21.15.112:0/835829895 shutdown_connections 2023-01-24T03:30:55.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.053+0000 7fac1551f700 1 -- 172.21.15.112:0/835829895 wait complete. 2023-01-24T03:30:55.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.054+0000 7fac1551f700 1 Processor -- start 2023-01-24T03:30:55.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.054+0000 7fac1551f700 1 -- start start 2023-01-24T03:30:55.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.055+0000 7fac1551f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fac1005e340 0x7fac101654d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:55.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.055+0000 7fac1551f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fac100c8e20 0x7fac10169430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:55.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.055+0000 7fac1551f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fac101569e0 0x7fac100bfa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:55.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.055+0000 7fac1551f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac10168a20 con 0x7fac1005e340 2023-01-24T03:30:55.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.055+0000 7fac1551f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac100c17d0 con 0x7fac100c8e20 2023-01-24T03:30:55.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.055+0000 7fac1551f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac10168fa0 con 0x7fac101569e0 2023-01-24T03:30:55.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.055+0000 7fac0effd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fac1005e340 0x7fac101654d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:55.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.055+0000 7fac0effd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fac1005e340 0x7fac101654d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51308/0 (socket says 172.21.15.112:51308) 2023-01-24T03:30:55.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.055+0000 7fac0effd700 1 -- 172.21.15.112:0/3817444164 learned_addr learned my addr 172.21.15.112:0/3817444164 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:55.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.055+0000 7fac0e7fc700 1 --2- 172.21.15.112:0/3817444164 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fac100c8e20 0x7fac10169430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:55.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.055+0000 7fac0f7fe700 1 --2- 172.21.15.112:0/3817444164 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fac101569e0 0x7fac100bfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:55.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.055+0000 7fac0effd700 1 -- 172.21.15.112:0/3817444164 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fac101569e0 msgr2=0x7fac100bfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:55.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.055+0000 7fac0effd700 1 --2- 172.21.15.112:0/3817444164 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fac101569e0 0x7fac100bfa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:55.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.055+0000 7fac0effd700 1 -- 172.21.15.112:0/3817444164 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fac100c8e20 msgr2=0x7fac10169430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:55.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.055+0000 7fac0effd700 1 --2- 172.21.15.112:0/3817444164 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fac100c8e20 0x7fac10169430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:55.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.055+0000 7fac0effd700 1 -- 172.21.15.112:0/3817444164 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac000092a0 con 0x7fac1005e340 2023-01-24T03:30:55.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.055+0000 7fac0e7fc700 1 --2- 172.21.15.112:0/3817444164 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fac100c8e20 0x7fac10169430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:30:55.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.055+0000 7fac0effd700 1 --2- 172.21.15.112:0/3817444164 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fac1005e340 0x7fac101654d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fac08009e80 tx=0x7fac0800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:55.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.055+0000 7fac06ffd700 1 -- 172.21.15.112:0/3817444164 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0800ccb0 con 0x7fac1005e340 2023-01-24T03:30:55.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.056+0000 7fac06ffd700 1 -- 172.21.15.112:0/3817444164 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac0800f040 con 0x7fac1005e340 2023-01-24T03:30:55.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.056+0000 7fac1551f700 1 -- 172.21.15.112:0/3817444164 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac100c1a20 con 0x7fac1005e340 2023-01-24T03:30:55.069 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.056+0000 7fac06ffd700 1 -- 172.21.15.112:0/3817444164 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac08014480 con 0x7fac1005e340 2023-01-24T03:30:55.069 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.056+0000 7fac1551f700 1 -- 172.21.15.112:0/3817444164 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac100c1fe0 con 0x7fac1005e340 2023-01-24T03:30:55.069 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.057+0000 7fac1551f700 1 -- 172.21.15.112:0/3817444164 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac1006f3a0 con 0x7fac1005e340 2023-01-24T03:30:55.069 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.057+0000 7fac06ffd700 1 -- 172.21.15.112:0/3817444164 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fac080147f0 con 0x7fac1005e340 2023-01-24T03:30:55.069 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.057+0000 7fac06ffd700 1 --2- 172.21.15.112:0/3817444164 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fabf40611c0 0x7fabf4063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:55.070 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.057+0000 7fac06ffd700 1 -- 172.21.15.112:0/3817444164 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fac0807e840 con 0x7fac1005e340 2023-01-24T03:30:55.070 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.057+0000 7fac0e7fc700 1 --2- 172.21.15.112:0/3817444164 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fabf40611c0 0x7fabf4063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:55.070 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.058+0000 7fac0e7fc700 1 --2- 172.21.15.112:0/3817444164 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fabf40611c0 0x7fabf4063670 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fabf8007fd0 tx=0x7fabf8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:55.070 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.061+0000 7fac06ffd700 1 -- 172.21.15.112:0/3817444164 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fac0804e680 con 0x7fac1005e340 2023-01-24T03:30:55.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.237+0000 7fac1551f700 1 -- 172.21.15.112:0/3817444164 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fac100680e0 con 0x7fac1005e340 2023-01-24T03:30:55.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.238+0000 7fac06ffd700 1 -- 172.21.15.112:0/3817444164 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7fac08051ca0 con 0x7fac1005e340 2023-01-24T03:30:55.240 INFO:teuthology.orchestra.run.smithi112.stdout:pg_num: 32 2023-01-24T03:30:55.242 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.239+0000 7fac1551f700 1 -- 172.21.15.112:0/3817444164 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fabf40611c0 msgr2=0x7fabf4063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:55.242 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.239+0000 7fac1551f700 1 --2- 172.21.15.112:0/3817444164 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fabf40611c0 0x7fabf4063670 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fabf8007fd0 tx=0x7fabf8009040 comp rx=0 tx=0).stop 2023-01-24T03:30:55.242 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.239+0000 7fac1551f700 1 -- 172.21.15.112:0/3817444164 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fac1005e340 msgr2=0x7fac101654d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:55.243 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.239+0000 7fac1551f700 1 --2- 172.21.15.112:0/3817444164 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fac1005e340 0x7fac101654d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fac08009e80 tx=0x7fac0800b040 comp rx=0 tx=0).stop 2023-01-24T03:30:55.243 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.239+0000 7fac1551f700 1 -- 172.21.15.112:0/3817444164 shutdown_connections 2023-01-24T03:30:55.243 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.239+0000 7fac1551f700 1 --2- 172.21.15.112:0/3817444164 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fabf40611c0 0x7fabf4063670 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:55.243 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.239+0000 7fac1551f700 1 --2- 172.21.15.112:0/3817444164 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fac1005e340 0x7fac101654d0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:55.244 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.239+0000 7fac1551f700 1 --2- 172.21.15.112:0/3817444164 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fac101569e0 0x7fac100bfa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:55.244 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.239+0000 7fac1551f700 1 --2- 172.21.15.112:0/3817444164 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fac100c8e20 0x7fac10169430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:55.244 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.239+0000 7fac1551f700 1 -- 172.21.15.112:0/3817444164 >> 172.21.15.112:0/3817444164 conn(0x7fac1005d830 msgr2=0x7fac1005dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:55.244 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.239+0000 7fac1551f700 1 -- 172.21.15.112:0/3817444164 shutdown_connections 2023-01-24T03:30:55.244 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.240+0000 7fac1551f700 1 -- 172.21.15.112:0/3817444164 wait complete. 2023-01-24T03:30:55.253 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-01-24T03:30:55.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.366+0000 7f1877279700 1 Processor -- start 2023-01-24T03:30:55.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.366+0000 7f1877279700 1 -- start start 2023-01-24T03:30:55.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.367+0000 7f1877279700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f18700c40a0 0x7f1870152fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:55.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.367+0000 7f1877279700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f18700c2570 0x7f18700c4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:55.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.367+0000 7f1877279700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18700c8c60 con 0x7f18700c49b0 2023-01-24T03:30:55.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.367+0000 7f1877279700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18700c9190 con 0x7f18700c2570 2023-01-24T03:30:55.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.367+0000 7f1877279700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f187006edd0 con 0x7f18700c40a0 2023-01-24T03:30:55.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.367+0000 7f1876a78700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f18700c49b0 0x7f1870066e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:60540/0 (socket says 172.21.15.112:60540) 2023-01-24T03:30:55.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.367+0000 7f1876a78700 1 -- 172.21.15.112:0/1449248867 learned_addr learned my addr 172.21.15.112:0/1449248867 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:55.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.367+0000 7f1875a76700 1 --2- 172.21.15.112:0/1449248867 >> v2:172.21.15.112:3300/0 conn(0x7f18700c2570 0x7f18700c4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:55.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.367+0000 7f1876277700 1 --2- 172.21.15.112:0/1449248867 >> v2:172.21.15.142:3300/0 conn(0x7f18700c40a0 0x7f1870152fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:55.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.367+0000 7f1875a76700 1 -- 172.21.15.112:0/1449248867 >> v2:172.21.15.142:3300/0 conn(0x7f18700c40a0 msgr2=0x7f1870152fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:55.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.367+0000 7f1875a76700 1 --2- 172.21.15.112:0/1449248867 >> v2:172.21.15.142:3300/0 conn(0x7f18700c40a0 0x7f1870152fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:55.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.367+0000 7f1875a76700 1 -- 172.21.15.112:0/1449248867 >> v1:172.21.15.112:6789/0 conn(0x7f18700c49b0 legacy=0x7f1870066e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:55.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.367+0000 7f1875a76700 1 -- 172.21.15.112:0/1449248867 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18700c8eb0 con 0x7f18700c2570 2023-01-24T03:30:55.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.367+0000 7f1875275700 1 -- 172.21.15.112:0/1449248867 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1434924286 0 0) 0x7f18700c8c60 con 0x7f18700c49b0 2023-01-24T03:30:55.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.367+0000 7f1875a76700 1 --2- 172.21.15.112:0/1449248867 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f18700c2570 0x7f18700c4470 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f187005e4d0 tx=0x7f1860009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf7da5ffa646cd9d server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:55.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.368+0000 7f1875275700 1 -- 172.21.15.112:0/1449248867 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1860011630 con 0x7f18700c2570 2023-01-24T03:30:55.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.368+0000 7f1875275700 1 -- 172.21.15.112:0/1449248867 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1860011790 con 0x7f18700c2570 2023-01-24T03:30:55.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.368+0000 7f1875275700 1 -- 172.21.15.112:0/1449248867 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1860011b50 con 0x7f18700c2570 2023-01-24T03:30:55.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.368+0000 7f1877279700 1 -- 172.21.15.112:0/1449248867 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f18700c2570 msgr2=0x7f18700c4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:55.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.368+0000 7f1877279700 1 --2- 172.21.15.112:0/1449248867 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f18700c2570 0x7f18700c4470 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f187005e4d0 tx=0x7f1860009060 comp rx=0 tx=0).stop 2023-01-24T03:30:55.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.368+0000 7f1877279700 1 -- 172.21.15.112:0/1449248867 shutdown_connections 2023-01-24T03:30:55.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.368+0000 7f1877279700 1 --2- 172.21.15.112:0/1449248867 >> v2:172.21.15.142:3300/0 conn(0x7f18700c40a0 0x7f1870152fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:55.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.368+0000 7f1877279700 1 --2- 172.21.15.112:0/1449248867 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f18700c2570 0x7f18700c4470 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:55.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.368+0000 7f1877279700 1 -- 172.21.15.112:0/1449248867 >> 172.21.15.112:0/1449248867 conn(0x7f18700689f0 msgr2=0x7f18700c8550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:55.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.368+0000 7f1877279700 1 -- 172.21.15.112:0/1449248867 shutdown_connections 2023-01-24T03:30:55.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.368+0000 7f1877279700 1 -- 172.21.15.112:0/1449248867 wait complete. 2023-01-24T03:30:55.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.369+0000 7f1877279700 1 Processor -- start 2023-01-24T03:30:55.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.369+0000 7f1877279700 1 -- start start 2023-01-24T03:30:55.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.370+0000 7f1877279700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f18700c40a0 0x7f1870151620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:55.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.370+0000 7f1877279700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f18700c49b0 0x7f18701530e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:55.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.370+0000 7f1877279700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f18700beed0 0x7f18700bdc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:55.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.370+0000 7f1877279700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18700bcf40 con 0x7f18700c49b0 2023-01-24T03:30:55.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.370+0000 7f1877279700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18700bf990 con 0x7f18700c40a0 2023-01-24T03:30:55.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.370+0000 7f1877279700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18700bd7a0 con 0x7f18700beed0 2023-01-24T03:30:55.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.370+0000 7f1875a76700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f18700c49b0 0x7f18701530e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:55.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.370+0000 7f1875a76700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f18700c49b0 0x7f18701530e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51324/0 (socket says 172.21.15.112:51324) 2023-01-24T03:30:55.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.370+0000 7f1875a76700 1 -- 172.21.15.112:0/3145925234 learned_addr learned my addr 172.21.15.112:0/3145925234 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:55.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.370+0000 7f1876277700 1 --2- 172.21.15.112:0/3145925234 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f18700c40a0 0x7f1870151620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:55.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.370+0000 7f1876a78700 1 --2- 172.21.15.112:0/3145925234 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f18700beed0 0x7f18700bdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:55.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.370+0000 7f1875a76700 1 -- 172.21.15.112:0/3145925234 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f18700beed0 msgr2=0x7f18700bdc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:55.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.370+0000 7f1875a76700 1 --2- 172.21.15.112:0/3145925234 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f18700beed0 0x7f18700bdc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:55.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.370+0000 7f1875a76700 1 -- 172.21.15.112:0/3145925234 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f18700c40a0 msgr2=0x7f1870151620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:55.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.370+0000 7f1875a76700 1 --2- 172.21.15.112:0/3145925234 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f18700c40a0 0x7f1870151620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:55.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.370+0000 7f1875a76700 1 -- 172.21.15.112:0/3145925234 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18600092a0 con 0x7f18700c49b0 2023-01-24T03:30:55.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.370+0000 7f1875a76700 1 --2- 172.21.15.112:0/3145925234 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f18700c49b0 0x7f18701530e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1860009160 tx=0x7f1860017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:55.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.370+0000 7f185e7fc700 1 -- 172.21.15.112:0/3145925234 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1860000bc0 con 0x7f18700c49b0 2023-01-24T03:30:55.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.370+0000 7f185e7fc700 1 -- 172.21.15.112:0/3145925234 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1860011900 con 0x7f18700c49b0 2023-01-24T03:30:55.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.370+0000 7f1877279700 1 -- 172.21.15.112:0/3145925234 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18700bc870 con 0x7f18700c49b0 2023-01-24T03:30:55.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.370+0000 7f185e7fc700 1 -- 172.21.15.112:0/3145925234 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1860025420 con 0x7f18700c49b0 2023-01-24T03:30:55.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.371+0000 7f1877279700 1 -- 172.21.15.112:0/3145925234 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18700bd150 con 0x7f18700c49b0 2023-01-24T03:30:55.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.371+0000 7f1877279700 1 -- 172.21.15.112:0/3145925234 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18700c5f50 con 0x7f18700c49b0 2023-01-24T03:30:55.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.372+0000 7f185e7fc700 1 -- 172.21.15.112:0/3145925234 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f186002c030 con 0x7f18700c49b0 2023-01-24T03:30:55.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.372+0000 7f185e7fc700 1 --2- 172.21.15.112:0/3145925234 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f18500611c0 0x7f1850063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:55.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.372+0000 7f185e7fc700 1 -- 172.21.15.112:0/3145925234 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f186007eab0 con 0x7f18700c49b0 2023-01-24T03:30:55.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.372+0000 7f1876277700 1 --2- 172.21.15.112:0/3145925234 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f18500611c0 0x7f1850063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:55.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.372+0000 7f1876277700 1 --2- 172.21.15.112:0/3145925234 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f18500611c0 0x7f1850063670 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f186c00bb30 tx=0x7f186c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:55.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.376+0000 7f185e7fc700 1 -- 172.21.15.112:0/3145925234 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f186004e9a0 con 0x7f18700c49b0 2023-01-24T03:30:55.555 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.553+0000 7f1877279700 1 -- 172.21.15.112:0/3145925234 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f187005ea20 con 0x7f18700c49b0 2023-01-24T03:30:55.556 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.554+0000 7f185e7fc700 1 -- 172.21.15.112:0/3145925234 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1860015020 con 0x7f18700c49b0 2023-01-24T03:30:55.556 INFO:teuthology.orchestra.run.smithi112.stdout:pg_num: 64 2023-01-24T03:30:55.558 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.555+0000 7f1877279700 1 -- 172.21.15.112:0/3145925234 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f18500611c0 msgr2=0x7f1850063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:55.558 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.555+0000 7f1877279700 1 --2- 172.21.15.112:0/3145925234 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f18500611c0 0x7f1850063670 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f186c00bb30 tx=0x7f186c008040 comp rx=0 tx=0).stop 2023-01-24T03:30:55.558 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.555+0000 7f1877279700 1 -- 172.21.15.112:0/3145925234 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f18700c49b0 msgr2=0x7f18701530e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:55.558 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.556+0000 7f1877279700 1 --2- 172.21.15.112:0/3145925234 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f18700c49b0 0x7f18701530e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1860009160 tx=0x7f1860017000 comp rx=0 tx=0).stop 2023-01-24T03:30:55.559 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.556+0000 7f1877279700 1 -- 172.21.15.112:0/3145925234 shutdown_connections 2023-01-24T03:30:55.559 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.556+0000 7f1877279700 1 --2- 172.21.15.112:0/3145925234 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f18500611c0 0x7f1850063670 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:55.559 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.556+0000 7f1877279700 1 --2- 172.21.15.112:0/3145925234 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f18700beed0 0x7f18700bdc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:55.559 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.556+0000 7f1877279700 1 --2- 172.21.15.112:0/3145925234 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f18700c40a0 0x7f1870151620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:55.559 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.556+0000 7f1877279700 1 --2- 172.21.15.112:0/3145925234 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f18700c49b0 0x7f18701530e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:55.560 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.556+0000 7f1877279700 1 -- 172.21.15.112:0/3145925234 >> 172.21.15.112:0/3145925234 conn(0x7f18700689f0 msgr2=0x7f187006b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:55.560 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.556+0000 7f1877279700 1 -- 172.21.15.112:0/3145925234 shutdown_connections 2023-01-24T03:30:55.560 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.556+0000 7f1877279700 1 -- 172.21.15.112:0/3145925234 wait complete. 2023-01-24T03:30:55.569 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-01-24T03:30:55.683 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.682+0000 7f947f4c4700 1 Processor -- start 2023-01-24T03:30:55.684 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.682+0000 7f947f4c4700 1 -- start start 2023-01-24T03:30:55.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.682+0000 7f947f4c4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f94780c6420 0x7f94780c4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:55.687 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.682+0000 7f947f4c4700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f9478155ba0 0x7f94780c8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:55.687 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.682+0000 7f947f4c4700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94780c82e0 con 0x7f947814f3f0 2023-01-24T03:30:55.687 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.682+0000 7f947f4c4700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94780c8cb0 con 0x7f9478155ba0 2023-01-24T03:30:55.688 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.682+0000 7f947f4c4700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94780c2980 con 0x7f94780c6420 2023-01-24T03:30:55.688 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.683+0000 7f947ecc3700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f9478155ba0 0x7f94780c8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:55.688 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.683+0000 7f947ecc3700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f9478155ba0 0x7f94780c8770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51334/0 (socket says 172.21.15.112:51334) 2023-01-24T03:30:55.689 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.683+0000 7f947ecc3700 1 -- 172.21.15.112:0/3471391012 learned_addr learned my addr 172.21.15.112:0/3471391012 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:55.689 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.683+0000 7f947dcc1700 1 --2- 172.21.15.112:0/3471391012 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f94780c6420 0x7f94780c4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:55.689 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.683+0000 7f947ecc3700 1 -- 172.21.15.112:0/3471391012 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f94780c6420 msgr2=0x7f94780c4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:55.689 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.683+0000 7f947ecc3700 1 --2- 172.21.15.112:0/3471391012 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f94780c6420 0x7f94780c4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:55.689 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.683+0000 7f947ecc3700 1 -- 172.21.15.112:0/3471391012 >> v1:172.21.15.142:6789/0 conn(0x7f947814f3f0 legacy=0x7f94780c2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:55.690 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.683+0000 7f947ecc3700 1 -- 172.21.15.112:0/3471391012 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94780c5290 con 0x7f9478155ba0 2023-01-24T03:30:55.690 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.683+0000 7f947dcc1700 1 --2- 172.21.15.112:0/3471391012 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f94780c6420 0x7f94780c4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:30:55.690 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.683+0000 7f947ecc3700 1 --2- 172.21.15.112:0/3471391012 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9478155ba0 0x7f94780c8770 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9470006fd0 tx=0x7f94700090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a2a12f817df01d3 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:55.690 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.683+0000 7f947d4c0700 1 -- 172.21.15.112:0/3471391012 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9470011630 con 0x7f9478155ba0 2023-01-24T03:30:55.690 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.683+0000 7f947d4c0700 1 -- 172.21.15.112:0/3471391012 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9470011790 con 0x7f9478155ba0 2023-01-24T03:30:55.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.683+0000 7f947d4c0700 1 -- 172.21.15.112:0/3471391012 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9470011b50 con 0x7f9478155ba0 2023-01-24T03:30:55.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.684+0000 7f947f4c4700 1 -- 172.21.15.112:0/3471391012 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9478155ba0 msgr2=0x7f94780c8770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:55.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.684+0000 7f947f4c4700 1 --2- 172.21.15.112:0/3471391012 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9478155ba0 0x7f94780c8770 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9470006fd0 tx=0x7f94700090f0 comp rx=0 tx=0).stop 2023-01-24T03:30:55.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.684+0000 7f947f4c4700 1 -- 172.21.15.112:0/3471391012 shutdown_connections 2023-01-24T03:30:55.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.684+0000 7f947f4c4700 1 --2- 172.21.15.112:0/3471391012 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9478155ba0 0x7f94780c8770 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:55.692 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.684+0000 7f947f4c4700 1 --2- 172.21.15.112:0/3471391012 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f94780c6420 0x7f94780c4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:55.692 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.684+0000 7f947f4c4700 1 -- 172.21.15.112:0/3471391012 >> 172.21.15.112:0/3471391012 conn(0x7f947805d830 msgr2=0x7f947805dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:55.692 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.684+0000 7f947f4c4700 1 -- 172.21.15.112:0/3471391012 shutdown_connections 2023-01-24T03:30:55.692 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.684+0000 7f947f4c4700 1 -- 172.21.15.112:0/3471391012 wait complete. 2023-01-24T03:30:55.692 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.685+0000 7f947f4c4700 1 Processor -- start 2023-01-24T03:30:55.693 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.685+0000 7f947f4c4700 1 -- start start 2023-01-24T03:30:55.695 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.685+0000 7f947f4c4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f94780c6420 0x7f947814cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:55.695 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.685+0000 7f947f4c4700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f947814f3f0 0x7f9478156360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:55.696 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.685+0000 7f947f4c4700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9478155ba0 0x7f9478148610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:55.696 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.685+0000 7f947f4c4700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94780c27c0 con 0x7f9478155ba0 2023-01-24T03:30:55.696 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.685+0000 7f947f4c4700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947814ddd0 con 0x7f947814f3f0 2023-01-24T03:30:55.696 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.685+0000 7f947f4c4700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f947814ec30 con 0x7f94780c6420 2023-01-24T03:30:55.696 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.686+0000 7f947ecc3700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9478155ba0 0x7f9478148610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:55.697 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.686+0000 7f947ecc3700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9478155ba0 0x7f9478148610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51340/0 (socket says 172.21.15.112:51340) 2023-01-24T03:30:55.697 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.686+0000 7f947ecc3700 1 -- 172.21.15.112:0/3812951518 learned_addr learned my addr 172.21.15.112:0/3812951518 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:55.697 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.686+0000 7f947e4c2700 1 --2- 172.21.15.112:0/3812951518 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f94780c6420 0x7f947814cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:55.697 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.686+0000 7f947dcc1700 1 --2- 172.21.15.112:0/3812951518 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f947814f3f0 0x7f9478156360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:55.697 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.686+0000 7f947ecc3700 1 -- 172.21.15.112:0/3812951518 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f94780c6420 msgr2=0x7f947814cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:55.698 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.686+0000 7f947ecc3700 1 --2- 172.21.15.112:0/3812951518 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f94780c6420 0x7f947814cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:55.698 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.686+0000 7f947ecc3700 1 -- 172.21.15.112:0/3812951518 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f947814f3f0 msgr2=0x7f9478156360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:55.698 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.686+0000 7f947ecc3700 1 --2- 172.21.15.112:0/3812951518 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f947814f3f0 0x7f9478156360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:55.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.686+0000 7f947ecc3700 1 -- 172.21.15.112:0/3812951518 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9470009370 con 0x7f9478155ba0 2023-01-24T03:30:55.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.686+0000 7f947ecc3700 1 --2- 172.21.15.112:0/3812951518 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9478155ba0 0x7f9478148610 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9470002000 tx=0x7f9470016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:55.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.686+0000 7f94767fc700 1 -- 172.21.15.112:0/3812951518 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f947001be60 con 0x7f9478155ba0 2023-01-24T03:30:55.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.686+0000 7f94767fc700 1 -- 172.21.15.112:0/3812951518 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9470020070 con 0x7f9478155ba0 2023-01-24T03:30:55.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.686+0000 7f947f4c4700 1 -- 172.21.15.112:0/3812951518 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94780c24e0 con 0x7f9478155ba0 2023-01-24T03:30:55.700 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.686+0000 7f94767fc700 1 -- 172.21.15.112:0/3812951518 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94700276f0 con 0x7f9478155ba0 2023-01-24T03:30:55.700 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.686+0000 7f947f4c4700 1 -- 172.21.15.112:0/3812951518 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f947814e020 con 0x7f9478155ba0 2023-01-24T03:30:55.700 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.687+0000 7f947f4c4700 1 -- 172.21.15.112:0/3812951518 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9478168ec0 con 0x7f9478155ba0 2023-01-24T03:30:55.700 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.687+0000 7f94767fc700 1 -- 172.21.15.112:0/3812951518 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f9470017020 con 0x7f9478155ba0 2023-01-24T03:30:55.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.688+0000 7f94767fc700 1 --2- 172.21.15.112:0/3812951518 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f94580611f0 0x7f94580636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:55.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.688+0000 7f94767fc700 1 -- 172.21.15.112:0/3812951518 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f9470022070 con 0x7f9478155ba0 2023-01-24T03:30:55.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.688+0000 7f947e4c2700 1 --2- 172.21.15.112:0/3812951518 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f94580611f0 0x7f94580636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:55.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.688+0000 7f947e4c2700 1 --2- 172.21.15.112:0/3812951518 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f94580611f0 0x7f94580636a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9468004290 tx=0x7f9468001560 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:55.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.692+0000 7f94767fc700 1 -- 172.21.15.112:0/3812951518 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f9478168ec0 con 0x7f9478155ba0 2023-01-24T03:30:55.871 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.869+0000 7f947f4c4700 1 -- 172.21.15.112:0/3812951518 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f9478168ec0 con 0x7f9478155ba0 2023-01-24T03:30:55.872 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.869+0000 7f94767fc700 1 -- 172.21.15.112:0/3812951518 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7f9478168ec0 con 0x7f9478155ba0 2023-01-24T03:30:55.872 INFO:teuthology.orchestra.run.smithi112.stdout:pg_num: 32 2023-01-24T03:30:55.874 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.871+0000 7f947f4c4700 1 -- 172.21.15.112:0/3812951518 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f94580611f0 msgr2=0x7f94580636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:55.874 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.871+0000 7f947f4c4700 1 --2- 172.21.15.112:0/3812951518 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f94580611f0 0x7f94580636a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9468004290 tx=0x7f9468001560 comp rx=0 tx=0).stop 2023-01-24T03:30:55.874 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.871+0000 7f947f4c4700 1 -- 172.21.15.112:0/3812951518 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9478155ba0 msgr2=0x7f9478148610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:55.874 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.871+0000 7f947f4c4700 1 --2- 172.21.15.112:0/3812951518 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9478155ba0 0x7f9478148610 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9470002000 tx=0x7f9470016000 comp rx=0 tx=0).stop 2023-01-24T03:30:55.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.871+0000 7f947f4c4700 1 -- 172.21.15.112:0/3812951518 shutdown_connections 2023-01-24T03:30:55.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.871+0000 7f947f4c4700 1 --2- 172.21.15.112:0/3812951518 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f94580611f0 0x7f94580636a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:55.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.871+0000 7f947f4c4700 1 --2- 172.21.15.112:0/3812951518 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9478155ba0 0x7f9478148610 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:55.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.871+0000 7f947f4c4700 1 --2- 172.21.15.112:0/3812951518 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f94780c6420 0x7f947814cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:55.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.871+0000 7f947f4c4700 1 --2- 172.21.15.112:0/3812951518 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f947814f3f0 0x7f9478156360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:55.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.871+0000 7f947f4c4700 1 -- 172.21.15.112:0/3812951518 >> 172.21.15.112:0/3812951518 conn(0x7f947805d830 msgr2=0x7f947814ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:55.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.871+0000 7f947f4c4700 1 -- 172.21.15.112:0/3812951518 shutdown_connections 2023-01-24T03:30:55.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.872+0000 7f947f4c4700 1 -- 172.21.15.112:0/3812951518 wait complete. 2023-01-24T03:30:55.885 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:30:55.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.995+0000 7f39fc9de700 1 Processor -- start 2023-01-24T03:30:55.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.995+0000 7f39fc9de700 1 -- start start 2023-01-24T03:30:56.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.996+0000 7f39fc9de700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39f80b0aa0 0x7f39f8155da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:56.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.996+0000 7f39fc9de700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39f80b0f40 con 0x7f39f80c7e40 2023-01-24T03:30:56.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.996+0000 7f39fc9de700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39f80b1170 con 0x7f39f80ca3e0 2023-01-24T03:30:56.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.996+0000 7f39fc9de700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39f80b1450 con 0x7f39f80b0aa0 2023-01-24T03:30:56.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.996+0000 7f39f659c700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f39f80c7e40 0x7f39f80ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:60564/0 (socket says 172.21.15.112:60564) 2023-01-24T03:30:56.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.996+0000 7f39f659c700 1 -- 172.21.15.112:0/609720426 learned_addr learned my addr 172.21.15.112:0/609720426 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:56.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.996+0000 7f39f5d9b700 1 --2- 172.21.15.112:0/609720426 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39f80b0aa0 0x7f39f8155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:56.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.996+0000 7f39f559a700 1 -- 172.21.15.112:0/609720426 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2702553555 0 0) 0x7f39f80b0f40 con 0x7f39f80c7e40 2023-01-24T03:30:56.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.996+0000 7f39f559a700 1 -- 172.21.15.112:0/609720426 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39d80033f0 con 0x7f39f80c7e40 2023-01-24T03:30:56.006 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.997+0000 7f39f559a700 1 -- 172.21.15.112:0/609720426 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 236735227 0 0) 0x7f39d80033f0 con 0x7f39f80c7e40 2023-01-24T03:30:56.006 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.997+0000 7f39f559a700 1 -- 172.21.15.112:0/609720426 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39f80b0aa0 msgr2=0x7f39f8155da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:56.006 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.997+0000 7f39f559a700 1 --2- 172.21.15.112:0/609720426 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39f80b0aa0 0x7f39f8155da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:56.006 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.997+0000 7f39f559a700 1 -- 172.21.15.112:0/609720426 >> v1:172.21.15.142:6789/0 conn(0x7f39f80ca3e0 legacy=0x7f39f80ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:56.006 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.997+0000 7f39f559a700 1 -- 172.21.15.112:0/609720426 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39f80b1730 con 0x7f39f80c7e40 2023-01-24T03:30:56.007 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.997+0000 7f39f559a700 1 -- 172.21.15.112:0/609720426 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7f39e80017f0 con 0x7f39f80c7e40 2023-01-24T03:30:56.007 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.997+0000 7f39f559a700 1 -- 172.21.15.112:0/609720426 >> v1:172.21.15.112:6789/0 conn(0x7f39f80c7e40 legacy=0x7f39f80ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:56.007 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.997+0000 7f39f559a700 1 --2- 172.21.15.112:0/609720426 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f39d8003910 0x7f39d8003d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:56.007 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.997+0000 7f39f559a700 1 --2- 172.21.15.112:0/609720426 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39d8004690 0x7f39d8004b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:56.007 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.997+0000 7f39f559a700 1 --2- 172.21.15.112:0/609720426 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39f80b0aa0 0x7f39d8005060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:56.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.997+0000 7f39f559a700 1 -- 172.21.15.112:0/609720426 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f80b1450 con 0x7f39d8004690 2023-01-24T03:30:56.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.997+0000 7f39f559a700 1 -- 172.21.15.112:0/609720426 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d800ba10 con 0x7f39d8003910 2023-01-24T03:30:56.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.997+0000 7f39f559a700 1 -- 172.21.15.112:0/609720426 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39d800bbd0 con 0x7f39f80b0aa0 2023-01-24T03:30:56.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.997+0000 7f39f5d9b700 1 --2- 172.21.15.112:0/609720426 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39d8004690 0x7f39d8004b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:56.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.997+0000 7f39f659c700 1 --2- 172.21.15.112:0/609720426 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f39d8003910 0x7f39d8003d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:56.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.997+0000 7f39f6d9d700 1 --2- 172.21.15.112:0/609720426 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39f80b0aa0 0x7f39d8005060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:56.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.997+0000 7f39f5d9b700 1 -- 172.21.15.112:0/609720426 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39f80b0aa0 msgr2=0x7f39d8005060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:56.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.997+0000 7f39f5d9b700 1 --2- 172.21.15.112:0/609720426 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39f80b0aa0 0x7f39d8005060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:56.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.997+0000 7f39f5d9b700 1 -- 172.21.15.112:0/609720426 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f39d8003910 msgr2=0x7f39d8003d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:56.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.997+0000 7f39f5d9b700 1 --2- 172.21.15.112:0/609720426 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f39d8003910 0x7f39d8003d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:56.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.997+0000 7f39f5d9b700 1 -- 172.21.15.112:0/609720426 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39d80033f0 con 0x7f39d8004690 2023-01-24T03:30:56.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.997+0000 7f39f6d9d700 1 --2- 172.21.15.112:0/609720426 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39f80b0aa0 0x7f39d8005060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:30:56.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.997+0000 7f39f5d9b700 1 --2- 172.21.15.112:0/609720426 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39d8004690 0x7f39d8004b00 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f39e0009a80 tx=0x7f39e0009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43503fe705e6f96b server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:56.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.997+0000 7f39f559a700 1 -- 172.21.15.112:0/609720426 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e0009430 con 0x7f39d8004690 2023-01-24T03:30:56.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.997+0000 7f39f559a700 1 -- 172.21.15.112:0/609720426 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39e000e640 con 0x7f39d8004690 2023-01-24T03:30:56.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.997+0000 7f39f559a700 1 -- 172.21.15.112:0/609720426 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e000ea00 con 0x7f39d8004690 2023-01-24T03:30:56.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.998+0000 7f39fc9de700 1 -- 172.21.15.112:0/609720426 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39d8004690 msgr2=0x7f39d8004b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:56.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.998+0000 7f39fc9de700 1 --2- 172.21.15.112:0/609720426 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39d8004690 0x7f39d8004b00 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f39e0009a80 tx=0x7f39e0009060 comp rx=0 tx=0).stop 2023-01-24T03:30:56.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.998+0000 7f39f6d9d700 1 -- 172.21.15.112:0/609720426 reap_dead start 2023-01-24T03:30:56.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.998+0000 7f39fc9de700 1 -- 172.21.15.112:0/609720426 shutdown_connections 2023-01-24T03:30:56.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.998+0000 7f39fc9de700 1 -- 172.21.15.112:0/609720426 >> 172.21.15.112:0/609720426 conn(0x7f39f805e2c0 msgr2=0x7f39f80cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:56.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.998+0000 7f39fc9de700 1 -- 172.21.15.112:0/609720426 shutdown_connections 2023-01-24T03:30:56.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.998+0000 7f39fc9de700 1 -- 172.21.15.112:0/609720426 wait complete. 2023-01-24T03:30:56.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.999+0000 7f39fc9de700 1 Processor -- start 2023-01-24T03:30:56.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:55.999+0000 7f39fc9de700 1 -- start start 2023-01-24T03:30:56.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.000+0000 7f39fc9de700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39d8004690 0x7f39f8165f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:56.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.000+0000 7f39fc9de700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39f8152990 0x7f39f8154350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:56.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.000+0000 7f39fc9de700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f39f8155090 0x7f39f80bffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:56.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.000+0000 7f39fc9de700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f8165780 con 0x7f39d8004690 2023-01-24T03:30:56.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.000+0000 7f39fc9de700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f8150030 con 0x7f39f8155090 2023-01-24T03:30:56.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.000+0000 7f39fc9de700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39f8150310 con 0x7f39f8152990 2023-01-24T03:30:56.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.000+0000 7f39f659c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39d8004690 0x7f39f8165f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:56.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.000+0000 7f39f659c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39d8004690 0x7f39f8165f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51356/0 (socket says 172.21.15.112:51356) 2023-01-24T03:30:56.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.000+0000 7f39f659c700 1 -- 172.21.15.112:0/3441435064 learned_addr learned my addr 172.21.15.112:0/3441435064 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:56.015 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.000+0000 7f39f5d9b700 1 --2- 172.21.15.112:0/3441435064 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39f8152990 0x7f39f8154350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:56.015 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.000+0000 7f39f6d9d700 1 --2- 172.21.15.112:0/3441435064 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f39f8155090 0x7f39f80bffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:56.015 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.000+0000 7f39f659c700 1 -- 172.21.15.112:0/3441435064 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39f8152990 msgr2=0x7f39f8154350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:56.015 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.000+0000 7f39f659c700 1 --2- 172.21.15.112:0/3441435064 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39f8152990 0x7f39f8154350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:56.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.000+0000 7f39f659c700 1 -- 172.21.15.112:0/3441435064 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f39f8155090 msgr2=0x7f39f80bffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:56.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.000+0000 7f39f659c700 1 --2- 172.21.15.112:0/3441435064 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f39f8155090 0x7f39f80bffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:56.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.000+0000 7f39f659c700 1 -- 172.21.15.112:0/3441435064 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39f81505f0 con 0x7f39d8004690 2023-01-24T03:30:56.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.000+0000 7f39f659c700 1 --2- 172.21.15.112:0/3441435064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39d8004690 0x7f39f8165f10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f39e8001920 tx=0x7f39e80018b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:56.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.000+0000 7f39ee7fc700 1 -- 172.21.15.112:0/3441435064 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e80079e0 con 0x7f39d8004690 2023-01-24T03:30:56.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.000+0000 7f39ee7fc700 1 -- 172.21.15.112:0/3441435064 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39e800fc30 con 0x7f39d8004690 2023-01-24T03:30:56.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.000+0000 7f39ee7fc700 1 -- 172.21.15.112:0/3441435064 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e80133f0 con 0x7f39d8004690 2023-01-24T03:30:56.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.000+0000 7f39fc9de700 1 -- 172.21.15.112:0/3441435064 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39f81659f0 con 0x7f39d8004690 2023-01-24T03:30:56.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.000+0000 7f39fc9de700 1 -- 172.21.15.112:0/3441435064 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39f814fc80 con 0x7f39d8004690 2023-01-24T03:30:56.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.001+0000 7f39fc9de700 1 -- 172.21.15.112:0/3441435064 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39f80b0f70 con 0x7f39d8004690 2023-01-24T03:30:56.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.001+0000 7f39ee7fc700 1 -- 172.21.15.112:0/3441435064 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f39e8013760 con 0x7f39d8004690 2023-01-24T03:30:56.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.002+0000 7f39ee7fc700 1 --2- 172.21.15.112:0/3441435064 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f39dc0611c0 0x7f39dc063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:56.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.002+0000 7f39ee7fc700 1 -- 172.21.15.112:0/3441435064 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f39e807de90 con 0x7f39d8004690 2023-01-24T03:30:56.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.002+0000 7f39f5d9b700 1 --2- 172.21.15.112:0/3441435064 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f39dc0611c0 0x7f39dc063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:56.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.002+0000 7f39f5d9b700 1 --2- 172.21.15.112:0/3441435064 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f39dc0611c0 0x7f39dc063670 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f39f80c2a30 tx=0x7f39e001a000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:56.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.006+0000 7f39ee7fc700 1 -- 172.21.15.112:0/3441435064 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f39e804dcd0 con 0x7f39d8004690 2023-01-24T03:30:56.229 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.227+0000 7f39fc9de700 1 -- 172.21.15.112:0/3441435064 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39f80c6df0 con 0x7f39d8004690 2023-01-24T03:30:56.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.228+0000 7f39ee7fc700 1 -- 172.21.15.112:0/3441435064 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6114 (secure 0 0 0) 0x7f39e80512f0 con 0x7f39d8004690 2023-01-24T03:30:56.230 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:30:56.230 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/869267413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":869267413},{"type":"v1","addr":"172.21.15.142:6834","nonce":869267413}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:30:37.600288+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.142:6835/2762621058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2762621058},{"type":"v1","addr":"172.21.15.142:6835","nonce":2762621058}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:30:56.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.230+0000 7f39fc9de700 1 -- 172.21.15.112:0/3441435064 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f39dc0611c0 msgr2=0x7f39dc063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:56.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.230+0000 7f39fc9de700 1 --2- 172.21.15.112:0/3441435064 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f39dc0611c0 0x7f39dc063670 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f39f80c2a30 tx=0x7f39e001a000 comp rx=0 tx=0).stop 2023-01-24T03:30:56.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.230+0000 7f39fc9de700 1 -- 172.21.15.112:0/3441435064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39d8004690 msgr2=0x7f39f8165f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:56.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.230+0000 7f39fc9de700 1 --2- 172.21.15.112:0/3441435064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39d8004690 0x7f39f8165f10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f39e8001920 tx=0x7f39e80018b0 comp rx=0 tx=0).stop 2023-01-24T03:30:56.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.230+0000 7f39fc9de700 1 -- 172.21.15.112:0/3441435064 shutdown_connections 2023-01-24T03:30:56.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.230+0000 7f39fc9de700 1 --2- 172.21.15.112:0/3441435064 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f39dc0611c0 0x7f39dc063670 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:56.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.230+0000 7f39fc9de700 1 --2- 172.21.15.112:0/3441435064 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39d8004690 0x7f39f8165f10 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:56.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.230+0000 7f39fc9de700 1 --2- 172.21.15.112:0/3441435064 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f39f8155090 0x7f39f80bffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:56.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.230+0000 7f39fc9de700 1 --2- 172.21.15.112:0/3441435064 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39f8152990 0x7f39f8154350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:56.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.230+0000 7f39fc9de700 1 -- 172.21.15.112:0/3441435064 >> 172.21.15.112:0/3441435064 conn(0x7f39f805e2c0 msgr2=0x7f39f80c8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:56.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.230+0000 7f39fc9de700 1 -- 172.21.15.112:0/3441435064 shutdown_connections 2023-01-24T03:30:56.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.230+0000 7f39fc9de700 1 -- 172.21.15.112:0/3441435064 wait complete. 2023-01-24T03:30:56.235 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 10 2023-01-24T03:30:56.244 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T03:30:56.355 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.353+0000 7fa292046700 1 Processor -- start 2023-01-24T03:30:56.355 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.353+0000 7fa292046700 1 -- start start 2023-01-24T03:30:56.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.354+0000 7fa292046700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fa28c0c7e40 0x7fa28c0b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:56.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.354+0000 7fa292046700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa28c156240 con 0x7fa28c155da0 2023-01-24T03:30:56.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.354+0000 7fa292046700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa28c0b1170 con 0x7fa28c0ca3e0 2023-01-24T03:30:56.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.354+0000 7fa292046700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa28c0b1450 con 0x7fa28c0c7e40 2023-01-24T03:30:56.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.354+0000 7fa28b7fe700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fa28c0c7e40 0x7fa28c0b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:56.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.354+0000 7fa28affd700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fa28c155da0 0x7fa28c0ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:60584/0 (socket says 172.21.15.112:60584) 2023-01-24T03:30:56.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.354+0000 7fa28affd700 1 -- 172.21.15.112:0/1621739320 learned_addr learned my addr 172.21.15.112:0/1621739320 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:30:56.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.354+0000 7fa28b7fe700 1 -- 172.21.15.112:0/1621739320 >> v1:172.21.15.142:6789/0 conn(0x7fa28c0ca3e0 legacy=0x7fa28c0ca7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:30:56.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.354+0000 7fa28b7fe700 1 -- 172.21.15.112:0/1621739320 >> v1:172.21.15.112:6789/0 conn(0x7fa28c155da0 legacy=0x7fa28c0ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:56.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.354+0000 7fa28b7fe700 1 -- 172.21.15.112:0/1621739320 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa28c0b1730 con 0x7fa28c0c7e40 2023-01-24T03:30:56.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.354+0000 7fa28a7fc700 1 -- 172.21.15.112:0/1621739320 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1599890330 0 0) 0x7fa28c156240 con 0x7fa28c155da0 2023-01-24T03:30:56.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.354+0000 7fa28b7fe700 1 --2- 172.21.15.112:0/1621739320 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa28c0c7e40 0x7fa28c0b0aa0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa27c009760 tx=0x7fa27c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5995eaf19c912272 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:56.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.354+0000 7fa28a7fc700 1 -- 172.21.15.112:0/1621739320 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27c011630 con 0x7fa28c0c7e40 2023-01-24T03:30:56.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.355+0000 7fa28a7fc700 1 -- 172.21.15.112:0/1621739320 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa27c011790 con 0x7fa28c0c7e40 2023-01-24T03:30:56.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.355+0000 7fa28a7fc700 1 -- 172.21.15.112:0/1621739320 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27c011b30 con 0x7fa28c0c7e40 2023-01-24T03:30:56.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.355+0000 7fa292046700 1 -- 172.21.15.112:0/1621739320 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa28c0c7e40 msgr2=0x7fa28c0b0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:56.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.355+0000 7fa292046700 1 --2- 172.21.15.112:0/1621739320 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa28c0c7e40 0x7fa28c0b0aa0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa27c009760 tx=0x7fa27c009060 comp rx=0 tx=0).stop 2023-01-24T03:30:56.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.355+0000 7fa292046700 1 -- 172.21.15.112:0/1621739320 shutdown_connections 2023-01-24T03:30:56.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.355+0000 7fa292046700 1 --2- 172.21.15.112:0/1621739320 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa28c0c7e40 0x7fa28c0b0aa0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:56.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.355+0000 7fa292046700 1 -- 172.21.15.112:0/1621739320 >> 172.21.15.112:0/1621739320 conn(0x7fa28c05e2c0 msgr2=0x7fa28c0cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:56.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.355+0000 7fa292046700 1 -- 172.21.15.112:0/1621739320 shutdown_connections 2023-01-24T03:30:56.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.355+0000 7fa292046700 1 -- 172.21.15.112:0/1621739320 wait complete. 2023-01-24T03:30:56.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.356+0000 7fa292046700 1 Processor -- start 2023-01-24T03:30:56.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.356+0000 7fa292046700 1 -- start start 2023-01-24T03:30:56.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.357+0000 7fa292046700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa28c0c7e40 0x7fa28c155100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:56.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.357+0000 7fa292046700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa28c0ca3e0 0x7fa28c14fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:56.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.357+0000 7fa292046700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa28c155da0 0x7fa28c168e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:56.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.357+0000 7fa292046700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28c168a30 con 0x7fa28c155da0 2023-01-24T03:30:56.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.357+0000 7fa292046700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28c154db0 con 0x7fa28c0c7e40 2023-01-24T03:30:56.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.357+0000 7fa292046700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa28c154230 con 0x7fa28c0ca3e0 2023-01-24T03:30:56.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.357+0000 7fa28bfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa28c155da0 0x7fa28c168e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:56.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.357+0000 7fa28bfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa28c155da0 0x7fa28c168e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51372/0 (socket says 172.21.15.112:51372) 2023-01-24T03:30:56.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.357+0000 7fa28bfff700 1 -- 172.21.15.112:0/200350668 learned_addr learned my addr 172.21.15.112:0/200350668 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:30:56.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.357+0000 7fa28b7fe700 1 --2- 172.21.15.112:0/200350668 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa28c0c7e40 0x7fa28c155100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:56.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.357+0000 7fa28affd700 1 --2- 172.21.15.112:0/200350668 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa28c0ca3e0 0x7fa28c14fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:56.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.357+0000 7fa28bfff700 1 -- 172.21.15.112:0/200350668 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa28c0ca3e0 msgr2=0x7fa28c14fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:56.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.357+0000 7fa28bfff700 1 --2- 172.21.15.112:0/200350668 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa28c0ca3e0 0x7fa28c14fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:56.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.357+0000 7fa28bfff700 1 -- 172.21.15.112:0/200350668 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa28c0c7e40 msgr2=0x7fa28c155100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:56.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.357+0000 7fa28bfff700 1 --2- 172.21.15.112:0/200350668 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa28c0c7e40 0x7fa28c155100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:56.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.357+0000 7fa28bfff700 1 -- 172.21.15.112:0/200350668 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa27c0092a0 con 0x7fa28c155da0 2023-01-24T03:30:56.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.358+0000 7fa28bfff700 1 --2- 172.21.15.112:0/200350668 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa28c155da0 0x7fa28c168e80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa28400b810 tx=0x7fa284009290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:56.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.358+0000 7fa2737fe700 1 -- 172.21.15.112:0/200350668 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2840105b0 con 0x7fa28c155da0 2023-01-24T03:30:56.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.358+0000 7fa2737fe700 1 -- 172.21.15.112:0/200350668 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa284010710 con 0x7fa28c155da0 2023-01-24T03:30:56.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.358+0000 7fa292046700 1 -- 172.21.15.112:0/200350668 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa28c154510 con 0x7fa28c155da0 2023-01-24T03:30:56.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.358+0000 7fa2737fe700 1 -- 172.21.15.112:0/200350668 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa284010ad0 con 0x7fa28c155da0 2023-01-24T03:30:56.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.358+0000 7fa292046700 1 -- 172.21.15.112:0/200350668 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa28c154ad0 con 0x7fa28c155da0 2023-01-24T03:30:56.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.359+0000 7fa292046700 1 -- 172.21.15.112:0/200350668 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa28c168ba0 con 0x7fa28c155da0 2023-01-24T03:30:56.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.359+0000 7fa2737fe700 1 -- 172.21.15.112:0/200350668 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fa28400f070 con 0x7fa28c155da0 2023-01-24T03:30:56.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.359+0000 7fa2737fe700 1 --2- 172.21.15.112:0/200350668 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa26c0611c0 0x7fa26c063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:30:56.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.359+0000 7fa2737fe700 1 -- 172.21.15.112:0/200350668 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6809+0+0 (secure 0 0 0) 0x7fa28407d680 con 0x7fa28c155da0 2023-01-24T03:30:56.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.359+0000 7fa28b7fe700 1 --2- 172.21.15.112:0/200350668 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa26c0611c0 0x7fa26c063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:30:56.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.360+0000 7fa28b7fe700 1 --2- 172.21.15.112:0/200350668 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa26c0611c0 0x7fa26c063670 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa27c011fd0 tx=0x7fa27c016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:30:56.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.363+0000 7fa2737fe700 1 -- 172.21.15.112:0/200350668 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fa28c168ba0 con 0x7fa28c155da0 2023-01-24T03:30:56.543 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.541+0000 7fa292046700 1 -- 172.21.15.112:0/200350668 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa28c168b70 con 0x7fa28c155da0 2023-01-24T03:30:56.544 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.542+0000 7fa2737fe700 1 -- 172.21.15.112:0/200350668 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18755 (secure 0 0 0) 0x7fa284050fa0 con 0x7fa28c155da0 2023-01-24T03:30:56.544 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:30:56.545 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":24,"fsid":"054b9ac9-b7b0-40e5-b932-6c8a955c5a72","created":"2023-01-24T03:30:03.798533+0000","modified":"2023-01-24T03:30:25.373340+0000","last_up_change":"2023-01-24T03:30:13.945316+0000","last_in_change":"2023-01-24T03:30:06.198433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-01-24T03:30:14.931166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-01-24T03:30:15.023730+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-01-24T03:30:19.060597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-01-24T03:30:19.943686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":5,"pool_name":"cephfs_data_ec","create_time":"2023-01-24T03:30:22.999036+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"fe6ad231-a39a-42eb-9a3a-49d008dd35e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6809","nonce":84512},{"type":"v1","addr":"172.21.15.112:6811","nonce":84512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":84512},{"type":"v1","addr":"172.21.15.112:6817","nonce":84512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6825","nonce":84512},{"type":"v1","addr":"172.21.15.112:6828","nonce":84512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6819","nonce":84512},{"type":"v1","addr":"172.21.15.112:6822","nonce":84512}]},"public_addr":"172.21.15.112:6811/84512","cluster_addr":"172.21.15.112:6817/84512","heartbeat_back_addr":"172.21.15.112:6828/84512","heartbeat_front_addr":"172.21.15.112:6822/84512","state":["exists","up"]},{"osd":1,"uuid":"8526a1d2-d540-4021-9766-c4bbb30317ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":84513},{"type":"v1","addr":"172.21.15.112:6812","nonce":84513}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6815","nonce":84513},{"type":"v1","addr":"172.21.15.112:6818","nonce":84513}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6827","nonce":84513},{"type":"v1","addr":"172.21.15.112:6830","nonce":84513}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6821","nonce":84513},{"type":"v1","addr":"172.21.15.112:6824","nonce":84513}]},"public_addr":"172.21.15.112:6812/84513","cluster_addr":"172.21.15.112:6818/84513","heartbeat_back_addr":"172.21.15.112:6830/84513","heartbeat_front_addr":"172.21.15.112:6824/84513","state":["exists","up"]},{"osd":2,"uuid":"041ef8e4-4858-432d-9476-cbdb3a0dc47c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":84511},{"type":"v1","addr":"172.21.15.112:6813","nonce":84511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":84511},{"type":"v1","addr":"172.21.15.112:6820","nonce":84511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6829","nonce":84511},{"type":"v1","addr":"172.21.15.112:6831","nonce":84511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6823","nonce":84511},{"type":"v1","addr":"172.21.15.112:6826","nonce":84511}]},"public_addr":"172.21.15.112:6813/84511","cluster_addr":"172.21.15.112:6820/84511","heartbeat_back_addr":"172.21.15.112:6831/84511","heartbeat_front_addr":"172.21.15.112:6826/84511","state":["exists","up"]},{"osd":3,"uuid":"987587cf-1357-4a70-ad85-67b189c67729","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":84510},{"type":"v1","addr":"172.21.15.112:6801","nonce":84510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":84510},{"type":"v1","addr":"172.21.15.112:6803","nonce":84510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":84510},{"type":"v1","addr":"172.21.15.112:6807","nonce":84510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":84510},{"type":"v1","addr":"172.21.15.112:6805","nonce":84510}]},"public_addr":"172.21.15.112:6801/84510","cluster_addr":"172.21.15.112:6803/84510","heartbeat_back_addr":"172.21.15.112:6807/84510","heartbeat_front_addr":"172.21.15.112:6805/84510","state":["exists","up"]},{"osd":4,"uuid":"8503512b-43db-40b8-8b2b-c86cf2e54e17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":84614},{"type":"v1","addr":"172.21.15.142:6801","nonce":84614}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":84614},{"type":"v1","addr":"172.21.15.142:6803","nonce":84614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":84614},{"type":"v1","addr":"172.21.15.142:6807","nonce":84614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":84614},{"type":"v1","addr":"172.21.15.142:6805","nonce":84614}]},"public_addr":"172.21.15.142:6801/84614","cluster_addr":"172.21.15.142:6803/84614","heartbeat_back_addr":"172.21.15.142:6807/84614","heartbeat_front_addr":"172.21.15.142:6805/84614","state":["exists","up"]},{"osd":5,"uuid":"d3bc0030-8b08-4645-9ec2-1f8a4e29ef36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":84638},{"type":"v1","addr":"172.21.15.142:6817","nonce":84638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":84638},{"type":"v1","addr":"172.21.15.142:6819","nonce":84638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":84638},{"type":"v1","addr":"172.21.15.142:6823","nonce":84638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":84638},{"type":"v1","addr":"172.21.15.142:6821","nonce":84638}]},"public_addr":"172.21.15.142:6817/84638","cluster_addr":"172.21.15.142:6819/84638","heartbeat_back_addr":"172.21.15.142:6823/84638","heartbeat_front_addr":"172.21.15.142:6821/84638","state":["exists","up"]},{"osd":6,"uuid":"c7649e3b-832d-4e6f-8866-b15e5d814476","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":84645},{"type":"v1","addr":"172.21.15.142:6825","nonce":84645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":84645},{"type":"v1","addr":"172.21.15.142:6827","nonce":84645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":84645},{"type":"v1","addr":"172.21.15.142:6831","nonce":84645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":84645},{"type":"v1","addr":"172.21.15.142:6829","nonce":84645}]},"public_addr":"172.21.15.142:6825/84645","cluster_addr":"172.21.15.142:6827/84645","heartbeat_back_addr":"172.21.15.142:6831/84645","heartbeat_front_addr":"172.21.15.142:6829/84645","state":["exists","up"]},{"osd":7,"uuid":"2ea5b3bf-884e-4b20-b572-1443efdc7477","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":84615},{"type":"v1","addr":"172.21.15.142:6809","nonce":84615}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":84615},{"type":"v1","addr":"172.21.15.142:6811","nonce":84615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":84615},{"type":"v1","addr":"172.21.15.142:6815","nonce":84615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":84615},{"type":"v1","addr":"172.21.15.142:6813","nonce":84615}]},"public_addr":"172.21.15.142:6809/84615","cluster_addr":"172.21.15.142:6811/84615","heartbeat_back_addr":"172.21.15.142:6815/84615","heartbeat_front_addr":"172.21.15.142:6813/84615","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.370027+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.335683+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.350667+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.350794+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.391119+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.422933+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.400439+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T03:30:11.354018+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T03:30:56.546 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.544+0000 7fa292046700 1 -- 172.21.15.112:0/200350668 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa26c0611c0 msgr2=0x7fa26c063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:56.547 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.544+0000 7fa292046700 1 --2- 172.21.15.112:0/200350668 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa26c0611c0 0x7fa26c063670 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa27c011fd0 tx=0x7fa27c016000 comp rx=0 tx=0).stop 2023-01-24T03:30:56.548 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.544+0000 7fa292046700 1 -- 172.21.15.112:0/200350668 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa28c155da0 msgr2=0x7fa28c168e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:30:56.549 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.544+0000 7fa292046700 1 --2- 172.21.15.112:0/200350668 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa28c155da0 0x7fa28c168e80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa28400b810 tx=0x7fa284009290 comp rx=0 tx=0).stop 2023-01-24T03:30:56.549 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.544+0000 7fa292046700 1 -- 172.21.15.112:0/200350668 shutdown_connections 2023-01-24T03:30:56.549 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.544+0000 7fa292046700 1 --2- 172.21.15.112:0/200350668 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa26c0611c0 0x7fa26c063670 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:56.549 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.544+0000 7fa292046700 1 --2- 172.21.15.112:0/200350668 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa28c155da0 0x7fa28c168e80 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:56.550 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.544+0000 7fa292046700 1 --2- 172.21.15.112:0/200350668 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa28c0c7e40 0x7fa28c155100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:56.550 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.544+0000 7fa292046700 1 --2- 172.21.15.112:0/200350668 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa28c0ca3e0 0x7fa28c14fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:30:56.550 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.544+0000 7fa292046700 1 -- 172.21.15.112:0/200350668 >> 172.21.15.112:0/200350668 conn(0x7fa28c05e2c0 msgr2=0x7fa28c069000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:30:56.550 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.544+0000 7fa292046700 1 -- 172.21.15.112:0/200350668 shutdown_connections 2023-01-24T03:30:56.550 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:30:56.544+0000 7fa292046700 1 -- 172.21.15.112:0/200350668 wait complete. 2023-01-24T03:30:56.558 DEBUG:tasks.mds_thrash:Yielding 2023-01-24T03:30:56.559 INFO:teuthology.run_tasks:Running task workunit... 2023-01-24T03:30:56.578 INFO:tasks.workunit:Pulling workunits from ref 8387d427d91a301da19bd82e090b96e1348bfadb 2023-01-24T03:30:56.579 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-01-24T03:30:56.579 INFO:tasks.workunit:timeout=3h 2023-01-24T03:30:56.580 INFO:tasks.workunit:cleanup=True 2023-01-24T03:30:56.580 DEBUG:teuthology.orchestra.run.smithi112:> stat -- /home/ubuntu/cephtest/mnt.0 2023-01-24T03:30:56.582 INFO:tasks.mds_thrash.fs.[cephfs]:starting mds_do_thrash for fs cephfs 2023-01-24T03:30:56.582 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 39 secs before thrashing 2023-01-24T03:30:56.597 INFO:teuthology.orchestra.run.smithi112.stdout: File: /home/ubuntu/cephtest/mnt.0 2023-01-24T03:30:56.598 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2023-01-24T03:30:56.598 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 2fh/47d Inode: 1 Links: 2 2023-01-24T03:30:56.598 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T03:30:56.598 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:cephfs_t:s0 2023-01-24T03:30:56.599 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2023-01-24T03:30:56.599 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2023-01-24 03:30:25.974494595 +0000 2023-01-24T03:30:56.599 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2023-01-24 03:30:34.374823904 +0000 2023-01-24T03:30:56.599 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: - 2023-01-24T03:30:56.600 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2023-01-24T03:30:56.600 DEBUG:teuthology.orchestra.run.smithi112:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2023-01-24T03:30:56.665 DEBUG:teuthology.orchestra.run.smithi112:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 8387d427d91a301da19bd82e090b96e1348bfadb 2023-01-24T03:30:56.720 INFO:tasks.workunit.client.0.smithi112.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-01-24T03:30:58.872 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:30:58.875 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:31:28.934 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:31:28.937 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:31:35.583 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:31:35.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.701+0000 7f999dae8700 1 Processor -- start 2023-01-24T03:31:35.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.701+0000 7f999dae8700 1 -- start start 2023-01-24T03:31:35.712 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.701+0000 7f999dae8700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f999814f300 0x7f99980b73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:35.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.701+0000 7f999dae8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f999814f6d0 0x7f99980c0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:35.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.701+0000 7f999dae8700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f999806ba90 con 0x7f99980c0e70 2023-01-24T03:31:35.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.702+0000 7f999dae8700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f999806da40 con 0x7f999814f300 2023-01-24T03:31:35.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.702+0000 7f999dae8700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f999806e1e0 con 0x7f999814f6d0 2023-01-24T03:31:35.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.702+0000 7f9997fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f999814f6d0 0x7f99980c0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:35.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.702+0000 7f999cae6700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f999814f300 0x7f99980b73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:35.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.702+0000 7f9997fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f999814f6d0 0x7f99980c0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.112:38702/0 (socket says 172.21.15.112:38702) 2023-01-24T03:31:35.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.702+0000 7f9997fff700 1 -- 172.21.15.112:0/2984436961 learned_addr learned my addr 172.21.15.112:0/2984436961 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:31:35.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.702+0000 7f999d2e7700 1 --1- 172.21.15.112:0/2984436961 >> v1:172.21.15.142:6789/0 conn(0x7f99980c0e70 0x7f9998151e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.112:46182/0 (socket says 172.21.15.112:46182) 2023-01-24T03:31:35.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.702+0000 7f9997fff700 1 -- 172.21.15.112:0/2984436961 >> v2:172.21.15.142:3300/0 conn(0x7f999814f300 msgr2=0x7f99980b73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:31:35.716 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.702+0000 7f9997fff700 1 --2- 172.21.15.112:0/2984436961 >> v2:172.21.15.142:3300/0 conn(0x7f999814f300 0x7f99980b73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:35.716 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.702+0000 7f9997fff700 1 -- 172.21.15.112:0/2984436961 >> v1:172.21.15.142:6789/0 conn(0x7f99980c0e70 legacy=0x7f9998151e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:35.716 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.702+0000 7f9997fff700 1 -- 172.21.15.112:0/2984436961 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99980c04a0 con 0x7f999814f6d0 2023-01-24T03:31:35.716 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.702+0000 7f9997fff700 1 --2- 172.21.15.112:0/2984436961 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f999814f6d0 0x7f99980c0930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9984006fd0 tx=0x7f99840090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2a33eee8c67f70d server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:31:35.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.703+0000 7f99977fe700 1 -- 172.21.15.112:0/2984436961 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9984011630 con 0x7f999814f6d0 2023-01-24T03:31:35.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.703+0000 7f99977fe700 1 -- 172.21.15.112:0/2984436961 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99840117d0 con 0x7f999814f6d0 2023-01-24T03:31:35.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.703+0000 7f99977fe700 1 -- 172.21.15.112:0/2984436961 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9984011b90 con 0x7f999814f6d0 2023-01-24T03:31:35.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.703+0000 7f999dae8700 1 -- 172.21.15.112:0/2984436961 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f999814f6d0 msgr2=0x7f99980c0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:35.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.703+0000 7f999dae8700 1 --2- 172.21.15.112:0/2984436961 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f999814f6d0 0x7f99980c0930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9984006fd0 tx=0x7f99840090f0 comp rx=0 tx=0).stop 2023-01-24T03:31:35.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.703+0000 7f999dae8700 1 -- 172.21.15.112:0/2984436961 shutdown_connections 2023-01-24T03:31:35.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.703+0000 7f999dae8700 1 --2- 172.21.15.112:0/2984436961 >> v2:172.21.15.142:3300/0 conn(0x7f999814f300 0x7f99980b73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:35.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.703+0000 7f999dae8700 1 --2- 172.21.15.112:0/2984436961 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f999814f6d0 0x7f99980c0930 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:35.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.703+0000 7f999dae8700 1 -- 172.21.15.112:0/2984436961 >> 172.21.15.112:0/2984436961 conn(0x7f9998066e10 msgr2=0x7f9998067210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:31:35.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.703+0000 7f999dae8700 1 -- 172.21.15.112:0/2984436961 shutdown_connections 2023-01-24T03:31:35.720 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.703+0000 7f999dae8700 1 -- 172.21.15.112:0/2984436961 wait complete. 2023-01-24T03:31:35.720 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.704+0000 7f999dae8700 1 Processor -- start 2023-01-24T03:31:35.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.704+0000 7f999dae8700 1 -- start start 2023-01-24T03:31:35.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.705+0000 7f999dae8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99980c0e70 0x7f999816a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:35.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.705+0000 7f999dae8700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f999814f300 0x7f9998150df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:35.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.705+0000 7f999dae8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9998152b20 0x7f9998148060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:35.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.705+0000 7f999dae8700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f999816a860 con 0x7f999814f300 2023-01-24T03:31:35.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.705+0000 7f999dae8700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9998151720 con 0x7f9998152b20 2023-01-24T03:31:35.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.705+0000 7f999dae8700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9998169e90 con 0x7f99980c0e70 2023-01-24T03:31:35.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.705+0000 7f9997fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f999814f300 0x7f9998150df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:35.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.705+0000 7f999cae6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99980c0e70 0x7f999816a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:35.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.705+0000 7f9997fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f999814f300 0x7f9998150df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51408/0 (socket says 172.21.15.112:51408) 2023-01-24T03:31:35.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.705+0000 7f9997fff700 1 -- 172.21.15.112:0/2476244039 learned_addr learned my addr 172.21.15.112:0/2476244039 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:31:35.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.705+0000 7f999d2e7700 1 --2- 172.21.15.112:0/2476244039 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9998152b20 0x7f9998148060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:35.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.706+0000 7f999cae6700 1 -- 172.21.15.112:0/2476244039 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9998152b20 msgr2=0x7f9998148060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:35.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.706+0000 7f999cae6700 1 --2- 172.21.15.112:0/2476244039 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9998152b20 0x7f9998148060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:35.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.706+0000 7f999cae6700 1 -- 172.21.15.112:0/2476244039 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f999814f300 msgr2=0x7f9998150df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:35.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.706+0000 7f999cae6700 1 --2- 172.21.15.112:0/2476244039 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f999814f300 0x7f9998150df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:35.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.706+0000 7f999cae6700 1 -- 172.21.15.112:0/2476244039 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9984009370 con 0x7f99980c0e70 2023-01-24T03:31:35.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.706+0000 7f9997fff700 1 --2- 172.21.15.112:0/2476244039 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f999814f300 0x7f9998150df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:31:35.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.706+0000 7f999cae6700 1 --2- 172.21.15.112:0/2476244039 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99980c0e70 0x7f999816a320 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f998c00e960 tx=0x7f998c009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:31:35.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.706+0000 7f9994ff9700 1 -- 172.21.15.112:0/2476244039 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998c013680 con 0x7f99980c0e70 2023-01-24T03:31:35.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.706+0000 7f9994ff9700 1 -- 172.21.15.112:0/2476244039 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f998c0137e0 con 0x7f99980c0e70 2023-01-24T03:31:35.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.706+0000 7f999dae8700 1 -- 172.21.15.112:0/2476244039 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99981698a0 con 0x7f99980c0e70 2023-01-24T03:31:35.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.706+0000 7f9994ff9700 1 -- 172.21.15.112:0/2476244039 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998c013b80 con 0x7f99980c0e70 2023-01-24T03:31:35.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.706+0000 7f999dae8700 1 -- 172.21.15.112:0/2476244039 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9998150bb0 con 0x7f99980c0e70 2023-01-24T03:31:35.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.707+0000 7f999dae8700 1 -- 172.21.15.112:0/2476244039 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f999805ea20 con 0x7f99980c0e70 2023-01-24T03:31:35.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.708+0000 7f9994ff9700 1 -- 172.21.15.112:0/2476244039 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f998c019030 con 0x7f99980c0e70 2023-01-24T03:31:35.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.708+0000 7f9994ff9700 1 --2- 172.21.15.112:0/2476244039 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f99800611c0 0x7f9980063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:35.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.708+0000 7f9994ff9700 1 -- 172.21.15.112:0/2476244039 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6867+0+0 (secure 0 0 0) 0x7f998c07d790 con 0x7f99980c0e70 2023-01-24T03:31:35.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.708+0000 7f9997fff700 1 --2- 172.21.15.112:0/2476244039 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f99800611c0 0x7f9980063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:35.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.709+0000 7f9997fff700 1 --2- 172.21.15.112:0/2476244039 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f99800611c0 0x7f9980063670 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f999806d710 tx=0x7f9984016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:31:35.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.713+0000 7f9994ff9700 1 -- 172.21.15.112:0/2476244039 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f998c04d590 con 0x7f99980c0e70 2023-01-24T03:31:35.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.948+0000 7f999dae8700 1 -- 172.21.15.112:0/2476244039 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f999804b340 con 0x7f99980c0e70 2023-01-24T03:31:35.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.949+0000 7f9994ff9700 1 -- 172.21.15.112:0/2476244039 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6114 (secure 0 0 0) 0x7f998c050bb0 con 0x7f99980c0e70 2023-01-24T03:31:35.951 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:31:35.951 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/869267413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":869267413},{"type":"v1","addr":"172.21.15.142:6834","nonce":869267413}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:30:37.600288+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.142:6835/2762621058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2762621058},{"type":"v1","addr":"172.21.15.142:6835","nonce":2762621058}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:31:35.954 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.951+0000 7f999dae8700 1 -- 172.21.15.112:0/2476244039 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f99800611c0 msgr2=0x7f9980063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:35.954 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.951+0000 7f999dae8700 1 --2- 172.21.15.112:0/2476244039 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f99800611c0 0x7f9980063670 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f999806d710 tx=0x7f9984016000 comp rx=0 tx=0).stop 2023-01-24T03:31:35.954 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.951+0000 7f999dae8700 1 -- 172.21.15.112:0/2476244039 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99980c0e70 msgr2=0x7f999816a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:35.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.951+0000 7f999dae8700 1 --2- 172.21.15.112:0/2476244039 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99980c0e70 0x7f999816a320 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f998c00e960 tx=0x7f998c009060 comp rx=0 tx=0).stop 2023-01-24T03:31:35.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.951+0000 7f999dae8700 1 -- 172.21.15.112:0/2476244039 shutdown_connections 2023-01-24T03:31:35.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.951+0000 7f999dae8700 1 --2- 172.21.15.112:0/2476244039 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f99800611c0 0x7f9980063670 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:35.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.951+0000 7f999dae8700 1 --2- 172.21.15.112:0/2476244039 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9998152b20 0x7f9998148060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:35.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.951+0000 7f999dae8700 1 --2- 172.21.15.112:0/2476244039 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f99980c0e70 0x7f999816a320 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:35.957 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.951+0000 7f999dae8700 1 --2- 172.21.15.112:0/2476244039 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f999814f300 0x7f9998150df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:35.957 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.951+0000 7f999dae8700 1 -- 172.21.15.112:0/2476244039 >> 172.21.15.112:0/2476244039 conn(0x7f9998066e10 msgr2=0x7f99980c66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:31:35.957 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.951+0000 7f999dae8700 1 -- 172.21.15.112:0/2476244039 shutdown_connections 2023-01-24T03:31:35.958 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:35.951+0000 7f999dae8700 1 -- 172.21.15.112:0/2476244039 wait complete. 2023-01-24T03:31:35.958 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 10 2023-01-24T03:31:35.965 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=0) 2023-01-24T03:31:35.965 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-01-24T03:31:35.965 INFO:teuthology.orchestra.run:waiting for 300 2023-01-24T03:31:35.977 INFO:tasks.ceph.mds.d:Stopped 2023-01-24T03:31:35.978 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:31:36.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.090+0000 7f7eaf226700 1 Processor -- start 2023-01-24T03:31:36.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.090+0000 7f7eaf226700 1 -- start start 2023-01-24T03:31:36.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.091+0000 7f7eaf226700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7ea806ee20 0x7f7ea8067a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:36.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.091+0000 7f7eaf226700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f7ea80c53c0 0x7f7ea8067fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:36.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.091+0000 7f7eaf226700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ea80c36b0 con 0x7f7ea806f1f0 2023-01-24T03:31:36.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.091+0000 7f7eaf226700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ea80c3d80 con 0x7f7ea80c53c0 2023-01-24T03:31:36.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.091+0000 7f7eaf226700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ea80c30f0 con 0x7f7ea806ee20 2023-01-24T03:31:36.095 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.091+0000 7f7eada23700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f7ea80c53c0 0x7f7ea8067fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:36.095 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.091+0000 7f7eada23700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f7ea80c53c0 0x7f7ea8067fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51416/0 (socket says 172.21.15.112:51416) 2023-01-24T03:31:36.095 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.091+0000 7f7eada23700 1 -- 172.21.15.112:0/1799361881 learned_addr learned my addr 172.21.15.112:0/1799361881 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:31:36.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.091+0000 7f7eae224700 1 --2- 172.21.15.112:0/1799361881 >> v2:172.21.15.142:3300/0 conn(0x7f7ea806ee20 0x7f7ea8067a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:36.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.091+0000 7f7eada23700 1 -- 172.21.15.112:0/1799361881 >> v2:172.21.15.142:3300/0 conn(0x7f7ea806ee20 msgr2=0x7f7ea8067a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:31:36.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.091+0000 7f7eada23700 1 --2- 172.21.15.112:0/1799361881 >> v2:172.21.15.142:3300/0 conn(0x7f7ea806ee20 0x7f7ea8067a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:36.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.091+0000 7f7eada23700 1 -- 172.21.15.112:0/1799361881 >> v1:172.21.15.142:6789/0 conn(0x7f7ea806f1f0 legacy=0x7f7ea8158950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:36.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.091+0000 7f7eada23700 1 -- 172.21.15.112:0/1799361881 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea80c33d0 con 0x7f7ea80c53c0 2023-01-24T03:31:36.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.091+0000 7f7eada23700 1 --2- 172.21.15.112:0/1799361881 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7ea80c53c0 0x7f7ea8067fd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7ea8070eb0 tx=0x7f7e980090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5706347954a9066 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:31:36.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.092+0000 7f7ead222700 1 -- 172.21.15.112:0/1799361881 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e98011630 con 0x7f7ea80c53c0 2023-01-24T03:31:36.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.092+0000 7f7ead222700 1 -- 172.21.15.112:0/1799361881 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e980117d0 con 0x7f7ea80c53c0 2023-01-24T03:31:36.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.092+0000 7f7ead222700 1 -- 172.21.15.112:0/1799361881 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e98011b90 con 0x7f7ea80c53c0 2023-01-24T03:31:36.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.092+0000 7f7eaf226700 1 -- 172.21.15.112:0/1799361881 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7ea80c53c0 msgr2=0x7f7ea8067fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:36.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.092+0000 7f7eaf226700 1 --2- 172.21.15.112:0/1799361881 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7ea80c53c0 0x7f7ea8067fd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7ea8070eb0 tx=0x7f7e980090f0 comp rx=0 tx=0).stop 2023-01-24T03:31:36.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.092+0000 7f7eaf226700 1 -- 172.21.15.112:0/1799361881 shutdown_connections 2023-01-24T03:31:36.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.092+0000 7f7eaf226700 1 --2- 172.21.15.112:0/1799361881 >> v2:172.21.15.142:3300/0 conn(0x7f7ea806ee20 0x7f7ea8067a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:36.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.092+0000 7f7eaf226700 1 --2- 172.21.15.112:0/1799361881 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7ea80c53c0 0x7f7ea8067fd0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:36.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.092+0000 7f7eaf226700 1 -- 172.21.15.112:0/1799361881 >> 172.21.15.112:0/1799361881 conn(0x7f7ea80c87b0 msgr2=0x7f7ea80c4780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:31:36.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.092+0000 7f7eaf226700 1 -- 172.21.15.112:0/1799361881 shutdown_connections 2023-01-24T03:31:36.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.093+0000 7f7eaf226700 1 -- 172.21.15.112:0/1799361881 wait complete. 2023-01-24T03:31:36.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.094+0000 7f7eaf226700 1 Processor -- start 2023-01-24T03:31:36.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.094+0000 7f7eaf226700 1 -- start start 2023-01-24T03:31:36.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.095+0000 7f7eaf226700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7ea806ee20 0x7f7ea8165d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:36.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.095+0000 7f7eaf226700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ea806f1f0 0x7f7ea8150660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:36.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.095+0000 7f7eaf226700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea80c53c0 0x7f7ea81555b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:36.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.095+0000 7f7eaf226700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea80ab6a0 con 0x7f7ea806ee20 2023-01-24T03:31:36.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.095+0000 7f7eaf226700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea81662c0 con 0x7f7ea806f1f0 2023-01-24T03:31:36.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.095+0000 7f7eaf226700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea8067030 con 0x7f7ea80c53c0 2023-01-24T03:31:36.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.095+0000 7f7eae224700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7ea806ee20 0x7f7ea8165d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:36.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.095+0000 7f7eae224700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7ea806ee20 0x7f7ea8165d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51420/0 (socket says 172.21.15.112:51420) 2023-01-24T03:31:36.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.095+0000 7f7eae224700 1 -- 172.21.15.112:0/558327551 learned_addr learned my addr 172.21.15.112:0/558327551 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:31:36.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.095+0000 7f7eada23700 1 --2- 172.21.15.112:0/558327551 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ea806f1f0 0x7f7ea8150660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:36.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.096+0000 7f7eaea25700 1 --2- 172.21.15.112:0/558327551 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea80c53c0 0x7f7ea81555b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:36.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.096+0000 7f7eae224700 1 -- 172.21.15.112:0/558327551 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea80c53c0 msgr2=0x7f7ea81555b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:36.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.096+0000 7f7eae224700 1 --2- 172.21.15.112:0/558327551 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea80c53c0 0x7f7ea81555b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:36.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.096+0000 7f7eae224700 1 -- 172.21.15.112:0/558327551 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ea806f1f0 msgr2=0x7f7ea8150660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:36.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.096+0000 7f7eae224700 1 --2- 172.21.15.112:0/558327551 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ea806f1f0 0x7f7ea8150660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:36.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.096+0000 7f7eae224700 1 -- 172.21.15.112:0/558327551 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e98009370 con 0x7f7ea806ee20 2023-01-24T03:31:36.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.096+0000 7f7eae224700 1 --2- 172.21.15.112:0/558327551 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7ea806ee20 0x7f7ea8165d80 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7ea400d890 tx=0x7f7ea4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:31:36.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.096+0000 7f7e967fc700 1 -- 172.21.15.112:0/558327551 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea40125f0 con 0x7f7ea806ee20 2023-01-24T03:31:36.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.096+0000 7f7e967fc700 1 -- 172.21.15.112:0/558327551 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ea4012750 con 0x7f7ea806ee20 2023-01-24T03:31:36.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.096+0000 7f7eaf226700 1 -- 172.21.15.112:0/558327551 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ea81501d0 con 0x7f7ea806ee20 2023-01-24T03:31:36.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.096+0000 7f7e967fc700 1 -- 172.21.15.112:0/558327551 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea4012b10 con 0x7f7ea806ee20 2023-01-24T03:31:36.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.096+0000 7f7eaf226700 1 -- 172.21.15.112:0/558327551 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ea81528a0 con 0x7f7ea806ee20 2023-01-24T03:31:36.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.097+0000 7f7eaf226700 1 -- 172.21.15.112:0/558327551 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ea805ea20 con 0x7f7ea806ee20 2023-01-24T03:31:36.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.098+0000 7f7e967fc700 1 -- 172.21.15.112:0/558327551 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f7ea401e460 con 0x7f7ea806ee20 2023-01-24T03:31:36.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.098+0000 7f7e967fc700 1 --2- 172.21.15.112:0/558327551 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7e880611f0 0x7f7e880636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:36.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.099+0000 7f7e967fc700 1 -- 172.21.15.112:0/558327551 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6867+0+0 (secure 0 0 0) 0x7f7ea407dd00 con 0x7f7ea806ee20 2023-01-24T03:31:36.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.099+0000 7f7eada23700 1 --2- 172.21.15.112:0/558327551 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7e880611f0 0x7f7e880636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:36.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.099+0000 7f7eada23700 1 --2- 172.21.15.112:0/558327551 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7e880611f0 0x7f7e880636a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7e98009340 tx=0x7f7e98016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:31:36.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.103+0000 7f7e967fc700 1 -- 172.21.15.112:0/558327551 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f7ea4081030 con 0x7f7ea806ee20 2023-01-24T03:31:36.281 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2023-01-24T03:31:36.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.331+0000 7f7eaf226700 1 -- 172.21.15.112:0/558327551 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ea80bf120 con 0x7f7ea806ee20 2023-01-24T03:31:36.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.332+0000 7f7e967fc700 1 -- 172.21.15.112:0/558327551 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6114 (secure 0 0 0) 0x7f7ea404daa0 con 0x7f7ea806ee20 2023-01-24T03:31:36.334 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:31:36.334 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/869267413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":869267413},{"type":"v1","addr":"172.21.15.142:6834","nonce":869267413}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:30:37.600288+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.142:6835/2762621058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2762621058},{"type":"v1","addr":"172.21.15.142:6835","nonce":2762621058}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:31:36.336 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.334+0000 7f7eaf226700 1 -- 172.21.15.112:0/558327551 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7e880611f0 msgr2=0x7f7e880636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:36.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.334+0000 7f7eaf226700 1 --2- 172.21.15.112:0/558327551 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7e880611f0 0x7f7e880636a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7e98009340 tx=0x7f7e98016000 comp rx=0 tx=0).stop 2023-01-24T03:31:36.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.334+0000 7f7eaf226700 1 -- 172.21.15.112:0/558327551 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7ea806ee20 msgr2=0x7f7ea8165d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:36.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.334+0000 7f7eaf226700 1 --2- 172.21.15.112:0/558327551 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7ea806ee20 0x7f7ea8165d80 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7ea400d890 tx=0x7f7ea4008040 comp rx=0 tx=0).stop 2023-01-24T03:31:36.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.334+0000 7f7eaf226700 1 -- 172.21.15.112:0/558327551 shutdown_connections 2023-01-24T03:31:36.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.334+0000 7f7eaf226700 1 --2- 172.21.15.112:0/558327551 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7e880611f0 0x7f7e880636a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:36.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.334+0000 7f7eaf226700 1 --2- 172.21.15.112:0/558327551 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7ea806ee20 0x7f7ea8165d80 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:36.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.334+0000 7f7eaf226700 1 --2- 172.21.15.112:0/558327551 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7ea80c53c0 0x7f7ea81555b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:36.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.334+0000 7f7eaf226700 1 --2- 172.21.15.112:0/558327551 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ea806f1f0 0x7f7ea8150660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:36.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.334+0000 7f7eaf226700 1 -- 172.21.15.112:0/558327551 >> 172.21.15.112:0/558327551 conn(0x7f7ea80c87b0 msgr2=0x7f7ea80c11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:31:36.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.334+0000 7f7eaf226700 1 -- 172.21.15.112:0/558327551 shutdown_connections 2023-01-24T03:31:36.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:36.334+0000 7f7eaf226700 1 -- 172.21.15.112:0/558327551 wait complete. 2023-01-24T03:31:36.339 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 10 2023-01-24T03:31:36.351 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-01-24T03:31:38.352 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:31:38.481 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.479+0000 7fc7582c8700 1 Processor -- start 2023-01-24T03:31:38.481 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.479+0000 7fc7582c8700 1 -- start start 2023-01-24T03:31:38.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.480+0000 7fc7582c8700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fc7500b0aa0 0x7fc750155da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:38.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.480+0000 7fc7582c8700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7500b0f40 con 0x7fc7500ca3e0 2023-01-24T03:31:38.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.480+0000 7fc7582c8700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7500b1170 con 0x7fc7500c7e40 2023-01-24T03:31:38.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.480+0000 7fc7582c8700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7500b1450 con 0x7fc7500b0aa0 2023-01-24T03:31:38.485 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.480+0000 7fc755863700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fc7500b0aa0 0x7fc750155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:38.485 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.480+0000 7fc755863700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fc7500b0aa0 0x7fc750155da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51442/0 (socket says 172.21.15.112:51442) 2023-01-24T03:31:38.485 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.480+0000 7fc756865700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fc7500ca3e0 0x7fc7500ca7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:60664/0 (socket says 172.21.15.112:60664) 2023-01-24T03:31:38.486 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.480+0000 7fc755863700 1 -- 172.21.15.112:0/2667353935 learned_addr learned my addr 172.21.15.112:0/2667353935 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:31:38.486 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.480+0000 7fc755863700 1 -- 172.21.15.112:0/2667353935 >> v1:172.21.15.142:6789/0 conn(0x7fc7500c7e40 legacy=0x7fc7500ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:38.486 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.480+0000 7fc755863700 1 -- 172.21.15.112:0/2667353935 >> v1:172.21.15.112:6789/0 conn(0x7fc7500ca3e0 legacy=0x7fc7500ca7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:38.486 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.480+0000 7fc755863700 1 -- 172.21.15.112:0/2667353935 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7500b1730 con 0x7fc7500b0aa0 2023-01-24T03:31:38.486 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.480+0000 7fc755863700 1 --2- 172.21.15.112:0/2667353935 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7500b0aa0 0x7fc750155da0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc740006fd0 tx=0x7fc7400090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85201a47e8bc8343 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:31:38.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.480+0000 7fc755062700 1 -- 172.21.15.112:0/2667353935 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc740011630 con 0x7fc7500b0aa0 2023-01-24T03:31:38.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.480+0000 7fc755062700 1 -- 172.21.15.112:0/2667353935 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7400117b0 con 0x7fc7500b0aa0 2023-01-24T03:31:38.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.480+0000 7fc755062700 1 -- 172.21.15.112:0/2667353935 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc740011b70 con 0x7fc7500b0aa0 2023-01-24T03:31:38.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.481+0000 7fc7582c8700 1 -- 172.21.15.112:0/2667353935 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7500b0aa0 msgr2=0x7fc750155da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:38.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.481+0000 7fc7582c8700 1 --2- 172.21.15.112:0/2667353935 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7500b0aa0 0x7fc750155da0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc740006fd0 tx=0x7fc7400090f0 comp rx=0 tx=0).stop 2023-01-24T03:31:38.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.481+0000 7fc7582c8700 1 -- 172.21.15.112:0/2667353935 shutdown_connections 2023-01-24T03:31:38.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.481+0000 7fc7582c8700 1 --2- 172.21.15.112:0/2667353935 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7500b0aa0 0x7fc750155da0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:38.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.481+0000 7fc7582c8700 1 -- 172.21.15.112:0/2667353935 >> 172.21.15.112:0/2667353935 conn(0x7fc75005e2c0 msgr2=0x7fc7500cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:31:38.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.481+0000 7fc7582c8700 1 -- 172.21.15.112:0/2667353935 shutdown_connections 2023-01-24T03:31:38.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.481+0000 7fc7582c8700 1 -- 172.21.15.112:0/2667353935 wait complete. 2023-01-24T03:31:38.489 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.482+0000 7fc7582c8700 1 Processor -- start 2023-01-24T03:31:38.489 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.482+0000 7fc7582c8700 1 -- start start 2023-01-24T03:31:38.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.483+0000 7fc7582c8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc7500b0aa0 0x7fc7500c2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:38.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.483+0000 7fc7582c8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc7500c7e40 0x7fc7500bf840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:38.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.483+0000 7fc7582c8700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7500ca3e0 0x7fc7500bfd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:38.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.483+0000 7fc7582c8700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc750169ef0 con 0x7fc7500ca3e0 2023-01-24T03:31:38.492 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.483+0000 7fc7582c8700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7501515b0 con 0x7fc7500c7e40 2023-01-24T03:31:38.492 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.483+0000 7fc7582c8700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc75016aa30 con 0x7fc7500b0aa0 2023-01-24T03:31:38.492 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.483+0000 7fc756865700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7500ca3e0 0x7fc7500bfd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:38.493 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.483+0000 7fc756865700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7500ca3e0 0x7fc7500bfd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51450/0 (socket says 172.21.15.112:51450) 2023-01-24T03:31:38.493 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.483+0000 7fc756865700 1 -- 172.21.15.112:0/4096993993 learned_addr learned my addr 172.21.15.112:0/4096993993 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:31:38.493 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.483+0000 7fc755863700 1 --2- 172.21.15.112:0/4096993993 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc7500c7e40 0x7fc7500bf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:38.493 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.483+0000 7fc756064700 1 --2- 172.21.15.112:0/4096993993 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc7500b0aa0 0x7fc7500c2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:38.494 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.483+0000 7fc756865700 1 -- 172.21.15.112:0/4096993993 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc7500b0aa0 msgr2=0x7fc7500c2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:38.495 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.483+0000 7fc756865700 1 --2- 172.21.15.112:0/4096993993 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc7500b0aa0 0x7fc7500c2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:38.495 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.483+0000 7fc756865700 1 -- 172.21.15.112:0/4096993993 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc7500c7e40 msgr2=0x7fc7500bf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:38.495 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.483+0000 7fc756865700 1 --2- 172.21.15.112:0/4096993993 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc7500c7e40 0x7fc7500bf840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:38.495 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.483+0000 7fc756865700 1 -- 172.21.15.112:0/4096993993 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc740009370 con 0x7fc7500ca3e0 2023-01-24T03:31:38.496 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.483+0000 7fc756865700 1 --2- 172.21.15.112:0/4096993993 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7500ca3e0 0x7fc7500bfd80 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc74800c7f0 tx=0x7fc748001510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:31:38.496 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.483+0000 7fc73e7fc700 1 -- 172.21.15.112:0/4096993993 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc748005c70 con 0x7fc7500ca3e0 2023-01-24T03:31:38.496 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.483+0000 7fc73e7fc700 1 -- 172.21.15.112:0/4096993993 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc748005dd0 con 0x7fc7500ca3e0 2023-01-24T03:31:38.496 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.484+0000 7fc73e7fc700 1 -- 172.21.15.112:0/4096993993 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc748007340 con 0x7fc7500ca3e0 2023-01-24T03:31:38.496 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.484+0000 7fc7582c8700 1 -- 172.21.15.112:0/4096993993 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7501568e0 con 0x7fc7500ca3e0 2023-01-24T03:31:38.497 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.484+0000 7fc7582c8700 1 -- 172.21.15.112:0/4096993993 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc75016a3e0 con 0x7fc7500ca3e0 2023-01-24T03:31:38.497 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.484+0000 7fc7582c8700 1 -- 172.21.15.112:0/4096993993 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc75004bb80 con 0x7fc7500ca3e0 2023-01-24T03:31:38.497 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.485+0000 7fc73e7fc700 1 -- 172.21.15.112:0/4096993993 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fc748010070 con 0x7fc7500ca3e0 2023-01-24T03:31:38.497 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.486+0000 7fc73e7fc700 1 --2- 172.21.15.112:0/4096993993 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc7300611c0 0x7fc730063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:38.497 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.486+0000 7fc756064700 1 --2- 172.21.15.112:0/4096993993 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc7300611c0 0x7fc730063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:38.498 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.486+0000 7fc73e7fc700 1 -- 172.21.15.112:0/4096993993 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6867+0+0 (secure 0 0 0) 0x7fc74807d260 con 0x7fc7500ca3e0 2023-01-24T03:31:38.498 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.486+0000 7fc756064700 1 --2- 172.21.15.112:0/4096993993 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc7300611c0 0x7fc730063670 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc74c000c00 tx=0x7fc74c007d20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:31:38.498 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.490+0000 7fc73e7fc700 1 -- 172.21.15.112:0/4096993993 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fc74804cff0 con 0x7fc7500ca3e0 2023-01-24T03:31:38.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.717+0000 7fc7582c8700 1 -- 172.21.15.112:0/4096993993 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc75016ab70 con 0x7fc7500ca3e0 2023-01-24T03:31:38.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.717+0000 7fc73e7fc700 1 -- 172.21.15.112:0/4096993993 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6114 (secure 0 0 0) 0x7fc748050610 con 0x7fc7500ca3e0 2023-01-24T03:31:38.720 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:31:38.720 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/869267413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":869267413},{"type":"v1","addr":"172.21.15.142:6834","nonce":869267413}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:30:37.600288+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.142:6835/2762621058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2762621058},{"type":"v1","addr":"172.21.15.142:6835","nonce":2762621058}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:31:38.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.720+0000 7fc7582c8700 1 -- 172.21.15.112:0/4096993993 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc7300611c0 msgr2=0x7fc730063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:38.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.720+0000 7fc7582c8700 1 --2- 172.21.15.112:0/4096993993 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc7300611c0 0x7fc730063670 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc74c000c00 tx=0x7fc74c007d20 comp rx=0 tx=0).stop 2023-01-24T03:31:38.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.720+0000 7fc7582c8700 1 -- 172.21.15.112:0/4096993993 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7500ca3e0 msgr2=0x7fc7500bfd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:38.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.720+0000 7fc7582c8700 1 --2- 172.21.15.112:0/4096993993 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7500ca3e0 0x7fc7500bfd80 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc74800c7f0 tx=0x7fc748001510 comp rx=0 tx=0).stop 2023-01-24T03:31:38.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.720+0000 7fc7582c8700 1 -- 172.21.15.112:0/4096993993 shutdown_connections 2023-01-24T03:31:38.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.720+0000 7fc7582c8700 1 --2- 172.21.15.112:0/4096993993 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc7300611c0 0x7fc730063670 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:38.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.720+0000 7fc7582c8700 1 --2- 172.21.15.112:0/4096993993 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7500ca3e0 0x7fc7500bfd80 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:38.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.720+0000 7fc7582c8700 1 --2- 172.21.15.112:0/4096993993 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc7500b0aa0 0x7fc7500c2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:38.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.720+0000 7fc7582c8700 1 --2- 172.21.15.112:0/4096993993 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc7500c7e40 0x7fc7500bf840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:38.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.720+0000 7fc7582c8700 1 -- 172.21.15.112:0/4096993993 >> 172.21.15.112:0/4096993993 conn(0x7fc75005e2c0 msgr2=0x7fc7500cb2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:31:38.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.720+0000 7fc7582c8700 1 -- 172.21.15.112:0/4096993993 shutdown_connections 2023-01-24T03:31:38.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:38.720+0000 7fc7582c8700 1 -- 172.21.15.112:0/4096993993 wait complete. 2023-01-24T03:31:38.724 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 10 2023-01-24T03:31:38.734 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-01-24T03:31:40.008 INFO:tasks.workunit.client.0.smithi112.stderr:Note: switching to '8387d427d91a301da19bd82e090b96e1348bfadb'. 2023-01-24T03:31:40.009 INFO:tasks.workunit.client.0.smithi112.stderr: 2023-01-24T03:31:40.009 INFO:tasks.workunit.client.0.smithi112.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-01-24T03:31:40.009 INFO:tasks.workunit.client.0.smithi112.stderr:changes and commit them, and you can discard any commits you make in this 2023-01-24T03:31:40.010 INFO:tasks.workunit.client.0.smithi112.stderr:state without impacting any branches by switching back to a branch. 2023-01-24T03:31:40.010 INFO:tasks.workunit.client.0.smithi112.stderr: 2023-01-24T03:31:40.010 INFO:tasks.workunit.client.0.smithi112.stderr:If you want to create a new branch to retain commits you create, you may 2023-01-24T03:31:40.010 INFO:tasks.workunit.client.0.smithi112.stderr:do so (now or later) by using -c with the switch command. Example: 2023-01-24T03:31:40.010 INFO:tasks.workunit.client.0.smithi112.stderr: 2023-01-24T03:31:40.011 INFO:tasks.workunit.client.0.smithi112.stderr: git switch -c 2023-01-24T03:31:40.011 INFO:tasks.workunit.client.0.smithi112.stderr: 2023-01-24T03:31:40.011 INFO:tasks.workunit.client.0.smithi112.stderr:Or undo this operation with: 2023-01-24T03:31:40.011 INFO:tasks.workunit.client.0.smithi112.stderr: 2023-01-24T03:31:40.011 INFO:tasks.workunit.client.0.smithi112.stderr: git switch - 2023-01-24T03:31:40.012 INFO:tasks.workunit.client.0.smithi112.stderr: 2023-01-24T03:31:40.012 INFO:tasks.workunit.client.0.smithi112.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-01-24T03:31:40.012 INFO:tasks.workunit.client.0.smithi112.stderr: 2023-01-24T03:31:40.012 INFO:tasks.workunit.client.0.smithi112.stderr:HEAD is now at 8387d427d91 Merge branch 'wip-58347' of https://github.com/vshankar/ceph into wip-yuri5-testing-2023-01-23-0928-quincy 2023-01-24T03:31:40.013 DEBUG:teuthology.orchestra.run.smithi112:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-01-24T03:31:40.046 INFO:tasks.workunit.client.0.smithi112.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-01-24T03:31:40.048 INFO:tasks.workunit.client.0.smithi112.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-01-24T03:31:40.048 INFO:tasks.workunit.client.0.smithi112.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-01-24T03:31:40.603 INFO:tasks.workunit.client.0.smithi112.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-01-24T03:31:40.654 INFO:tasks.workunit.client.0.smithi112.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-01-24T03:31:40.690 INFO:tasks.workunit.client.0.smithi112.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-01-24T03:31:40.692 INFO:tasks.workunit.client.0.smithi112.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-01-24T03:31:40.692 INFO:tasks.workunit.client.0.smithi112.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-01-24T03:31:40.727 INFO:tasks.workunit.client.0.smithi112.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-01-24T03:31:40.730 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-01-24T03:31:40.730 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-01-24T03:31:40.735 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:31:40.785 INFO:tasks.workunit:Running workunits matching fs/snaps on client.0... 2023-01-24T03:31:40.786 INFO:tasks.workunit:Running workunit fs/snaps/snap-rm-diff.sh... 2023-01-24T03:31:40.786 DEBUG:teuthology.orchestra.run.smithi112:workunit test fs/snaps/snap-rm-diff.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8387d427d91a301da19bd82e090b96e1348bfadb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snap-rm-diff.sh 2023-01-24T03:31:40.886 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.883+0000 7f1becea3700 1 Processor -- start 2023-01-24T03:31:40.886 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.883+0000 7f1becea3700 1 -- start start 2023-01-24T03:31:40.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.884+0000 7f1becea3700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f1be814f2c0 0x7f1be80c4f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:40.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.884+0000 7f1becea3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1be80c40e0 0x7f1be81592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:40.891 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.884+0000 7f1becea3700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1be805df40 con 0x7f1be80c4790 2023-01-24T03:31:40.891 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.884+0000 7f1becea3700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1be8155920 con 0x7f1be814f2c0 2023-01-24T03:31:40.891 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.884+0000 7f1becea3700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1be81552f0 con 0x7f1be80c40e0 2023-01-24T03:31:40.891 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.884+0000 7f1be659c700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f1be814f2c0 0x7f1be80c4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:40.892 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.884+0000 7f1be659c700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f1be814f2c0 0x7f1be80c4f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51460/0 (socket says 172.21.15.112:51460) 2023-01-24T03:31:40.892 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.884+0000 7f1be6d9d700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f1be80c4790 0x7f1be8067f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:60684/0 (socket says 172.21.15.112:60684) 2023-01-24T03:31:40.892 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.884+0000 7f1be659c700 1 -- 172.21.15.112:0/3828091906 learned_addr learned my addr 172.21.15.112:0/3828091906 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:31:40.893 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.885+0000 7f1be5d9b700 1 --2- 172.21.15.112:0/3828091906 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1be80c40e0 0x7f1be81592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:40.893 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.885+0000 7f1be659c700 1 -- 172.21.15.112:0/3828091906 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1be80c40e0 msgr2=0x7f1be81592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:31:40.893 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.885+0000 7f1be659c700 1 --2- 172.21.15.112:0/3828091906 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1be80c40e0 0x7f1be81592d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:40.893 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.885+0000 7f1be659c700 1 -- 172.21.15.112:0/3828091906 >> v1:172.21.15.112:6789/0 conn(0x7f1be80c4790 legacy=0x7f1be8067f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:40.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.885+0000 7f1be659c700 1 -- 172.21.15.112:0/3828091906 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1be805e190 con 0x7f1be814f2c0 2023-01-24T03:31:40.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.885+0000 7f1be659c700 1 --2- 172.21.15.112:0/3828091906 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1be814f2c0 0x7f1be80c4f10 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1bd8009f50 tx=0x7f1bd8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=441dd35a73c7b421 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:31:40.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.885+0000 7f1be559a700 1 -- 172.21.15.112:0/3828091906 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd800e070 con 0x7f1be814f2c0 2023-01-24T03:31:40.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.885+0000 7f1be559a700 1 -- 172.21.15.112:0/3828091906 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bd800ba70 con 0x7f1be814f2c0 2023-01-24T03:31:40.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.885+0000 7f1be559a700 1 -- 172.21.15.112:0/3828091906 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd80092e0 con 0x7f1be814f2c0 2023-01-24T03:31:40.895 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.885+0000 7f1becea3700 1 -- 172.21.15.112:0/3828091906 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1be814f2c0 msgr2=0x7f1be80c4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:40.895 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.885+0000 7f1becea3700 1 --2- 172.21.15.112:0/3828091906 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1be814f2c0 0x7f1be80c4f10 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1bd8009f50 tx=0x7f1bd8009840 comp rx=0 tx=0).stop 2023-01-24T03:31:40.895 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.886+0000 7f1becea3700 1 -- 172.21.15.112:0/3828091906 shutdown_connections 2023-01-24T03:31:40.895 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.886+0000 7f1becea3700 1 --2- 172.21.15.112:0/3828091906 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1be814f2c0 0x7f1be80c4f10 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:40.896 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.886+0000 7f1becea3700 1 --2- 172.21.15.112:0/3828091906 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1be80c40e0 0x7f1be81592d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:40.896 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.886+0000 7f1becea3700 1 -- 172.21.15.112:0/3828091906 >> 172.21.15.112:0/3828091906 conn(0x7f1be806daf0 msgr2=0x7f1be805d830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:31:40.896 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.886+0000 7f1becea3700 1 -- 172.21.15.112:0/3828091906 shutdown_connections 2023-01-24T03:31:40.896 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.886+0000 7f1becea3700 1 -- 172.21.15.112:0/3828091906 wait complete. 2023-01-24T03:31:40.897 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.887+0000 7f1becea3700 1 Processor -- start 2023-01-24T03:31:40.897 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.887+0000 7f1becea3700 1 -- start start 2023-01-24T03:31:40.897 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.887+0000 7f1becea3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1be80c40e0 0x7f1be81569d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:40.897 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.887+0000 7f1becea3700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1be80c4790 0x7f1be80c5e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:40.898 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.887+0000 7f1becea3700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1be814f2c0 0x7f1be81653c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:40.898 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.887+0000 7f1becea3700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be81547c0 con 0x7f1be80c4790 2023-01-24T03:31:40.898 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.887+0000 7f1becea3700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be8169510 con 0x7f1be814f2c0 2023-01-24T03:31:40.898 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.887+0000 7f1becea3700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1be8157180 con 0x7f1be80c40e0 2023-01-24T03:31:40.898 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.887+0000 7f1be5d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1be80c4790 0x7f1be80c5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:40.899 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.887+0000 7f1be5d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1be80c4790 0x7f1be80c5e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51468/0 (socket says 172.21.15.112:51468) 2023-01-24T03:31:40.899 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.887+0000 7f1be5d9b700 1 -- 172.21.15.112:0/2105578244 learned_addr learned my addr 172.21.15.112:0/2105578244 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:31:40.899 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.887+0000 7f1be659c700 1 --2- 172.21.15.112:0/2105578244 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1be80c40e0 0x7f1be81569d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:40.899 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.887+0000 7f1be6d9d700 1 --2- 172.21.15.112:0/2105578244 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1be814f2c0 0x7f1be81653c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:40.899 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.888+0000 7f1be5d9b700 1 -- 172.21.15.112:0/2105578244 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1be80c40e0 msgr2=0x7f1be81569d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:40.900 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.888+0000 7f1be5d9b700 1 --2- 172.21.15.112:0/2105578244 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1be80c40e0 0x7f1be81569d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:40.900 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.888+0000 7f1be5d9b700 1 -- 172.21.15.112:0/2105578244 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1be814f2c0 msgr2=0x7f1be81653c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:40.900 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.888+0000 7f1be5d9b700 1 --2- 172.21.15.112:0/2105578244 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1be814f2c0 0x7f1be81653c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:40.900 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.888+0000 7f1be5d9b700 1 -- 172.21.15.112:0/2105578244 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd8009ae0 con 0x7f1be80c4790 2023-01-24T03:31:40.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.888+0000 7f1be5d9b700 1 --2- 172.21.15.112:0/2105578244 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1be80c4790 0x7f1be80c5e00 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f1bd000d890 tx=0x7f1bd0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:31:40.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.888+0000 7f1bde7fc700 1 -- 172.21.15.112:0/2105578244 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd00125f0 con 0x7f1be80c4790 2023-01-24T03:31:40.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.888+0000 7f1bde7fc700 1 -- 172.21.15.112:0/2105578244 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bd0012750 con 0x7f1be80c4790 2023-01-24T03:31:40.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.888+0000 7f1becea3700 1 -- 172.21.15.112:0/2105578244 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1be80ca020 con 0x7f1be80c4790 2023-01-24T03:31:40.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.888+0000 7f1bde7fc700 1 -- 172.21.15.112:0/2105578244 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd0012af0 con 0x7f1be80c4790 2023-01-24T03:31:40.902 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.888+0000 7f1becea3700 1 -- 172.21.15.112:0/2105578244 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1be8168890 con 0x7f1be80c4790 2023-01-24T03:31:40.903 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.889+0000 7f1becea3700 1 -- 172.21.15.112:0/2105578244 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1be804bb80 con 0x7f1be80c4790 2023-01-24T03:31:40.903 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.889+0000 7f1bde7fc700 1 -- 172.21.15.112:0/2105578244 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f1bd001e460 con 0x7f1be80c4790 2023-01-24T03:31:40.903 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.890+0000 7f1bde7fc700 1 --2- 172.21.15.112:0/2105578244 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f1bcc0611c0 0x7f1bcc063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:40.903 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.890+0000 7f1bde7fc700 1 -- 172.21.15.112:0/2105578244 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6867+0+0 (secure 0 0 0) 0x7f1bd007dcd0 con 0x7f1be80c4790 2023-01-24T03:31:40.904 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.890+0000 7f1be659c700 1 --2- 172.21.15.112:0/2105578244 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f1bcc0611c0 0x7f1bcc063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:40.904 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.890+0000 7f1be659c700 1 --2- 172.21.15.112:0/2105578244 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f1bcc0611c0 0x7f1bcc063670 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1bd8009880 tx=0x7f1bd8018000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:31:40.904 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:40.894+0000 7f1bde7fc700 1 -- 172.21.15.112:0/2105578244 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f1bd004da70 con 0x7f1be80c4790 2023-01-24T03:31:41.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:41.116+0000 7f1becea3700 1 -- 172.21.15.112:0/2105578244 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1be80c7170 con 0x7f1be80c4790 2023-01-24T03:31:41.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:41.116+0000 7f1bde7fc700 1 -- 172.21.15.112:0/2105578244 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6114 (secure 0 0 0) 0x7f1bd0051090 con 0x7f1be80c4790 2023-01-24T03:31:41.119 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:31:41.119 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/869267413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":869267413},{"type":"v1","addr":"172.21.15.142:6834","nonce":869267413}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:30:37.600288+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.142:6835/2762621058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2762621058},{"type":"v1","addr":"172.21.15.142:6835","nonce":2762621058}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:31:41.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:41.118+0000 7f1becea3700 1 -- 172.21.15.112:0/2105578244 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f1bcc0611c0 msgr2=0x7f1bcc063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:41.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:41.118+0000 7f1becea3700 1 --2- 172.21.15.112:0/2105578244 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f1bcc0611c0 0x7f1bcc063670 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1bd8009880 tx=0x7f1bd8018000 comp rx=0 tx=0).stop 2023-01-24T03:31:41.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:41.118+0000 7f1becea3700 1 -- 172.21.15.112:0/2105578244 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1be80c4790 msgr2=0x7f1be80c5e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:41.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:41.118+0000 7f1becea3700 1 --2- 172.21.15.112:0/2105578244 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1be80c4790 0x7f1be80c5e00 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f1bd000d890 tx=0x7f1bd0008040 comp rx=0 tx=0).stop 2023-01-24T03:31:41.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:41.118+0000 7f1becea3700 1 -- 172.21.15.112:0/2105578244 shutdown_connections 2023-01-24T03:31:41.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:41.118+0000 7f1becea3700 1 --2- 172.21.15.112:0/2105578244 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f1bcc0611c0 0x7f1bcc063670 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:41.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:41.118+0000 7f1becea3700 1 --2- 172.21.15.112:0/2105578244 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1be814f2c0 0x7f1be81653c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:41.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:41.118+0000 7f1becea3700 1 --2- 172.21.15.112:0/2105578244 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1be80c40e0 0x7f1be81569d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:41.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:41.118+0000 7f1becea3700 1 --2- 172.21.15.112:0/2105578244 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1be80c4790 0x7f1be80c5e00 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:41.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:41.118+0000 7f1becea3700 1 -- 172.21.15.112:0/2105578244 >> 172.21.15.112:0/2105578244 conn(0x7f1be806daf0 msgr2=0x7f1be814ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:31:41.124 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:41.118+0000 7f1becea3700 1 -- 172.21.15.112:0/2105578244 shutdown_connections 2023-01-24T03:31:41.124 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:41.118+0000 7f1becea3700 1 -- 172.21.15.112:0/2105578244 wait complete. 2023-01-24T03:31:41.124 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 10 2023-01-24T03:31:41.132 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-01-24T03:31:43.096 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2023-01-24T03:31:43.132 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:31:43.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.257+0000 7f30209e9700 1 Processor -- start 2023-01-24T03:31:43.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.257+0000 7f30209e9700 1 -- start start 2023-01-24T03:31:43.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.258+0000 7f30209e9700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f301c06dbb0 0x7f301c06e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:43.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.258+0000 7f30209e9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f301c06f240 0x7f301c06e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:43.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.258+0000 7f30209e9700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f301c06c200 con 0x7f301c14bff0 2023-01-24T03:31:43.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.258+0000 7f30209e9700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f301c06f010 con 0x7f301c06dbb0 2023-01-24T03:31:43.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.258+0000 7f30209e9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f301c151ec0 con 0x7f301c06f240 2023-01-24T03:31:43.263 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.258+0000 7f301affd700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f301c06dbb0 0x7f301c06e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:43.263 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.258+0000 7f301affd700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f301c06dbb0 0x7f301c06e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51482/0 (socket says 172.21.15.112:51482) 2023-01-24T03:31:43.263 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.258+0000 7f301affd700 1 -- 172.21.15.112:0/231699576 learned_addr learned my addr 172.21.15.112:0/231699576 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:31:43.263 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.258+0000 7f301bfff700 1 --2- 172.21.15.112:0/231699576 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f301c06f240 0x7f301c06e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:43.263 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.259+0000 7f301affd700 1 -- 172.21.15.112:0/231699576 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f301c06f240 msgr2=0x7f301c06e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:31:43.264 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.259+0000 7f301affd700 1 --2- 172.21.15.112:0/231699576 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f301c06f240 0x7f301c06e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:43.264 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.259+0000 7f301affd700 1 -- 172.21.15.112:0/231699576 >> v1:172.21.15.142:6789/0 conn(0x7f301c14bff0 legacy=0x7f301c0c0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:43.264 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.259+0000 7f301a7fc700 1 -- 172.21.15.112:0/231699576 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3600751685 0 0) 0x7f301c06c200 con 0x7f301c14bff0 2023-01-24T03:31:43.264 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.259+0000 7f301affd700 1 -- 172.21.15.112:0/231699576 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f301c06b8d0 con 0x7f301c06dbb0 2023-01-24T03:31:43.265 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.259+0000 7f301affd700 1 --2- 172.21.15.112:0/231699576 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f301c06dbb0 0x7f301c06e1b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3010006fd0 tx=0x7f30100090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f19f6d716d1a814a server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:31:43.265 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.259+0000 7f301a7fc700 1 -- 172.21.15.112:0/231699576 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3010011630 con 0x7f301c06dbb0 2023-01-24T03:31:43.265 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.259+0000 7f301a7fc700 1 -- 172.21.15.112:0/231699576 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30100117d0 con 0x7f301c06dbb0 2023-01-24T03:31:43.265 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.259+0000 7f301a7fc700 1 -- 172.21.15.112:0/231699576 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3010011b90 con 0x7f301c06dbb0 2023-01-24T03:31:43.266 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.259+0000 7f30209e9700 1 -- 172.21.15.112:0/231699576 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f301c06dbb0 msgr2=0x7f301c06e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:43.266 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.259+0000 7f30209e9700 1 --2- 172.21.15.112:0/231699576 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f301c06dbb0 0x7f301c06e1b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3010006fd0 tx=0x7f30100090f0 comp rx=0 tx=0).stop 2023-01-24T03:31:43.266 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.259+0000 7f30209e9700 1 -- 172.21.15.112:0/231699576 shutdown_connections 2023-01-24T03:31:43.266 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.259+0000 7f30209e9700 1 --2- 172.21.15.112:0/231699576 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f301c06f240 0x7f301c06e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:43.266 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.259+0000 7f30209e9700 1 --2- 172.21.15.112:0/231699576 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f301c06dbb0 0x7f301c06e1b0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:43.267 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.259+0000 7f30209e9700 1 -- 172.21.15.112:0/231699576 >> 172.21.15.112:0/231699576 conn(0x7f301c0689b0 msgr2=0x7f301c0c8540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:31:43.267 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.259+0000 7f30209e9700 1 -- 172.21.15.112:0/231699576 shutdown_connections 2023-01-24T03:31:43.267 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.259+0000 7f30209e9700 1 -- 172.21.15.112:0/231699576 wait complete. 2023-01-24T03:31:43.270 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.261+0000 7f30209e9700 1 Processor -- start 2023-01-24T03:31:43.270 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.261+0000 7f30209e9700 1 -- start start 2023-01-24T03:31:43.270 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.261+0000 7f30209e9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f301c06dbb0 0x7f301c0c86e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:43.270 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.261+0000 7f30209e9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f301c06f240 0x7f301c0bc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:43.271 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.261+0000 7f30209e9700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f301c14bff0 0x7f301c0bcd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:43.271 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.261+0000 7f30209e9700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301c14dba0 con 0x7f301c06f240 2023-01-24T03:31:43.271 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.261+0000 7f30209e9700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301c14e330 con 0x7f301c14bff0 2023-01-24T03:31:43.271 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.261+0000 7f30209e9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f301c14e5f0 con 0x7f301c06dbb0 2023-01-24T03:31:43.271 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.261+0000 7f301affd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f301c06f240 0x7f301c0bc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:43.272 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.261+0000 7f301affd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f301c06f240 0x7f301c0bc7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51486/0 (socket says 172.21.15.112:51486) 2023-01-24T03:31:43.272 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.261+0000 7f301affd700 1 -- 172.21.15.112:0/4194171715 learned_addr learned my addr 172.21.15.112:0/4194171715 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:31:43.272 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.262+0000 7f301bfff700 1 --2- 172.21.15.112:0/4194171715 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f301c14bff0 0x7f301c0bcd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:43.272 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.262+0000 7f301b7fe700 1 --2- 172.21.15.112:0/4194171715 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f301c06dbb0 0x7f301c0c86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:43.272 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.262+0000 7f301affd700 1 -- 172.21.15.112:0/4194171715 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f301c06dbb0 msgr2=0x7f301c0c86e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:43.273 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.262+0000 7f301affd700 1 --2- 172.21.15.112:0/4194171715 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f301c06dbb0 0x7f301c0c86e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:43.273 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.262+0000 7f301affd700 1 -- 172.21.15.112:0/4194171715 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f301c14bff0 msgr2=0x7f301c0bcd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:43.273 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.262+0000 7f301affd700 1 --2- 172.21.15.112:0/4194171715 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f301c14bff0 0x7f301c0bcd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:43.273 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.262+0000 7f301affd700 1 -- 172.21.15.112:0/4194171715 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3010009370 con 0x7f301c06f240 2023-01-24T03:31:43.273 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.262+0000 7f301affd700 1 --2- 172.21.15.112:0/4194171715 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f301c06f240 0x7f301c0bc7e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3010009ea0 tx=0x7f3010016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:31:43.274 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.262+0000 7f30037fe700 1 -- 172.21.15.112:0/4194171715 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3010011b50 con 0x7f301c06f240 2023-01-24T03:31:43.274 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.262+0000 7f30037fe700 1 -- 172.21.15.112:0/4194171715 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3010000cc0 con 0x7f301c06f240 2023-01-24T03:31:43.274 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.262+0000 7f30209e9700 1 -- 172.21.15.112:0/4194171715 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f301c0bd780 con 0x7f301c06f240 2023-01-24T03:31:43.274 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.262+0000 7f30037fe700 1 -- 172.21.15.112:0/4194171715 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301001c6a0 con 0x7f301c06f240 2023-01-24T03:31:43.274 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.262+0000 7f30209e9700 1 -- 172.21.15.112:0/4194171715 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f301c14e860 con 0x7f301c06f240 2023-01-24T03:31:43.274 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.263+0000 7f30209e9700 1 -- 172.21.15.112:0/4194171715 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f301c0bffc0 con 0x7f301c06f240 2023-01-24T03:31:43.275 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.263+0000 7f30037fe700 1 -- 172.21.15.112:0/4194171715 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f301002d070 con 0x7f301c06f240 2023-01-24T03:31:43.275 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.264+0000 7f30037fe700 1 --2- 172.21.15.112:0/4194171715 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2ffc0611c0 0x7f2ffc063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:43.275 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.264+0000 7f30037fe700 1 -- 172.21.15.112:0/4194171715 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6867+0+0 (secure 0 0 0) 0x7f301007eeb0 con 0x7f301c06f240 2023-01-24T03:31:43.276 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.264+0000 7f301b7fe700 1 --2- 172.21.15.112:0/4194171715 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2ffc0611c0 0x7f2ffc063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:43.276 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.264+0000 7f301b7fe700 1 --2- 172.21.15.112:0/4194171715 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2ffc0611c0 0x7f2ffc063670 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f301c14f910 tx=0x7f300c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:31:43.276 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.268+0000 7f30037fe700 1 -- 172.21.15.112:0/4194171715 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f301004ecb0 con 0x7f301c06f240 2023-01-24T03:31:43.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.489+0000 7f30209e9700 1 -- 172.21.15.112:0/4194171715 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f301c14ee70 con 0x7f301c06f240 2023-01-24T03:31:43.492 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.490+0000 7f30037fe700 1 -- 172.21.15.112:0/4194171715 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6114 (secure 0 0 0) 0x7f30100522d0 con 0x7f301c06f240 2023-01-24T03:31:43.492 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:31:43.493 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/869267413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":869267413},{"type":"v1","addr":"172.21.15.142:6834","nonce":869267413}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:30:37.600288+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.142:6835/2762621058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2762621058},{"type":"v1","addr":"172.21.15.142:6835","nonce":2762621058}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:31:43.494 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.492+0000 7f30209e9700 1 -- 172.21.15.112:0/4194171715 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2ffc0611c0 msgr2=0x7f2ffc063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:43.494 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.492+0000 7f30209e9700 1 --2- 172.21.15.112:0/4194171715 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2ffc0611c0 0x7f2ffc063670 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f301c14f910 tx=0x7f300c009040 comp rx=0 tx=0).stop 2023-01-24T03:31:43.494 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.492+0000 7f30209e9700 1 -- 172.21.15.112:0/4194171715 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f301c06f240 msgr2=0x7f301c0bc7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:43.494 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.492+0000 7f30209e9700 1 --2- 172.21.15.112:0/4194171715 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f301c06f240 0x7f301c0bc7e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3010009ea0 tx=0x7f3010016000 comp rx=0 tx=0).stop 2023-01-24T03:31:43.495 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.492+0000 7f30209e9700 1 -- 172.21.15.112:0/4194171715 shutdown_connections 2023-01-24T03:31:43.495 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.492+0000 7f30209e9700 1 --2- 172.21.15.112:0/4194171715 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2ffc0611c0 0x7f2ffc063670 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:43.495 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.492+0000 7f30209e9700 1 --2- 172.21.15.112:0/4194171715 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f301c14bff0 0x7f301c0bcd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:43.495 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.492+0000 7f30209e9700 1 --2- 172.21.15.112:0/4194171715 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f301c06dbb0 0x7f301c0c86e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:43.495 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.492+0000 7f30209e9700 1 --2- 172.21.15.112:0/4194171715 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f301c06f240 0x7f301c0bc7e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:43.496 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.492+0000 7f30209e9700 1 -- 172.21.15.112:0/4194171715 >> 172.21.15.112:0/4194171715 conn(0x7f301c0689b0 msgr2=0x7f301c150530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:31:43.496 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.492+0000 7f30209e9700 1 -- 172.21.15.112:0/4194171715 shutdown_connections 2023-01-24T03:31:43.496 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:43.492+0000 7f30209e9700 1 -- 172.21.15.112:0/4194171715 wait complete. 2023-01-24T03:31:43.496 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 10 2023-01-24T03:31:43.505 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-01-24T03:31:45.506 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:31:45.631 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.629+0000 7f338f373700 1 Processor -- start 2023-01-24T03:31:45.631 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.629+0000 7f338f373700 1 -- start start 2023-01-24T03:31:45.634 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.630+0000 7f338f373700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f33880c7e40 0x7f33880b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:45.634 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.630+0000 7f338f373700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f33880ca3e0 0x7f33880ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:45.635 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.630+0000 7f338f373700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3388156240 con 0x7f3388155da0 2023-01-24T03:31:45.635 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.630+0000 7f338f373700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33880b1170 con 0x7f33880c7e40 2023-01-24T03:31:45.635 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.630+0000 7f338f373700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33880b1450 con 0x7f33880ca3e0 2023-01-24T03:31:45.635 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.630+0000 7f338d10f700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f33880c7e40 0x7f33880b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:45.636 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.630+0000 7f338d10f700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f33880c7e40 0x7f33880b0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51496/0 (socket says 172.21.15.112:51496) 2023-01-24T03:31:45.636 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.630+0000 7f338d10f700 1 -- 172.21.15.112:0/2275432346 learned_addr learned my addr 172.21.15.112:0/2275432346 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:31:45.636 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.630+0000 7f338d910700 1 --2- 172.21.15.112:0/2275432346 >> v2:172.21.15.142:3300/0 conn(0x7f33880ca3e0 0x7f33880ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:45.636 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.630+0000 7f338d10f700 1 -- 172.21.15.112:0/2275432346 >> v2:172.21.15.142:3300/0 conn(0x7f33880ca3e0 msgr2=0x7f33880ca7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:31:45.636 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.630+0000 7f338d10f700 1 --2- 172.21.15.112:0/2275432346 >> v2:172.21.15.142:3300/0 conn(0x7f33880ca3e0 0x7f33880ca7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:45.637 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.630+0000 7f338d10f700 1 -- 172.21.15.112:0/2275432346 >> v1:172.21.15.142:6789/0 conn(0x7f3388155da0 legacy=0x7f33880ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:31:45.637 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.630+0000 7f338d10f700 1 -- 172.21.15.112:0/2275432346 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33880b1730 con 0x7f33880c7e40 2023-01-24T03:31:45.637 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.631+0000 7f338d10f700 1 --2- 172.21.15.112:0/2275432346 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33880c7e40 0x7f33880b0aa0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f338805e810 tx=0x7f3378009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17df7ccb43768f77 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:31:45.637 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.631+0000 7f3387fff700 1 -- 172.21.15.112:0/2275432346 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f337800e070 con 0x7f33880c7e40 2023-01-24T03:31:45.637 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.631+0000 7f3387fff700 1 -- 172.21.15.112:0/2275432346 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f337800ba10 con 0x7f33880c7e40 2023-01-24T03:31:45.638 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.631+0000 7f3387fff700 1 -- 172.21.15.112:0/2275432346 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33780092e0 con 0x7f33880c7e40 2023-01-24T03:31:45.638 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.631+0000 7f338f373700 1 -- 172.21.15.112:0/2275432346 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33880c7e40 msgr2=0x7f33880b0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:45.638 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.631+0000 7f338f373700 1 --2- 172.21.15.112:0/2275432346 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33880c7e40 0x7f33880b0aa0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f338805e810 tx=0x7f3378009840 comp rx=0 tx=0).stop 2023-01-24T03:31:45.638 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.631+0000 7f338f373700 1 -- 172.21.15.112:0/2275432346 shutdown_connections 2023-01-24T03:31:45.638 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.631+0000 7f338f373700 1 --2- 172.21.15.112:0/2275432346 >> v2:172.21.15.142:3300/0 conn(0x7f33880ca3e0 0x7f33880ca7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:45.639 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.631+0000 7f338f373700 1 --2- 172.21.15.112:0/2275432346 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33880c7e40 0x7f33880b0aa0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:45.639 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.631+0000 7f338f373700 1 -- 172.21.15.112:0/2275432346 >> 172.21.15.112:0/2275432346 conn(0x7f338805e2c0 msgr2=0x7f33880cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:31:45.639 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.631+0000 7f338f373700 1 -- 172.21.15.112:0/2275432346 shutdown_connections 2023-01-24T03:31:45.639 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.631+0000 7f338f373700 1 -- 172.21.15.112:0/2275432346 wait complete. 2023-01-24T03:31:45.639 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.632+0000 7f338f373700 1 Processor -- start 2023-01-24T03:31:45.640 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.632+0000 7f338f373700 1 -- start start 2023-01-24T03:31:45.642 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.633+0000 7f338f373700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33880c7e40 0x7f3388154f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:45.643 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.633+0000 7f338f373700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33880ca3e0 0x7f33880c00c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:45.643 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.633+0000 7f338f373700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3388155da0 0x7f3388168f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:45.643 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.633+0000 7f338f373700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33880bf840 con 0x7f33880ca3e0 2023-01-24T03:31:45.643 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.633+0000 7f338f373700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3388168af0 con 0x7f33880c7e40 2023-01-24T03:31:45.643 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.633+0000 7f338f373700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f338814fc80 con 0x7f3388155da0 2023-01-24T03:31:45.644 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.633+0000 7f338c90e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33880ca3e0 0x7f33880c00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:45.644 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.633+0000 7f338c90e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33880ca3e0 0x7f33880c00c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51504/0 (socket says 172.21.15.112:51504) 2023-01-24T03:31:45.644 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.633+0000 7f338c90e700 1 -- 172.21.15.112:0/856198285 learned_addr learned my addr 172.21.15.112:0/856198285 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:31:45.644 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.634+0000 7f338d10f700 1 --2- 172.21.15.112:0/856198285 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33880c7e40 0x7f3388154f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:45.644 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.634+0000 7f338d910700 1 --2- 172.21.15.112:0/856198285 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3388155da0 0x7f3388168f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:45.645 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.634+0000 7f338c90e700 1 -- 172.21.15.112:0/856198285 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3388155da0 msgr2=0x7f3388168f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:45.645 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.634+0000 7f338c90e700 1 --2- 172.21.15.112:0/856198285 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3388155da0 0x7f3388168f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:45.645 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.634+0000 7f338c90e700 1 -- 172.21.15.112:0/856198285 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33880c7e40 msgr2=0x7f3388154f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:45.645 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.634+0000 7f338c90e700 1 --2- 172.21.15.112:0/856198285 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33880c7e40 0x7f3388154f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:45.645 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.634+0000 7f338c90e700 1 -- 172.21.15.112:0/856198285 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3378009ae0 con 0x7f33880ca3e0 2023-01-24T03:31:45.646 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.634+0000 7f338d910700 1 --2- 172.21.15.112:0/856198285 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3388155da0 0x7f3388168f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-01-24T03:31:45.646 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.634+0000 7f338c90e700 1 --2- 172.21.15.112:0/856198285 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33880ca3e0 0x7f33880c00c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f337c00c440 tx=0x7f337c007de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:31:45.646 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.634+0000 7f33857fa700 1 -- 172.21.15.112:0/856198285 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f337c011590 con 0x7f33880ca3e0 2023-01-24T03:31:45.646 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.634+0000 7f33857fa700 1 -- 172.21.15.112:0/856198285 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f337c011710 con 0x7f33880ca3e0 2023-01-24T03:31:45.646 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.634+0000 7f338f373700 1 -- 172.21.15.112:0/856198285 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f338814ff40 con 0x7f33880ca3e0 2023-01-24T03:31:45.647 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.634+0000 7f33857fa700 1 -- 172.21.15.112:0/856198285 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f337c011ad0 con 0x7f33880ca3e0 2023-01-24T03:31:45.647 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.634+0000 7f338f373700 1 -- 172.21.15.112:0/856198285 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3388150500 con 0x7f33880ca3e0 2023-01-24T03:31:45.647 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.635+0000 7f338f373700 1 -- 172.21.15.112:0/856198285 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33881688c0 con 0x7f33880ca3e0 2023-01-24T03:31:45.647 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.635+0000 7f33857fa700 1 -- 172.21.15.112:0/856198285 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f337c010050 con 0x7f33880ca3e0 2023-01-24T03:31:45.647 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.636+0000 7f33857fa700 1 --2- 172.21.15.112:0/856198285 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f33680611c0 0x7f3368063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:45.648 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.636+0000 7f33857fa700 1 -- 172.21.15.112:0/856198285 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6867+0+0 (secure 0 0 0) 0x7f337c07d350 con 0x7f33880ca3e0 2023-01-24T03:31:45.648 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.636+0000 7f338d10f700 1 --2- 172.21.15.112:0/856198285 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f33680611c0 0x7f3368063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:45.648 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.636+0000 7f338d10f700 1 --2- 172.21.15.112:0/856198285 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f33680611c0 0x7f3368063670 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3378009880 tx=0x7f3378017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:31:45.648 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.640+0000 7f33857fa700 1 -- 172.21.15.112:0/856198285 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f33881688c0 con 0x7f33880ca3e0 2023-01-24T03:31:45.863 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.861+0000 7f338f373700 1 -- 172.21.15.112:0/856198285 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33881688c0 con 0x7f33880ca3e0 2023-01-24T03:31:45.864 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.862+0000 7f33857fa700 1 -- 172.21.15.112:0/856198285 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6114 (secure 0 0 0) 0x7f33881688c0 con 0x7f33880ca3e0 2023-01-24T03:31:45.864 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:31:45.864 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/869267413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":869267413},{"type":"v1","addr":"172.21.15.142:6834","nonce":869267413}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:30:37.600288+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.142:6835/2762621058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2762621058},{"type":"v1","addr":"172.21.15.142:6835","nonce":2762621058}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:31:45.866 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.863+0000 7f338f373700 1 -- 172.21.15.112:0/856198285 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f33680611c0 msgr2=0x7f3368063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:45.866 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.863+0000 7f338f373700 1 --2- 172.21.15.112:0/856198285 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f33680611c0 0x7f3368063670 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3378009880 tx=0x7f3378017000 comp rx=0 tx=0).stop 2023-01-24T03:31:45.866 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.864+0000 7f338f373700 1 -- 172.21.15.112:0/856198285 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33880ca3e0 msgr2=0x7f33880c00c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:45.866 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.864+0000 7f338f373700 1 --2- 172.21.15.112:0/856198285 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33880ca3e0 0x7f33880c00c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f337c00c440 tx=0x7f337c007de0 comp rx=0 tx=0).stop 2023-01-24T03:31:45.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.864+0000 7f338f373700 1 -- 172.21.15.112:0/856198285 shutdown_connections 2023-01-24T03:31:45.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.864+0000 7f338f373700 1 --2- 172.21.15.112:0/856198285 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f33680611c0 0x7f3368063670 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:45.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.864+0000 7f338f373700 1 --2- 172.21.15.112:0/856198285 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3388155da0 0x7f3388168f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:45.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.864+0000 7f338f373700 1 --2- 172.21.15.112:0/856198285 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33880c7e40 0x7f3388154f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:45.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.864+0000 7f338f373700 1 --2- 172.21.15.112:0/856198285 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33880ca3e0 0x7f33880c00c0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:45.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.864+0000 7f338f373700 1 -- 172.21.15.112:0/856198285 >> 172.21.15.112:0/856198285 conn(0x7f338805e2c0 msgr2=0x7f33880699f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:31:45.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.864+0000 7f338f373700 1 -- 172.21.15.112:0/856198285 shutdown_connections 2023-01-24T03:31:45.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:45.864+0000 7f338f373700 1 -- 172.21.15.112:0/856198285 wait complete. 2023-01-24T03:31:45.869 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 10 2023-01-24T03:31:45.877 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-01-24T03:31:47.879 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:31:48.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.001+0000 7fe8cdb4c700 1 Processor -- start 2023-01-24T03:31:48.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.001+0000 7fe8cdb4c700 1 -- start start 2023-01-24T03:31:48.006 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.002+0000 7fe8cdb4c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe8c80c5680 0x7fe8c8156730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:48.006 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.002+0000 7fe8cdb4c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8c80c5a50 0x7fe8c8156c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:48.006 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.002+0000 7fe8cdb4c700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8c80c3d10 con 0x7fe8c8155e20 2023-01-24T03:31:48.006 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.002+0000 7fe8cdb4c700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8c80c4eb0 con 0x7fe8c80c5680 2023-01-24T03:31:48.006 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.002+0000 7fe8cdb4c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c80c2e20 con 0x7fe8c80c5a50 2023-01-24T03:31:48.007 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.002+0000 7fe8ccb4a700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe8c80c5680 0x7fe8c8156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:48.007 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.002+0000 7fe8cd34b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8c80c5a50 0x7fe8c8156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:48.007 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.002+0000 7fe8ccb4a700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe8c80c5680 0x7fe8c8156730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.112:44004/0 (socket says 172.21.15.112:44004) 2023-01-24T03:31:48.007 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.002+0000 7fe8cd34b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8c80c5a50 0x7fe8c8156c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.112:38816/0 (socket says 172.21.15.112:38816) 2023-01-24T03:31:48.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.002+0000 7fe8ccb4a700 1 -- 172.21.15.112:0/351060607 learned_addr learned my addr 172.21.15.112:0/351060607 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:31:48.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.002+0000 7fe8c7fff700 1 --1- 172.21.15.112:0/351060607 >> v1:172.21.15.142:6789/0 conn(0x7fe8c8155e20 0x7fe8c80cb870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.112:46288/0 (socket says 172.21.15.112:46288) 2023-01-24T03:31:48.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.003+0000 7fe8cd34b700 1 -- 172.21.15.112:0/351060607 >> v2:172.21.15.142:3300/0 conn(0x7fe8c80c5680 msgr2=0x7fe8c8156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:31:48.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.003+0000 7fe8cd34b700 1 --2- 172.21.15.112:0/351060607 >> v2:172.21.15.142:3300/0 conn(0x7fe8c80c5680 0x7fe8c8156730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:48.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.003+0000 7fe8cd34b700 1 -- 172.21.15.112:0/351060607 >> v1:172.21.15.142:6789/0 conn(0x7fe8c8155e20 legacy=0x7fe8c80cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:48.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.003+0000 7fe8cd34b700 1 -- 172.21.15.112:0/351060607 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8c80c3100 con 0x7fe8c80c5a50 2023-01-24T03:31:48.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.003+0000 7fe8ccb4a700 1 --2- 172.21.15.112:0/351060607 >> v2:172.21.15.142:3300/0 conn(0x7fe8c80c5680 0x7fe8c8156730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:31:48.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.003+0000 7fe8cd34b700 1 --2- 172.21.15.112:0/351060607 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8c80c5a50 0x7fe8c8156c70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe8c0009760 tx=0x7fe8c0009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4898093a48aca9d3 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:31:48.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.003+0000 7fe8c77fe700 1 -- 172.21.15.112:0/351060607 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c0011630 con 0x7fe8c80c5a50 2023-01-24T03:31:48.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.003+0000 7fe8c77fe700 1 -- 172.21.15.112:0/351060607 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8c0011790 con 0x7fe8c80c5a50 2023-01-24T03:31:48.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.003+0000 7fe8c77fe700 1 -- 172.21.15.112:0/351060607 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c0011b50 con 0x7fe8c80c5a50 2023-01-24T03:31:48.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.004+0000 7fe8cdb4c700 1 -- 172.21.15.112:0/351060607 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8c80c5a50 msgr2=0x7fe8c8156c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:48.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.004+0000 7fe8cdb4c700 1 --2- 172.21.15.112:0/351060607 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8c80c5a50 0x7fe8c8156c70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe8c0009760 tx=0x7fe8c0009060 comp rx=0 tx=0).stop 2023-01-24T03:31:48.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.004+0000 7fe8cdb4c700 1 -- 172.21.15.112:0/351060607 shutdown_connections 2023-01-24T03:31:48.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.004+0000 7fe8cdb4c700 1 --2- 172.21.15.112:0/351060607 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8c80c5a50 0x7fe8c8156c70 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:48.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.004+0000 7fe8cdb4c700 1 --2- 172.21.15.112:0/351060607 >> v2:172.21.15.142:3300/0 conn(0x7fe8c80c5680 0x7fe8c8156730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:48.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.004+0000 7fe8cdb4c700 1 -- 172.21.15.112:0/351060607 >> 172.21.15.112:0/351060607 conn(0x7fe8c80685b0 msgr2=0x7fe8c80689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:31:48.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.004+0000 7fe8cdb4c700 1 -- 172.21.15.112:0/351060607 shutdown_connections 2023-01-24T03:31:48.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.004+0000 7fe8cdb4c700 1 -- 172.21.15.112:0/351060607 wait complete. 2023-01-24T03:31:48.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.005+0000 7fe8cdb4c700 1 Processor -- start 2023-01-24T03:31:48.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.005+0000 7fe8cdb4c700 1 -- start start 2023-01-24T03:31:48.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.006+0000 7fe8cdb4c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe8c80c5680 0x7fe8c815a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:48.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.006+0000 7fe8cdb4c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8c80c5a50 0x7fe8c815acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:48.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.006+0000 7fe8cdb4c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8c8155e20 0x7fe8c814bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:48.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.006+0000 7fe8cdb4c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c8165b80 con 0x7fe8c80c5680 2023-01-24T03:31:48.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.006+0000 7fe8cdb4c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c81692a0 con 0x7fe8c8155e20 2023-01-24T03:31:48.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.006+0000 7fe8cdb4c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c8166010 con 0x7fe8c80c5a50 2023-01-24T03:31:48.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.006+0000 7fe8ccb4a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe8c80c5680 0x7fe8c815a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:48.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.006+0000 7fe8ccb4a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe8c80c5680 0x7fe8c815a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51516/0 (socket says 172.21.15.112:51516) 2023-01-24T03:31:48.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.006+0000 7fe8ccb4a700 1 -- 172.21.15.112:0/1183281300 learned_addr learned my addr 172.21.15.112:0/1183281300 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:31:48.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.006+0000 7fe8cd34b700 1 --2- 172.21.15.112:0/1183281300 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8c8155e20 0x7fe8c814bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:48.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.006+0000 7fe8c7fff700 1 --2- 172.21.15.112:0/1183281300 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8c80c5a50 0x7fe8c815acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:48.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.006+0000 7fe8ccb4a700 1 -- 172.21.15.112:0/1183281300 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8c80c5a50 msgr2=0x7fe8c815acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:48.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.006+0000 7fe8ccb4a700 1 --2- 172.21.15.112:0/1183281300 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8c80c5a50 0x7fe8c815acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:48.020 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.006+0000 7fe8ccb4a700 1 -- 172.21.15.112:0/1183281300 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8c8155e20 msgr2=0x7fe8c814bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:48.020 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.006+0000 7fe8ccb4a700 1 --2- 172.21.15.112:0/1183281300 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8c8155e20 0x7fe8c814bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:48.020 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.006+0000 7fe8ccb4a700 1 -- 172.21.15.112:0/1183281300 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b8009370 con 0x7fe8c80c5680 2023-01-24T03:31:48.020 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.006+0000 7fe8c7fff700 1 --2- 172.21.15.112:0/1183281300 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8c80c5a50 0x7fe8c815acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-01-24T03:31:48.020 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.006+0000 7fe8ccb4a700 1 --2- 172.21.15.112:0/1183281300 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe8c80c5680 0x7fe8c815a6c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe8b8006d00 tx=0x7fe8b80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:31:48.021 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.006+0000 7fe8c4ff9700 1 -- 172.21.15.112:0/1183281300 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b800eee0 con 0x7fe8c80c5680 2023-01-24T03:31:48.021 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.006+0000 7fe8cdb4c700 1 -- 172.21.15.112:0/1183281300 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8c00092a0 con 0x7fe8c80c5680 2023-01-24T03:31:48.021 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.006+0000 7fe8c4ff9700 1 -- 172.21.15.112:0/1183281300 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8b800c8e0 con 0x7fe8c80c5680 2023-01-24T03:31:48.021 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.006+0000 7fe8c4ff9700 1 -- 172.21.15.112:0/1183281300 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b800cc60 con 0x7fe8c80c5680 2023-01-24T03:31:48.021 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.006+0000 7fe8cdb4c700 1 -- 172.21.15.112:0/1183281300 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8c8165250 con 0x7fe8c80c5680 2023-01-24T03:31:48.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.007+0000 7fe8cdb4c700 1 -- 172.21.15.112:0/1183281300 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8c8150580 con 0x7fe8c80c5680 2023-01-24T03:31:48.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.008+0000 7fe8c4ff9700 1 -- 172.21.15.112:0/1183281300 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fe8b8023070 con 0x7fe8c80c5680 2023-01-24T03:31:48.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.008+0000 7fe8c4ff9700 1 --2- 172.21.15.112:0/1183281300 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fe8ac0611f0 0x7fe8ac0636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:48.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.008+0000 7fe8c4ff9700 1 -- 172.21.15.112:0/1183281300 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6867+0+0 (secure 0 0 0) 0x7fe8b807dd80 con 0x7fe8c80c5680 2023-01-24T03:31:48.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.008+0000 7fe8c7fff700 1 --2- 172.21.15.112:0/1183281300 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fe8ac0611f0 0x7fe8ac0636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:48.023 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.009+0000 7fe8c7fff700 1 --2- 172.21.15.112:0/1183281300 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fe8ac0611f0 0x7fe8ac0636a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe8c80c9220 tx=0x7fe8b0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:31:48.023 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.012+0000 7fe8c4ff9700 1 -- 172.21.15.112:0/1183281300 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fe8c8150580 con 0x7fe8c80c5680 2023-01-24T03:31:48.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.234+0000 7fe8cdb4c700 1 -- 172.21.15.112:0/1183281300 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8c8150580 con 0x7fe8c80c5680 2023-01-24T03:31:48.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.235+0000 7fe8c4ff9700 1 -- 172.21.15.112:0/1183281300 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6114 (secure 0 0 0) 0x7fe8c8150580 con 0x7fe8c80c5680 2023-01-24T03:31:48.237 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:31:48.238 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/869267413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":869267413},{"type":"v1","addr":"172.21.15.142:6834","nonce":869267413}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:30:37.600288+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.142:6835/2762621058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2762621058},{"type":"v1","addr":"172.21.15.142:6835","nonce":2762621058}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:31:48.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.237+0000 7fe8cdb4c700 1 -- 172.21.15.112:0/1183281300 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fe8ac0611f0 msgr2=0x7fe8ac0636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:48.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.237+0000 7fe8cdb4c700 1 --2- 172.21.15.112:0/1183281300 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fe8ac0611f0 0x7fe8ac0636a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe8c80c9220 tx=0x7fe8b0009040 comp rx=0 tx=0).stop 2023-01-24T03:31:48.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.237+0000 7fe8cdb4c700 1 -- 172.21.15.112:0/1183281300 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe8c80c5680 msgr2=0x7fe8c815a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:48.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.237+0000 7fe8cdb4c700 1 --2- 172.21.15.112:0/1183281300 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe8c80c5680 0x7fe8c815a6c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe8b8006d00 tx=0x7fe8b80090f0 comp rx=0 tx=0).stop 2023-01-24T03:31:48.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.237+0000 7fe8cdb4c700 1 -- 172.21.15.112:0/1183281300 shutdown_connections 2023-01-24T03:31:48.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.237+0000 7fe8cdb4c700 1 --2- 172.21.15.112:0/1183281300 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fe8ac0611f0 0x7fe8ac0636a0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:48.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.237+0000 7fe8cdb4c700 1 --2- 172.21.15.112:0/1183281300 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe8c80c5680 0x7fe8c815a6c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:48.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.237+0000 7fe8cdb4c700 1 --2- 172.21.15.112:0/1183281300 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe8c8155e20 0x7fe8c814bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:48.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.237+0000 7fe8cdb4c700 1 --2- 172.21.15.112:0/1183281300 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe8c80c5a50 0x7fe8c815acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:48.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.237+0000 7fe8cdb4c700 1 -- 172.21.15.112:0/1183281300 >> 172.21.15.112:0/1183281300 conn(0x7fe8c80685b0 msgr2=0x7fe8c8153b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:31:48.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.237+0000 7fe8cdb4c700 1 -- 172.21.15.112:0/1183281300 shutdown_connections 2023-01-24T03:31:48.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:48.237+0000 7fe8cdb4c700 1 -- 172.21.15.112:0/1183281300 wait complete. 2023-01-24T03:31:48.242 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 10 2023-01-24T03:31:48.251 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-01-24T03:31:50.252 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:31:50.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.381+0000 7ff766a47700 1 Processor -- start 2023-01-24T03:31:50.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.381+0000 7ff766a47700 1 -- start start 2023-01-24T03:31:50.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.382+0000 7ff766a47700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7ff7600b0aa0 0x7ff760155da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:50.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.382+0000 7ff766a47700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff7600ca3e0 0x7ff7600ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:50.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.382+0000 7ff766a47700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7600b0f40 con 0x7ff7600c7e40 2023-01-24T03:31:50.393 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.382+0000 7ff766a47700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7600b1170 con 0x7ff7600b0aa0 2023-01-24T03:31:50.393 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.382+0000 7ff766a47700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7600b1450 con 0x7ff7600ca3e0 2023-01-24T03:31:50.393 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.383+0000 7ff75f7fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7ff7600b0aa0 0x7ff760155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:50.393 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.383+0000 7ff764fe4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff7600ca3e0 0x7ff7600ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:50.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.383+0000 7ff75ffff700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7ff7600c7e40 0x7ff7600ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.112:46304/0 (socket says 172.21.15.112:46304) 2023-01-24T03:31:50.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.383+0000 7ff75ffff700 1 -- 172.21.15.112:0/2972729482 learned_addr learned my addr 172.21.15.112:0/2972729482 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:31:50.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.383+0000 7ff75effd700 1 -- 172.21.15.112:0/2972729482 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2313653742 0 0) 0x7ff7600b0f40 con 0x7ff7600c7e40 2023-01-24T03:31:50.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.383+0000 7ff75effd700 1 -- 172.21.15.112:0/2972729482 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff74c0033f0 con 0x7ff7600c7e40 2023-01-24T03:31:50.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.383+0000 7ff75f7fe700 1 -- 172.21.15.112:0/2972729482 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff7600ca3e0 msgr2=0x7ff7600ca7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:31:50.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.383+0000 7ff75f7fe700 1 --2- 172.21.15.112:0/2972729482 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff7600ca3e0 0x7ff7600ca7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:50.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.383+0000 7ff75f7fe700 1 -- 172.21.15.112:0/2972729482 >> v1:172.21.15.142:6789/0 conn(0x7ff7600c7e40 legacy=0x7ff7600ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:50.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.383+0000 7ff75f7fe700 1 -- 172.21.15.112:0/2972729482 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7600b1730 con 0x7ff7600b0aa0 2023-01-24T03:31:50.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.383+0000 7ff764fe4700 1 --2- 172.21.15.112:0/2972729482 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff7600ca3e0 0x7ff7600ca7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:31:50.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.383+0000 7ff75f7fe700 1 --2- 172.21.15.112:0/2972729482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff7600b0aa0 0x7ff760155da0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff754009760 tx=0x7ff754009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8d98fc9c37ab931 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:31:50.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.383+0000 7ff75effd700 1 -- 172.21.15.112:0/2972729482 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7540105b0 con 0x7ff7600b0aa0 2023-01-24T03:31:50.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.383+0000 7ff75effd700 1 -- 172.21.15.112:0/2972729482 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff754010710 con 0x7ff7600b0aa0 2023-01-24T03:31:50.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.383+0000 7ff75effd700 1 -- 172.21.15.112:0/2972729482 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff754010ab0 con 0x7ff7600b0aa0 2023-01-24T03:31:50.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.384+0000 7ff766a47700 1 -- 172.21.15.112:0/2972729482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff7600b0aa0 msgr2=0x7ff760155da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:50.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.384+0000 7ff766a47700 1 --2- 172.21.15.112:0/2972729482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff7600b0aa0 0x7ff760155da0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff754009760 tx=0x7ff754009060 comp rx=0 tx=0).stop 2023-01-24T03:31:50.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.384+0000 7ff766a47700 1 -- 172.21.15.112:0/2972729482 shutdown_connections 2023-01-24T03:31:50.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.384+0000 7ff766a47700 1 --2- 172.21.15.112:0/2972729482 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff7600ca3e0 0x7ff7600ca7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:50.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.384+0000 7ff766a47700 1 --2- 172.21.15.112:0/2972729482 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff7600b0aa0 0x7ff760155da0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:50.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.384+0000 7ff766a47700 1 -- 172.21.15.112:0/2972729482 >> 172.21.15.112:0/2972729482 conn(0x7ff76005e2c0 msgr2=0x7ff7600cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:31:50.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.384+0000 7ff766a47700 1 -- 172.21.15.112:0/2972729482 shutdown_connections 2023-01-24T03:31:50.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.384+0000 7ff766a47700 1 -- 172.21.15.112:0/2972729482 wait complete. 2023-01-24T03:31:50.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.385+0000 7ff766a47700 1 Processor -- start 2023-01-24T03:31:50.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.385+0000 7ff766a47700 1 -- start start 2023-01-24T03:31:50.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.386+0000 7ff766a47700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff7600b0aa0 0x7ff76014b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:50.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.386+0000 7ff766a47700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff7600c7e40 0x7ff760168590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:50.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.386+0000 7ff766a47700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff7600ca3e0 0x7ff760168ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:50.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.386+0000 7ff766a47700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7600c1640 con 0x7ff7600ca3e0 2023-01-24T03:31:50.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.386+0000 7ff766a47700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff760155580 con 0x7ff7600c7e40 2023-01-24T03:31:50.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.386+0000 7ff766a47700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7600bf840 con 0x7ff7600b0aa0 2023-01-24T03:31:50.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.386+0000 7ff764fe4700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff7600ca3e0 0x7ff760168ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:50.400 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.386+0000 7ff764fe4700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff7600ca3e0 0x7ff760168ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51536/0 (socket says 172.21.15.112:51536) 2023-01-24T03:31:50.400 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.386+0000 7ff764fe4700 1 -- 172.21.15.112:0/3722932811 learned_addr learned my addr 172.21.15.112:0/3722932811 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:31:50.400 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.386+0000 7ff75f7fe700 1 --2- 172.21.15.112:0/3722932811 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff7600c7e40 0x7ff760168590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:50.400 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.386+0000 7ff75ffff700 1 --2- 172.21.15.112:0/3722932811 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff7600b0aa0 0x7ff76014b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:50.400 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.386+0000 7ff764fe4700 1 -- 172.21.15.112:0/3722932811 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff7600b0aa0 msgr2=0x7ff76014b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:50.401 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.386+0000 7ff764fe4700 1 --2- 172.21.15.112:0/3722932811 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff7600b0aa0 0x7ff76014b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:50.401 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.386+0000 7ff764fe4700 1 -- 172.21.15.112:0/3722932811 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff7600c7e40 msgr2=0x7ff760168590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:50.401 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.386+0000 7ff764fe4700 1 --2- 172.21.15.112:0/3722932811 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff7600c7e40 0x7ff760168590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:50.401 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.386+0000 7ff764fe4700 1 -- 172.21.15.112:0/3722932811 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff758009a20 con 0x7ff7600ca3e0 2023-01-24T03:31:50.401 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.386+0000 7ff764fe4700 1 --2- 172.21.15.112:0/3722932811 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff7600ca3e0 0x7ff760168ad0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff75800c0f0 tx=0x7ff75800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:31:50.402 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.387+0000 7ff747fff700 1 -- 172.21.15.112:0/3722932811 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75800cc30 con 0x7ff7600ca3e0 2023-01-24T03:31:50.402 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.387+0000 7ff747fff700 1 -- 172.21.15.112:0/3722932811 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff75800f040 con 0x7ff7600ca3e0 2023-01-24T03:31:50.402 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.387+0000 7ff766a47700 1 -- 172.21.15.112:0/3722932811 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7540092a0 con 0x7ff7600ca3e0 2023-01-24T03:31:50.402 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.387+0000 7ff747fff700 1 -- 172.21.15.112:0/3722932811 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff758014480 con 0x7ff7600ca3e0 2023-01-24T03:31:50.402 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.387+0000 7ff766a47700 1 -- 172.21.15.112:0/3722932811 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7600c00e0 con 0x7ff7600ca3e0 2023-01-24T03:31:50.402 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.388+0000 7ff766a47700 1 -- 172.21.15.112:0/3722932811 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7600c0270 con 0x7ff7600ca3e0 2023-01-24T03:31:50.403 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.388+0000 7ff747fff700 1 -- 172.21.15.112:0/3722932811 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7ff758014830 con 0x7ff7600ca3e0 2023-01-24T03:31:50.403 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.388+0000 7ff747fff700 1 --2- 172.21.15.112:0/3722932811 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff7400611c0 0x7ff740063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:50.403 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.388+0000 7ff747fff700 1 -- 172.21.15.112:0/3722932811 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6867+0+0 (secure 0 0 0) 0x7ff75807e6f0 con 0x7ff7600ca3e0 2023-01-24T03:31:50.403 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.388+0000 7ff75ffff700 1 --2- 172.21.15.112:0/3722932811 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff7400611c0 0x7ff740063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:50.403 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.389+0000 7ff75ffff700 1 --2- 172.21.15.112:0/3722932811 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff7400611c0 0x7ff740063670 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff750005750 tx=0x7ff750009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:31:50.404 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.393+0000 7ff747fff700 1 -- 172.21.15.112:0/3722932811 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7ff75804e4f0 con 0x7ff7600ca3e0 2023-01-24T03:31:50.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.615+0000 7ff766a47700 1 -- 172.21.15.112:0/3722932811 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7600c0470 con 0x7ff7600ca3e0 2023-01-24T03:31:50.619 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.616+0000 7ff747fff700 1 -- 172.21.15.112:0/3722932811 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6114 (secure 0 0 0) 0x7ff758051b10 con 0x7ff7600ca3e0 2023-01-24T03:31:50.619 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:31:50.619 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/869267413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":869267413},{"type":"v1","addr":"172.21.15.142:6834","nonce":869267413}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:30:37.600288+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.142:6835/2762621058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2762621058},{"type":"v1","addr":"172.21.15.142:6835","nonce":2762621058}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:31:50.621 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.618+0000 7ff766a47700 1 -- 172.21.15.112:0/3722932811 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff7400611c0 msgr2=0x7ff740063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:50.622 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.618+0000 7ff766a47700 1 --2- 172.21.15.112:0/3722932811 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff7400611c0 0x7ff740063670 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff750005750 tx=0x7ff750009040 comp rx=0 tx=0).stop 2023-01-24T03:31:50.622 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.618+0000 7ff766a47700 1 -- 172.21.15.112:0/3722932811 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff7600ca3e0 msgr2=0x7ff760168ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:50.622 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.618+0000 7ff766a47700 1 --2- 172.21.15.112:0/3722932811 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff7600ca3e0 0x7ff760168ad0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff75800c0f0 tx=0x7ff75800b040 comp rx=0 tx=0).stop 2023-01-24T03:31:50.622 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.618+0000 7ff766a47700 1 -- 172.21.15.112:0/3722932811 shutdown_connections 2023-01-24T03:31:50.622 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.618+0000 7ff766a47700 1 --2- 172.21.15.112:0/3722932811 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff7400611c0 0x7ff740063670 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:50.623 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.618+0000 7ff766a47700 1 --2- 172.21.15.112:0/3722932811 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff7600ca3e0 0x7ff760168ad0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:50.623 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.618+0000 7ff766a47700 1 --2- 172.21.15.112:0/3722932811 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff7600b0aa0 0x7ff76014b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:50.623 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.618+0000 7ff766a47700 1 --2- 172.21.15.112:0/3722932811 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff7600c7e40 0x7ff760168590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:50.623 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.618+0000 7ff766a47700 1 -- 172.21.15.112:0/3722932811 >> 172.21.15.112:0/3722932811 conn(0x7ff76005e2c0 msgr2=0x7ff7600680c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:31:50.623 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.618+0000 7ff766a47700 1 -- 172.21.15.112:0/3722932811 shutdown_connections 2023-01-24T03:31:50.624 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:50.619+0000 7ff766a47700 1 -- 172.21.15.112:0/3722932811 wait complete. 2023-01-24T03:31:50.624 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 10 2023-01-24T03:31:50.632 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-01-24T03:31:52.633 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:31:52.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.755+0000 7f3a9d2a1700 1 Processor -- start 2023-01-24T03:31:52.759 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.755+0000 7f3a9d2a1700 1 -- start start 2023-01-24T03:31:52.761 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.756+0000 7f3a9d2a1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3a980aeb90 0x7f3a980cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:52.761 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.756+0000 7f3a9d2a1700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f3a98068b10 0x7f3a980b14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:52.762 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.756+0000 7f3a9d2a1700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a981560e0 con 0x7f3a980aef60 2023-01-24T03:31:52.762 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.756+0000 7f3a9d2a1700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a98155530 con 0x7f3a98068b10 2023-01-24T03:31:52.762 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.756+0000 7f3a9d2a1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a98069ac0 con 0x7f3a980aeb90 2023-01-24T03:31:52.763 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.756+0000 7f3a977fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f3a98068b10 0x7f3a980b14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:52.763 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.756+0000 7f3a96ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3a980aeb90 0x7f3a980cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:52.763 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.756+0000 7f3a977fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f3a98068b10 0x7f3a980b14c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.112:44038/0 (socket says 172.21.15.112:44038) 2023-01-24T03:31:52.763 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.756+0000 7f3a96ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3a980aeb90 0x7f3a980cc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.112:38844/0 (socket says 172.21.15.112:38844) 2023-01-24T03:31:52.763 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.756+0000 7f3a967fc700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f3a980aef60 0x7f3a98068400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.112:46322/0 (socket says 172.21.15.112:46322) 2023-01-24T03:31:52.764 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.756+0000 7f3a977fe700 1 -- 172.21.15.112:0/1512776013 learned_addr learned my addr 172.21.15.112:0/1512776013 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:31:52.764 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.757+0000 7f3a977fe700 1 -- 172.21.15.112:0/1512776013 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3a980aeb90 msgr2=0x7f3a980cc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:31:52.764 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.757+0000 7f3a977fe700 1 --2- 172.21.15.112:0/1512776013 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3a980aeb90 0x7f3a980cc3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:52.764 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.757+0000 7f3a977fe700 1 -- 172.21.15.112:0/1512776013 >> v1:172.21.15.142:6789/0 conn(0x7f3a980aef60 legacy=0x7f3a98068400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:52.764 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.757+0000 7f3a977fe700 1 -- 172.21.15.112:0/1512776013 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a98069d80 con 0x7f3a98068b10 2023-01-24T03:31:52.765 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.757+0000 7f3a95ffb700 1 -- 172.21.15.112:0/1512776013 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4253277757 0 0) 0x7f3a981560e0 con 0x7f3a980aef60 2023-01-24T03:31:52.765 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.757+0000 7f3a96ffd700 1 --2- 172.21.15.112:0/1512776013 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3a980aeb90 0x7f3a980cc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:31:52.765 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.757+0000 7f3a977fe700 1 --2- 172.21.15.112:0/1512776013 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a98068b10 0x7f3a980b14c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3a90006fd0 tx=0x7f3a900090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c22a3c203cfd18e7 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:31:52.765 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.757+0000 7f3a95ffb700 1 -- 172.21.15.112:0/1512776013 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a90011630 con 0x7f3a98068b10 2023-01-24T03:31:52.765 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.757+0000 7f3a95ffb700 1 -- 172.21.15.112:0/1512776013 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a900117b0 con 0x7f3a98068b10 2023-01-24T03:31:52.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.757+0000 7f3a95ffb700 1 -- 172.21.15.112:0/1512776013 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a90011b70 con 0x7f3a98068b10 2023-01-24T03:31:52.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.758+0000 7f3a9d2a1700 1 -- 172.21.15.112:0/1512776013 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a98068b10 msgr2=0x7f3a980b14c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:52.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.758+0000 7f3a9d2a1700 1 --2- 172.21.15.112:0/1512776013 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a98068b10 0x7f3a980b14c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3a90006fd0 tx=0x7f3a900090f0 comp rx=0 tx=0).stop 2023-01-24T03:31:52.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.758+0000 7f3a9d2a1700 1 -- 172.21.15.112:0/1512776013 shutdown_connections 2023-01-24T03:31:52.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.758+0000 7f3a9d2a1700 1 --2- 172.21.15.112:0/1512776013 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a98068b10 0x7f3a980b14c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:52.767 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.758+0000 7f3a9d2a1700 1 --2- 172.21.15.112:0/1512776013 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3a980aeb90 0x7f3a980cc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:52.767 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.758+0000 7f3a9d2a1700 1 -- 172.21.15.112:0/1512776013 >> 172.21.15.112:0/1512776013 conn(0x7f3a9805e300 msgr2=0x7f3a980cba20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:31:52.767 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.758+0000 7f3a9d2a1700 1 -- 172.21.15.112:0/1512776013 shutdown_connections 2023-01-24T03:31:52.767 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.758+0000 7f3a9d2a1700 1 -- 172.21.15.112:0/1512776013 wait complete. 2023-01-24T03:31:52.770 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.759+0000 7f3a9d2a1700 1 Processor -- start 2023-01-24T03:31:52.770 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.759+0000 7f3a9d2a1700 1 -- start start 2023-01-24T03:31:52.770 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.760+0000 7f3a9d2a1700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3a98068b10 0x7f3a980cb020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:52.771 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.760+0000 7f3a9d2a1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3a980aeb90 0x7f3a9814f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:52.771 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.760+0000 7f3a9d2a1700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a980aef60 0x7f3a981650c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:52.771 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.760+0000 7f3a9d2a1700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a98168690 con 0x7f3a98068b10 2023-01-24T03:31:52.771 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.760+0000 7f3a9d2a1700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9814cf70 con 0x7f3a980aef60 2023-01-24T03:31:52.771 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.760+0000 7f3a9d2a1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a980bf9b0 con 0x7f3a980aeb90 2023-01-24T03:31:52.772 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.760+0000 7f3a96ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3a98068b10 0x7f3a980cb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:52.772 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.760+0000 7f3a96ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3a98068b10 0x7f3a980cb020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51548/0 (socket says 172.21.15.112:51548) 2023-01-24T03:31:52.772 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.760+0000 7f3a96ffd700 1 -- 172.21.15.112:0/2530059678 learned_addr learned my addr 172.21.15.112:0/2530059678 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:31:52.772 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.760+0000 7f3a967fc700 1 --2- 172.21.15.112:0/2530059678 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3a980aeb90 0x7f3a9814f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:52.773 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.760+0000 7f3a977fe700 1 --2- 172.21.15.112:0/2530059678 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a980aef60 0x7f3a981650c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:52.773 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.760+0000 7f3a96ffd700 1 -- 172.21.15.112:0/2530059678 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3a980aeb90 msgr2=0x7f3a9814f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:52.773 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.760+0000 7f3a96ffd700 1 --2- 172.21.15.112:0/2530059678 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3a980aeb90 0x7f3a9814f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:52.773 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.760+0000 7f3a96ffd700 1 -- 172.21.15.112:0/2530059678 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a980aef60 msgr2=0x7f3a981650c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:52.774 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.760+0000 7f3a96ffd700 1 --2- 172.21.15.112:0/2530059678 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a980aef60 0x7f3a981650c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:52.774 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.760+0000 7f3a96ffd700 1 -- 172.21.15.112:0/2530059678 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a88009a20 con 0x7f3a98068b10 2023-01-24T03:31:52.774 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.760+0000 7f3a96ffd700 1 --2- 172.21.15.112:0/2530059678 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3a98068b10 0x7f3a980cb020 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3a8800c170 tx=0x7f3a8800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:31:52.774 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.760+0000 7f3a8effd700 1 -- 172.21.15.112:0/2530059678 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8800ccb0 con 0x7f3a98068b10 2023-01-24T03:31:52.774 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.760+0000 7f3a8effd700 1 -- 172.21.15.112:0/2530059678 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a8800f040 con 0x7f3a98068b10 2023-01-24T03:31:52.775 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.760+0000 7f3a9d2a1700 1 -- 172.21.15.112:0/2530059678 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a90009370 con 0x7f3a98068b10 2023-01-24T03:31:52.775 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.760+0000 7f3a8effd700 1 -- 172.21.15.112:0/2530059678 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a88014480 con 0x7f3a98068b10 2023-01-24T03:31:52.775 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.760+0000 7f3a9d2a1700 1 -- 172.21.15.112:0/2530059678 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a981688e0 con 0x7f3a98068b10 2023-01-24T03:31:52.775 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.761+0000 7f3a9d2a1700 1 -- 172.21.15.112:0/2530059678 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a9804bb80 con 0x7f3a98068b10 2023-01-24T03:31:52.775 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.762+0000 7f3a8effd700 1 -- 172.21.15.112:0/2530059678 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f3a880147f0 con 0x7f3a98068b10 2023-01-24T03:31:52.776 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.762+0000 7f3a8effd700 1 --2- 172.21.15.112:0/2530059678 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3a7c0611c0 0x7f3a7c063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:52.776 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.762+0000 7f3a8effd700 1 -- 172.21.15.112:0/2530059678 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6867+0+0 (secure 0 0 0) 0x7f3a8807e880 con 0x7f3a98068b10 2023-01-24T03:31:52.776 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.762+0000 7f3a967fc700 1 --2- 172.21.15.112:0/2530059678 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3a7c0611c0 0x7f3a7c063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:52.776 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.762+0000 7f3a967fc700 1 --2- 172.21.15.112:0/2530059678 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3a7c0611c0 0x7f3a7c063670 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3a98162ca0 tx=0x7f3a80007bd0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:31:52.776 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.766+0000 7f3a8effd700 1 -- 172.21.15.112:0/2530059678 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f3a8804e680 con 0x7f3a98068b10 2023-01-24T03:31:52.990 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.987+0000 7f3a9d2a1700 1 -- 172.21.15.112:0/2530059678 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a9806efe0 con 0x7f3a98068b10 2023-01-24T03:31:52.990 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.987+0000 7f3a8effd700 1 -- 172.21.15.112:0/2530059678 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+6114 (secure 0 0 0) 0x7f3a88051ca0 con 0x7f3a98068b10 2023-01-24T03:31:52.991 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:31:52.991 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":10,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4288,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/869267413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":869267413},{"type":"v1","addr":"172.21.15.142:6834","nonce":869267413}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":10,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:30:37.600288+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.142:6835/2762621058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2762621058},{"type":"v1","addr":"172.21.15.142:6835","nonce":2762621058}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:31:52.992 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.989+0000 7f3a9d2a1700 1 -- 172.21.15.112:0/2530059678 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3a7c0611c0 msgr2=0x7f3a7c063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:52.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.989+0000 7f3a9d2a1700 1 --2- 172.21.15.112:0/2530059678 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3a7c0611c0 0x7f3a7c063670 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3a98162ca0 tx=0x7f3a80007bd0 comp rx=0 tx=0).stop 2023-01-24T03:31:52.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.989+0000 7f3a9d2a1700 1 -- 172.21.15.112:0/2530059678 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3a98068b10 msgr2=0x7f3a980cb020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:52.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.989+0000 7f3a9d2a1700 1 --2- 172.21.15.112:0/2530059678 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3a98068b10 0x7f3a980cb020 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3a8800c170 tx=0x7f3a8800b040 comp rx=0 tx=0).stop 2023-01-24T03:31:52.994 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.989+0000 7f3a9d2a1700 1 -- 172.21.15.112:0/2530059678 shutdown_connections 2023-01-24T03:31:52.994 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.989+0000 7f3a9d2a1700 1 --2- 172.21.15.112:0/2530059678 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3a7c0611c0 0x7f3a7c063670 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:52.994 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.989+0000 7f3a9d2a1700 1 --2- 172.21.15.112:0/2530059678 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3a98068b10 0x7f3a980cb020 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:52.994 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.989+0000 7f3a9d2a1700 1 --2- 172.21.15.112:0/2530059678 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3a980aef60 0x7f3a981650c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:52.994 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.989+0000 7f3a9d2a1700 1 --2- 172.21.15.112:0/2530059678 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3a980aeb90 0x7f3a9814f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:52.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.989+0000 7f3a9d2a1700 1 -- 172.21.15.112:0/2530059678 >> 172.21.15.112:0/2530059678 conn(0x7f3a9805e300 msgr2=0x7f3a98068ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:31:52.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.989+0000 7f3a9d2a1700 1 -- 172.21.15.112:0/2530059678 shutdown_connections 2023-01-24T03:31:52.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:52.989+0000 7f3a9d2a1700 1 -- 172.21.15.112:0/2530059678 wait complete. 2023-01-24T03:31:52.995 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 10 2023-01-24T03:31:53.004 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-01-24T03:31:55.005 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:31:55.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.127+0000 7f5e751cf700 1 Processor -- start 2023-01-24T03:31:55.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.127+0000 7f5e751cf700 1 -- start start 2023-01-24T03:31:55.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.128+0000 7f5e751cf700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e700c5680 0x7f5e70156730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:55.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.128+0000 7f5e751cf700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e700c3d10 con 0x7f5e70155e20 2023-01-24T03:31:55.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.128+0000 7f5e751cf700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e700c2e20 con 0x7f5e700c5a50 2023-01-24T03:31:55.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.128+0000 7f5e751cf700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e700c4eb0 con 0x7f5e700c5680 2023-01-24T03:31:55.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.128+0000 7f5e6f7fe700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f5e70155e20 0x7f5e700cb870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:60788/0 (socket says 172.21.15.112:60788) 2023-01-24T03:31:55.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.128+0000 7f5e6f7fe700 1 -- 172.21.15.112:0/1474658387 learned_addr learned my addr 172.21.15.112:0/1474658387 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:31:55.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.128+0000 7f5e6ffff700 1 --2- 172.21.15.112:0/1474658387 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e700c5680 0x7f5e70156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:55.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.128+0000 7f5e6effd700 1 -- 172.21.15.112:0/1474658387 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1027638087 0 0) 0x7f5e700c3d10 con 0x7f5e70155e20 2023-01-24T03:31:55.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.128+0000 7f5e6effd700 1 -- 172.21.15.112:0/1474658387 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e5c0033f0 con 0x7f5e70155e20 2023-01-24T03:31:55.139 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.128+0000 7f5e6effd700 1 -- 172.21.15.112:0/1474658387 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1129535988 0 0) 0x7f5e5c0033f0 con 0x7f5e70155e20 2023-01-24T03:31:55.139 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.128+0000 7f5e6effd700 1 -- 172.21.15.112:0/1474658387 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e700c5680 msgr2=0x7f5e70156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:31:55.139 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.128+0000 7f5e6effd700 1 --2- 172.21.15.112:0/1474658387 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e700c5680 0x7f5e70156730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:55.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.128+0000 7f5e6effd700 1 -- 172.21.15.112:0/1474658387 >> v1:172.21.15.142:6789/0 conn(0x7f5e700c5a50 legacy=0x7f5e7014ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:55.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.128+0000 7f5e6effd700 1 -- 172.21.15.112:0/1474658387 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e700c3100 con 0x7f5e70155e20 2023-01-24T03:31:55.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.128+0000 7f5e6effd700 1 -- 172.21.15.112:0/1474658387 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7f5e64001470 con 0x7f5e70155e20 2023-01-24T03:31:55.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.129+0000 7f5e6effd700 1 -- 172.21.15.112:0/1474658387 >> v1:172.21.15.112:6789/0 conn(0x7f5e70155e20 legacy=0x7f5e700cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:55.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.129+0000 7f5e6effd700 1 --2- 172.21.15.112:0/1474658387 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e700c5680 0x7f5e5c003910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:55.141 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.129+0000 7f5e6effd700 1 --2- 172.21.15.112:0/1474658387 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e5c003e50 0x7f5e5c004690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:55.141 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.129+0000 7f5e6effd700 1 --2- 172.21.15.112:0/1474658387 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5e5c004bd0 0x7f5e5c005040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:55.141 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.129+0000 7f5e6effd700 1 -- 172.21.15.112:0/1474658387 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e700c4eb0 con 0x7f5e5c004bd0 2023-01-24T03:31:55.141 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.129+0000 7f5e6effd700 1 -- 172.21.15.112:0/1474658387 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c00b9c0 con 0x7f5e5c003e50 2023-01-24T03:31:55.141 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.129+0000 7f5e6effd700 1 -- 172.21.15.112:0/1474658387 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c00bb80 con 0x7f5e700c5680 2023-01-24T03:31:55.142 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.129+0000 7f5e749ce700 1 --2- 172.21.15.112:0/1474658387 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5e5c004bd0 0x7f5e5c005040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:55.142 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.129+0000 7f5e6ffff700 1 --2- 172.21.15.112:0/1474658387 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e700c5680 0x7f5e5c003910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:55.142 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.129+0000 7f5e6f7fe700 1 --2- 172.21.15.112:0/1474658387 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e5c003e50 0x7f5e5c004690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:55.142 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.129+0000 7f5e749ce700 1 -- 172.21.15.112:0/1474658387 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e700c5680 msgr2=0x7f5e5c003910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:31:55.142 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.129+0000 7f5e749ce700 1 --2- 172.21.15.112:0/1474658387 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e700c5680 0x7f5e5c003910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:55.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.129+0000 7f5e749ce700 1 -- 172.21.15.112:0/1474658387 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e5c003e50 msgr2=0x7f5e5c004690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:31:55.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.129+0000 7f5e749ce700 1 --2- 172.21.15.112:0/1474658387 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e5c003e50 0x7f5e5c004690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:55.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.129+0000 7f5e749ce700 1 -- 172.21.15.112:0/1474658387 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5c0033f0 con 0x7f5e5c004bd0 2023-01-24T03:31:55.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.129+0000 7f5e749ce700 1 --2- 172.21.15.112:0/1474658387 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5e5c004bd0 0x7f5e5c005040 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5e68001f20 tx=0x7f5e68007f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5c6e76813ec87d5 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:31:55.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.129+0000 7f5e6effd700 1 -- 172.21.15.112:0/1474658387 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e680017e0 con 0x7f5e5c004bd0 2023-01-24T03:31:55.144 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.129+0000 7f5e6effd700 1 -- 172.21.15.112:0/1474658387 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e6800c6d0 con 0x7f5e5c004bd0 2023-01-24T03:31:55.144 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.129+0000 7f5e6effd700 1 -- 172.21.15.112:0/1474658387 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6800ca90 con 0x7f5e5c004bd0 2023-01-24T03:31:55.144 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.130+0000 7f5e751cf700 1 -- 172.21.15.112:0/1474658387 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5e5c004bd0 msgr2=0x7f5e5c005040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:55.144 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.130+0000 7f5e751cf700 1 --2- 172.21.15.112:0/1474658387 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5e5c004bd0 0x7f5e5c005040 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5e68001f20 tx=0x7f5e68007f10 comp rx=0 tx=0).stop 2023-01-24T03:31:55.144 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.130+0000 7f5e749ce700 1 -- 172.21.15.112:0/1474658387 reap_dead start 2023-01-24T03:31:55.145 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.130+0000 7f5e751cf700 1 -- 172.21.15.112:0/1474658387 shutdown_connections 2023-01-24T03:31:55.145 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.130+0000 7f5e751cf700 1 -- 172.21.15.112:0/1474658387 >> 172.21.15.112:0/1474658387 conn(0x7f5e700685b0 msgr2=0x7f5e700689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:31:55.145 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.130+0000 7f5e751cf700 1 -- 172.21.15.112:0/1474658387 shutdown_connections 2023-01-24T03:31:55.145 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.130+0000 7f5e751cf700 1 -- 172.21.15.112:0/1474658387 wait complete. 2023-01-24T03:31:55.145 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.131+0000 7f5e751cf700 1 Processor -- start 2023-01-24T03:31:55.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.131+0000 7f5e751cf700 1 -- start start 2023-01-24T03:31:55.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.132+0000 7f5e751cf700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e70168110 0x7f5e70150570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:55.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.132+0000 7f5e751cf700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e700c8150 0x7f5e70149700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:55.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.132+0000 7f5e751cf700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5e701684e0 0x7f5e70149c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:55.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.132+0000 7f5e751cf700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e70152a70 con 0x7f5e701684e0 2023-01-24T03:31:55.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.132+0000 7f5e751cf700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e700b18d0 con 0x7f5e70168110 2023-01-24T03:31:55.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.132+0000 7f5e751cf700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e70168dc0 con 0x7f5e700c8150 2023-01-24T03:31:55.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.132+0000 7f5e749ce700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5e701684e0 0x7f5e70149c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:55.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.132+0000 7f5e749ce700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5e701684e0 0x7f5e70149c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51584/0 (socket says 172.21.15.112:51584) 2023-01-24T03:31:55.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.132+0000 7f5e749ce700 1 -- 172.21.15.112:0/746262910 learned_addr learned my addr 172.21.15.112:0/746262910 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:31:55.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.132+0000 7f5e6f7fe700 1 --2- 172.21.15.112:0/746262910 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e700c8150 0x7f5e70149700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:55.148 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.132+0000 7f5e6ffff700 1 --2- 172.21.15.112:0/746262910 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e70168110 0x7f5e70150570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:55.148 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.132+0000 7f5e749ce700 1 -- 172.21.15.112:0/746262910 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e700c8150 msgr2=0x7f5e70149700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:55.148 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.132+0000 7f5e749ce700 1 --2- 172.21.15.112:0/746262910 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e700c8150 0x7f5e70149700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:55.148 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.132+0000 7f5e749ce700 1 -- 172.21.15.112:0/746262910 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e70168110 msgr2=0x7f5e70150570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:55.148 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.132+0000 7f5e749ce700 1 --2- 172.21.15.112:0/746262910 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e70168110 0x7f5e70150570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:55.149 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.132+0000 7f5e749ce700 1 -- 172.21.15.112:0/746262910 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e70168a90 con 0x7f5e701684e0 2023-01-24T03:31:55.149 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.132+0000 7f5e749ce700 1 --2- 172.21.15.112:0/746262910 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5e701684e0 0x7f5e70149c40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5e68012f50 tx=0x7f5e68016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:31:55.149 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.132+0000 7f5e57fff700 1 -- 172.21.15.112:0/746262910 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e680017e0 con 0x7f5e701684e0 2023-01-24T03:31:55.149 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.132+0000 7f5e57fff700 1 -- 172.21.15.112:0/746262910 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e68022070 con 0x7f5e701684e0 2023-01-24T03:31:55.150 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.132+0000 7f5e751cf700 1 -- 172.21.15.112:0/746262910 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e700c77e0 con 0x7f5e701684e0 2023-01-24T03:31:55.150 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.132+0000 7f5e57fff700 1 -- 172.21.15.112:0/746262910 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6800c840 con 0x7f5e701684e0 2023-01-24T03:31:55.150 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.132+0000 7f5e751cf700 1 -- 172.21.15.112:0/746262910 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e701521f0 con 0x7f5e701684e0 2023-01-24T03:31:55.150 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.133+0000 7f5e751cf700 1 -- 172.21.15.112:0/746262910 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e700c7e70 con 0x7f5e701684e0 2023-01-24T03:31:55.151 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.134+0000 7f5e57fff700 1 -- 172.21.15.112:0/746262910 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f5e6802e070 con 0x7f5e701684e0 2023-01-24T03:31:55.151 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.134+0000 7f5e57fff700 1 --2- 172.21.15.112:0/746262910 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5e500611f0 0x7f5e500636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:55.151 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.134+0000 7f5e57fff700 1 -- 172.21.15.112:0/746262910 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6953+0+0 (secure 0 0 0) 0x7f5e6807f2f0 con 0x7f5e701684e0 2023-01-24T03:31:55.151 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.134+0000 7f5e6ffff700 1 --2- 172.21.15.112:0/746262910 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5e500611f0 0x7f5e500636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:55.151 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.135+0000 7f5e6ffff700 1 --2- 172.21.15.112:0/746262910 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5e500611f0 0x7f5e500636a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5e60006d80 tx=0x7f5e60009060 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:31:55.152 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.138+0000 7f5e57fff700 1 -- 172.21.15.112:0/746262910 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f5e700c7e70 con 0x7f5e701684e0 2023-01-24T03:31:55.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.360+0000 7f5e751cf700 1 -- 172.21.15.112:0/746262910 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e701523e0 con 0x7f5e701684e0 2023-01-24T03:31:55.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.360+0000 7f5e57fff700 1 -- 172.21.15.112:0/746262910 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+5357 (secure 0 0 0) 0x7f5e6804f0a0 con 0x7f5e701684e0 2023-01-24T03:31:55.364 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:31:55.364 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:31:53.596607+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4288},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":0,"incarnation":11,"state":"up:replay","state_seq":1,"addr":"172.21.15.142:6834/869267413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":869267413},{"type":"v1","addr":"172.21.15.142:6834","nonce":869267413}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:31:55.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.362+0000 7f5e751cf700 1 -- 172.21.15.112:0/746262910 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5e500611f0 msgr2=0x7f5e500636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:55.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.362+0000 7f5e751cf700 1 --2- 172.21.15.112:0/746262910 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5e500611f0 0x7f5e500636a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5e60006d80 tx=0x7f5e60009060 comp rx=0 tx=0).stop 2023-01-24T03:31:55.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.362+0000 7f5e751cf700 1 -- 172.21.15.112:0/746262910 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5e701684e0 msgr2=0x7f5e70149c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:55.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.362+0000 7f5e751cf700 1 --2- 172.21.15.112:0/746262910 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5e701684e0 0x7f5e70149c40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5e68012f50 tx=0x7f5e68016000 comp rx=0 tx=0).stop 2023-01-24T03:31:55.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.362+0000 7f5e751cf700 1 -- 172.21.15.112:0/746262910 shutdown_connections 2023-01-24T03:31:55.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.362+0000 7f5e751cf700 1 --2- 172.21.15.112:0/746262910 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5e500611f0 0x7f5e500636a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:55.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.362+0000 7f5e751cf700 1 --2- 172.21.15.112:0/746262910 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5e701684e0 0x7f5e70149c40 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:55.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.362+0000 7f5e751cf700 1 --2- 172.21.15.112:0/746262910 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e70168110 0x7f5e70150570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:55.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.362+0000 7f5e751cf700 1 --2- 172.21.15.112:0/746262910 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e700c8150 0x7f5e70149700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:55.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.363+0000 7f5e751cf700 1 -- 172.21.15.112:0/746262910 >> 172.21.15.112:0/746262910 conn(0x7f5e700685b0 msgr2=0x7f5e7005d830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:31:55.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.363+0000 7f5e751cf700 1 -- 172.21.15.112:0/746262910 shutdown_connections 2023-01-24T03:31:55.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.363+0000 7f5e751cf700 1 -- 172.21.15.112:0/746262910 wait complete. 2023-01-24T03:31:55.369 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 11 2023-01-24T03:31:55.377 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2023-01-24T03:31:55.377 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-01-24T03:31:55.378 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:31:55.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.488+0000 7fed887e7700 1 Processor -- start 2023-01-24T03:31:55.492 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.488+0000 7fed887e7700 1 -- start start 2023-01-24T03:31:55.497 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.489+0000 7fed887e7700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fed800c7e40 0x7fed800b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:55.497 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.489+0000 7fed887e7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed80155da0 0x7fed800ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:55.498 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.489+0000 7fed887e7700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fed800aef20 0x7fed800ca3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:55.498 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.489+0000 7fed887e7700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed80156240 con 0x7fed800c7e40 2023-01-24T03:31:55.498 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.489+0000 7fed887e7700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed800b1170 con 0x7fed800aef20 2023-01-24T03:31:55.498 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.489+0000 7fed887e7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed800b1450 con 0x7fed80155da0 2023-01-24T03:31:55.499 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.489+0000 7fed86583700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fed800c7e40 0x7fed800b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:55.499 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.490+0000 7fed86583700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fed800c7e40 0x7fed800b0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51588/0 (socket says 172.21.15.112:51588) 2023-01-24T03:31:55.499 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.490+0000 7fed86583700 1 -- 172.21.15.112:0/3986262502 learned_addr learned my addr 172.21.15.112:0/3986262502 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:31:55.499 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.490+0000 7fed86d84700 1 --2- 172.21.15.112:0/3986262502 >> v2:172.21.15.142:3300/0 conn(0x7fed800aef20 0x7fed800ca3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:55.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.490+0000 7fed85d82700 1 --2- 172.21.15.112:0/3986262502 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed80155da0 0x7fed800ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:55.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.490+0000 7fed86583700 1 -- 172.21.15.112:0/3986262502 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed80155da0 msgr2=0x7fed800ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:31:55.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.490+0000 7fed86583700 1 --2- 172.21.15.112:0/3986262502 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed80155da0 0x7fed800ae9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:55.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.490+0000 7fed86583700 1 -- 172.21.15.112:0/3986262502 >> v2:172.21.15.142:3300/0 conn(0x7fed800aef20 msgr2=0x7fed800ca3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:31:55.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.490+0000 7fed86583700 1 --2- 172.21.15.112:0/3986262502 >> v2:172.21.15.142:3300/0 conn(0x7fed800aef20 0x7fed800ca3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:55.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.490+0000 7fed86583700 1 -- 172.21.15.112:0/3986262502 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed800b1730 con 0x7fed800c7e40 2023-01-24T03:31:55.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.490+0000 7fed86583700 1 --2- 172.21.15.112:0/3986262502 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed800c7e40 0x7fed800b0aa0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fed80089ca0 tx=0x7fed7c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb43cda3d3d72aaf server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:31:55.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.490+0000 7fed85581700 1 -- 172.21.15.112:0/3986262502 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7c011630 con 0x7fed800c7e40 2023-01-24T03:31:55.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.490+0000 7fed85581700 1 -- 172.21.15.112:0/3986262502 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed7c011790 con 0x7fed800c7e40 2023-01-24T03:31:55.502 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.490+0000 7fed85581700 1 -- 172.21.15.112:0/3986262502 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7c011b50 con 0x7fed800c7e40 2023-01-24T03:31:55.502 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.491+0000 7fed887e7700 1 -- 172.21.15.112:0/3986262502 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed800c7e40 msgr2=0x7fed800b0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:55.502 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.491+0000 7fed887e7700 1 --2- 172.21.15.112:0/3986262502 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed800c7e40 0x7fed800b0aa0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fed80089ca0 tx=0x7fed7c0090f0 comp rx=0 tx=0).stop 2023-01-24T03:31:55.502 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.491+0000 7fed887e7700 1 -- 172.21.15.112:0/3986262502 shutdown_connections 2023-01-24T03:31:55.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.491+0000 7fed887e7700 1 --2- 172.21.15.112:0/3986262502 >> v2:172.21.15.142:3300/0 conn(0x7fed800aef20 0x7fed800ca3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:55.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.491+0000 7fed887e7700 1 --2- 172.21.15.112:0/3986262502 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed800c7e40 0x7fed800b0aa0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:55.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.491+0000 7fed887e7700 1 --2- 172.21.15.112:0/3986262502 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed80155da0 0x7fed800ae9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:55.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.491+0000 7fed887e7700 1 -- 172.21.15.112:0/3986262502 >> 172.21.15.112:0/3986262502 conn(0x7fed8005e2c0 msgr2=0x7fed800cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:31:55.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.491+0000 7fed887e7700 1 -- 172.21.15.112:0/3986262502 shutdown_connections 2023-01-24T03:31:55.504 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.491+0000 7fed887e7700 1 -- 172.21.15.112:0/3986262502 wait complete. 2023-01-24T03:31:55.504 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.492+0000 7fed887e7700 1 Processor -- start 2023-01-24T03:31:55.504 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.492+0000 7fed887e7700 1 -- start start 2023-01-24T03:31:55.504 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.492+0000 7fed887e7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed800aef20 0x7fed800c1b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:55.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.492+0000 7fed887e7700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed800c7e40 0x7fed800c2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:55.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.492+0000 7fed887e7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fed80155da0 0x7fed800c3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:55.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.492+0000 7fed887e7700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed801512c0 con 0x7fed800c7e40 2023-01-24T03:31:55.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.492+0000 7fed887e7700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed801527c0 con 0x7fed80155da0 2023-01-24T03:31:55.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.492+0000 7fed887e7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed800c5150 con 0x7fed800aef20 2023-01-24T03:31:55.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.492+0000 7fed85d82700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed800c7e40 0x7fed800c2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:55.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.493+0000 7fed85d82700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed800c7e40 0x7fed800c2ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51596/0 (socket says 172.21.15.112:51596) 2023-01-24T03:31:55.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.493+0000 7fed85d82700 1 -- 172.21.15.112:0/1907936218 learned_addr learned my addr 172.21.15.112:0/1907936218 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:31:55.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.493+0000 7fed86583700 1 --2- 172.21.15.112:0/1907936218 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed800aef20 0x7fed800c1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:55.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.493+0000 7fed86d84700 1 --2- 172.21.15.112:0/1907936218 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fed80155da0 0x7fed800c3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:55.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.493+0000 7fed85d82700 1 -- 172.21.15.112:0/1907936218 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed800aef20 msgr2=0x7fed800c1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:55.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.493+0000 7fed85d82700 1 --2- 172.21.15.112:0/1907936218 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed800aef20 0x7fed800c1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:55.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.493+0000 7fed85d82700 1 -- 172.21.15.112:0/1907936218 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fed80155da0 msgr2=0x7fed800c3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:55.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.493+0000 7fed85d82700 1 --2- 172.21.15.112:0/1907936218 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fed80155da0 0x7fed800c3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:55.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.493+0000 7fed85d82700 1 -- 172.21.15.112:0/1907936218 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed7c009370 con 0x7fed800c7e40 2023-01-24T03:31:55.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.493+0000 7fed85d82700 1 --2- 172.21.15.112:0/1907936218 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed800c7e40 0x7fed800c2ff0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fed7000e960 tx=0x7fed70009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:31:55.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.493+0000 7fed6e7fc700 1 -- 172.21.15.112:0/1907936218 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed70013680 con 0x7fed800c7e40 2023-01-24T03:31:55.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.493+0000 7fed6e7fc700 1 -- 172.21.15.112:0/1907936218 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed700137e0 con 0x7fed800c7e40 2023-01-24T03:31:55.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.493+0000 7fed887e7700 1 -- 172.21.15.112:0/1907936218 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed80152a30 con 0x7fed800c7e40 2023-01-24T03:31:55.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.493+0000 7fed6e7fc700 1 -- 172.21.15.112:0/1907936218 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed70013b60 con 0x7fed800c7e40 2023-01-24T03:31:55.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.493+0000 7fed887e7700 1 -- 172.21.15.112:0/1907936218 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed800c2ad0 con 0x7fed800c7e40 2023-01-24T03:31:55.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.494+0000 7fed887e7700 1 -- 172.21.15.112:0/1907936218 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed8004bb80 con 0x7fed800c7e40 2023-01-24T03:31:55.510 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.494+0000 7fed6e7fc700 1 -- 172.21.15.112:0/1907936218 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fed70019030 con 0x7fed800c7e40 2023-01-24T03:31:55.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.495+0000 7fed6e7fc700 1 --2- 172.21.15.112:0/1907936218 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fed600611c0 0x7fed60063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:55.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.495+0000 7fed6e7fc700 1 -- 172.21.15.112:0/1907936218 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6953+0+0 (secure 0 0 0) 0x7fed7007d730 con 0x7fed800c7e40 2023-01-24T03:31:55.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.495+0000 7fed86583700 1 --2- 172.21.15.112:0/1907936218 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fed600611c0 0x7fed60063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:55.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.495+0000 7fed86583700 1 --2- 172.21.15.112:0/1907936218 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fed600611c0 0x7fed60063670 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fed80089ca0 tx=0x7fed7c016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:31:55.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.499+0000 7fed6e7fc700 1 -- 172.21.15.112:0/1907936218 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fed7004d4e0 con 0x7fed800c7e40 2023-01-24T03:31:55.720 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.717+0000 7fed887e7700 1 -- 172.21.15.112:0/1907936218 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed800c5df0 con 0x7fed800c7e40 2023-01-24T03:31:55.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.718+0000 7fed6e7fc700 1 -- 172.21.15.112:0/1907936218 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+5357 (secure 0 0 0) 0x7fed70050b00 con 0x7fed800c7e40 2023-01-24T03:31:55.721 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:31:55.722 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:31:53.596607+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4288},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":0,"incarnation":11,"state":"up:replay","state_seq":1,"addr":"172.21.15.142:6834/869267413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":869267413},{"type":"v1","addr":"172.21.15.142:6834","nonce":869267413}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:31:55.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.720+0000 7fed887e7700 1 -- 172.21.15.112:0/1907936218 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fed600611c0 msgr2=0x7fed60063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:55.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.720+0000 7fed887e7700 1 --2- 172.21.15.112:0/1907936218 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fed600611c0 0x7fed60063670 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fed80089ca0 tx=0x7fed7c016000 comp rx=0 tx=0).stop 2023-01-24T03:31:55.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.720+0000 7fed887e7700 1 -- 172.21.15.112:0/1907936218 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed800c7e40 msgr2=0x7fed800c2ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:55.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.720+0000 7fed887e7700 1 --2- 172.21.15.112:0/1907936218 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed800c7e40 0x7fed800c2ff0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fed7000e960 tx=0x7fed70009060 comp rx=0 tx=0).stop 2023-01-24T03:31:55.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.720+0000 7fed887e7700 1 -- 172.21.15.112:0/1907936218 shutdown_connections 2023-01-24T03:31:55.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.720+0000 7fed887e7700 1 --2- 172.21.15.112:0/1907936218 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fed600611c0 0x7fed60063670 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:55.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.720+0000 7fed887e7700 1 --2- 172.21.15.112:0/1907936218 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fed80155da0 0x7fed800c3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:55.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.720+0000 7fed887e7700 1 --2- 172.21.15.112:0/1907936218 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed800aef20 0x7fed800c1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:55.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.720+0000 7fed887e7700 1 --2- 172.21.15.112:0/1907936218 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed800c7e40 0x7fed800c2ff0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:55.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.720+0000 7fed887e7700 1 -- 172.21.15.112:0/1907936218 >> 172.21.15.112:0/1907936218 conn(0x7fed8005e2c0 msgr2=0x7fed80156f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:31:55.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.720+0000 7fed887e7700 1 -- 172.21.15.112:0/1907936218 shutdown_connections 2023-01-24T03:31:55.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:55.720+0000 7fed887e7700 1 -- 172.21.15.112:0/1907936218 wait complete. 2023-01-24T03:31:55.726 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 11 2023-01-24T03:31:55.736 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 11, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-01-24T03:30:20.971710+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 11, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4288": { "addr": "172.21.15.142:6834/869267413", "addrs": { "addrvec": [ { "addr": "172.21.15.142:6832", "nonce": 869267413, "type": "v2" }, { "addr": "172.21.15.142:6834", "nonce": 869267413, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4288, "incarnation": 11, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 27, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-01-24T03:31:53.596607+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4288 } } } ], "standbys": [ { "addr": "172.21.15.142:6837/2181345869", "addrs": { "addrvec": [ { "addr": "172.21.15.142:6836", "nonce": 2181345869, "type": "v2" }, { "addr": "172.21.15.142:6837", "nonce": 2181345869, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6835/330343927", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6834", "nonce": 330343927, "type": "v2" }, { "addr": "172.21.15.112:6835", "nonce": 330343927, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6837/12176817", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6836", "nonce": 12176817, "type": "v2" }, { "addr": "172.21.15.112:6837", "nonce": 12176817, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4276, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6839/1110983841", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6838", "nonce": 1110983841, "type": "v2" }, { "addr": "172.21.15.112:6839", "nonce": 1110983841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4281, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-01-24T03:31:57.737 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:31:57.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.858+0000 7fc7550c6700 1 Processor -- start 2023-01-24T03:31:57.862 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.858+0000 7fc7550c6700 1 -- start start 2023-01-24T03:31:57.864 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.859+0000 7fc7550c6700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fc7500c7e40 0x7fc7500b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:57.864 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.859+0000 7fc7550c6700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc750156240 con 0x7fc7500ca3e0 2023-01-24T03:31:57.864 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.859+0000 7fc7550c6700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7500b1170 con 0x7fc750155da0 2023-01-24T03:31:57.864 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.859+0000 7fc7550c6700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7500b1450 con 0x7fc7500c7e40 2023-01-24T03:31:57.865 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.859+0000 7fc74ed9d700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fc7500c7e40 0x7fc7500b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:57.865 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.859+0000 7fc74ed9d700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fc7500c7e40 0x7fc7500b0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51612/0 (socket says 172.21.15.112:51612) 2023-01-24T03:31:57.865 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.859+0000 7fc74ed9d700 1 -- 172.21.15.112:0/4050498050 learned_addr learned my addr 172.21.15.112:0/4050498050 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:31:57.866 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.859+0000 7fc74ed9d700 1 -- 172.21.15.112:0/4050498050 >> v1:172.21.15.142:6789/0 conn(0x7fc750155da0 legacy=0x7fc7500ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:31:57.866 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.859+0000 7fc74ed9d700 1 -- 172.21.15.112:0/4050498050 >> v1:172.21.15.112:6789/0 conn(0x7fc7500ca3e0 legacy=0x7fc7500ca7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:57.866 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.859+0000 7fc74ed9d700 1 -- 172.21.15.112:0/4050498050 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7500b1730 con 0x7fc7500c7e40 2023-01-24T03:31:57.866 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.860+0000 7fc74ed9d700 1 --2- 172.21.15.112:0/4050498050 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7500c7e40 0x7fc7500b0aa0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc740009f20 tx=0x7fc740009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=513ec42e1a89993b server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:31:57.866 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.860+0000 7fc74dd9b700 1 -- 172.21.15.112:0/4050498050 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74000e070 con 0x7fc7500c7e40 2023-01-24T03:31:57.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.860+0000 7fc74dd9b700 1 -- 172.21.15.112:0/4050498050 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc74000ba40 con 0x7fc7500c7e40 2023-01-24T03:31:57.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.860+0000 7fc74dd9b700 1 -- 172.21.15.112:0/4050498050 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7400092e0 con 0x7fc7500c7e40 2023-01-24T03:31:57.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.860+0000 7fc7550c6700 1 -- 172.21.15.112:0/4050498050 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7500c7e40 msgr2=0x7fc7500b0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:57.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.860+0000 7fc7550c6700 1 --2- 172.21.15.112:0/4050498050 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7500c7e40 0x7fc7500b0aa0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc740009f20 tx=0x7fc740009840 comp rx=0 tx=0).stop 2023-01-24T03:31:57.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.860+0000 7fc7550c6700 1 -- 172.21.15.112:0/4050498050 shutdown_connections 2023-01-24T03:31:57.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.860+0000 7fc7550c6700 1 --2- 172.21.15.112:0/4050498050 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7500c7e40 0x7fc7500b0aa0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:57.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.860+0000 7fc7550c6700 1 -- 172.21.15.112:0/4050498050 >> 172.21.15.112:0/4050498050 conn(0x7fc75005e2c0 msgr2=0x7fc7500cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:31:57.869 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.860+0000 7fc7550c6700 1 -- 172.21.15.112:0/4050498050 shutdown_connections 2023-01-24T03:31:57.869 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.860+0000 7fc7550c6700 1 -- 172.21.15.112:0/4050498050 wait complete. 2023-01-24T03:31:57.871 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.862+0000 7fc7550c6700 1 Processor -- start 2023-01-24T03:31:57.872 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.862+0000 7fc7550c6700 1 -- start start 2023-01-24T03:31:57.872 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.862+0000 7fc7550c6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7500c7e40 0x7fc750152660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:57.872 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.862+0000 7fc7550c6700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc7500ca3e0 0x7fc75014fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:57.872 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.862+0000 7fc7550c6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc750155da0 0x7fc7500bfcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:57.872 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.862+0000 7fc7550c6700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc750150e00 con 0x7fc7500c7e40 2023-01-24T03:31:57.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.862+0000 7fc7550c6700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7501655d0 con 0x7fc7500ca3e0 2023-01-24T03:31:57.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.862+0000 7fc7550c6700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7500bf840 con 0x7fc750155da0 2023-01-24T03:31:57.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.862+0000 7fc74ed9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7500c7e40 0x7fc750152660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:57.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.862+0000 7fc74ed9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7500c7e40 0x7fc750152660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51618/0 (socket says 172.21.15.112:51618) 2023-01-24T03:31:57.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.862+0000 7fc74ed9d700 1 -- 172.21.15.112:0/162696014 learned_addr learned my addr 172.21.15.112:0/162696014 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:31:57.874 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.863+0000 7fc74ed9d700 1 -- 172.21.15.112:0/162696014 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc750155da0 msgr2=0x7fc7500bfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:57.874 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.863+0000 7fc74f59e700 1 --2- 172.21.15.112:0/162696014 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc750155da0 0x7fc7500bfcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:57.874 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.863+0000 7fc74e59c700 1 --2- 172.21.15.112:0/162696014 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc7500ca3e0 0x7fc75014fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:57.874 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.863+0000 7fc74ed9d700 1 --2- 172.21.15.112:0/162696014 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc750155da0 0x7fc7500bfcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:57.874 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.863+0000 7fc74ed9d700 1 -- 172.21.15.112:0/162696014 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc7500ca3e0 msgr2=0x7fc75014fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:57.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.863+0000 7fc74ed9d700 1 --2- 172.21.15.112:0/162696014 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc7500ca3e0 0x7fc75014fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:57.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.863+0000 7fc74ed9d700 1 -- 172.21.15.112:0/162696014 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc740009ae0 con 0x7fc7500c7e40 2023-01-24T03:31:57.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.863+0000 7fc74ed9d700 1 --2- 172.21.15.112:0/162696014 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7500c7e40 0x7fc750152660 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc74000b6c0 tx=0x7fc740017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:31:57.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.863+0000 7fc746ffd700 1 -- 172.21.15.112:0/162696014 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc740012950 con 0x7fc7500c7e40 2023-01-24T03:31:57.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.863+0000 7fc746ffd700 1 -- 172.21.15.112:0/162696014 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc740012ab0 con 0x7fc7500c7e40 2023-01-24T03:31:57.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.863+0000 7fc7550c6700 1 -- 172.21.15.112:0/162696014 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7501657e0 con 0x7fc7500c7e40 2023-01-24T03:31:57.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.863+0000 7fc746ffd700 1 -- 172.21.15.112:0/162696014 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc740012e50 con 0x7fc7500c7e40 2023-01-24T03:31:57.877 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.863+0000 7fc7550c6700 1 -- 172.21.15.112:0/162696014 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc750165da0 con 0x7fc7500c7e40 2023-01-24T03:31:57.877 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.864+0000 7fc7550c6700 1 -- 172.21.15.112:0/162696014 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7500bfb20 con 0x7fc7500c7e40 2023-01-24T03:31:57.877 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.864+0000 7fc746ffd700 1 -- 172.21.15.112:0/162696014 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fc74002c070 con 0x7fc7500c7e40 2023-01-24T03:31:57.877 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.865+0000 7fc746ffd700 1 --2- 172.21.15.112:0/162696014 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc7340611c0 0x7fc734063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:31:57.877 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.865+0000 7fc74e59c700 1 --2- 172.21.15.112:0/162696014 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc7340611c0 0x7fc734063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:31:57.878 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.865+0000 7fc746ffd700 1 -- 172.21.15.112:0/162696014 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6953+0+0 (secure 0 0 0) 0x7fc74007ef70 con 0x7fc7500c7e40 2023-01-24T03:31:57.878 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.865+0000 7fc74e59c700 1 --2- 172.21.15.112:0/162696014 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc7340611c0 0x7fc734063670 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc738006fd0 tx=0x7fc738008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:31:57.878 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:57.869+0000 7fc746ffd700 1 -- 172.21.15.112:0/162696014 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fc7400af1d0 con 0x7fc7500c7e40 2023-01-24T03:31:58.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:58.087+0000 7fc7550c6700 1 -- 172.21.15.112:0/162696014 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7500c5df0 con 0x7fc7500c7e40 2023-01-24T03:31:58.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:58.087+0000 7fc746ffd700 1 -- 172.21.15.112:0/162696014 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+5357 (secure 0 0 0) 0x7fc74004ed20 con 0x7fc7500c7e40 2023-01-24T03:31:58.091 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:31:58.091 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:31:53.596607+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4288},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":0,"incarnation":11,"state":"up:replay","state_seq":1,"addr":"172.21.15.142:6834/869267413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":869267413},{"type":"v1","addr":"172.21.15.142:6834","nonce":869267413}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:31:58.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:58.089+0000 7fc7550c6700 1 -- 172.21.15.112:0/162696014 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc7340611c0 msgr2=0x7fc734063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:58.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:58.089+0000 7fc7550c6700 1 --2- 172.21.15.112:0/162696014 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc7340611c0 0x7fc734063670 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc738006fd0 tx=0x7fc738008040 comp rx=0 tx=0).stop 2023-01-24T03:31:58.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:58.089+0000 7fc7550c6700 1 -- 172.21.15.112:0/162696014 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7500c7e40 msgr2=0x7fc750152660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:31:58.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:58.089+0000 7fc7550c6700 1 --2- 172.21.15.112:0/162696014 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7500c7e40 0x7fc750152660 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc74000b6c0 tx=0x7fc740017000 comp rx=0 tx=0).stop 2023-01-24T03:31:58.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:58.089+0000 7fc7550c6700 1 -- 172.21.15.112:0/162696014 shutdown_connections 2023-01-24T03:31:58.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:58.089+0000 7fc7550c6700 1 --2- 172.21.15.112:0/162696014 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc7340611c0 0x7fc734063670 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:58.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:58.089+0000 7fc7550c6700 1 --2- 172.21.15.112:0/162696014 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7500c7e40 0x7fc750152660 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:58.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:58.089+0000 7fc7550c6700 1 --2- 172.21.15.112:0/162696014 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc750155da0 0x7fc7500bfcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:58.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:58.089+0000 7fc7550c6700 1 --2- 172.21.15.112:0/162696014 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc7500ca3e0 0x7fc75014fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:31:58.095 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:58.089+0000 7fc7550c6700 1 -- 172.21.15.112:0/162696014 >> 172.21.15.112:0/162696014 conn(0x7fc75005e2c0 msgr2=0x7fc7500ca7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:31:58.095 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:58.090+0000 7fc7550c6700 1 -- 172.21.15.112:0/162696014 shutdown_connections 2023-01-24T03:31:58.095 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:31:58.090+0000 7fc7550c6700 1 -- 172.21.15.112:0/162696014 wait complete. 2023-01-24T03:31:58.095 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 11 2023-01-24T03:31:58.104 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:31:58.999 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:31:59.003 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:32:00.105 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:32:00.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.229+0000 7fa323c35700 1 Processor -- start 2023-01-24T03:32:00.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.230+0000 7fa323c35700 1 -- start start 2023-01-24T03:32:00.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.230+0000 7fa323c35700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa31c06a890 0x7fa31c0c6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:00.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.230+0000 7fa323c35700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa31c06ac60 0x7fa31c06b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:00.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.230+0000 7fa323c35700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa31c0c2010 con 0x7fa31c0c24a0 2023-01-24T03:32:00.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.230+0000 7fa323c35700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa31c0c5e10 con 0x7fa31c06ac60 2023-01-24T03:32:00.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.230+0000 7fa323c35700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa31c0c4fe0 con 0x7fa31c06a890 2023-01-24T03:32:00.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.230+0000 7fa3221d2700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fa31c0c24a0 0x7fa31c151f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:60854/0 (socket says 172.21.15.112:60854) 2023-01-24T03:32:00.238 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.230+0000 7fa3221d2700 1 -- 172.21.15.112:0/891074356 learned_addr learned my addr 172.21.15.112:0/891074356 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:32:00.238 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.231+0000 7fa3219d1700 1 --2- 172.21.15.112:0/891074356 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa31c06a890 0x7fa31c0c6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:00.238 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.231+0000 7fa3211d0700 1 --2- 172.21.15.112:0/891074356 >> v2:172.21.15.142:3300/0 conn(0x7fa31c06ac60 0x7fa31c06b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:00.238 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.231+0000 7fa3209cf700 1 -- 172.21.15.112:0/891074356 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 954199770 0 0) 0x7fa31c0c2010 con 0x7fa31c0c24a0 2023-01-24T03:32:00.238 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.231+0000 7fa3209cf700 1 -- 172.21.15.112:0/891074356 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3080033f0 con 0x7fa31c0c24a0 2023-01-24T03:32:00.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.231+0000 7fa3209cf700 1 -- 172.21.15.112:0/891074356 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1677040126 0 0) 0x7fa3080033f0 con 0x7fa31c0c24a0 2023-01-24T03:32:00.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.231+0000 7fa3209cf700 1 -- 172.21.15.112:0/891074356 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa31c06a890 msgr2=0x7fa31c0c6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:32:00.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.231+0000 7fa3209cf700 1 --2- 172.21.15.112:0/891074356 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa31c06a890 0x7fa31c0c6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:00.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.231+0000 7fa3209cf700 1 -- 172.21.15.112:0/891074356 >> v2:172.21.15.142:3300/0 conn(0x7fa31c06ac60 msgr2=0x7fa31c06b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:32:00.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.231+0000 7fa3209cf700 1 --2- 172.21.15.112:0/891074356 >> v2:172.21.15.142:3300/0 conn(0x7fa31c06ac60 0x7fa31c06b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:00.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.231+0000 7fa3209cf700 1 -- 172.21.15.112:0/891074356 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa31c0c4a60 con 0x7fa31c0c24a0 2023-01-24T03:32:00.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.231+0000 7fa3209cf700 1 -- 172.21.15.112:0/891074356 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7fa314001470 con 0x7fa31c0c24a0 2023-01-24T03:32:00.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.231+0000 7fa3209cf700 1 -- 172.21.15.112:0/891074356 >> v1:172.21.15.112:6789/0 conn(0x7fa31c0c24a0 legacy=0x7fa31c151f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:00.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.231+0000 7fa3209cf700 1 --2- 172.21.15.112:0/891074356 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa3080038e0 0x7fa308003cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:00.242 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.231+0000 7fa3209cf700 1 --2- 172.21.15.112:0/891074356 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa308004690 0x7fa308004b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:00.242 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.231+0000 7fa3209cf700 1 --2- 172.21.15.112:0/891074356 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa308005040 0x7fa30800b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:00.242 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.231+0000 7fa3209cf700 1 -- 172.21.15.112:0/891074356 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31c0c5e10 con 0x7fa3080038e0 2023-01-24T03:32:00.242 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.231+0000 7fa3209cf700 1 -- 172.21.15.112:0/891074356 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31c0c4fe0 con 0x7fa308004690 2023-01-24T03:32:00.242 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.231+0000 7fa3209cf700 1 -- 172.21.15.112:0/891074356 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa30800be20 con 0x7fa308005040 2023-01-24T03:32:00.243 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.231+0000 7fa3211d0700 1 --2- 172.21.15.112:0/891074356 >> v2:172.21.15.142:3300/0 conn(0x7fa31c06ac60 0x7fa31c06b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:32:00.243 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.231+0000 7fa3219d1700 1 --2- 172.21.15.112:0/891074356 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa31c06a890 0x7fa31c0c6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:32:00.243 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.231+0000 7fa3219d1700 1 --2- 172.21.15.112:0/891074356 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa3080038e0 0x7fa308003cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:00.243 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.231+0000 7fa3221d2700 1 --2- 172.21.15.112:0/891074356 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa308005040 0x7fa30800b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:00.243 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.231+0000 7fa3211d0700 1 --2- 172.21.15.112:0/891074356 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa308004690 0x7fa308004b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:00.244 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.231+0000 7fa3219d1700 1 -- 172.21.15.112:0/891074356 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa308005040 msgr2=0x7fa30800b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:32:00.244 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.231+0000 7fa3219d1700 1 --2- 172.21.15.112:0/891074356 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa308005040 0x7fa30800b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:00.244 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.232+0000 7fa3219d1700 1 -- 172.21.15.112:0/891074356 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa308004690 msgr2=0x7fa308004b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:32:00.244 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.232+0000 7fa3219d1700 1 --2- 172.21.15.112:0/891074356 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa308004690 0x7fa308004b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:00.244 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.232+0000 7fa3219d1700 1 -- 172.21.15.112:0/891074356 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3080033f0 con 0x7fa3080038e0 2023-01-24T03:32:00.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.232+0000 7fa3219d1700 1 --2- 172.21.15.112:0/891074356 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa3080038e0 0x7fa308003cf0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa31c06c880 tx=0x7fa30c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b39bda46abe69d1 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:00.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.232+0000 7fa3209cf700 1 -- 172.21.15.112:0/891074356 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30c00eaa0 con 0x7fa3080038e0 2023-01-24T03:32:00.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.232+0000 7fa3209cf700 1 -- 172.21.15.112:0/891074356 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa30c00f040 con 0x7fa3080038e0 2023-01-24T03:32:00.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.232+0000 7fa3209cf700 1 -- 172.21.15.112:0/891074356 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30c00ee30 con 0x7fa3080038e0 2023-01-24T03:32:00.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.232+0000 7fa323c35700 1 -- 172.21.15.112:0/891074356 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa3080038e0 msgr2=0x7fa308003cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:00.246 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.232+0000 7fa323c35700 1 --2- 172.21.15.112:0/891074356 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa3080038e0 0x7fa308003cf0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa31c06c880 tx=0x7fa30c00b040 comp rx=0 tx=0).stop 2023-01-24T03:32:00.246 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.232+0000 7fa3221d2700 1 -- 172.21.15.112:0/891074356 reap_dead start 2023-01-24T03:32:00.246 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.232+0000 7fa323c35700 1 -- 172.21.15.112:0/891074356 shutdown_connections 2023-01-24T03:32:00.246 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.232+0000 7fa323c35700 1 -- 172.21.15.112:0/891074356 >> 172.21.15.112:0/891074356 conn(0x7fa31c066e10 msgr2=0x7fa31c067210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:32:00.246 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.232+0000 7fa323c35700 1 -- 172.21.15.112:0/891074356 shutdown_connections 2023-01-24T03:32:00.247 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.232+0000 7fa323c35700 1 -- 172.21.15.112:0/891074356 wait complete. 2023-01-24T03:32:00.247 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.233+0000 7fa323c35700 1 Processor -- start 2023-01-24T03:32:00.247 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.234+0000 7fa323c35700 1 -- start start 2023-01-24T03:32:00.249 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.234+0000 7fa323c35700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa3080038e0 0x7fa31c0b7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:00.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.234+0000 7fa323c35700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa31c1653a0 0x7fa31c0c7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:00.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.234+0000 7fa323c35700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa31c14fa90 0x7fa31c14fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:00.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.234+0000 7fa323c35700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31c0b6db0 con 0x7fa31c14fa90 2023-01-24T03:32:00.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.234+0000 7fa323c35700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31c0b79a0 con 0x7fa31c1653a0 2023-01-24T03:32:00.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.234+0000 7fa323c35700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa31c0be340 con 0x7fa3080038e0 2023-01-24T03:32:00.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.234+0000 7fa3221d2700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa31c14fa90 0x7fa31c14fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:00.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.234+0000 7fa3221d2700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa31c14fa90 0x7fa31c14fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51648/0 (socket says 172.21.15.112:51648) 2023-01-24T03:32:00.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.234+0000 7fa3221d2700 1 -- 172.21.15.112:0/2859331113 learned_addr learned my addr 172.21.15.112:0/2859331113 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:32:00.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.234+0000 7fa3211d0700 1 --2- 172.21.15.112:0/2859331113 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa31c1653a0 0x7fa31c0c7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:00.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.234+0000 7fa3219d1700 1 --2- 172.21.15.112:0/2859331113 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa3080038e0 0x7fa31c0b7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:00.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.235+0000 7fa3221d2700 1 -- 172.21.15.112:0/2859331113 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa3080038e0 msgr2=0x7fa31c0b7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:00.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.235+0000 7fa3221d2700 1 --2- 172.21.15.112:0/2859331113 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa3080038e0 0x7fa31c0b7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:00.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.235+0000 7fa3221d2700 1 -- 172.21.15.112:0/2859331113 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa31c1653a0 msgr2=0x7fa31c0c7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:00.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.235+0000 7fa3221d2700 1 --2- 172.21.15.112:0/2859331113 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa31c1653a0 0x7fa31c0c7fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:00.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.235+0000 7fa3221d2700 1 -- 172.21.15.112:0/2859331113 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa31c14f600 con 0x7fa31c14fa90 2023-01-24T03:32:00.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.235+0000 7fa3221d2700 1 --2- 172.21.15.112:0/2859331113 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa31c14fa90 0x7fa31c14fec0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa31c0c4080 tx=0x7fa31c06d260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:00.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.235+0000 7fa319ffb700 1 -- 172.21.15.112:0/2859331113 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa314011590 con 0x7fa31c14fa90 2023-01-24T03:32:00.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.235+0000 7fa319ffb700 1 -- 172.21.15.112:0/2859331113 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3140116f0 con 0x7fa31c14fa90 2023-01-24T03:32:00.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.235+0000 7fa323c35700 1 -- 172.21.15.112:0/2859331113 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa31c150400 con 0x7fa31c14fa90 2023-01-24T03:32:00.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.235+0000 7fa319ffb700 1 -- 172.21.15.112:0/2859331113 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa314011ab0 con 0x7fa31c14fa90 2023-01-24T03:32:00.254 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.235+0000 7fa323c35700 1 -- 172.21.15.112:0/2859331113 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa31c0be060 con 0x7fa31c14fa90 2023-01-24T03:32:00.254 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.236+0000 7fa323c35700 1 -- 172.21.15.112:0/2859331113 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa31c14f790 con 0x7fa31c14fa90 2023-01-24T03:32:00.254 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.236+0000 7fa319ffb700 1 -- 172.21.15.112:0/2859331113 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fa31401c030 con 0x7fa31c14fa90 2023-01-24T03:32:00.254 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.236+0000 7fa319ffb700 1 --2- 172.21.15.112:0/2859331113 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa2fc0611c0 0x7fa2fc063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:00.254 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.237+0000 7fa319ffb700 1 -- 172.21.15.112:0/2859331113 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6953+0+0 (secure 0 0 0) 0x7fa31407d1a0 con 0x7fa31c14fa90 2023-01-24T03:32:00.255 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.237+0000 7fa3219d1700 1 --2- 172.21.15.112:0/2859331113 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa2fc0611c0 0x7fa2fc063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:00.255 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.237+0000 7fa3219d1700 1 --2- 172.21.15.112:0/2859331113 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa2fc0611c0 0x7fa2fc063670 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa30c007140 tx=0x7fa30c016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:00.255 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.241+0000 7fa319ffb700 1 -- 172.21.15.112:0/2859331113 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fa31404cf50 con 0x7fa31c14fa90 2023-01-24T03:32:00.463 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.460+0000 7fa323c35700 1 -- 172.21.15.112:0/2859331113 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa31c05e3f0 con 0x7fa31c14fa90 2023-01-24T03:32:00.464 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.461+0000 7fa319ffb700 1 -- 172.21.15.112:0/2859331113 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+5361 (secure 0 0 0) 0x7fa31401a070 con 0x7fa31c14fa90 2023-01-24T03:32:00.464 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:32:00.464 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:31:59.598951+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4288},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":0,"incarnation":11,"state":"up:reconnect","state_seq":27,"addr":"172.21.15.142:6834/869267413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":869267413},{"type":"v1","addr":"172.21.15.142:6834","nonce":869267413}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:32:00.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.463+0000 7fa323c35700 1 -- 172.21.15.112:0/2859331113 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa2fc0611c0 msgr2=0x7fa2fc063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:00.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.463+0000 7fa323c35700 1 --2- 172.21.15.112:0/2859331113 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa2fc0611c0 0x7fa2fc063670 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa30c007140 tx=0x7fa30c016000 comp rx=0 tx=0).stop 2023-01-24T03:32:00.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.463+0000 7fa323c35700 1 -- 172.21.15.112:0/2859331113 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa31c14fa90 msgr2=0x7fa31c14fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:00.467 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.463+0000 7fa323c35700 1 --2- 172.21.15.112:0/2859331113 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa31c14fa90 0x7fa31c14fec0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa31c0c4080 tx=0x7fa31c06d260 comp rx=0 tx=0).stop 2023-01-24T03:32:00.467 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.463+0000 7fa323c35700 1 -- 172.21.15.112:0/2859331113 shutdown_connections 2023-01-24T03:32:00.467 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.463+0000 7fa323c35700 1 --2- 172.21.15.112:0/2859331113 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa2fc0611c0 0x7fa2fc063670 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:00.467 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.463+0000 7fa323c35700 1 --2- 172.21.15.112:0/2859331113 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa31c14fa90 0x7fa31c14fec0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:00.467 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.463+0000 7fa323c35700 1 --2- 172.21.15.112:0/2859331113 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa3080038e0 0x7fa31c0b7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:00.468 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.463+0000 7fa323c35700 1 --2- 172.21.15.112:0/2859331113 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa31c1653a0 0x7fa31c0c7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:00.468 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.463+0000 7fa323c35700 1 -- 172.21.15.112:0/2859331113 >> 172.21.15.112:0/2859331113 conn(0x7fa31c066e10 msgr2=0x7fa31c0ba810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:32:00.468 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.463+0000 7fa323c35700 1 -- 172.21.15.112:0/2859331113 shutdown_connections 2023-01-24T03:32:00.468 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:00.463+0000 7fa323c35700 1 -- 172.21.15.112:0/2859331113 wait complete. 2023-01-24T03:32:00.469 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 12 2023-01-24T03:32:00.479 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:32:02.480 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:32:02.605 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.602+0000 7fb732422700 1 Processor -- start 2023-01-24T03:32:02.606 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.602+0000 7fb732422700 1 -- start start 2023-01-24T03:32:02.607 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.603+0000 7fb732422700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb72c0c7e40 0x7fb72c0b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:02.607 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.603+0000 7fb732422700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb72c0ca3e0 0x7fb72c0ca820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:02.607 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.603+0000 7fb732422700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb72c156210 con 0x7fb72c155da0 2023-01-24T03:32:02.608 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.603+0000 7fb732422700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb72c0cad60 con 0x7fb72c0ca3e0 2023-01-24T03:32:02.608 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.603+0000 7fb732422700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c0b1170 con 0x7fb72c0c7e40 2023-01-24T03:32:02.608 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.603+0000 7fb72b7fe700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fb72c155da0 0x7fb72c0ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:60874/0 (socket says 172.21.15.112:60874) 2023-01-24T03:32:02.608 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.603+0000 7fb72b7fe700 1 -- 172.21.15.112:0/3305107225 learned_addr learned my addr 172.21.15.112:0/3305107225 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:32:02.608 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.603+0000 7fb72bfff700 1 --2- 172.21.15.112:0/3305107225 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb72c0c7e40 0x7fb72c0b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:02.609 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.603+0000 7fb7309bf700 1 --2- 172.21.15.112:0/3305107225 >> v2:172.21.15.142:3300/0 conn(0x7fb72c0ca3e0 0x7fb72c0ca820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:02.609 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.604+0000 7fb7309bf700 1 -- 172.21.15.112:0/3305107225 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb72c0c7e40 msgr2=0x7fb72c0b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:32:02.610 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.604+0000 7fb7309bf700 1 --2- 172.21.15.112:0/3305107225 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb72c0c7e40 0x7fb72c0b0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:02.610 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.604+0000 7fb7309bf700 1 -- 172.21.15.112:0/3305107225 >> v1:172.21.15.112:6789/0 conn(0x7fb72c155da0 legacy=0x7fb72c0ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:02.610 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.604+0000 7fb7309bf700 1 -- 172.21.15.112:0/3305107225 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72c0b1450 con 0x7fb72c0ca3e0 2023-01-24T03:32:02.610 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.604+0000 7fb72bfff700 1 --2- 172.21.15.112:0/3305107225 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb72c0c7e40 0x7fb72c0b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:32:02.611 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.604+0000 7fb7309bf700 1 --2- 172.21.15.112:0/3305107225 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb72c0ca3e0 0x7fb72c0ca820 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb724009e60 tx=0x7fb724009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9ed6be21cd890d2 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:02.614 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.606+0000 7fb72affd700 1 -- 172.21.15.112:0/3305107225 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72400e070 con 0x7fb72c0ca3e0 2023-01-24T03:32:02.614 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.606+0000 7fb72affd700 1 -- 172.21.15.112:0/3305107225 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb72400bb60 con 0x7fb72c0ca3e0 2023-01-24T03:32:02.614 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.606+0000 7fb72affd700 1 -- 172.21.15.112:0/3305107225 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb724012430 con 0x7fb72c0ca3e0 2023-01-24T03:32:02.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.607+0000 7fb732422700 1 -- 172.21.15.112:0/3305107225 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb72c0ca3e0 msgr2=0x7fb72c0ca820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:02.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.607+0000 7fb732422700 1 --2- 172.21.15.112:0/3305107225 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb72c0ca3e0 0x7fb72c0ca820 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb724009e60 tx=0x7fb724009780 comp rx=0 tx=0).stop 2023-01-24T03:32:02.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.607+0000 7fb732422700 1 -- 172.21.15.112:0/3305107225 shutdown_connections 2023-01-24T03:32:02.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.607+0000 7fb732422700 1 --2- 172.21.15.112:0/3305107225 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb72c0ca3e0 0x7fb72c0ca820 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:02.616 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.607+0000 7fb732422700 1 --2- 172.21.15.112:0/3305107225 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb72c0c7e40 0x7fb72c0b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:02.616 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.607+0000 7fb732422700 1 -- 172.21.15.112:0/3305107225 >> 172.21.15.112:0/3305107225 conn(0x7fb72c05e2c0 msgr2=0x7fb72c0cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:32:02.616 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.607+0000 7fb732422700 1 -- 172.21.15.112:0/3305107225 shutdown_connections 2023-01-24T03:32:02.616 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.607+0000 7fb732422700 1 -- 172.21.15.112:0/3305107225 wait complete. 2023-01-24T03:32:02.616 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.608+0000 7fb732422700 1 Processor -- start 2023-01-24T03:32:02.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.608+0000 7fb732422700 1 -- start start 2023-01-24T03:32:02.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.609+0000 7fb732422700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb72c0c7e40 0x7fb72c1571b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:02.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.609+0000 7fb732422700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb72c0ca3e0 0x7fb72c0c5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:02.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.609+0000 7fb732422700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb72c155da0 0x7fb72c165400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:02.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.609+0000 7fb732422700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c0c20f0 con 0x7fb72c155da0 2023-01-24T03:32:02.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.609+0000 7fb732422700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c0c1ac0 con 0x7fb72c0ca3e0 2023-01-24T03:32:02.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.609+0000 7fb732422700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c0c1120 con 0x7fb72c0c7e40 2023-01-24T03:32:02.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.609+0000 7fb7309bf700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb72c155da0 0x7fb72c165400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:02.619 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.609+0000 7fb7309bf700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb72c155da0 0x7fb72c165400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51662/0 (socket says 172.21.15.112:51662) 2023-01-24T03:32:02.619 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.609+0000 7fb7309bf700 1 -- 172.21.15.112:0/694922777 learned_addr learned my addr 172.21.15.112:0/694922777 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:32:02.619 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.609+0000 7fb72bfff700 1 --2- 172.21.15.112:0/694922777 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb72c0c7e40 0x7fb72c1571b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:02.619 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.609+0000 7fb72b7fe700 1 --2- 172.21.15.112:0/694922777 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb72c0ca3e0 0x7fb72c0c5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:02.620 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.609+0000 7fb7309bf700 1 -- 172.21.15.112:0/694922777 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb72c0c7e40 msgr2=0x7fb72c1571b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:02.620 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.609+0000 7fb7309bf700 1 --2- 172.21.15.112:0/694922777 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb72c0c7e40 0x7fb72c1571b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:02.620 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.609+0000 7fb7309bf700 1 -- 172.21.15.112:0/694922777 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb72c0ca3e0 msgr2=0x7fb72c0c5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:02.620 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.609+0000 7fb7309bf700 1 --2- 172.21.15.112:0/694922777 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb72c0ca3e0 0x7fb72c0c5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:02.620 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.609+0000 7fb7309bf700 1 -- 172.21.15.112:0/694922777 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb71c0092a0 con 0x7fb72c155da0 2023-01-24T03:32:02.621 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.610+0000 7fb7309bf700 1 --2- 172.21.15.112:0/694922777 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb72c155da0 0x7fb72c165400 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb72c089340 tx=0x7fb724019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:02.621 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.610+0000 7fb713fff700 1 -- 172.21.15.112:0/694922777 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb724012ea0 con 0x7fb72c155da0 2023-01-24T03:32:02.621 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.610+0000 7fb713fff700 1 -- 172.21.15.112:0/694922777 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7240032f0 con 0x7fb72c155da0 2023-01-24T03:32:02.621 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.610+0000 7fb732422700 1 -- 172.21.15.112:0/694922777 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb724009a20 con 0x7fb72c155da0 2023-01-24T03:32:02.621 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.610+0000 7fb713fff700 1 -- 172.21.15.112:0/694922777 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7240276a0 con 0x7fb72c155da0 2023-01-24T03:32:02.622 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.610+0000 7fb732422700 1 -- 172.21.15.112:0/694922777 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb72c0c16e0 con 0x7fb72c155da0 2023-01-24T03:32:02.622 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.611+0000 7fb732422700 1 -- 172.21.15.112:0/694922777 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb72c0c22a0 con 0x7fb72c155da0 2023-01-24T03:32:02.622 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.611+0000 7fb713fff700 1 -- 172.21.15.112:0/694922777 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fb72401d420 con 0x7fb72c155da0 2023-01-24T03:32:02.622 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.611+0000 7fb713fff700 1 --2- 172.21.15.112:0/694922777 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb70c0611f0 0x7fb70c0636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:02.622 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.611+0000 7fb713fff700 1 -- 172.21.15.112:0/694922777 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6953+0+0 (secure 0 0 0) 0x7fb724080240 con 0x7fb72c155da0 2023-01-24T03:32:02.623 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.611+0000 7fb72bfff700 1 --2- 172.21.15.112:0/694922777 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb70c0611f0 0x7fb70c0636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:02.623 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.612+0000 7fb72bfff700 1 --2- 172.21.15.112:0/694922777 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb70c0611f0 0x7fb70c0636a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb71c016fd0 tx=0x7fb71c018040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:02.624 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.615+0000 7fb713fff700 1 -- 172.21.15.112:0/694922777 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fb72404ff90 con 0x7fb72c155da0 2023-01-24T03:32:02.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.835+0000 7fb732422700 1 -- 172.21.15.112:0/694922777 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb72c068360 con 0x7fb72c155da0 2023-01-24T03:32:02.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.835+0000 7fb713fff700 1 -- 172.21.15.112:0/694922777 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+5361 (secure 0 0 0) 0x7fb7240535b0 con 0x7fb72c155da0 2023-01-24T03:32:02.839 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:32:02.839 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:31:59.598951+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4288},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":0,"incarnation":11,"state":"up:reconnect","state_seq":27,"addr":"172.21.15.142:6834/869267413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":869267413},{"type":"v1","addr":"172.21.15.142:6834","nonce":869267413}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:32:02.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.837+0000 7fb732422700 1 -- 172.21.15.112:0/694922777 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb70c0611f0 msgr2=0x7fb70c0636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:02.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.837+0000 7fb732422700 1 --2- 172.21.15.112:0/694922777 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb70c0611f0 0x7fb70c0636a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb71c016fd0 tx=0x7fb71c018040 comp rx=0 tx=0).stop 2023-01-24T03:32:02.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.837+0000 7fb732422700 1 -- 172.21.15.112:0/694922777 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb72c155da0 msgr2=0x7fb72c165400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:02.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.837+0000 7fb732422700 1 --2- 172.21.15.112:0/694922777 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb72c155da0 0x7fb72c165400 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb72c089340 tx=0x7fb724019000 comp rx=0 tx=0).stop 2023-01-24T03:32:02.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.837+0000 7fb732422700 1 -- 172.21.15.112:0/694922777 shutdown_connections 2023-01-24T03:32:02.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.837+0000 7fb732422700 1 --2- 172.21.15.112:0/694922777 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb70c0611f0 0x7fb70c0636a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:02.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.837+0000 7fb732422700 1 --2- 172.21.15.112:0/694922777 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb72c155da0 0x7fb72c165400 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:02.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.837+0000 7fb732422700 1 --2- 172.21.15.112:0/694922777 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb72c0c7e40 0x7fb72c1571b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:02.843 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.837+0000 7fb732422700 1 --2- 172.21.15.112:0/694922777 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb72c0ca3e0 0x7fb72c0c5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:02.843 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.837+0000 7fb732422700 1 -- 172.21.15.112:0/694922777 >> 172.21.15.112:0/694922777 conn(0x7fb72c05e2c0 msgr2=0x7fb72c157db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:32:02.843 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.837+0000 7fb732422700 1 -- 172.21.15.112:0/694922777 shutdown_connections 2023-01-24T03:32:02.843 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:02.837+0000 7fb732422700 1 -- 172.21.15.112:0/694922777 wait complete. 2023-01-24T03:32:02.843 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 12 2023-01-24T03:32:02.853 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:32:04.856 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:32:04.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.978+0000 7f77b0a7b700 1 Processor -- start 2023-01-24T03:32:04.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.978+0000 7f77b0a7b700 1 -- start start 2023-01-24T03:32:04.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.979+0000 7f77b0a7b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f77ac0b0aa0 0x7f77ac155da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:04.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.979+0000 7f77b0a7b700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f77ac0ca3e0 0x7f77ac0ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:04.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.979+0000 7f77b0a7b700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77ac0b0f40 con 0x7f77ac0c7e40 2023-01-24T03:32:04.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.979+0000 7f77b0a7b700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77ac0b1170 con 0x7f77ac0ca3e0 2023-01-24T03:32:04.985 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.979+0000 7f77b0a7b700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77ac0b1450 con 0x7f77ac0b0aa0 2023-01-24T03:32:04.985 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.979+0000 7f77aad9d700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f77ac0ca3e0 0x7f77ac0ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:04.985 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.979+0000 7f77aad9d700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f77ac0ca3e0 0x7f77ac0ca7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51670/0 (socket says 172.21.15.112:51670) 2023-01-24T03:32:04.985 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.979+0000 7f77aad9d700 1 -- 172.21.15.112:0/127334417 learned_addr learned my addr 172.21.15.112:0/127334417 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:32:04.985 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.979+0000 7f77aa59c700 1 --1- 172.21.15.112:0/127334417 >> v1:172.21.15.112:6789/0 conn(0x7f77ac0c7e40 0x7f77ac0ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:60886/0 (socket says 172.21.15.112:60886) 2023-01-24T03:32:04.986 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.979+0000 7f77aad9d700 1 -- 172.21.15.112:0/127334417 >> v2:172.21.15.142:3300/0 conn(0x7f77ac0b0aa0 msgr2=0x7f77ac155da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:32:04.986 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.979+0000 7f77a9d9b700 1 --2- 172.21.15.112:0/127334417 >> v2:172.21.15.142:3300/0 conn(0x7f77ac0b0aa0 0x7f77ac155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:04.986 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.979+0000 7f77aad9d700 1 --2- 172.21.15.112:0/127334417 >> v2:172.21.15.142:3300/0 conn(0x7f77ac0b0aa0 0x7f77ac155da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:04.987 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.979+0000 7f77aad9d700 1 -- 172.21.15.112:0/127334417 >> v1:172.21.15.112:6789/0 conn(0x7f77ac0c7e40 legacy=0x7f77ac0ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:04.987 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.979+0000 7f77aad9d700 1 -- 172.21.15.112:0/127334417 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77ac0b1730 con 0x7f77ac0ca3e0 2023-01-24T03:32:04.987 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.979+0000 7f77aad9d700 1 --2- 172.21.15.112:0/127334417 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f77ac0ca3e0 0x7f77ac0ca7e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f77ac152ea0 tx=0x7f779c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19b70741979ef976 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:04.987 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.979+0000 7f77a959a700 1 -- 172.21.15.112:0/127334417 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f779c011630 con 0x7f77ac0ca3e0 2023-01-24T03:32:04.987 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.979+0000 7f77a959a700 1 -- 172.21.15.112:0/127334417 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f779c0117d0 con 0x7f77ac0ca3e0 2023-01-24T03:32:04.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.980+0000 7f77a959a700 1 -- 172.21.15.112:0/127334417 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f779c011b90 con 0x7f77ac0ca3e0 2023-01-24T03:32:04.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.980+0000 7f77b0a7b700 1 -- 172.21.15.112:0/127334417 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f77ac0ca3e0 msgr2=0x7f77ac0ca7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:04.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.980+0000 7f77b0a7b700 1 --2- 172.21.15.112:0/127334417 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f77ac0ca3e0 0x7f77ac0ca7e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f77ac152ea0 tx=0x7f779c0090f0 comp rx=0 tx=0).stop 2023-01-24T03:32:04.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.980+0000 7f77b0a7b700 1 -- 172.21.15.112:0/127334417 shutdown_connections 2023-01-24T03:32:04.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.980+0000 7f77b0a7b700 1 --2- 172.21.15.112:0/127334417 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f77ac0ca3e0 0x7f77ac0ca7e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:04.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.980+0000 7f77b0a7b700 1 --2- 172.21.15.112:0/127334417 >> v2:172.21.15.142:3300/0 conn(0x7f77ac0b0aa0 0x7f77ac155da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:04.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.980+0000 7f77b0a7b700 1 -- 172.21.15.112:0/127334417 >> 172.21.15.112:0/127334417 conn(0x7f77ac05e2c0 msgr2=0x7f77ac0cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:32:04.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.980+0000 7f77b0a7b700 1 -- 172.21.15.112:0/127334417 shutdown_connections 2023-01-24T03:32:04.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.980+0000 7f77b0a7b700 1 -- 172.21.15.112:0/127334417 wait complete. 2023-01-24T03:32:04.992 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.981+0000 7f77b0a7b700 1 Processor -- start 2023-01-24T03:32:04.992 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.981+0000 7f77b0a7b700 1 -- start start 2023-01-24T03:32:04.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.982+0000 7f77b0a7b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f77ac0b0aa0 0x7f77ac14c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:04.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.982+0000 7f77b0a7b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f77ac0c7e40 0x7f77ac1650d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:04.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.982+0000 7f77b0a7b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f77ac0ca3e0 0x7f77ac154120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:04.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.982+0000 7f77b0a7b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ac14c800 con 0x7f77ac0ca3e0 2023-01-24T03:32:04.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.982+0000 7f77b0a7b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ac169070 con 0x7f77ac0c7e40 2023-01-24T03:32:04.994 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.982+0000 7f77b0a7b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77ac169500 con 0x7f77ac0b0aa0 2023-01-24T03:32:04.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.982+0000 7f77aad9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f77ac0ca3e0 0x7f77ac154120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:04.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.982+0000 7f77aad9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f77ac0ca3e0 0x7f77ac154120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51676/0 (socket says 172.21.15.112:51676) 2023-01-24T03:32:04.996 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.982+0000 7f77aad9d700 1 -- 172.21.15.112:0/919544764 learned_addr learned my addr 172.21.15.112:0/919544764 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:32:04.996 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.982+0000 7f77a9d9b700 1 --2- 172.21.15.112:0/919544764 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f77ac0c7e40 0x7f77ac1650d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:04.996 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.982+0000 7f77aa59c700 1 --2- 172.21.15.112:0/919544764 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f77ac0b0aa0 0x7f77ac14c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:04.996 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.982+0000 7f77aad9d700 1 -- 172.21.15.112:0/919544764 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f77ac0b0aa0 msgr2=0x7f77ac14c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:04.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.982+0000 7f77aad9d700 1 --2- 172.21.15.112:0/919544764 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f77ac0b0aa0 0x7f77ac14c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:04.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.982+0000 7f77aad9d700 1 -- 172.21.15.112:0/919544764 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f77ac0c7e40 msgr2=0x7f77ac1650d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:04.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.982+0000 7f77aad9d700 1 --2- 172.21.15.112:0/919544764 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f77ac0c7e40 0x7f77ac1650d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:04.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.982+0000 7f77aad9d700 1 -- 172.21.15.112:0/919544764 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f779c009370 con 0x7f77ac0ca3e0 2023-01-24T03:32:04.998 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.982+0000 7f77aad9d700 1 --2- 172.21.15.112:0/919544764 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f77ac0ca3e0 0x7f77ac154120 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f779c002000 tx=0x7f779c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:04.998 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.982+0000 7f779a7fc700 1 -- 172.21.15.112:0/919544764 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f779c011b50 con 0x7f77ac0ca3e0 2023-01-24T03:32:04.998 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.982+0000 7f779a7fc700 1 -- 172.21.15.112:0/919544764 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f779c020070 con 0x7f77ac0ca3e0 2023-01-24T03:32:04.998 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.982+0000 7f77b0a7b700 1 -- 172.21.15.112:0/919544764 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77ac155820 con 0x7f77ac0ca3e0 2023-01-24T03:32:04.998 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.982+0000 7f779a7fc700 1 -- 172.21.15.112:0/919544764 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f779c0276b0 con 0x7f77ac0ca3e0 2023-01-24T03:32:04.999 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.983+0000 7f77b0a7b700 1 -- 172.21.15.112:0/919544764 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77ac0c3110 con 0x7f77ac0ca3e0 2023-01-24T03:32:04.999 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.983+0000 7f77b0a7b700 1 -- 172.21.15.112:0/919544764 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77ac1691e0 con 0x7f77ac0ca3e0 2023-01-24T03:32:04.999 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.984+0000 7f779a7fc700 1 -- 172.21.15.112:0/919544764 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f779c017020 con 0x7f77ac0ca3e0 2023-01-24T03:32:04.999 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.984+0000 7f779a7fc700 1 --2- 172.21.15.112:0/919544764 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f77880611c0 0x7f7788063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:04.999 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.984+0000 7f779a7fc700 1 -- 172.21.15.112:0/919544764 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6953+0+0 (secure 0 0 0) 0x7f779c022070 con 0x7f77ac0ca3e0 2023-01-24T03:32:05.000 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.984+0000 7f77aa59c700 1 --2- 172.21.15.112:0/919544764 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f77880611c0 0x7f7788063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:05.000 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.984+0000 7f77aa59c700 1 --2- 172.21.15.112:0/919544764 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f77880611c0 0x7f7788063670 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f77a4007fd0 tx=0x7f77a4007d30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:05.000 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:04.988+0000 7f779a7fc700 1 -- 172.21.15.112:0/919544764 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f779c04f710 con 0x7f77ac0ca3e0 2023-01-24T03:32:05.212 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:05.209+0000 7f77b0a7b700 1 -- 172.21.15.112:0/919544764 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77ac155600 con 0x7f77ac0ca3e0 2023-01-24T03:32:05.213 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:05.210+0000 7f779a7fc700 1 -- 172.21.15.112:0/919544764 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+5358 (secure 0 0 0) 0x7f779c052d30 con 0x7f77ac0ca3e0 2023-01-24T03:32:05.213 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:32:05.213 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:32:04.610259+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4288},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":0,"incarnation":11,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.142:6834/869267413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":869267413},{"type":"v1","addr":"172.21.15.142:6834","nonce":869267413}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:32:05.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:05.211+0000 7f77b0a7b700 1 -- 172.21.15.112:0/919544764 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f77880611c0 msgr2=0x7f7788063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:05.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:05.211+0000 7f77b0a7b700 1 --2- 172.21.15.112:0/919544764 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f77880611c0 0x7f7788063670 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f77a4007fd0 tx=0x7f77a4007d30 comp rx=0 tx=0).stop 2023-01-24T03:32:05.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:05.211+0000 7f77b0a7b700 1 -- 172.21.15.112:0/919544764 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f77ac0ca3e0 msgr2=0x7f77ac154120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:05.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:05.211+0000 7f77b0a7b700 1 --2- 172.21.15.112:0/919544764 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f77ac0ca3e0 0x7f77ac154120 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f779c002000 tx=0x7f779c016000 comp rx=0 tx=0).stop 2023-01-24T03:32:05.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:05.212+0000 7f77b0a7b700 1 -- 172.21.15.112:0/919544764 shutdown_connections 2023-01-24T03:32:05.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:05.212+0000 7f77b0a7b700 1 --2- 172.21.15.112:0/919544764 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f77880611c0 0x7f7788063670 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:05.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:05.212+0000 7f77b0a7b700 1 --2- 172.21.15.112:0/919544764 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f77ac0ca3e0 0x7f77ac154120 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:05.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:05.212+0000 7f77b0a7b700 1 --2- 172.21.15.112:0/919544764 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f77ac0b0aa0 0x7f77ac14c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:05.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:05.212+0000 7f77b0a7b700 1 --2- 172.21.15.112:0/919544764 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f77ac0c7e40 0x7f77ac1650d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:05.217 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:05.212+0000 7f77b0a7b700 1 -- 172.21.15.112:0/919544764 >> 172.21.15.112:0/919544764 conn(0x7f77ac05e2c0 msgr2=0x7f77ac157ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:32:05.217 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:05.212+0000 7f77b0a7b700 1 -- 172.21.15.112:0/919544764 shutdown_connections 2023-01-24T03:32:05.217 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:05.212+0000 7f77b0a7b700 1 -- 172.21.15.112:0/919544764 wait complete. 2023-01-24T03:32:05.218 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 13 2023-01-24T03:32:05.226 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:32:07.228 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:32:07.352 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.350+0000 7f397b0b5700 1 Processor -- start 2023-01-24T03:32:07.353 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.350+0000 7f397b0b5700 1 -- start start 2023-01-24T03:32:07.355 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.351+0000 7f397b0b5700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f39740c3080 0x7f397405df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:07.355 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.351+0000 7f397b0b5700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39741566c0 con 0x7f39740c15b0 2023-01-24T03:32:07.356 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.351+0000 7f397b0b5700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39741569a0 con 0x7f397414d5d0 2023-01-24T03:32:07.356 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.351+0000 7f397b0b5700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3974156c80 con 0x7f39740c3080 2023-01-24T03:32:07.356 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.351+0000 7f3978e51700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f39740c3080 0x7f397405df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:07.356 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.351+0000 7f3978e51700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f39740c3080 0x7f397405df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51684/0 (socket says 172.21.15.112:51684) 2023-01-24T03:32:07.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.351+0000 7f3978e51700 1 -- 172.21.15.112:0/1383374882 learned_addr learned my addr 172.21.15.112:0/1383374882 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:32:07.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.351+0000 7f3973fff700 1 --1- 172.21.15.112:0/1383374882 >> v1:172.21.15.112:6789/0 conn(0x7f39740c15b0 0x7f397414cec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:60904/0 (socket says 172.21.15.112:60904) 2023-01-24T03:32:07.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.352+0000 7f3978e51700 1 -- 172.21.15.112:0/1383374882 >> v1:172.21.15.142:6789/0 conn(0x7f397414d5d0 legacy=0x7f3974153400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:32:07.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.352+0000 7f3978e51700 1 -- 172.21.15.112:0/1383374882 >> v1:172.21.15.112:6789/0 conn(0x7f39740c15b0 legacy=0x7f397414cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:07.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.352+0000 7f3978e51700 1 -- 172.21.15.112:0/1383374882 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3974156f60 con 0x7f39740c3080 2023-01-24T03:32:07.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.352+0000 7f3978e51700 1 --2- 172.21.15.112:0/1383374882 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39740c3080 0x7f397405df40 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3964006fd0 tx=0x7f39640090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6154de1891348 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:07.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.352+0000 7f39737fe700 1 -- 172.21.15.112:0/1383374882 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3964011630 con 0x7f39740c3080 2023-01-24T03:32:07.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.352+0000 7f39737fe700 1 -- 172.21.15.112:0/1383374882 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3964011790 con 0x7f39740c3080 2023-01-24T03:32:07.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.352+0000 7f39737fe700 1 -- 172.21.15.112:0/1383374882 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3964011b50 con 0x7f39740c3080 2023-01-24T03:32:07.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.353+0000 7f397b0b5700 1 -- 172.21.15.112:0/1383374882 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39740c3080 msgr2=0x7f397405df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:07.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.353+0000 7f397b0b5700 1 --2- 172.21.15.112:0/1383374882 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39740c3080 0x7f397405df40 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3964006fd0 tx=0x7f39640090f0 comp rx=0 tx=0).stop 2023-01-24T03:32:07.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.353+0000 7f397b0b5700 1 -- 172.21.15.112:0/1383374882 shutdown_connections 2023-01-24T03:32:07.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.353+0000 7f397b0b5700 1 --2- 172.21.15.112:0/1383374882 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39740c3080 0x7f397405df40 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:07.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.353+0000 7f397b0b5700 1 -- 172.21.15.112:0/1383374882 >> 172.21.15.112:0/1383374882 conn(0x7f397406daf0 msgr2=0x7f397405d830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:32:07.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.353+0000 7f397b0b5700 1 -- 172.21.15.112:0/1383374882 shutdown_connections 2023-01-24T03:32:07.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.353+0000 7f397b0b5700 1 -- 172.21.15.112:0/1383374882 wait complete. 2023-01-24T03:32:07.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.354+0000 7f397b0b5700 1 Processor -- start 2023-01-24T03:32:07.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.354+0000 7f397b0b5700 1 -- start start 2023-01-24T03:32:07.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.355+0000 7f397b0b5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39740c15b0 0x7f3974163b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:07.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.355+0000 7f397b0b5700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39740c3080 0x7f39741499f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:07.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.355+0000 7f397b0b5700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f397414d5d0 0x7f3974149f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:07.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.355+0000 7f397b0b5700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397414a470 con 0x7f39740c3080 2023-01-24T03:32:07.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.355+0000 7f397b0b5700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397414fcc0 con 0x7f397414d5d0 2023-01-24T03:32:07.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.355+0000 7f397b0b5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3974163340 con 0x7f39740c15b0 2023-01-24T03:32:07.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.355+0000 7f3973fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39740c3080 0x7f39741499f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:07.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.355+0000 7f3973fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39740c3080 0x7f39741499f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51692/0 (socket says 172.21.15.112:51692) 2023-01-24T03:32:07.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.355+0000 7f3973fff700 1 -- 172.21.15.112:0/1925206509 learned_addr learned my addr 172.21.15.112:0/1925206509 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:32:07.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.355+0000 7f3978e51700 1 --2- 172.21.15.112:0/1925206509 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39740c15b0 0x7f3974163b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:07.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.355+0000 7f3979652700 1 --2- 172.21.15.112:0/1925206509 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f397414d5d0 0x7f3974149f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:07.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.355+0000 7f3973fff700 1 -- 172.21.15.112:0/1925206509 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39740c15b0 msgr2=0x7f3974163b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:07.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.355+0000 7f3973fff700 1 --2- 172.21.15.112:0/1925206509 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39740c15b0 0x7f3974163b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:07.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.355+0000 7f3973fff700 1 -- 172.21.15.112:0/1925206509 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f397414d5d0 msgr2=0x7f3974149f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:07.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.355+0000 7f3973fff700 1 --2- 172.21.15.112:0/1925206509 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f397414d5d0 0x7f3974149f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:07.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.355+0000 7f3973fff700 1 -- 172.21.15.112:0/1925206509 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3964009370 con 0x7f39740c3080 2023-01-24T03:32:07.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.355+0000 7f3973fff700 1 --2- 172.21.15.112:0/1925206509 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39740c3080 0x7f39741499f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f396800c440 tx=0x7f3968007dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:07.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.355+0000 7f3970ff9700 1 -- 172.21.15.112:0/1925206509 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39680073b0 con 0x7f39740c3080 2023-01-24T03:32:07.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.355+0000 7f3970ff9700 1 -- 172.21.15.112:0/1925206509 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3968007530 con 0x7f39740c3080 2023-01-24T03:32:07.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.355+0000 7f397b0b5700 1 -- 172.21.15.112:0/1925206509 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f397414ff10 con 0x7f39740c3080 2023-01-24T03:32:07.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.355+0000 7f3970ff9700 1 -- 172.21.15.112:0/1925206509 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3968004900 con 0x7f39740c3080 2023-01-24T03:32:07.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.355+0000 7f397b0b5700 1 -- 172.21.15.112:0/1925206509 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39741504d0 con 0x7f39740c3080 2023-01-24T03:32:07.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.356+0000 7f397b0b5700 1 -- 172.21.15.112:0/1925206509 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f397404bb80 con 0x7f39740c3080 2023-01-24T03:32:07.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.357+0000 7f3970ff9700 1 -- 172.21.15.112:0/1925206509 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f3968010070 con 0x7f39740c3080 2023-01-24T03:32:07.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.357+0000 7f3970ff9700 1 --2- 172.21.15.112:0/1925206509 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f39540611c0 0x7f3954063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:07.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.357+0000 7f3970ff9700 1 -- 172.21.15.112:0/1925206509 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6953+0+0 (secure 0 0 0) 0x7f396807ca30 con 0x7f39740c3080 2023-01-24T03:32:07.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.357+0000 7f3978e51700 1 --2- 172.21.15.112:0/1925206509 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f39540611c0 0x7f3954063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:07.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.357+0000 7f3978e51700 1 --2- 172.21.15.112:0/1925206509 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f39540611c0 0x7f3954063670 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3964006df0 tx=0x7f3964017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:07.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.361+0000 7f3970ff9700 1 -- 172.21.15.112:0/1925206509 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f396804a1d0 con 0x7f39740c3080 2023-01-24T03:32:07.580 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.579+0000 7f397b0b5700 1 -- 172.21.15.112:0/1925206509 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39740c7170 con 0x7f39740c3080 2023-01-24T03:32:07.581 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.579+0000 7f3970ff9700 1 -- 172.21.15.112:0/1925206509 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+5358 (secure 0 0 0) 0x7f396804c7e0 con 0x7f39740c3080 2023-01-24T03:32:07.582 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:32:07.582 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:32:04.610259+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4288},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":0,"incarnation":11,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.142:6834/869267413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":869267413},{"type":"v1","addr":"172.21.15.142:6834","nonce":869267413}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:32:07.583 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.581+0000 7f397b0b5700 1 -- 172.21.15.112:0/1925206509 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f39540611c0 msgr2=0x7f3954063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:07.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.581+0000 7f397b0b5700 1 --2- 172.21.15.112:0/1925206509 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f39540611c0 0x7f3954063670 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3964006df0 tx=0x7f3964017000 comp rx=0 tx=0).stop 2023-01-24T03:32:07.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.581+0000 7f397b0b5700 1 -- 172.21.15.112:0/1925206509 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39740c3080 msgr2=0x7f39741499f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:07.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.581+0000 7f397b0b5700 1 --2- 172.21.15.112:0/1925206509 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39740c3080 0x7f39741499f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f396800c440 tx=0x7f3968007dc0 comp rx=0 tx=0).stop 2023-01-24T03:32:07.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.581+0000 7f397b0b5700 1 -- 172.21.15.112:0/1925206509 shutdown_connections 2023-01-24T03:32:07.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.581+0000 7f397b0b5700 1 --2- 172.21.15.112:0/1925206509 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f39540611c0 0x7f3954063670 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:07.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.581+0000 7f397b0b5700 1 --2- 172.21.15.112:0/1925206509 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f397414d5d0 0x7f3974149f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:07.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.581+0000 7f397b0b5700 1 --2- 172.21.15.112:0/1925206509 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39740c15b0 0x7f3974163b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:07.586 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.581+0000 7f397b0b5700 1 --2- 172.21.15.112:0/1925206509 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39740c3080 0x7f39741499f0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:07.586 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.581+0000 7f397b0b5700 1 -- 172.21.15.112:0/1925206509 >> 172.21.15.112:0/1925206509 conn(0x7f397406daf0 msgr2=0x7f397414b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:32:07.586 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.581+0000 7f397b0b5700 1 -- 172.21.15.112:0/1925206509 shutdown_connections 2023-01-24T03:32:07.586 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:07.581+0000 7f397b0b5700 1 -- 172.21.15.112:0/1925206509 wait complete. 2023-01-24T03:32:07.587 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 13 2023-01-24T03:32:07.596 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 13, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-01-24T03:30:20.971710+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 13, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4288": { "addr": "172.21.15.142:6834/869267413", "addrs": { "addrvec": [ { "addr": "172.21.15.142:6832", "nonce": 869267413, "type": "v2" }, { "addr": "172.21.15.142:6834", "nonce": 869267413, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4288, "incarnation": 11, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:rejoin", "state_seq": 29 } }, "last_failure": 0, "last_failure_osd_epoch": 27, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-01-24T03:32:04.610259+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4288 } } } ], "standbys": [ { "addr": "172.21.15.142:6837/2181345869", "addrs": { "addrvec": [ { "addr": "172.21.15.142:6836", "nonce": 2181345869, "type": "v2" }, { "addr": "172.21.15.142:6837", "nonce": 2181345869, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6835/330343927", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6834", "nonce": 330343927, "type": "v2" }, { "addr": "172.21.15.112:6835", "nonce": 330343927, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6837/12176817", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6836", "nonce": 12176817, "type": "v2" }, { "addr": "172.21.15.112:6837", "nonce": 12176817, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4276, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6839/1110983841", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6838", "nonce": 1110983841, "type": "v2" }, { "addr": "172.21.15.112:6839", "nonce": 1110983841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4281, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-01-24T03:32:09.597 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:32:09.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.720+0000 7fea4e693700 1 Processor -- start 2023-01-24T03:32:09.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.721+0000 7fea4e693700 1 -- start start 2023-01-24T03:32:09.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.721+0000 7fea4e693700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fea480c7e40 0x7fea480b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:09.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.721+0000 7fea4e693700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fea48155da0 0x7fea480ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:09.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.721+0000 7fea4e693700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea48156240 con 0x7fea480aef20 2023-01-24T03:32:09.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.721+0000 7fea4e693700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea480af320 con 0x7fea480c7e40 2023-01-24T03:32:09.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.721+0000 7fea4e693700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea480b1170 con 0x7fea48155da0 2023-01-24T03:32:09.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.721+0000 7fea47fff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fea480c7e40 0x7fea480b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:09.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.721+0000 7fea47fff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fea480c7e40 0x7fea480b0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51704/0 (socket says 172.21.15.112:51704) 2023-01-24T03:32:09.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.721+0000 7fea47fff700 1 -- 172.21.15.112:0/1015460310 learned_addr learned my addr 172.21.15.112:0/1015460310 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:32:09.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.722+0000 7fea3ffff700 1 --2- 172.21.15.112:0/1015460310 >> v2:172.21.15.142:3300/0 conn(0x7fea48155da0 0x7fea480ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:09.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.722+0000 7fea47fff700 1 -- 172.21.15.112:0/1015460310 >> v2:172.21.15.142:3300/0 conn(0x7fea48155da0 msgr2=0x7fea480ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:32:09.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.722+0000 7fea47fff700 1 --2- 172.21.15.112:0/1015460310 >> v2:172.21.15.142:3300/0 conn(0x7fea48155da0 0x7fea480ae9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:09.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.722+0000 7fea47fff700 1 -- 172.21.15.112:0/1015460310 >> v1:172.21.15.112:6789/0 conn(0x7fea480aef20 legacy=0x7fea480ca3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:09.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.722+0000 7fea47fff700 1 -- 172.21.15.112:0/1015460310 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea480b1450 con 0x7fea480c7e40 2023-01-24T03:32:09.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.722+0000 7fea47fff700 1 --2- 172.21.15.112:0/1015460310 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea480c7e40 0x7fea480b0aa0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fea48152ea0 tx=0x7fea380090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10194e4d9a629d11 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:09.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.722+0000 7fea477fe700 1 -- 172.21.15.112:0/1015460310 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea38011630 con 0x7fea480c7e40 2023-01-24T03:32:09.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.722+0000 7fea477fe700 1 -- 172.21.15.112:0/1015460310 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea380117d0 con 0x7fea480c7e40 2023-01-24T03:32:09.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.722+0000 7fea477fe700 1 -- 172.21.15.112:0/1015460310 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea38011b90 con 0x7fea480c7e40 2023-01-24T03:32:09.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.723+0000 7fea4e693700 1 -- 172.21.15.112:0/1015460310 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea480c7e40 msgr2=0x7fea480b0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:09.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.723+0000 7fea4e693700 1 --2- 172.21.15.112:0/1015460310 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea480c7e40 0x7fea480b0aa0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fea48152ea0 tx=0x7fea380090f0 comp rx=0 tx=0).stop 2023-01-24T03:32:09.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.723+0000 7fea4e693700 1 -- 172.21.15.112:0/1015460310 shutdown_connections 2023-01-24T03:32:09.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.723+0000 7fea4e693700 1 --2- 172.21.15.112:0/1015460310 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea480c7e40 0x7fea480b0aa0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:09.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.723+0000 7fea4e693700 1 --2- 172.21.15.112:0/1015460310 >> v2:172.21.15.142:3300/0 conn(0x7fea48155da0 0x7fea480ae9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:09.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.723+0000 7fea4e693700 1 -- 172.21.15.112:0/1015460310 >> 172.21.15.112:0/1015460310 conn(0x7fea4805e2c0 msgr2=0x7fea480cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:32:09.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.723+0000 7fea4e693700 1 -- 172.21.15.112:0/1015460310 shutdown_connections 2023-01-24T03:32:09.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.723+0000 7fea4e693700 1 -- 172.21.15.112:0/1015460310 wait complete. 2023-01-24T03:32:09.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.724+0000 7fea4e693700 1 Processor -- start 2023-01-24T03:32:09.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.724+0000 7fea4e693700 1 -- start start 2023-01-24T03:32:09.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.724+0000 7fea4e693700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea480aef20 0x7fea48165fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:09.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.724+0000 7fea4e693700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea480c7e40 0x7fea4814f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:09.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.724+0000 7fea4e693700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea48155da0 0x7fea48150310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:09.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.724+0000 7fea4e693700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea48166510 con 0x7fea480c7e40 2023-01-24T03:32:09.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.724+0000 7fea4e693700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea481522a0 con 0x7fea480aef20 2023-01-24T03:32:09.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.724+0000 7fea4e693700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea48152580 con 0x7fea48155da0 2023-01-24T03:32:09.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.724+0000 7fea3ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea480c7e40 0x7fea4814f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:09.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.724+0000 7fea3ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea480c7e40 0x7fea4814f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51712/0 (socket says 172.21.15.112:51712) 2023-01-24T03:32:09.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.725+0000 7fea3ffff700 1 -- 172.21.15.112:0/3011115196 learned_addr learned my addr 172.21.15.112:0/3011115196 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:32:09.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.725+0000 7fea47fff700 1 --2- 172.21.15.112:0/3011115196 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea480aef20 0x7fea48165fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:09.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.725+0000 7fea4cc30700 1 --2- 172.21.15.112:0/3011115196 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea48155da0 0x7fea48150310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:09.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.725+0000 7fea3ffff700 1 -- 172.21.15.112:0/3011115196 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea48155da0 msgr2=0x7fea48150310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:09.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.725+0000 7fea3ffff700 1 --2- 172.21.15.112:0/3011115196 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea48155da0 0x7fea48150310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:09.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.725+0000 7fea3ffff700 1 -- 172.21.15.112:0/3011115196 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea480aef20 msgr2=0x7fea48165fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:09.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.725+0000 7fea3ffff700 1 --2- 172.21.15.112:0/3011115196 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea480aef20 0x7fea48165fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:09.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.725+0000 7fea3ffff700 1 -- 172.21.15.112:0/3011115196 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea38009370 con 0x7fea480c7e40 2023-01-24T03:32:09.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.725+0000 7fea47fff700 1 --2- 172.21.15.112:0/3011115196 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea480aef20 0x7fea48165fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:32:09.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.725+0000 7fea3ffff700 1 --2- 172.21.15.112:0/3011115196 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea480c7e40 0x7fea4814f0e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fea3000c850 tx=0x7fea30008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:09.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.725+0000 7fea44ff9700 1 -- 172.21.15.112:0/3011115196 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea30011600 con 0x7fea480c7e40 2023-01-24T03:32:09.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.725+0000 7fea44ff9700 1 -- 172.21.15.112:0/3011115196 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea30011760 con 0x7fea480c7e40 2023-01-24T03:32:09.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.725+0000 7fea4e693700 1 -- 172.21.15.112:0/3011115196 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea48152860 con 0x7fea480c7e40 2023-01-24T03:32:09.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.725+0000 7fea44ff9700 1 -- 172.21.15.112:0/3011115196 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea30011ae0 con 0x7fea480c7e40 2023-01-24T03:32:09.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.725+0000 7fea4e693700 1 -- 172.21.15.112:0/3011115196 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea4814fdf0 con 0x7fea480c7e40 2023-01-24T03:32:09.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.726+0000 7fea4e693700 1 -- 172.21.15.112:0/3011115196 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea4804bb80 con 0x7fea480c7e40 2023-01-24T03:32:09.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.726+0000 7fea44ff9700 1 -- 172.21.15.112:0/3011115196 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fea30020070 con 0x7fea480c7e40 2023-01-24T03:32:09.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.727+0000 7fea44ff9700 1 --2- 172.21.15.112:0/3011115196 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fea2c0611c0 0x7fea2c063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:09.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.727+0000 7fea44ff9700 1 -- 172.21.15.112:0/3011115196 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6953+0+0 (secure 0 0 0) 0x7fea3007d3f0 con 0x7fea480c7e40 2023-01-24T03:32:09.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.727+0000 7fea47fff700 1 --2- 172.21.15.112:0/3011115196 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fea2c0611c0 0x7fea2c063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:09.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.727+0000 7fea47fff700 1 --2- 172.21.15.112:0/3011115196 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fea2c0611c0 0x7fea2c063670 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fea38009340 tx=0x7fea38016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:09.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.731+0000 7fea44ff9700 1 -- 172.21.15.112:0/3011115196 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fea3004d1a0 con 0x7fea480c7e40 2023-01-24T03:32:09.953 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.951+0000 7fea4e693700 1 -- 172.21.15.112:0/3011115196 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea481586d0 con 0x7fea480c7e40 2023-01-24T03:32:09.954 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.952+0000 7fea44ff9700 1 -- 172.21.15.112:0/3011115196 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+5358 (secure 0 0 0) 0x7fea300507c0 con 0x7fea480c7e40 2023-01-24T03:32:09.954 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:32:09.955 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:32:09.626128+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4288},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":0,"incarnation":11,"state":"up:active","state_seq":31,"addr":"172.21.15.142:6834/869267413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":869267413},{"type":"v1","addr":"172.21.15.142:6834","nonce":869267413}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:32:09.957 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.954+0000 7fea4e693700 1 -- 172.21.15.112:0/3011115196 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fea2c0611c0 msgr2=0x7fea2c063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:09.957 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.954+0000 7fea4e693700 1 --2- 172.21.15.112:0/3011115196 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fea2c0611c0 0x7fea2c063670 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fea38009340 tx=0x7fea38016000 comp rx=0 tx=0).stop 2023-01-24T03:32:09.957 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.954+0000 7fea4e693700 1 -- 172.21.15.112:0/3011115196 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea480c7e40 msgr2=0x7fea4814f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:09.957 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.954+0000 7fea4e693700 1 --2- 172.21.15.112:0/3011115196 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea480c7e40 0x7fea4814f0e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fea3000c850 tx=0x7fea30008040 comp rx=0 tx=0).stop 2023-01-24T03:32:09.957 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.954+0000 7fea4e693700 1 -- 172.21.15.112:0/3011115196 shutdown_connections 2023-01-24T03:32:09.958 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.954+0000 7fea4e693700 1 --2- 172.21.15.112:0/3011115196 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fea2c0611c0 0x7fea2c063670 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:09.958 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.954+0000 7fea4e693700 1 --2- 172.21.15.112:0/3011115196 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fea48155da0 0x7fea48150310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:09.958 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.954+0000 7fea4e693700 1 --2- 172.21.15.112:0/3011115196 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea480aef20 0x7fea48165fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:09.958 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.954+0000 7fea4e693700 1 --2- 172.21.15.112:0/3011115196 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea480c7e40 0x7fea4814f0e0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:09.958 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.954+0000 7fea4e693700 1 -- 172.21.15.112:0/3011115196 >> 172.21.15.112:0/3011115196 conn(0x7fea4805e2c0 msgr2=0x7fea48156700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:32:09.959 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.954+0000 7fea4e693700 1 -- 172.21.15.112:0/3011115196 shutdown_connections 2023-01-24T03:32:09.959 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:09.954+0000 7fea4e693700 1 -- 172.21.15.112:0/3011115196 wait complete. 2023-01-24T03:32:09.959 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 14 2023-01-24T03:32:09.968 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=4293 2023-01-24T03:32:09.968 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.d 2023-01-24T03:32:13.969 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2023-01-24T03:32:13.970 INFO:tasks.ceph.mds.d:Restarting daemon 2023-01-24T03:32:13.970 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-01-24T03:32:13.974 INFO:tasks.ceph.mds.d:Started 2023-01-24T03:32:13.974 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:32:14.054 INFO:tasks.ceph.mds.d.smithi142.stdout:starting mds.d at 2023-01-24T03:32:14.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.098+0000 7fbca1135700 1 Processor -- start 2023-01-24T03:32:14.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.098+0000 7fbca1135700 1 -- start start 2023-01-24T03:32:14.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.099+0000 7fbca1135700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fbc9c0c6f60 0x7fbc9c0c24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:14.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.099+0000 7fbca1135700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc9c0c2010 con 0x7fbc9c0c29e0 2023-01-24T03:32:14.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.099+0000 7fbca1135700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc9c0c4a60 con 0x7fbc9c06a890 2023-01-24T03:32:14.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.099+0000 7fbca1135700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc9c0c5e10 con 0x7fbc9c0c6f60 2023-01-24T03:32:14.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.099+0000 7fbc9b59e700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fbc9c0c29e0 0x7fbc9c151f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:60942/0 (socket says 172.21.15.112:60942) 2023-01-24T03:32:14.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.099+0000 7fbc9b59e700 1 -- 172.21.15.112:0/857361388 learned_addr learned my addr 172.21.15.112:0/857361388 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:32:14.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.099+0000 7fbc9a59c700 1 --2- 172.21.15.112:0/857361388 >> v2:172.21.15.142:3300/0 conn(0x7fbc9c0c6f60 0x7fbc9c0c24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:14.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.100+0000 7fbc99d9b700 1 -- 172.21.15.112:0/857361388 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2467734481 0 0) 0x7fbc9c0c2010 con 0x7fbc9c0c29e0 2023-01-24T03:32:14.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.100+0000 7fbc99d9b700 1 -- 172.21.15.112:0/857361388 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc880033f0 con 0x7fbc9c0c29e0 2023-01-24T03:32:14.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.100+0000 7fbc99d9b700 1 -- 172.21.15.112:0/857361388 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 126067516 0 0) 0x7fbc9c0c4a60 con 0x7fbc9c06a890 2023-01-24T03:32:14.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.100+0000 7fbc99d9b700 1 -- 172.21.15.112:0/857361388 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc9c0c2010 con 0x7fbc9c06a890 2023-01-24T03:32:14.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.100+0000 7fbc9a59c700 1 -- 172.21.15.112:0/857361388 >> v1:172.21.15.142:6789/0 conn(0x7fbc9c06a890 legacy=0x7fbc9c06ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:14.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.100+0000 7fbc9a59c700 1 -- 172.21.15.112:0/857361388 >> v1:172.21.15.112:6789/0 conn(0x7fbc9c0c29e0 legacy=0x7fbc9c151f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:14.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.100+0000 7fbc9a59c700 1 -- 172.21.15.112:0/857361388 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc9c0c2e40 con 0x7fbc9c0c6f60 2023-01-24T03:32:14.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.100+0000 7fbc9a59c700 1 --2- 172.21.15.112:0/857361388 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc9c0c6f60 0x7fbc9c0c24a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fbc90009760 tx=0x7fbc90009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7306da25c1cf3a2 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:14.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.100+0000 7fbc99d9b700 1 -- 172.21.15.112:0/857361388 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc900105b0 con 0x7fbc9c0c6f60 2023-01-24T03:32:14.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.100+0000 7fbc99d9b700 1 -- 172.21.15.112:0/857361388 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc90010710 con 0x7fbc9c0c6f60 2023-01-24T03:32:14.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.100+0000 7fbc99d9b700 1 -- 172.21.15.112:0/857361388 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc90010a90 con 0x7fbc9c0c6f60 2023-01-24T03:32:14.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.101+0000 7fbca1135700 1 -- 172.21.15.112:0/857361388 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc9c0c6f60 msgr2=0x7fbc9c0c24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:14.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.101+0000 7fbca1135700 1 --2- 172.21.15.112:0/857361388 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc9c0c6f60 0x7fbc9c0c24a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fbc90009760 tx=0x7fbc90009060 comp rx=0 tx=0).stop 2023-01-24T03:32:14.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.101+0000 7fbca1135700 1 -- 172.21.15.112:0/857361388 shutdown_connections 2023-01-24T03:32:14.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.101+0000 7fbca1135700 1 --2- 172.21.15.112:0/857361388 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc9c0c6f60 0x7fbc9c0c24a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:14.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.101+0000 7fbca1135700 1 -- 172.21.15.112:0/857361388 >> 172.21.15.112:0/857361388 conn(0x7fbc9c066e10 msgr2=0x7fbc9c067210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:32:14.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.101+0000 7fbca1135700 1 -- 172.21.15.112:0/857361388 shutdown_connections 2023-01-24T03:32:14.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.101+0000 7fbca1135700 1 -- 172.21.15.112:0/857361388 wait complete. 2023-01-24T03:32:14.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.102+0000 7fbca1135700 1 Processor -- start 2023-01-24T03:32:14.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.102+0000 7fbca1135700 1 -- start start 2023-01-24T03:32:14.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.103+0000 7fbca1135700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc9c06a890 0x7fbc9c165d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:14.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.103+0000 7fbca1135700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc9c0c29e0 0x7fbc9c150b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:14.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.103+0000 7fbca1135700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbc9c0c6f60 0x7fbc9c1510d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:14.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.103+0000 7fbca1135700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9c0c4fe0 con 0x7fbc9c0c29e0 2023-01-24T03:32:14.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.103+0000 7fbca1135700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9c150700 con 0x7fbc9c06a890 2023-01-24T03:32:14.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.103+0000 7fbca1135700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9c14fae0 con 0x7fbc9c0c6f60 2023-01-24T03:32:14.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.103+0000 7fbc9a59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc9c0c29e0 0x7fbc9c150b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:14.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.103+0000 7fbc9a59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc9c0c29e0 0x7fbc9c150b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51726/0 (socket says 172.21.15.112:51726) 2023-01-24T03:32:14.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.103+0000 7fbc9a59c700 1 -- 172.21.15.112:0/1481509484 learned_addr learned my addr 172.21.15.112:0/1481509484 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:32:14.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.103+0000 7fbc9b59e700 1 --2- 172.21.15.112:0/1481509484 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbc9c0c6f60 0x7fbc9c1510d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:14.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.103+0000 7fbc9ad9d700 1 --2- 172.21.15.112:0/1481509484 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc9c06a890 0x7fbc9c165d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:14.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.103+0000 7fbc9a59c700 1 -- 172.21.15.112:0/1481509484 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbc9c0c6f60 msgr2=0x7fbc9c1510d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:14.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.103+0000 7fbc9a59c700 1 --2- 172.21.15.112:0/1481509484 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbc9c0c6f60 0x7fbc9c1510d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:14.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.103+0000 7fbc9a59c700 1 -- 172.21.15.112:0/1481509484 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc9c06a890 msgr2=0x7fbc9c165d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:14.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.103+0000 7fbc9a59c700 1 --2- 172.21.15.112:0/1481509484 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc9c06a890 0x7fbc9c165d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:14.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.103+0000 7fbc9a59c700 1 -- 172.21.15.112:0/1481509484 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc900092a0 con 0x7fbc9c0c29e0 2023-01-24T03:32:14.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.103+0000 7fbc9a59c700 1 --2- 172.21.15.112:0/1481509484 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc9c0c29e0 0x7fbc9c150b90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fbc900030c0 tx=0x7fbc90016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:14.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.103+0000 7fbc82ffd700 1 -- 172.21.15.112:0/1481509484 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc90011720 con 0x7fbc9c0c29e0 2023-01-24T03:32:14.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.103+0000 7fbc82ffd700 1 -- 172.21.15.112:0/1481509484 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc90011880 con 0x7fbc9c0c29e0 2023-01-24T03:32:14.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.103+0000 7fbca1135700 1 -- 172.21.15.112:0/1481509484 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc9c1503b0 con 0x7fbc9c0c29e0 2023-01-24T03:32:14.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.103+0000 7fbc82ffd700 1 -- 172.21.15.112:0/1481509484 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc90011c40 con 0x7fbc9c0c29e0 2023-01-24T03:32:14.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.103+0000 7fbca1135700 1 -- 172.21.15.112:0/1481509484 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc9c150030 con 0x7fbc9c0c29e0 2023-01-24T03:32:14.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.104+0000 7fbca1135700 1 -- 172.21.15.112:0/1481509484 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc9c04bb80 con 0x7fbc9c0c29e0 2023-01-24T03:32:14.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.105+0000 7fbc82ffd700 1 -- 172.21.15.112:0/1481509484 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fbc9000f070 con 0x7fbc9c0c29e0 2023-01-24T03:32:14.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.105+0000 7fbc82ffd700 1 --2- 172.21.15.112:0/1481509484 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fbc7c0611c0 0x7fbc7c063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:14.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.105+0000 7fbc82ffd700 1 -- 172.21.15.112:0/1481509484 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6953+0+0 (secure 0 0 0) 0x7fbc9007ecf0 con 0x7fbc9c0c29e0 2023-01-24T03:32:14.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.105+0000 7fbc9ad9d700 1 --2- 172.21.15.112:0/1481509484 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fbc7c0611c0 0x7fbc7c063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:14.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.106+0000 7fbc9ad9d700 1 --2- 172.21.15.112:0/1481509484 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fbc7c0611c0 0x7fbc7c063670 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbc94003fc0 tx=0x7fbc94007f50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:14.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.109+0000 7fbc82ffd700 1 -- 172.21.15.112:0/1481509484 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fbc9004ea40 con 0x7fbc9c0c29e0 2023-01-24T03:32:14.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.332+0000 7fbca1135700 1 -- 172.21.15.112:0/1481509484 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc9c150540 con 0x7fbc9c0c29e0 2023-01-24T03:32:14.335 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.333+0000 7fbc82ffd700 1 -- 172.21.15.112:0/1481509484 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+5358 (secure 0 0 0) 0x7fbc9c150540 con 0x7fbc9c0c29e0 2023-01-24T03:32:14.335 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:32:14.335 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:32:09.626128+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4288},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":0,"incarnation":11,"state":"up:active","state_seq":31,"addr":"172.21.15.142:6834/869267413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":869267413},{"type":"v1","addr":"172.21.15.142:6834","nonce":869267413}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:32:14.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.335+0000 7fbca1135700 1 -- 172.21.15.112:0/1481509484 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fbc7c0611c0 msgr2=0x7fbc7c063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:14.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.335+0000 7fbca1135700 1 --2- 172.21.15.112:0/1481509484 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fbc7c0611c0 0x7fbc7c063670 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbc94003fc0 tx=0x7fbc94007f50 comp rx=0 tx=0).stop 2023-01-24T03:32:14.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.335+0000 7fbca1135700 1 -- 172.21.15.112:0/1481509484 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc9c0c29e0 msgr2=0x7fbc9c150b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:14.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.335+0000 7fbca1135700 1 --2- 172.21.15.112:0/1481509484 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc9c0c29e0 0x7fbc9c150b90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fbc900030c0 tx=0x7fbc90016000 comp rx=0 tx=0).stop 2023-01-24T03:32:14.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.335+0000 7fbca1135700 1 -- 172.21.15.112:0/1481509484 shutdown_connections 2023-01-24T03:32:14.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.335+0000 7fbca1135700 1 --2- 172.21.15.112:0/1481509484 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fbc7c0611c0 0x7fbc7c063670 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:14.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.335+0000 7fbca1135700 1 --2- 172.21.15.112:0/1481509484 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbc9c0c6f60 0x7fbc9c1510d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:14.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.335+0000 7fbca1135700 1 --2- 172.21.15.112:0/1481509484 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc9c06a890 0x7fbc9c165d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:14.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.335+0000 7fbca1135700 1 --2- 172.21.15.112:0/1481509484 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc9c0c29e0 0x7fbc9c150b90 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:14.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.335+0000 7fbca1135700 1 -- 172.21.15.112:0/1481509484 >> 172.21.15.112:0/1481509484 conn(0x7fbc9c066e10 msgr2=0x7fbc9c0671e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:32:14.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.335+0000 7fbca1135700 1 -- 172.21.15.112:0/1481509484 shutdown_connections 2023-01-24T03:32:14.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:14.335+0000 7fbca1135700 1 -- 172.21.15.112:0/1481509484 wait complete. 2023-01-24T03:32:14.340 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 14 2023-01-24T03:32:14.349 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2023-01-24T03:32:14.912 INFO:tasks.workunit.client.0.smithi112.stderr:+ wget -q http://download.ceph.com/qa/linux-2.6.33.tar.bz2 2023-01-24T03:32:16.351 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:32:16.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.465+0000 7fbc79773700 1 Processor -- start 2023-01-24T03:32:16.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.465+0000 7fbc79773700 1 -- start start 2023-01-24T03:32:16.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.466+0000 7fbc79773700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fbc740c1040 0x7fbc74151e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:16.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.466+0000 7fbc79773700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc7406ba90 con 0x7fbc7414f6d0 2023-01-24T03:32:16.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.466+0000 7fbc79773700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc7406e1e0 con 0x7fbc7414f300 2023-01-24T03:32:16.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.466+0000 7fbc79773700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc7406da40 con 0x7fbc740c1040 2023-01-24T03:32:16.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.466+0000 7fbc737fe700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fbc7414f6d0 0x7fbc740c0930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:60960/0 (socket says 172.21.15.112:60960) 2023-01-24T03:32:16.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.466+0000 7fbc737fe700 1 -- 172.21.15.112:0/3207962652 learned_addr learned my addr 172.21.15.112:0/3207962652 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:32:16.471 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.466+0000 7fbc78f72700 1 --2- 172.21.15.112:0/3207962652 >> v2:172.21.15.142:3300/0 conn(0x7fbc740c1040 0x7fbc74151e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:16.471 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.466+0000 7fbc72ffd700 1 -- 172.21.15.112:0/3207962652 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1025524300 0 0) 0x7fbc7406ba90 con 0x7fbc7414f6d0 2023-01-24T03:32:16.471 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.466+0000 7fbc72ffd700 1 -- 172.21.15.112:0/3207962652 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc600033f0 con 0x7fbc7414f6d0 2023-01-24T03:32:16.471 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.466+0000 7fbc72ffd700 1 -- 172.21.15.112:0/3207962652 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1708083379 0 0) 0x7fbc7406e1e0 con 0x7fbc7414f300 2023-01-24T03:32:16.471 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.466+0000 7fbc72ffd700 1 -- 172.21.15.112:0/3207962652 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc7406ba90 con 0x7fbc7414f300 2023-01-24T03:32:16.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.466+0000 7fbc72ffd700 1 -- 172.21.15.112:0/3207962652 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1517559148 0 0) 0x7fbc600033f0 con 0x7fbc7414f6d0 2023-01-24T03:32:16.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.466+0000 7fbc78f72700 1 -- 172.21.15.112:0/3207962652 >> v1:172.21.15.142:6789/0 conn(0x7fbc7414f300 legacy=0x7fbc740b73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:16.476 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.466+0000 7fbc78f72700 1 -- 172.21.15.112:0/3207962652 >> v1:172.21.15.112:6789/0 conn(0x7fbc7414f6d0 legacy=0x7fbc740c0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:16.476 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.466+0000 7fbc78f72700 1 -- 172.21.15.112:0/3207962652 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc740c04a0 con 0x7fbc740c1040 2023-01-24T03:32:16.476 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.467+0000 7fbc78f72700 1 --2- 172.21.15.112:0/3207962652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc740c1040 0x7fbc74151e20 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c009e60 tx=0x7fbc6c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf8c2165139b6983 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:16.476 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.467+0000 7fbc72ffd700 1 -- 172.21.15.112:0/3207962652 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c00e070 con 0x7fbc740c1040 2023-01-24T03:32:16.476 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.467+0000 7fbc72ffd700 1 -- 172.21.15.112:0/3207962652 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc6c00b830 con 0x7fbc740c1040 2023-01-24T03:32:16.477 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.467+0000 7fbc72ffd700 1 -- 172.21.15.112:0/3207962652 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc64001470 con 0x7fbc740c1040 2023-01-24T03:32:16.477 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.467+0000 7fbc79773700 1 -- 172.21.15.112:0/3207962652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc740c1040 msgr2=0x7fbc74151e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:16.477 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.467+0000 7fbc79773700 1 --2- 172.21.15.112:0/3207962652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc740c1040 0x7fbc74151e20 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c009e60 tx=0x7fbc6c009780 comp rx=0 tx=0).stop 2023-01-24T03:32:16.477 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.467+0000 7fbc79773700 1 -- 172.21.15.112:0/3207962652 shutdown_connections 2023-01-24T03:32:16.477 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.467+0000 7fbc79773700 1 --2- 172.21.15.112:0/3207962652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc740c1040 0x7fbc74151e20 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:16.478 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.467+0000 7fbc79773700 1 -- 172.21.15.112:0/3207962652 >> 172.21.15.112:0/3207962652 conn(0x7fbc74066e10 msgr2=0x7fbc74067210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:32:16.478 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.467+0000 7fbc79773700 1 -- 172.21.15.112:0/3207962652 shutdown_connections 2023-01-24T03:32:16.478 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.467+0000 7fbc79773700 1 -- 172.21.15.112:0/3207962652 wait complete. 2023-01-24T03:32:16.478 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.468+0000 7fbc79773700 1 Processor -- start 2023-01-24T03:32:16.478 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.468+0000 7fbc79773700 1 -- start start 2023-01-24T03:32:16.479 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.469+0000 7fbc79773700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc740c1040 0x7fbc740b6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:16.479 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.469+0000 7fbc79773700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc7414f300 0x7fbc740c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:16.479 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.469+0000 7fbc79773700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbc740c49e0 0x7fbc740b3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:16.479 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.469+0000 7fbc79773700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc740ba4f0 con 0x7fbc7414f300 2023-01-24T03:32:16.479 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.469+0000 7fbc79773700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc74164d80 con 0x7fbc740c1040 2023-01-24T03:32:16.479 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.469+0000 7fbc79773700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc740b6ca0 con 0x7fbc740c49e0 2023-01-24T03:32:16.480 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.469+0000 7fbc737fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc7414f300 0x7fbc740c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:16.480 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.469+0000 7fbc737fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc7414f300 0x7fbc740c5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51746/0 (socket says 172.21.15.112:51746) 2023-01-24T03:32:16.480 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.469+0000 7fbc737fe700 1 -- 172.21.15.112:0/2072012478 learned_addr learned my addr 172.21.15.112:0/2072012478 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:32:16.480 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.469+0000 7fbc78f72700 1 --2- 172.21.15.112:0/2072012478 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbc740c49e0 0x7fbc740b3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:16.480 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.469+0000 7fbc73fff700 1 --2- 172.21.15.112:0/2072012478 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc740c1040 0x7fbc740b6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:16.481 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.470+0000 7fbc737fe700 1 -- 172.21.15.112:0/2072012478 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbc740c49e0 msgr2=0x7fbc740b3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:16.481 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.470+0000 7fbc737fe700 1 --2- 172.21.15.112:0/2072012478 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbc740c49e0 0x7fbc740b3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:16.481 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.470+0000 7fbc737fe700 1 -- 172.21.15.112:0/2072012478 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc740c1040 msgr2=0x7fbc740b6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:16.481 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.470+0000 7fbc737fe700 1 --2- 172.21.15.112:0/2072012478 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc740c1040 0x7fbc740b6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:16.481 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.470+0000 7fbc737fe700 1 -- 172.21.15.112:0/2072012478 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc6c009a20 con 0x7fbc7414f300 2023-01-24T03:32:16.482 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.470+0000 7fbc737fe700 1 --2- 172.21.15.112:0/2072012478 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc7414f300 0x7fbc740c5d60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fbc6400c800 tx=0x7fbc64001b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:16.482 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.470+0000 7fbc5bfff700 1 -- 172.21.15.112:0/2072012478 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc640073c0 con 0x7fbc7414f300 2023-01-24T03:32:16.482 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.470+0000 7fbc5bfff700 1 -- 172.21.15.112:0/2072012478 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc64007520 con 0x7fbc7414f300 2023-01-24T03:32:16.482 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.470+0000 7fbc79773700 1 -- 172.21.15.112:0/2072012478 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc74164450 con 0x7fbc7414f300 2023-01-24T03:32:16.482 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.470+0000 7fbc5bfff700 1 -- 172.21.15.112:0/2072012478 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc640048e0 con 0x7fbc7414f300 2023-01-24T03:32:16.483 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.470+0000 7fbc79773700 1 -- 172.21.15.112:0/2072012478 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc74164a10 con 0x7fbc7414f300 2023-01-24T03:32:16.483 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.471+0000 7fbc79773700 1 -- 172.21.15.112:0/2072012478 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc74164f30 con 0x7fbc7414f300 2023-01-24T03:32:16.483 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.471+0000 7fbc5bfff700 1 -- 172.21.15.112:0/2072012478 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fbc6401a020 con 0x7fbc7414f300 2023-01-24T03:32:16.483 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.472+0000 7fbc5bfff700 1 --2- 172.21.15.112:0/2072012478 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fbc540611c0 0x7fbc54063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:16.483 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.472+0000 7fbc5bfff700 1 -- 172.21.15.112:0/2072012478 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6953+0+0 (secure 0 0 0) 0x7fbc6407d5f0 con 0x7fbc7414f300 2023-01-24T03:32:16.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.472+0000 7fbc73fff700 1 --2- 172.21.15.112:0/2072012478 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fbc540611c0 0x7fbc54063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:16.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.472+0000 7fbc73fff700 1 --2- 172.21.15.112:0/2072012478 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fbc540611c0 0x7fbc54063670 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbc68003930 tx=0x7fbc68007bd0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:16.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.476+0000 7fbc5bfff700 1 -- 172.21.15.112:0/2072012478 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fbc74164f30 con 0x7fbc7414f300 2023-01-24T03:32:16.707 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.705+0000 7fbc79773700 1 -- 172.21.15.112:0/2072012478 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc74164ef0 con 0x7fbc7414f300 2023-01-24T03:32:16.708 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.706+0000 7fbc5bfff700 1 -- 172.21.15.112:0/2072012478 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+6112 (secure 0 0 0) 0x7fbc6404d3a0 con 0x7fbc7414f300 2023-01-24T03:32:16.708 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:32:16.710 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4897,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/11022091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":11022091},{"type":"v1","addr":"172.21.15.142:6835","nonce":11022091}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:32:09.626128+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4288},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":0,"incarnation":11,"state":"up:active","state_seq":31,"addr":"172.21.15.142:6834/869267413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":869267413},{"type":"v1","addr":"172.21.15.142:6834","nonce":869267413}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:32:16.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.708+0000 7fbc79773700 1 -- 172.21.15.112:0/2072012478 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fbc540611c0 msgr2=0x7fbc54063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:16.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.708+0000 7fbc79773700 1 --2- 172.21.15.112:0/2072012478 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fbc540611c0 0x7fbc54063670 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbc68003930 tx=0x7fbc68007bd0 comp rx=0 tx=0).stop 2023-01-24T03:32:16.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.708+0000 7fbc79773700 1 -- 172.21.15.112:0/2072012478 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc7414f300 msgr2=0x7fbc740c5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:16.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.708+0000 7fbc79773700 1 --2- 172.21.15.112:0/2072012478 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc7414f300 0x7fbc740c5d60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fbc6400c800 tx=0x7fbc64001b50 comp rx=0 tx=0).stop 2023-01-24T03:32:16.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.708+0000 7fbc79773700 1 -- 172.21.15.112:0/2072012478 shutdown_connections 2023-01-24T03:32:16.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.708+0000 7fbc79773700 1 --2- 172.21.15.112:0/2072012478 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fbc540611c0 0x7fbc54063670 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:16.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.708+0000 7fbc79773700 1 --2- 172.21.15.112:0/2072012478 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbc740c49e0 0x7fbc740b3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:16.716 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.708+0000 7fbc79773700 1 --2- 172.21.15.112:0/2072012478 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbc740c1040 0x7fbc740b6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:16.716 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.708+0000 7fbc79773700 1 --2- 172.21.15.112:0/2072012478 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc7414f300 0x7fbc740c5d60 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:16.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.708+0000 7fbc79773700 1 -- 172.21.15.112:0/2072012478 >> 172.21.15.112:0/2072012478 conn(0x7fbc74066e10 msgr2=0x7fbc7414bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:32:16.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.708+0000 7fbc79773700 1 -- 172.21.15.112:0/2072012478 shutdown_connections 2023-01-24T03:32:16.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:16.708+0000 7fbc79773700 1 -- 172.21.15.112:0/2072012478 wait complete. 2023-01-24T03:32:16.718 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 15 2023-01-24T03:32:16.722 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2023-01-24T03:32:16.723 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 22 secs before thrashing 2023-01-24T03:32:17.194 INFO:tasks.ceph.mds.b.smithi142.stderr:2023-01-24T03:32:17.192+0000 7f9e0c0cc700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-01-24T03:32:17.793 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir foo 2023-01-24T03:32:17.799 INFO:tasks.workunit.client.0.smithi112.stderr:+ cp linux-2.6.33.tar.bz2 foo 2023-01-24T03:32:17.832 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir foo/.snap/barsnap 2023-01-24T03:32:17.855 INFO:tasks.workunit.client.0.smithi112.stderr:+ rm foo/linux-2.6.33.tar.bz2 2023-01-24T03:32:17.863 INFO:tasks.workunit.client.0.smithi112.stderr:+ diff -q foo/.snap/barsnap/linux-2.6.33.tar.bz2 linux-2.6.33.tar.bz2 2023-01-24T03:32:18.279 INFO:tasks.workunit.client.0.smithi112.stderr:+ echo 'passed: files are identical' 2023-01-24T03:32:18.279 INFO:tasks.workunit.client.0.smithi112.stdout:passed: files are identical 2023-01-24T03:32:18.279 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir foo/.snap/barsnap 2023-01-24T03:32:18.287 INFO:tasks.workunit.client.0.smithi112.stdout:OK 2023-01-24T03:32:18.288 INFO:tasks.workunit.client.0.smithi112.stderr:+ echo OK 2023-01-24T03:32:18.289 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-01-24T03:32:18.289 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-01-24T03:32:18.324 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-1.sh... 2023-01-24T03:32:18.324 DEBUG:teuthology.orchestra.run.smithi112:workunit test fs/snaps/snaptest-1.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8387d427d91a301da19bd82e090b96e1348bfadb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-1.sh 2023-01-24T03:32:18.384 INFO:tasks.workunit.client.0.smithi112.stderr:+ echo 1 2023-01-24T03:32:18.385 INFO:tasks.workunit.client.0.smithi112.stderr:+ echo 2 2023-01-24T03:32:18.386 INFO:tasks.workunit.client.0.smithi112.stderr:+ echo 3 2023-01-24T03:32:18.387 INFO:tasks.workunit.client.0.smithi112.stderr:+ '[' -e file4 ']' 2023-01-24T03:32:18.387 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir .snap/snap1 2023-01-24T03:32:18.411 INFO:tasks.workunit.client.0.smithi112.stderr:+ echo 4 2023-01-24T03:32:18.412 INFO:tasks.workunit.client.0.smithi112.stderr:++ ls 2023-01-24T03:32:18.416 INFO:tasks.workunit.client.0.smithi112.stderr:+ now='file1 2023-01-24T03:32:18.416 INFO:tasks.workunit.client.0.smithi112.stderr:file2 2023-01-24T03:32:18.416 INFO:tasks.workunit.client.0.smithi112.stderr:file3 2023-01-24T03:32:18.417 INFO:tasks.workunit.client.0.smithi112.stderr:file4' 2023-01-24T03:32:18.417 INFO:tasks.workunit.client.0.smithi112.stderr:++ ls .snap/snap1 2023-01-24T03:32:18.419 INFO:tasks.workunit.client.0.smithi112.stderr:+ then='file1 2023-01-24T03:32:18.419 INFO:tasks.workunit.client.0.smithi112.stderr:file2 2023-01-24T03:32:18.420 INFO:tasks.workunit.client.0.smithi112.stderr:file3' 2023-01-24T03:32:18.420 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir .snap/snap1 2023-01-24T03:32:18.424 INFO:tasks.workunit.client.0.smithi112.stderr:+ '[' 'file1 2023-01-24T03:32:18.424 INFO:tasks.workunit.client.0.smithi112.stderr:file2 2023-01-24T03:32:18.424 INFO:tasks.workunit.client.0.smithi112.stderr:file3 2023-01-24T03:32:18.425 INFO:tasks.workunit.client.0.smithi112.stderr:file4' = 'file1 2023-01-24T03:32:18.425 INFO:tasks.workunit.client.0.smithi112.stderr:file2 2023-01-24T03:32:18.425 INFO:tasks.workunit.client.0.smithi112.stderr:file3' ']' 2023-01-24T03:32:18.425 INFO:tasks.workunit.client.0.smithi112.stderr:+ echo 1 2023-01-24T03:32:18.429 INFO:tasks.workunit.client.0.smithi112.stderr:+ echo 2 2023-01-24T03:32:18.646 INFO:tasks.workunit.client.0.smithi112.stderr:+ echo 3 2023-01-24T03:32:19.376 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir .snap/snap1 2023-01-24T03:32:19.655 INFO:tasks.workunit.client.0.smithi112.stderr:+ echo 4 2023-01-24T03:32:19.666 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir .snap/snap1 2023-01-24T03:32:19.673 INFO:tasks.workunit.client.0.smithi112.stderr:+ rm file1 file2 file3 file4 2023-01-24T03:32:19.679 INFO:tasks.workunit.client.0.smithi112.stdout:OK 2023-01-24T03:32:19.681 INFO:tasks.workunit.client.0.smithi112.stderr:+ echo OK 2023-01-24T03:32:19.681 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-01-24T03:32:19.681 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-01-24T03:32:19.756 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-2.sh... 2023-01-24T03:32:19.756 DEBUG:teuthology.orchestra.run.smithi112:workunit test fs/snaps/snaptest-2.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8387d427d91a301da19bd82e090b96e1348bfadb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-2.sh 2023-01-24T03:32:19.817 INFO:tasks.workunit.client.0.smithi112.stdout:Create dir 100 to 199 ... 2023-01-24T03:32:19.824 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 100 2023-01-24T03:32:19.835 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 101 2023-01-24T03:32:19.848 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 102 2023-01-24T03:32:19.859 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 103 2023-01-24T03:32:19.871 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 104 2023-01-24T03:32:20.146 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 105 2023-01-24T03:32:20.159 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 106 2023-01-24T03:32:20.173 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 107 2023-01-24T03:32:20.185 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 108 2023-01-24T03:32:20.198 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 109 2023-01-24T03:32:20.211 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 110 2023-01-24T03:32:20.222 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 111 2023-01-24T03:32:20.234 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 112 2023-01-24T03:32:20.245 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 113 2023-01-24T03:32:20.257 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 114 2023-01-24T03:32:20.270 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 115 2023-01-24T03:32:20.281 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 116 2023-01-24T03:32:20.295 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 117 2023-01-24T03:32:20.312 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 118 2023-01-24T03:32:20.322 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 119 2023-01-24T03:32:20.335 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 120 2023-01-24T03:32:20.347 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 121 2023-01-24T03:32:20.358 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 122 2023-01-24T03:32:20.370 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 123 2023-01-24T03:32:20.382 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 124 2023-01-24T03:32:21.378 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 125 2023-01-24T03:32:21.392 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 126 2023-01-24T03:32:21.405 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 127 2023-01-24T03:32:21.419 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 128 2023-01-24T03:32:21.433 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 129 2023-01-24T03:32:21.447 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 130 2023-01-24T03:32:22.395 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 131 2023-01-24T03:32:22.409 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 132 2023-01-24T03:32:22.423 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 133 2023-01-24T03:32:22.971 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 134 2023-01-24T03:32:22.984 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 135 2023-01-24T03:32:22.998 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 136 2023-01-24T03:32:23.012 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 137 2023-01-24T03:32:23.536 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 138 2023-01-24T03:32:23.550 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 139 2023-01-24T03:32:23.563 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 140 2023-01-24T03:32:23.577 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 141 2023-01-24T03:32:23.591 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 142 2023-01-24T03:32:23.609 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 143 2023-01-24T03:32:23.618 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 144 2023-01-24T03:32:23.631 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 145 2023-01-24T03:32:23.644 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 146 2023-01-24T03:32:23.655 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 147 2023-01-24T03:32:24.276 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 148 2023-01-24T03:32:24.291 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 149 2023-01-24T03:32:24.304 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 150 2023-01-24T03:32:24.580 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 151 2023-01-24T03:32:24.596 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 152 2023-01-24T03:32:24.611 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 153 2023-01-24T03:32:24.628 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 154 2023-01-24T03:32:24.646 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 155 2023-01-24T03:32:24.662 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 156 2023-01-24T03:32:24.676 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 157 2023-01-24T03:32:24.687 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 158 2023-01-24T03:32:24.699 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 159 2023-01-24T03:32:24.711 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 160 2023-01-24T03:32:24.722 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 161 2023-01-24T03:32:24.734 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 162 2023-01-24T03:32:24.745 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 163 2023-01-24T03:32:24.757 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 164 2023-01-24T03:32:24.770 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 165 2023-01-24T03:32:24.781 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 166 2023-01-24T03:32:24.792 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 167 2023-01-24T03:32:24.803 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 168 2023-01-24T03:32:24.814 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 169 2023-01-24T03:32:24.826 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 170 2023-01-24T03:32:24.836 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 171 2023-01-24T03:32:24.848 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 172 2023-01-24T03:32:24.859 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 173 2023-01-24T03:32:24.871 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 174 2023-01-24T03:32:25.663 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 175 2023-01-24T03:32:26.452 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 176 2023-01-24T03:32:26.466 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 177 2023-01-24T03:32:26.480 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 178 2023-01-24T03:32:26.492 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 179 2023-01-24T03:32:26.505 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 180 2023-01-24T03:32:26.518 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 181 2023-01-24T03:32:26.530 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 182 2023-01-24T03:32:26.543 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 183 2023-01-24T03:32:26.556 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 184 2023-01-24T03:32:26.568 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 185 2023-01-24T03:32:26.581 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 186 2023-01-24T03:32:26.593 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 187 2023-01-24T03:32:26.606 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 188 2023-01-24T03:32:26.618 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 189 2023-01-24T03:32:26.631 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 190 2023-01-24T03:32:26.908 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 191 2023-01-24T03:32:26.920 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 192 2023-01-24T03:32:26.933 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 193 2023-01-24T03:32:26.945 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 194 2023-01-24T03:32:27.217 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 195 2023-01-24T03:32:27.497 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 196 2023-01-24T03:32:27.774 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 197 2023-01-24T03:32:27.788 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 198 2023-01-24T03:32:27.802 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 199 2023-01-24T03:32:27.816 INFO:tasks.workunit.client.0.smithi112.stdout:Take first snapshot .snap/test1 2023-01-24T03:32:28.421 INFO:tasks.workunit.client.0.smithi112.stdout:Create dir 200 to 299 ... 2023-01-24T03:32:28.422 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 200 2023-01-24T03:32:28.435 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 201 2023-01-24T03:32:28.448 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 202 2023-01-24T03:32:28.460 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 203 2023-01-24T03:32:28.472 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 204 2023-01-24T03:32:28.484 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 205 2023-01-24T03:32:28.497 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 206 2023-01-24T03:32:28.507 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 207 2023-01-24T03:32:29.067 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 208 2023-01-24T03:32:29.080 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:32:29.081 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 209 2023-01-24T03:32:29.083 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:32:29.092 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 210 2023-01-24T03:32:29.102 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 211 2023-01-24T03:32:29.115 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 212 2023-01-24T03:32:29.364 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 213 2023-01-24T03:32:30.007 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 214 2023-01-24T03:32:30.018 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 215 2023-01-24T03:32:30.033 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 216 2023-01-24T03:32:30.700 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 217 2023-01-24T03:32:30.712 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 218 2023-01-24T03:32:30.725 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 219 2023-01-24T03:32:30.738 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 220 2023-01-24T03:32:31.012 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 221 2023-01-24T03:32:31.026 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 222 2023-01-24T03:32:31.039 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 223 2023-01-24T03:32:31.052 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 224 2023-01-24T03:32:31.065 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 225 2023-01-24T03:32:31.080 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 226 2023-01-24T03:32:31.089 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 227 2023-01-24T03:32:31.100 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 228 2023-01-24T03:32:31.113 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 229 2023-01-24T03:32:31.384 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 230 2023-01-24T03:32:31.395 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 231 2023-01-24T03:32:31.406 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 232 2023-01-24T03:32:31.418 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 233 2023-01-24T03:32:31.430 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 234 2023-01-24T03:32:31.443 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 235 2023-01-24T03:32:31.455 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 236 2023-01-24T03:32:31.467 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 237 2023-01-24T03:32:31.479 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 238 2023-01-24T03:32:31.491 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 239 2023-01-24T03:32:31.502 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 240 2023-01-24T03:32:31.514 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 241 2023-01-24T03:32:31.525 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 242 2023-01-24T03:32:31.537 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 243 2023-01-24T03:32:31.548 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 244 2023-01-24T03:32:31.559 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 245 2023-01-24T03:32:31.569 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 246 2023-01-24T03:32:31.580 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 247 2023-01-24T03:32:31.591 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 248 2023-01-24T03:32:31.602 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 249 2023-01-24T03:32:31.612 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 250 2023-01-24T03:32:31.628 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 251 2023-01-24T03:32:34.932 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 252 2023-01-24T03:32:34.949 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 253 2023-01-24T03:32:34.964 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 254 2023-01-24T03:32:35.236 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 255 2023-01-24T03:32:35.249 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 256 2023-01-24T03:32:35.261 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 257 2023-01-24T03:32:35.274 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 258 2023-01-24T03:32:35.683 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 259 2023-01-24T03:32:35.696 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 260 2023-01-24T03:32:35.709 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 261 2023-01-24T03:32:35.722 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 262 2023-01-24T03:32:35.733 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 263 2023-01-24T03:32:35.745 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 264 2023-01-24T03:32:35.756 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 265 2023-01-24T03:32:35.767 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 266 2023-01-24T03:32:36.570 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 267 2023-01-24T03:32:36.584 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 268 2023-01-24T03:32:36.597 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 269 2023-01-24T03:32:36.610 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 270 2023-01-24T03:32:36.622 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 271 2023-01-24T03:32:36.634 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 272 2023-01-24T03:32:36.647 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 273 2023-01-24T03:32:36.660 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 274 2023-01-24T03:32:37.163 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 275 2023-01-24T03:32:37.177 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 276 2023-01-24T03:32:37.191 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 277 2023-01-24T03:32:37.204 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 278 2023-01-24T03:32:37.217 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 279 2023-01-24T03:32:37.230 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 280 2023-01-24T03:32:37.243 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 281 2023-01-24T03:32:37.256 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 282 2023-01-24T03:32:37.269 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 283 2023-01-24T03:32:37.282 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 284 2023-01-24T03:32:37.295 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 285 2023-01-24T03:32:37.308 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 286 2023-01-24T03:32:37.321 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 287 2023-01-24T03:32:37.334 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 288 2023-01-24T03:32:37.347 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 289 2023-01-24T03:32:37.359 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 290 2023-01-24T03:32:37.371 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 291 2023-01-24T03:32:37.829 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 292 2023-01-24T03:32:37.979 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 293 2023-01-24T03:32:37.992 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 294 2023-01-24T03:32:38.005 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 295 2023-01-24T03:32:38.018 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 296 2023-01-24T03:32:38.031 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 297 2023-01-24T03:32:38.044 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 298 2023-01-24T03:32:38.058 INFO:tasks.workunit.client.0.smithi112.stdout: create dir 299 2023-01-24T03:32:38.071 INFO:tasks.workunit.client.0.smithi112.stdout:Create a snapshot in every first level dir ... 2023-01-24T03:32:38.395 INFO:tasks.workunit.client.0.smithi112.stdout: create 100/.snap/snap-subdir-test 2023-01-24T03:32:38.431 INFO:tasks.workunit.client.0.smithi112.stdout: create 100/30 file after the snapshot 2023-01-24T03:32:38.432 INFO:tasks.workunit.client.0.smithi112.stdout: create 100/31 file after the snapshot 2023-01-24T03:32:38.433 INFO:tasks.workunit.client.0.smithi112.stdout: create 100/32 file after the snapshot 2023-01-24T03:32:38.434 INFO:tasks.workunit.client.0.smithi112.stdout: create 100/33 file after the snapshot 2023-01-24T03:32:38.435 INFO:tasks.workunit.client.0.smithi112.stdout: create 100/34 file after the snapshot 2023-01-24T03:32:38.436 INFO:tasks.workunit.client.0.smithi112.stdout: create 100/35 file after the snapshot 2023-01-24T03:32:38.437 INFO:tasks.workunit.client.0.smithi112.stdout: create 100/36 file after the snapshot 2023-01-24T03:32:38.438 INFO:tasks.workunit.client.0.smithi112.stdout: create 100/37 file after the snapshot 2023-01-24T03:32:38.439 INFO:tasks.workunit.client.0.smithi112.stdout: create 100/38 file after the snapshot 2023-01-24T03:32:38.440 INFO:tasks.workunit.client.0.smithi112.stdout: create 100/39 file after the snapshot 2023-01-24T03:32:38.441 INFO:tasks.workunit.client.0.smithi112.stdout: create 101/.snap/snap-subdir-test 2023-01-24T03:32:38.479 INFO:tasks.workunit.client.0.smithi112.stdout: create 101/30 file after the snapshot 2023-01-24T03:32:38.480 INFO:tasks.workunit.client.0.smithi112.stdout: create 101/31 file after the snapshot 2023-01-24T03:32:38.481 INFO:tasks.workunit.client.0.smithi112.stdout: create 101/32 file after the snapshot 2023-01-24T03:32:38.482 INFO:tasks.workunit.client.0.smithi112.stdout: create 101/33 file after the snapshot 2023-01-24T03:32:38.483 INFO:tasks.workunit.client.0.smithi112.stdout: create 101/34 file after the snapshot 2023-01-24T03:32:38.484 INFO:tasks.workunit.client.0.smithi112.stdout: create 101/35 file after the snapshot 2023-01-24T03:32:38.485 INFO:tasks.workunit.client.0.smithi112.stdout: create 101/36 file after the snapshot 2023-01-24T03:32:38.486 INFO:tasks.workunit.client.0.smithi112.stdout: create 101/37 file after the snapshot 2023-01-24T03:32:38.487 INFO:tasks.workunit.client.0.smithi112.stdout: create 101/38 file after the snapshot 2023-01-24T03:32:38.487 INFO:tasks.workunit.client.0.smithi112.stdout: create 101/39 file after the snapshot 2023-01-24T03:32:38.488 INFO:tasks.workunit.client.0.smithi112.stdout: create 102/.snap/snap-subdir-test 2023-01-24T03:32:38.530 INFO:tasks.workunit.client.0.smithi112.stdout: create 102/30 file after the snapshot 2023-01-24T03:32:38.531 INFO:tasks.workunit.client.0.smithi112.stdout: create 102/31 file after the snapshot 2023-01-24T03:32:38.531 INFO:tasks.workunit.client.0.smithi112.stdout: create 102/32 file after the snapshot 2023-01-24T03:32:38.532 INFO:tasks.workunit.client.0.smithi112.stdout: create 102/33 file after the snapshot 2023-01-24T03:32:38.533 INFO:tasks.workunit.client.0.smithi112.stdout: create 102/34 file after the snapshot 2023-01-24T03:32:38.534 INFO:tasks.workunit.client.0.smithi112.stdout: create 102/35 file after the snapshot 2023-01-24T03:32:38.535 INFO:tasks.workunit.client.0.smithi112.stdout: create 102/36 file after the snapshot 2023-01-24T03:32:38.536 INFO:tasks.workunit.client.0.smithi112.stdout: create 102/37 file after the snapshot 2023-01-24T03:32:38.537 INFO:tasks.workunit.client.0.smithi112.stdout: create 102/38 file after the snapshot 2023-01-24T03:32:38.538 INFO:tasks.workunit.client.0.smithi112.stdout: create 102/39 file after the snapshot 2023-01-24T03:32:38.539 INFO:tasks.workunit.client.0.smithi112.stdout: create 103/.snap/snap-subdir-test 2023-01-24T03:32:38.577 INFO:tasks.workunit.client.0.smithi112.stdout: create 103/30 file after the snapshot 2023-01-24T03:32:38.578 INFO:tasks.workunit.client.0.smithi112.stdout: create 103/31 file after the snapshot 2023-01-24T03:32:38.579 INFO:tasks.workunit.client.0.smithi112.stdout: create 103/32 file after the snapshot 2023-01-24T03:32:38.724 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:32:38.844 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.841+0000 7f7a5cbc8700 1 Processor -- start 2023-01-24T03:32:38.844 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.841+0000 7f7a5cbc8700 1 -- start start 2023-01-24T03:32:38.847 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.842+0000 7f7a5cbc8700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f7a580c7e40 0x7f7a580b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:38.847 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.842+0000 7f7a5cbc8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a58155da0 0x7f7a580ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:38.847 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.842+0000 7f7a5cbc8700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f7a580aef20 0x7f7a580ca3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:38.847 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.842+0000 7f7a5cbc8700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a58156240 con 0x7f7a580c7e40 2023-01-24T03:32:38.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.842+0000 7f7a5cbc8700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a580b1170 con 0x7f7a580aef20 2023-01-24T03:32:38.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.842+0000 7f7a5cbc8700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a580b1450 con 0x7f7a58155da0 2023-01-24T03:32:38.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.842+0000 7f7a5659c700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f7a580c7e40 0x7f7a580b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:38.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.842+0000 7f7a5659c700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f7a580c7e40 0x7f7a580b0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51826/0 (socket says 172.21.15.112:51826) 2023-01-24T03:32:38.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.842+0000 7f7a5659c700 1 -- 172.21.15.112:0/1748154287 learned_addr learned my addr 172.21.15.112:0/1748154287 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:32:38.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.842+0000 7f7a55d9b700 1 --2- 172.21.15.112:0/1748154287 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a58155da0 0x7f7a580ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:38.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.842+0000 7f7a56d9d700 1 --2- 172.21.15.112:0/1748154287 >> v2:172.21.15.142:3300/0 conn(0x7f7a580aef20 0x7f7a580ca3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:38.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.842+0000 7f7a5659c700 1 -- 172.21.15.112:0/1748154287 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a58155da0 msgr2=0x7f7a580ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:32:38.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.842+0000 7f7a5659c700 1 --2- 172.21.15.112:0/1748154287 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a58155da0 0x7f7a580ae9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:38.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.842+0000 7f7a5659c700 1 -- 172.21.15.112:0/1748154287 >> v2:172.21.15.142:3300/0 conn(0x7f7a580aef20 msgr2=0x7f7a580ca3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:32:38.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.842+0000 7f7a5659c700 1 --2- 172.21.15.112:0/1748154287 >> v2:172.21.15.142:3300/0 conn(0x7f7a580aef20 0x7f7a580ca3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:38.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.843+0000 7f7a5659c700 1 -- 172.21.15.112:0/1748154287 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a580b1730 con 0x7f7a580c7e40 2023-01-24T03:32:38.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.843+0000 7f7a5659c700 1 --2- 172.21.15.112:0/1748154287 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7a580c7e40 0x7f7a580b0aa0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7a58089ca0 tx=0x7f7a480090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48ecc24a3bdd58ca server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:38.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.843+0000 7f7a5559a700 1 -- 172.21.15.112:0/1748154287 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a48011630 con 0x7f7a580c7e40 2023-01-24T03:32:38.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.843+0000 7f7a5559a700 1 -- 172.21.15.112:0/1748154287 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a480117d0 con 0x7f7a580c7e40 2023-01-24T03:32:38.851 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.843+0000 7f7a5559a700 1 -- 172.21.15.112:0/1748154287 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a48011b90 con 0x7f7a580c7e40 2023-01-24T03:32:38.851 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.843+0000 7f7a5cbc8700 1 -- 172.21.15.112:0/1748154287 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7a580c7e40 msgr2=0x7f7a580b0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:38.851 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.843+0000 7f7a5cbc8700 1 --2- 172.21.15.112:0/1748154287 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7a580c7e40 0x7f7a580b0aa0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7a58089ca0 tx=0x7f7a480090f0 comp rx=0 tx=0).stop 2023-01-24T03:32:38.851 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.843+0000 7f7a5cbc8700 1 -- 172.21.15.112:0/1748154287 shutdown_connections 2023-01-24T03:32:38.851 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.843+0000 7f7a5cbc8700 1 --2- 172.21.15.112:0/1748154287 >> v2:172.21.15.142:3300/0 conn(0x7f7a580aef20 0x7f7a580ca3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:38.852 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.843+0000 7f7a5cbc8700 1 --2- 172.21.15.112:0/1748154287 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7a580c7e40 0x7f7a580b0aa0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:38.852 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.843+0000 7f7a5cbc8700 1 --2- 172.21.15.112:0/1748154287 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a58155da0 0x7f7a580ae9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:38.852 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.843+0000 7f7a5cbc8700 1 -- 172.21.15.112:0/1748154287 >> 172.21.15.112:0/1748154287 conn(0x7f7a5805e2c0 msgr2=0x7f7a580cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:32:38.852 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.843+0000 7f7a5cbc8700 1 -- 172.21.15.112:0/1748154287 shutdown_connections 2023-01-24T03:32:38.852 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.843+0000 7f7a5cbc8700 1 -- 172.21.15.112:0/1748154287 wait complete. 2023-01-24T03:32:38.856 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.844+0000 7f7a5cbc8700 1 Processor -- start 2023-01-24T03:32:38.856 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.844+0000 7f7a5cbc8700 1 -- start start 2023-01-24T03:32:38.856 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.845+0000 7f7a5cbc8700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7a580aef20 0x7f7a581521e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:38.856 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.845+0000 7f7a5cbc8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a580c7e40 0x7f7a580c1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:38.856 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.845+0000 7f7a5cbc8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a58155da0 0x7f7a580c2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:38.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.845+0000 7f7a5cbc8700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a58151b70 con 0x7f7a580aef20 2023-01-24T03:32:38.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.845+0000 7f7a5cbc8700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a58169900 con 0x7f7a58155da0 2023-01-24T03:32:38.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.845+0000 7f7a5cbc8700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a58168d80 con 0x7f7a580c7e40 2023-01-24T03:32:38.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.845+0000 7f7a5659c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7a580aef20 0x7f7a581521e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:38.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.845+0000 7f7a5659c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7a580aef20 0x7f7a581521e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51834/0 (socket says 172.21.15.112:51834) 2023-01-24T03:32:38.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.845+0000 7f7a5659c700 1 -- 172.21.15.112:0/2165624774 learned_addr learned my addr 172.21.15.112:0/2165624774 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:32:38.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.845+0000 7f7a55d9b700 1 --2- 172.21.15.112:0/2165624774 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a580c7e40 0x7f7a580c1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:38.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.845+0000 7f7a56d9d700 1 --2- 172.21.15.112:0/2165624774 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a58155da0 0x7f7a580c2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:38.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.845+0000 7f7a5659c700 1 -- 172.21.15.112:0/2165624774 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a580c7e40 msgr2=0x7f7a580c1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:38.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.845+0000 7f7a5659c700 1 --2- 172.21.15.112:0/2165624774 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a580c7e40 0x7f7a580c1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:38.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.845+0000 7f7a5659c700 1 -- 172.21.15.112:0/2165624774 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a58155da0 msgr2=0x7f7a580c2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:38.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.845+0000 7f7a5659c700 1 --2- 172.21.15.112:0/2165624774 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a58155da0 0x7f7a580c2520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:38.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.845+0000 7f7a5659c700 1 -- 172.21.15.112:0/2165624774 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a48009370 con 0x7f7a580aef20 2023-01-24T03:32:38.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.845+0000 7f7a55d9b700 1 --2- 172.21.15.112:0/2165624774 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a580c7e40 0x7f7a580c1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:32:38.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.846+0000 7f7a5659c700 1 --2- 172.21.15.112:0/2165624774 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7a580aef20 0x7f7a581521e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7a48002000 tx=0x7f7a48016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:38.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.846+0000 7f7a467fc700 1 -- 172.21.15.112:0/2165624774 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a48011b50 con 0x7f7a580aef20 2023-01-24T03:32:38.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.846+0000 7f7a467fc700 1 -- 172.21.15.112:0/2165624774 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a48020070 con 0x7f7a580aef20 2023-01-24T03:32:38.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.846+0000 7f7a467fc700 1 -- 172.21.15.112:0/2165624774 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a480276b0 con 0x7f7a580aef20 2023-01-24T03:32:38.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.846+0000 7f7a5cbc8700 1 -- 172.21.15.112:0/2165624774 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a58169060 con 0x7f7a580aef20 2023-01-24T03:32:38.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.846+0000 7f7a5cbc8700 1 -- 172.21.15.112:0/2165624774 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a58169620 con 0x7f7a580aef20 2023-01-24T03:32:38.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.847+0000 7f7a5cbc8700 1 -- 172.21.15.112:0/2165624774 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a580c0800 con 0x7f7a580aef20 2023-01-24T03:32:38.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.847+0000 7f7a467fc700 1 -- 172.21.15.112:0/2165624774 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f7a48017020 con 0x7f7a580aef20 2023-01-24T03:32:38.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.847+0000 7f7a467fc700 1 --2- 172.21.15.112:0/2165624774 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7a340611c0 0x7f7a34063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:38.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.847+0000 7f7a467fc700 1 -- 172.21.15.112:0/2165624774 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 7009+0+0 (secure 0 0 0) 0x7f7a48022070 con 0x7f7a580aef20 2023-01-24T03:32:38.862 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.847+0000 7f7a55d9b700 1 --2- 172.21.15.112:0/2165624774 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7a340611c0 0x7f7a34063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:38.862 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.848+0000 7f7a55d9b700 1 --2- 172.21.15.112:0/2165624774 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7a340611c0 0x7f7a34063670 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7a580cb290 tx=0x7f7a4c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:38.862 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:38.851+0000 7f7a467fc700 1 -- 172.21.15.112:0/2165624774 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f7a4804f460 con 0x7f7a580aef20 2023-01-24T03:32:39.075 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.072+0000 7f7a5cbc8700 1 -- 172.21.15.112:0/2165624774 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a580692e0 con 0x7f7a580aef20 2023-01-24T03:32:39.075 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.072+0000 7f7a467fc700 1 -- 172.21.15.112:0/2165624774 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+6112 (secure 0 0 0) 0x7f7a48052a80 con 0x7f7a580aef20 2023-01-24T03:32:39.076 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:32:39.076 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4897,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/11022091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":11022091},{"type":"v1","addr":"172.21.15.142:6835","nonce":11022091}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:32:09.626128+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4288},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":0,"incarnation":11,"state":"up:active","state_seq":31,"addr":"172.21.15.142:6834/869267413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":869267413},{"type":"v1","addr":"172.21.15.142:6834","nonce":869267413}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:32:39.078 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.074+0000 7f7a5cbc8700 1 -- 172.21.15.112:0/2165624774 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7a340611c0 msgr2=0x7f7a34063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:39.078 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.075+0000 7f7a5cbc8700 1 --2- 172.21.15.112:0/2165624774 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7a340611c0 0x7f7a34063670 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7a580cb290 tx=0x7f7a4c008040 comp rx=0 tx=0).stop 2023-01-24T03:32:39.078 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.075+0000 7f7a5cbc8700 1 -- 172.21.15.112:0/2165624774 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7a580aef20 msgr2=0x7f7a581521e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:39.078 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.075+0000 7f7a5cbc8700 1 --2- 172.21.15.112:0/2165624774 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7a580aef20 0x7f7a581521e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7a48002000 tx=0x7f7a48016000 comp rx=0 tx=0).stop 2023-01-24T03:32:39.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.075+0000 7f7a5cbc8700 1 -- 172.21.15.112:0/2165624774 shutdown_connections 2023-01-24T03:32:39.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.075+0000 7f7a5cbc8700 1 --2- 172.21.15.112:0/2165624774 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f7a340611c0 0x7f7a34063670 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:39.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.075+0000 7f7a5cbc8700 1 --2- 172.21.15.112:0/2165624774 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7a580aef20 0x7f7a581521e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:39.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.075+0000 7f7a5cbc8700 1 --2- 172.21.15.112:0/2165624774 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7a58155da0 0x7f7a580c2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:39.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.075+0000 7f7a5cbc8700 1 --2- 172.21.15.112:0/2165624774 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f7a580c7e40 0x7f7a580c1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:39.080 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.075+0000 7f7a5cbc8700 1 -- 172.21.15.112:0/2165624774 >> 172.21.15.112:0/2165624774 conn(0x7f7a5805e2c0 msgr2=0x7f7a581551d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:32:39.080 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.075+0000 7f7a5cbc8700 1 -- 172.21.15.112:0/2165624774 shutdown_connections 2023-01-24T03:32:39.080 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.075+0000 7f7a5cbc8700 1 -- 172.21.15.112:0/2165624774 wait complete. 2023-01-24T03:32:39.081 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 15 2023-01-24T03:32:39.089 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2023-01-24T03:32:39.089 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-01-24T03:32:39.090 INFO:teuthology.orchestra.run:waiting for 300 2023-01-24T03:32:39.100 INFO:tasks.ceph.mds.b:Stopped 2023-01-24T03:32:39.100 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:32:39.212 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.209+0000 7f8d73ca8700 1 Processor -- start 2023-01-24T03:32:39.212 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.209+0000 7f8d73ca8700 1 -- start start 2023-01-24T03:32:39.218 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.210+0000 7f8d73ca8700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f8d6c0b0aa0 0x7f8d6c155da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:39.218 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.210+0000 7f8d73ca8700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d6c0b0f40 con 0x7f8d6c0c7e40 2023-01-24T03:32:39.218 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.210+0000 7f8d73ca8700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d6c0b1170 con 0x7f8d6c0ca3e0 2023-01-24T03:32:39.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.210+0000 7f8d73ca8700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d6c0b1450 con 0x7f8d6c0b0aa0 2023-01-24T03:32:39.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.210+0000 7f8d71a44700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f8d6c0c7e40 0x7f8d6c0ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:32828/0 (socket says 172.21.15.112:32828) 2023-01-24T03:32:39.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.210+0000 7f8d71a44700 1 -- 172.21.15.112:0/3779256576 learned_addr learned my addr 172.21.15.112:0/3779256576 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:32:39.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.210+0000 7f8d71243700 1 --2- 172.21.15.112:0/3779256576 >> v2:172.21.15.142:3300/0 conn(0x7f8d6c0b0aa0 0x7f8d6c155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:39.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.210+0000 7f8d70a42700 1 -- 172.21.15.112:0/3779256576 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3367022454 0 0) 0x7f8d6c0b0f40 con 0x7f8d6c0c7e40 2023-01-24T03:32:39.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.210+0000 7f8d70a42700 1 -- 172.21.15.112:0/3779256576 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d580033f0 con 0x7f8d6c0c7e40 2023-01-24T03:32:39.221 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.211+0000 7f8d70a42700 1 -- 172.21.15.112:0/3779256576 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 811931617 0 0) 0x7f8d580033f0 con 0x7f8d6c0c7e40 2023-01-24T03:32:39.221 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.211+0000 7f8d70a42700 1 -- 172.21.15.112:0/3779256576 >> v2:172.21.15.142:3300/0 conn(0x7f8d6c0b0aa0 msgr2=0x7f8d6c155da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:32:39.221 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.211+0000 7f8d70a42700 1 --2- 172.21.15.112:0/3779256576 >> v2:172.21.15.142:3300/0 conn(0x7f8d6c0b0aa0 0x7f8d6c155da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:39.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.211+0000 7f8d70a42700 1 -- 172.21.15.112:0/3779256576 >> v1:172.21.15.142:6789/0 conn(0x7f8d6c0ca3e0 legacy=0x7f8d6c0ca7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:39.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.211+0000 7f8d70a42700 1 -- 172.21.15.112:0/3779256576 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d6c0b1730 con 0x7f8d6c0c7e40 2023-01-24T03:32:39.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.211+0000 7f8d70a42700 1 -- 172.21.15.112:0/3779256576 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7f8d5c0017f0 con 0x7f8d6c0c7e40 2023-01-24T03:32:39.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.211+0000 7f8d70a42700 1 -- 172.21.15.112:0/3779256576 >> v1:172.21.15.112:6789/0 conn(0x7f8d6c0c7e40 legacy=0x7f8d6c0ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:39.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.211+0000 7f8d70a42700 1 --2- 172.21.15.112:0/3779256576 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8d580038e0 0x7f8d58003cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:39.223 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.211+0000 7f8d70a42700 1 --2- 172.21.15.112:0/3779256576 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d58004690 0x7f8d58004b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:39.223 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.211+0000 7f8d70a42700 1 --2- 172.21.15.112:0/3779256576 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8d58005040 0x7f8d5800b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:39.223 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.211+0000 7f8d70a42700 1 -- 172.21.15.112:0/3779256576 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d6c0b1450 con 0x7f8d58004690 2023-01-24T03:32:39.223 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.211+0000 7f8d70a42700 1 -- 172.21.15.112:0/3779256576 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5800bfe0 con 0x7f8d58005040 2023-01-24T03:32:39.223 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.211+0000 7f8d70a42700 1 -- 172.21.15.112:0/3779256576 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5800c1a0 con 0x7f8d580038e0 2023-01-24T03:32:39.224 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.211+0000 7f8d71243700 1 --2- 172.21.15.112:0/3779256576 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d58004690 0x7f8d58004b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:39.224 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.211+0000 7f8d72245700 1 --2- 172.21.15.112:0/3779256576 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8d58005040 0x7f8d5800b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:39.224 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.211+0000 7f8d71a44700 1 --2- 172.21.15.112:0/3779256576 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8d580038e0 0x7f8d58003cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:39.224 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.211+0000 7f8d71243700 1 -- 172.21.15.112:0/3779256576 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8d580038e0 msgr2=0x7f8d58003cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:32:39.224 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.211+0000 7f8d71243700 1 --2- 172.21.15.112:0/3779256576 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8d580038e0 0x7f8d58003cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:39.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.211+0000 7f8d71243700 1 -- 172.21.15.112:0/3779256576 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8d58005040 msgr2=0x7f8d5800b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:32:39.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.211+0000 7f8d71243700 1 --2- 172.21.15.112:0/3779256576 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8d58005040 0x7f8d5800b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:39.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.211+0000 7f8d71243700 1 -- 172.21.15.112:0/3779256576 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d580033f0 con 0x7f8d58004690 2023-01-24T03:32:39.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.211+0000 7f8d72245700 1 -- 172.21.15.112:0/3779256576 reap_dead start 2023-01-24T03:32:39.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.211+0000 7f8d71a44700 1 --2- 172.21.15.112:0/3779256576 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8d580038e0 0x7f8d58003cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:32:39.226 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.211+0000 7f8d71243700 1 --2- 172.21.15.112:0/3779256576 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d58004690 0x7f8d58004b00 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8d60009a80 tx=0x7f8d60009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed7809ee91cbb029 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:39.226 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.211+0000 7f8d70a42700 1 -- 172.21.15.112:0/3779256576 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d60009430 con 0x7f8d58004690 2023-01-24T03:32:39.226 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.211+0000 7f8d70a42700 1 -- 172.21.15.112:0/3779256576 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d6000e640 con 0x7f8d58004690 2023-01-24T03:32:39.226 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.211+0000 7f8d70a42700 1 -- 172.21.15.112:0/3779256576 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6000ea00 con 0x7f8d58004690 2023-01-24T03:32:39.226 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.212+0000 7f8d73ca8700 1 -- 172.21.15.112:0/3779256576 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d58004690 msgr2=0x7f8d58004b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:39.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.212+0000 7f8d73ca8700 1 --2- 172.21.15.112:0/3779256576 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d58004690 0x7f8d58004b00 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8d60009a80 tx=0x7f8d60009060 comp rx=0 tx=0).stop 2023-01-24T03:32:39.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.212+0000 7f8d73ca8700 1 -- 172.21.15.112:0/3779256576 shutdown_connections 2023-01-24T03:32:39.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.212+0000 7f8d73ca8700 1 --2- 172.21.15.112:0/3779256576 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d58004690 0x7f8d58004b00 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:39.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.212+0000 7f8d73ca8700 1 -- 172.21.15.112:0/3779256576 >> 172.21.15.112:0/3779256576 conn(0x7f8d6c05e2c0 msgr2=0x7f8d6c0cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:32:39.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.212+0000 7f8d73ca8700 1 -- 172.21.15.112:0/3779256576 shutdown_connections 2023-01-24T03:32:39.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.212+0000 7f8d73ca8700 1 -- 172.21.15.112:0/3779256576 wait complete. 2023-01-24T03:32:39.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.213+0000 7f8d73ca8700 1 Processor -- start 2023-01-24T03:32:39.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.213+0000 7f8d73ca8700 1 -- start start 2023-01-24T03:32:39.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.214+0000 7f8d73ca8700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d58004690 0x7f8d6c1524b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:39.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.214+0000 7f8d73ca8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8d6c0aeba0 0x7f8d6c0bfe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:39.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.214+0000 7f8d73ca8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8d6c0aeff0 0x7f8d6c0c0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:39.229 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.214+0000 7f8d73ca8700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d6c0bf970 con 0x7f8d58004690 2023-01-24T03:32:39.229 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.214+0000 7f8d73ca8700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d6c0c7050 con 0x7f8d6c0aeff0 2023-01-24T03:32:39.229 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.214+0000 7f8d73ca8700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d6c1659d0 con 0x7f8d6c0aeba0 2023-01-24T03:32:39.229 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.214+0000 7f8d71a44700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d58004690 0x7f8d6c1524b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:39.229 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.214+0000 7f8d71a44700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d58004690 0x7f8d6c1524b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51852/0 (socket says 172.21.15.112:51852) 2023-01-24T03:32:39.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.214+0000 7f8d71a44700 1 -- 172.21.15.112:0/3648390491 learned_addr learned my addr 172.21.15.112:0/3648390491 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:32:39.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.214+0000 7f8d71243700 1 --2- 172.21.15.112:0/3648390491 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8d6c0aeba0 0x7f8d6c0bfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:39.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.214+0000 7f8d72245700 1 --2- 172.21.15.112:0/3648390491 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8d6c0aeff0 0x7f8d6c0c0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:39.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.214+0000 7f8d71a44700 1 -- 172.21.15.112:0/3648390491 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8d6c0aeba0 msgr2=0x7f8d6c0bfe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:39.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.214+0000 7f8d71a44700 1 --2- 172.21.15.112:0/3648390491 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8d6c0aeba0 0x7f8d6c0bfe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:39.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.214+0000 7f8d71a44700 1 -- 172.21.15.112:0/3648390491 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8d6c0aeff0 msgr2=0x7f8d6c0c0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:39.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.214+0000 7f8d71a44700 1 --2- 172.21.15.112:0/3648390491 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8d6c0aeff0 0x7f8d6c0c0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:39.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.214+0000 7f8d71a44700 1 -- 172.21.15.112:0/3648390491 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d6c165cb0 con 0x7f8d58004690 2023-01-24T03:32:39.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.214+0000 7f8d71243700 1 --2- 172.21.15.112:0/3648390491 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8d6c0aeba0 0x7f8d6c0bfe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:32:39.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.214+0000 7f8d71a44700 1 --2- 172.21.15.112:0/3648390491 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d58004690 0x7f8d6c1524b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c007700 tx=0x7f8d5c001880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:39.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.214+0000 7f8d69ffb700 1 -- 172.21.15.112:0/3648390491 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c00e960 con 0x7f8d58004690 2023-01-24T03:32:39.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.214+0000 7f8d69ffb700 1 -- 172.21.15.112:0/3648390491 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d5c00eac0 con 0x7f8d58004690 2023-01-24T03:32:39.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.214+0000 7f8d73ca8700 1 -- 172.21.15.112:0/3648390491 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d6c165f90 con 0x7f8d58004690 2023-01-24T03:32:39.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.214+0000 7f8d69ffb700 1 -- 172.21.15.112:0/3648390491 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c00ee40 con 0x7f8d58004690 2023-01-24T03:32:39.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.215+0000 7f8d73ca8700 1 -- 172.21.15.112:0/3648390491 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d6c166550 con 0x7f8d58004690 2023-01-24T03:32:39.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.215+0000 7f8d73ca8700 1 -- 172.21.15.112:0/3648390491 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d6c0bfc50 con 0x7f8d58004690 2023-01-24T03:32:39.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.216+0000 7f8d69ffb700 1 -- 172.21.15.112:0/3648390491 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f8d5c013630 con 0x7f8d58004690 2023-01-24T03:32:39.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.216+0000 7f8d69ffb700 1 --2- 172.21.15.112:0/3648390491 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f8d4c0611c0 0x7f8d4c063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:39.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.216+0000 7f8d69ffb700 1 -- 172.21.15.112:0/3648390491 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 7009+0+0 (secure 0 0 0) 0x7f8d5c07dda0 con 0x7f8d58004690 2023-01-24T03:32:39.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.216+0000 7f8d71243700 1 --2- 172.21.15.112:0/3648390491 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f8d4c0611c0 0x7f8d4c063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:39.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.216+0000 7f8d71243700 1 --2- 172.21.15.112:0/3648390491 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f8d4c0611c0 0x7f8d4c063670 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c0c16f0 tx=0x7f8d6001a000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:39.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.220+0000 7f8d69ffb700 1 -- 172.21.15.112:0/3648390491 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f8d5c04dbd0 con 0x7f8d58004690 2023-01-24T03:32:39.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.441+0000 7f8d73ca8700 1 -- 172.21.15.112:0/3648390491 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d6c0bf760 con 0x7f8d58004690 2023-01-24T03:32:39.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.442+0000 7f8d69ffb700 1 -- 172.21.15.112:0/3648390491 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+6112 (secure 0 0 0) 0x7f8d5c0511f0 con 0x7f8d58004690 2023-01-24T03:32:39.445 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:32:39.446 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4897,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/11022091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":11022091},{"type":"v1","addr":"172.21.15.142:6835","nonce":11022091}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:32:09.626128+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4288},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":0,"incarnation":11,"state":"up:active","state_seq":31,"addr":"172.21.15.142:6834/869267413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":869267413},{"type":"v1","addr":"172.21.15.142:6834","nonce":869267413}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:32:39.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.444+0000 7f8d73ca8700 1 -- 172.21.15.112:0/3648390491 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f8d4c0611c0 msgr2=0x7f8d4c063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:39.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.444+0000 7f8d73ca8700 1 --2- 172.21.15.112:0/3648390491 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f8d4c0611c0 0x7f8d4c063670 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c0c16f0 tx=0x7f8d6001a000 comp rx=0 tx=0).stop 2023-01-24T03:32:39.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.444+0000 7f8d73ca8700 1 -- 172.21.15.112:0/3648390491 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d58004690 msgr2=0x7f8d6c1524b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:39.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.444+0000 7f8d73ca8700 1 --2- 172.21.15.112:0/3648390491 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d58004690 0x7f8d6c1524b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c007700 tx=0x7f8d5c001880 comp rx=0 tx=0).stop 2023-01-24T03:32:39.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.444+0000 7f8d73ca8700 1 -- 172.21.15.112:0/3648390491 shutdown_connections 2023-01-24T03:32:39.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.444+0000 7f8d73ca8700 1 --2- 172.21.15.112:0/3648390491 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f8d4c0611c0 0x7f8d4c063670 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:39.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.444+0000 7f8d73ca8700 1 --2- 172.21.15.112:0/3648390491 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d58004690 0x7f8d6c1524b0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:39.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.444+0000 7f8d73ca8700 1 --2- 172.21.15.112:0/3648390491 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8d6c0aeff0 0x7f8d6c0c0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:39.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.444+0000 7f8d73ca8700 1 --2- 172.21.15.112:0/3648390491 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8d6c0aeba0 0x7f8d6c0bfe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:39.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.444+0000 7f8d73ca8700 1 -- 172.21.15.112:0/3648390491 >> 172.21.15.112:0/3648390491 conn(0x7f8d6c05e2c0 msgr2=0x7f8d6c0684b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:32:39.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.444+0000 7f8d73ca8700 1 -- 172.21.15.112:0/3648390491 shutdown_connections 2023-01-24T03:32:39.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:39.444+0000 7f8d73ca8700 1 -- 172.21.15.112:0/3648390491 wait complete. 2023-01-24T03:32:39.450 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 15 2023-01-24T03:32:39.458 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-01-24T03:32:41.459 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:32:41.583 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.580+0000 7f49dea7b700 1 Processor -- start 2023-01-24T03:32:41.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.580+0000 7f49dea7b700 1 -- start start 2023-01-24T03:32:41.587 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.581+0000 7f49dea7b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f49d80b0aa0 0x7f49d8155da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:41.587 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.581+0000 7f49dea7b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49d80ca3e0 0x7f49d80ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:41.588 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.581+0000 7f49dea7b700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49d80b0f40 con 0x7f49d80c7e40 2023-01-24T03:32:41.588 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.581+0000 7f49dea7b700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49d80b1170 con 0x7f49d80b0aa0 2023-01-24T03:32:41.588 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.581+0000 7f49dea7b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49d80b1450 con 0x7f49d80ca3e0 2023-01-24T03:32:41.589 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.581+0000 7f49dc817700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f49d80c7e40 0x7f49d80ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:32856/0 (socket says 172.21.15.112:32856) 2023-01-24T03:32:41.589 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.581+0000 7f49dc817700 1 -- 172.21.15.112:0/2711217905 learned_addr learned my addr 172.21.15.112:0/2711217905 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:32:41.589 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.581+0000 7f49d77fe700 1 -- 172.21.15.112:0/2711217905 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1803471969 0 0) 0x7f49d80b0f40 con 0x7f49d80c7e40 2023-01-24T03:32:41.589 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.581+0000 7f49dd018700 1 --2- 172.21.15.112:0/2711217905 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49d80ca3e0 0x7f49d80ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:41.589 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.581+0000 7f49d7fff700 1 --2- 172.21.15.112:0/2711217905 >> v2:172.21.15.142:3300/0 conn(0x7f49d80b0aa0 0x7f49d8155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:41.590 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.581+0000 7f49d77fe700 1 -- 172.21.15.112:0/2711217905 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49b80033f0 con 0x7f49d80c7e40 2023-01-24T03:32:41.590 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.582+0000 7f49d77fe700 1 -- 172.21.15.112:0/2711217905 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 792635674 0 0) 0x7f49b80033f0 con 0x7f49d80c7e40 2023-01-24T03:32:41.590 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.582+0000 7f49d77fe700 1 -- 172.21.15.112:0/2711217905 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49d80ca3e0 msgr2=0x7f49d80ca7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:32:41.590 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.582+0000 7f49d77fe700 1 --2- 172.21.15.112:0/2711217905 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49d80ca3e0 0x7f49d80ca7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:41.591 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.582+0000 7f49d77fe700 1 -- 172.21.15.112:0/2711217905 >> v2:172.21.15.142:3300/0 conn(0x7f49d80b0aa0 msgr2=0x7f49d8155da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:32:41.591 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.582+0000 7f49d77fe700 1 --2- 172.21.15.112:0/2711217905 >> v2:172.21.15.142:3300/0 conn(0x7f49d80b0aa0 0x7f49d8155da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:41.591 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.582+0000 7f49d77fe700 1 -- 172.21.15.112:0/2711217905 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d80b1730 con 0x7f49d80c7e40 2023-01-24T03:32:41.591 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.582+0000 7f49d77fe700 1 -- 172.21.15.112:0/2711217905 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7f49c0001470 con 0x7f49d80c7e40 2023-01-24T03:32:41.591 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.582+0000 7f49d77fe700 1 -- 172.21.15.112:0/2711217905 >> v1:172.21.15.112:6789/0 conn(0x7f49d80c7e40 legacy=0x7f49d80ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:41.592 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.582+0000 7f49d77fe700 1 --2- 172.21.15.112:0/2711217905 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49d80ca3e0 0x7f49b80038e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:41.592 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.582+0000 7f49d77fe700 1 --2- 172.21.15.112:0/2711217905 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f49b8003e60 0x7f49b8004690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:41.592 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.582+0000 7f49d77fe700 1 --2- 172.21.15.112:0/2711217905 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49b8004bd0 0x7f49b8005040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:41.592 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.582+0000 7f49d77fe700 1 -- 172.21.15.112:0/2711217905 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d80b1170 con 0x7f49b8003e60 2023-01-24T03:32:41.592 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.582+0000 7f49d77fe700 1 -- 172.21.15.112:0/2711217905 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d80b1450 con 0x7f49b8004bd0 2023-01-24T03:32:41.593 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.582+0000 7f49d77fe700 1 -- 172.21.15.112:0/2711217905 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49b800b910 con 0x7f49d80ca3e0 2023-01-24T03:32:41.593 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.582+0000 7f49d7fff700 1 --2- 172.21.15.112:0/2711217905 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f49b8003e60 0x7f49b8004690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:41.593 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.582+0000 7f49dc817700 1 --2- 172.21.15.112:0/2711217905 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49d80ca3e0 0x7f49b80038e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:41.593 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.582+0000 7f49dd018700 1 --2- 172.21.15.112:0/2711217905 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49b8004bd0 0x7f49b8005040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:41.593 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.582+0000 7f49d7fff700 1 -- 172.21.15.112:0/2711217905 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49d80ca3e0 msgr2=0x7f49b80038e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:32:41.594 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.582+0000 7f49d7fff700 1 --2- 172.21.15.112:0/2711217905 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49d80ca3e0 0x7f49b80038e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:41.594 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.582+0000 7f49d7fff700 1 -- 172.21.15.112:0/2711217905 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49b8004bd0 msgr2=0x7f49b8005040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:32:41.594 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.582+0000 7f49d7fff700 1 --2- 172.21.15.112:0/2711217905 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49b8004bd0 0x7f49b8005040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:41.594 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.582+0000 7f49d7fff700 1 -- 172.21.15.112:0/2711217905 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b80033f0 con 0x7f49b8003e60 2023-01-24T03:32:41.594 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.582+0000 7f49dd018700 1 --2- 172.21.15.112:0/2711217905 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49b8004bd0 0x7f49b8005040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:32:41.595 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.582+0000 7f49d7fff700 1 --2- 172.21.15.112:0/2711217905 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f49b8003e60 0x7f49b8004690 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f49c800e210 tx=0x7f49c800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c61c981f661f31a9 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:41.595 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.583+0000 7f49d77fe700 1 -- 172.21.15.112:0/2711217905 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c800ea60 con 0x7f49b8003e60 2023-01-24T03:32:41.595 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.583+0000 7f49d77fe700 1 -- 172.21.15.112:0/2711217905 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c800f040 con 0x7f49b8003e60 2023-01-24T03:32:41.595 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.583+0000 7f49d77fe700 1 -- 172.21.15.112:0/2711217905 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c800ee10 con 0x7f49b8003e60 2023-01-24T03:32:41.595 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.583+0000 7f49dea7b700 1 -- 172.21.15.112:0/2711217905 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f49b8003e60 msgr2=0x7f49b8004690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:41.596 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.583+0000 7f49dea7b700 1 --2- 172.21.15.112:0/2711217905 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f49b8003e60 0x7f49b8004690 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f49c800e210 tx=0x7f49c800b040 comp rx=0 tx=0).stop 2023-01-24T03:32:41.596 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.583+0000 7f49dd018700 1 -- 172.21.15.112:0/2711217905 reap_dead start 2023-01-24T03:32:41.596 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.583+0000 7f49dea7b700 1 -- 172.21.15.112:0/2711217905 shutdown_connections 2023-01-24T03:32:41.596 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.583+0000 7f49dea7b700 1 -- 172.21.15.112:0/2711217905 >> 172.21.15.112:0/2711217905 conn(0x7f49d805e2c0 msgr2=0x7f49d80cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:32:41.596 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.583+0000 7f49dea7b700 1 -- 172.21.15.112:0/2711217905 shutdown_connections 2023-01-24T03:32:41.597 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.583+0000 7f49dea7b700 1 -- 172.21.15.112:0/2711217905 wait complete. 2023-01-24T03:32:41.597 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.584+0000 7f49dea7b700 1 Processor -- start 2023-01-24T03:32:41.597 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.584+0000 7f49dea7b700 1 -- start start 2023-01-24T03:32:41.600 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.585+0000 7f49dea7b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f49b8003e60 0x7f49d8164f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:41.600 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.585+0000 7f49dea7b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49d80c58d0 0x7f49d8152690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:41.600 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.585+0000 7f49dea7b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49d80c2420 0x7f49d80ca970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:41.600 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.585+0000 7f49dea7b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d81654c0 con 0x7f49b8003e60 2023-01-24T03:32:41.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.585+0000 7f49dea7b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d80c19d0 con 0x7f49d80c58d0 2023-01-24T03:32:41.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.585+0000 7f49dea7b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49d80c2b70 con 0x7f49d80c2420 2023-01-24T03:32:41.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.585+0000 7f49dc817700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f49b8003e60 0x7f49d8164f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:41.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.585+0000 7f49dc817700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f49b8003e60 0x7f49d8164f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51880/0 (socket says 172.21.15.112:51880) 2023-01-24T03:32:41.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.585+0000 7f49dc817700 1 -- 172.21.15.112:0/3395609995 learned_addr learned my addr 172.21.15.112:0/3395609995 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:32:41.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.585+0000 7f49dd018700 1 --2- 172.21.15.112:0/3395609995 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49d80c2420 0x7f49d80ca970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:41.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.585+0000 7f49d7fff700 1 --2- 172.21.15.112:0/3395609995 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49d80c58d0 0x7f49d8152690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:41.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.585+0000 7f49dc817700 1 -- 172.21.15.112:0/3395609995 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49d80c2420 msgr2=0x7f49d80ca970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:41.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.585+0000 7f49dc817700 1 --2- 172.21.15.112:0/3395609995 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49d80c2420 0x7f49d80ca970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:41.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.585+0000 7f49dc817700 1 -- 172.21.15.112:0/3395609995 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49d80c58d0 msgr2=0x7f49d8152690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:41.603 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.585+0000 7f49dc817700 1 --2- 172.21.15.112:0/3395609995 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49d80c58d0 0x7f49d8152690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:41.603 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.585+0000 7f49dc817700 1 -- 172.21.15.112:0/3395609995 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d80c2e30 con 0x7f49b8003e60 2023-01-24T03:32:41.603 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.586+0000 7f49dc817700 1 --2- 172.21.15.112:0/3395609995 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f49b8003e60 0x7f49d8164f80 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f49c0004710 tx=0x7f49c0001e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:41.603 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.586+0000 7f49d4ff9700 1 -- 172.21.15.112:0/3395609995 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c000f840 con 0x7f49b8003e60 2023-01-24T03:32:41.603 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.586+0000 7f49d4ff9700 1 -- 172.21.15.112:0/3395609995 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c000f9a0 con 0x7f49b8003e60 2023-01-24T03:32:41.604 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.586+0000 7f49dea7b700 1 -- 172.21.15.112:0/3395609995 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49d80c3110 con 0x7f49b8003e60 2023-01-24T03:32:41.604 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.586+0000 7f49d4ff9700 1 -- 172.21.15.112:0/3395609995 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c000fd60 con 0x7f49b8003e60 2023-01-24T03:32:41.604 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.586+0000 7f49dea7b700 1 -- 172.21.15.112:0/3395609995 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49d80c1f00 con 0x7f49b8003e60 2023-01-24T03:32:41.604 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.587+0000 7f49dea7b700 1 -- 172.21.15.112:0/3395609995 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49d804bb80 con 0x7f49b8003e60 2023-01-24T03:32:41.604 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.587+0000 7f49d4ff9700 1 -- 172.21.15.112:0/3395609995 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f49c001c030 con 0x7f49b8003e60 2023-01-24T03:32:41.605 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.588+0000 7f49d4ff9700 1 --2- 172.21.15.112:0/3395609995 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f49bc0611c0 0x7f49bc063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:41.605 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.588+0000 7f49d4ff9700 1 -- 172.21.15.112:0/3395609995 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 7009+0+0 (secure 0 0 0) 0x7f49c007d5f0 con 0x7f49b8003e60 2023-01-24T03:32:41.605 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.588+0000 7f49d7fff700 1 --2- 172.21.15.112:0/3395609995 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f49bc0611c0 0x7f49bc063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:41.605 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.588+0000 7f49d7fff700 1 --2- 172.21.15.112:0/3395609995 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f49bc0611c0 0x7f49bc063670 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f49c800afd0 tx=0x7f49c801a000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:41.605 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.592+0000 7f49d4ff9700 1 -- 172.21.15.112:0/3395609995 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f49c004d360 con 0x7f49b8003e60 2023-01-24T03:32:41.814 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.811+0000 7f49dea7b700 1 -- 172.21.15.112:0/3395609995 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49d80c5df0 con 0x7f49b8003e60 2023-01-24T03:32:41.815 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.812+0000 7f49d4ff9700 1 -- 172.21.15.112:0/3395609995 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+6112 (secure 0 0 0) 0x7f49c001a020 con 0x7f49b8003e60 2023-01-24T03:32:41.815 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:32:41.816 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4897,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/11022091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":11022091},{"type":"v1","addr":"172.21.15.142:6835","nonce":11022091}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:32:09.626128+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4288},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":0,"incarnation":11,"state":"up:active","state_seq":31,"addr":"172.21.15.142:6834/869267413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":869267413},{"type":"v1","addr":"172.21.15.142:6834","nonce":869267413}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:32:41.817 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.814+0000 7f49dea7b700 1 -- 172.21.15.112:0/3395609995 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f49bc0611c0 msgr2=0x7f49bc063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:41.817 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.814+0000 7f49dea7b700 1 --2- 172.21.15.112:0/3395609995 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f49bc0611c0 0x7f49bc063670 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f49c800afd0 tx=0x7f49c801a000 comp rx=0 tx=0).stop 2023-01-24T03:32:41.817 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.814+0000 7f49dea7b700 1 -- 172.21.15.112:0/3395609995 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f49b8003e60 msgr2=0x7f49d8164f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:41.818 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.814+0000 7f49dea7b700 1 --2- 172.21.15.112:0/3395609995 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f49b8003e60 0x7f49d8164f80 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f49c0004710 tx=0x7f49c0001e90 comp rx=0 tx=0).stop 2023-01-24T03:32:41.818 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.814+0000 7f49dea7b700 1 -- 172.21.15.112:0/3395609995 shutdown_connections 2023-01-24T03:32:41.818 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.814+0000 7f49dea7b700 1 --2- 172.21.15.112:0/3395609995 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f49bc0611c0 0x7f49bc063670 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:41.818 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.814+0000 7f49dea7b700 1 --2- 172.21.15.112:0/3395609995 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f49b8003e60 0x7f49d8164f80 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:41.818 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.814+0000 7f49dea7b700 1 --2- 172.21.15.112:0/3395609995 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f49d80c2420 0x7f49d80ca970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:41.819 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.814+0000 7f49dea7b700 1 --2- 172.21.15.112:0/3395609995 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f49d80c58d0 0x7f49d8152690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:41.819 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.814+0000 7f49dea7b700 1 -- 172.21.15.112:0/3395609995 >> 172.21.15.112:0/3395609995 conn(0x7f49d805e2c0 msgr2=0x7f49d80aef20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:32:41.819 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.814+0000 7f49dea7b700 1 -- 172.21.15.112:0/3395609995 shutdown_connections 2023-01-24T03:32:41.820 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:41.814+0000 7f49dea7b700 1 -- 172.21.15.112:0/3395609995 wait complete. 2023-01-24T03:32:41.820 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 15 2023-01-24T03:32:41.828 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-01-24T03:32:43.829 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:32:43.953 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.950+0000 7f3d40ce2700 1 Processor -- start 2023-01-24T03:32:43.953 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.950+0000 7f3d40ce2700 1 -- start start 2023-01-24T03:32:43.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.952+0000 7f3d40ce2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d3c0c7e40 0x7f3d3c0b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:43.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.952+0000 7f3d40ce2700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d3c156210 con 0x7f3d3c0ca3e0 2023-01-24T03:32:43.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.952+0000 7f3d40ce2700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d3c0b1170 con 0x7f3d3c155da0 2023-01-24T03:32:43.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.952+0000 7f3d40ce2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c0b1450 con 0x7f3d3c0c7e40 2023-01-24T03:32:43.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.952+0000 7f3d3ad9d700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f3d3c0ca3e0 0x7f3d3c0ca820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:32882/0 (socket says 172.21.15.112:32882) 2023-01-24T03:32:43.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.952+0000 7f3d3ad9d700 1 -- 172.21.15.112:0/1527006381 learned_addr learned my addr 172.21.15.112:0/1527006381 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:32:43.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.952+0000 7f3d3a59c700 1 --2- 172.21.15.112:0/1527006381 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d3c0c7e40 0x7f3d3c0b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:43.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.952+0000 7f3d3959a700 1 -- 172.21.15.112:0/1527006381 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1042098108 0 0) 0x7f3d3c156210 con 0x7f3d3c0ca3e0 2023-01-24T03:32:43.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.952+0000 7f3d3959a700 1 -- 172.21.15.112:0/1527006381 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d1c0033f0 con 0x7f3d3c0ca3e0 2023-01-24T03:32:43.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.952+0000 7f3d3959a700 1 -- 172.21.15.112:0/1527006381 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2759216238 0 0) 0x7f3d3c0b1170 con 0x7f3d3c155da0 2023-01-24T03:32:43.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.952+0000 7f3d3959a700 1 -- 172.21.15.112:0/1527006381 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d3c156210 con 0x7f3d3c155da0 2023-01-24T03:32:43.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.952+0000 7f3d3959a700 1 -- 172.21.15.112:0/1527006381 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3566204969 0 0) 0x7f3d1c0033f0 con 0x7f3d3c0ca3e0 2023-01-24T03:32:43.965 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.952+0000 7f3d3959a700 1 -- 172.21.15.112:0/1527006381 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d3c0c7e40 msgr2=0x7f3d3c0b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:32:43.965 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.952+0000 7f3d3959a700 1 --2- 172.21.15.112:0/1527006381 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d3c0c7e40 0x7f3d3c0b0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:43.965 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.952+0000 7f3d3959a700 1 -- 172.21.15.112:0/1527006381 >> v1:172.21.15.142:6789/0 conn(0x7f3d3c155da0 legacy=0x7f3d3c0ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:43.965 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.952+0000 7f3d3959a700 1 -- 172.21.15.112:0/1527006381 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d3c0b1730 con 0x7f3d3c0ca3e0 2023-01-24T03:32:43.965 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.952+0000 7f3d3959a700 1 -- 172.21.15.112:0/1527006381 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7f3d34001470 con 0x7f3d3c0ca3e0 2023-01-24T03:32:43.966 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.952+0000 7f3d3959a700 1 -- 172.21.15.112:0/1527006381 >> v1:172.21.15.112:6789/0 conn(0x7f3d3c0ca3e0 legacy=0x7f3d3c0ca820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:43.966 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.952+0000 7f3d3959a700 1 --2- 172.21.15.112:0/1527006381 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d1c003aa0 0x7f3d1c003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:43.966 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.952+0000 7f3d3959a700 1 --2- 172.21.15.112:0/1527006381 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d3c0c7e40 0x7f3d1c004690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:43.966 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.952+0000 7f3d3959a700 1 --2- 172.21.15.112:0/1527006381 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d1c004bd0 0x7f3d1c005040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:43.966 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.952+0000 7f3d3959a700 1 -- 172.21.15.112:0/1527006381 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c0b1450 con 0x7f3d1c003aa0 2023-01-24T03:32:43.967 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.952+0000 7f3d3959a700 1 -- 172.21.15.112:0/1527006381 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c00b9c0 con 0x7f3d1c004bd0 2023-01-24T03:32:43.967 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.952+0000 7f3d3959a700 1 -- 172.21.15.112:0/1527006381 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c00bb80 con 0x7f3d3c0c7e40 2023-01-24T03:32:43.967 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.952+0000 7f3d3a59c700 1 --2- 172.21.15.112:0/1527006381 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d1c003aa0 0x7f3d1c003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:43.967 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.952+0000 7f3d39d9b700 1 --2- 172.21.15.112:0/1527006381 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d3c0c7e40 0x7f3d1c004690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:43.968 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.952+0000 7f3d3ad9d700 1 --2- 172.21.15.112:0/1527006381 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d1c004bd0 0x7f3d1c005040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:43.968 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.952+0000 7f3d3a59c700 1 -- 172.21.15.112:0/1527006381 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d3c0c7e40 msgr2=0x7f3d1c004690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:32:43.968 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.952+0000 7f3d3a59c700 1 --2- 172.21.15.112:0/1527006381 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d3c0c7e40 0x7f3d1c004690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:43.968 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.952+0000 7f3d3a59c700 1 -- 172.21.15.112:0/1527006381 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d1c004bd0 msgr2=0x7f3d1c005040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:32:43.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.952+0000 7f3d3a59c700 1 --2- 172.21.15.112:0/1527006381 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d1c004bd0 0x7f3d1c005040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:43.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.952+0000 7f3d3a59c700 1 -- 172.21.15.112:0/1527006381 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1c0033f0 con 0x7f3d1c003aa0 2023-01-24T03:32:43.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.952+0000 7f3d39d9b700 1 --2- 172.21.15.112:0/1527006381 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d3c0c7e40 0x7f3d1c004690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:32:43.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.952+0000 7f3d3ad9d700 1 --2- 172.21.15.112:0/1527006381 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d1c004bd0 0x7f3d1c005040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:32:43.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.952+0000 7f3d3a59c700 1 --2- 172.21.15.112:0/1527006381 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d1c003aa0 0x7f3d1c003e90 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c00e210 tx=0x7f3d2c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92eb15b2c41140fd server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:43.970 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.952+0000 7f3d3959a700 1 -- 172.21.15.112:0/1527006381 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c00eac0 con 0x7f3d1c003aa0 2023-01-24T03:32:43.970 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.952+0000 7f3d3959a700 1 -- 172.21.15.112:0/1527006381 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d2c00f040 con 0x7f3d1c003aa0 2023-01-24T03:32:43.970 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.952+0000 7f3d3959a700 1 -- 172.21.15.112:0/1527006381 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c00ee70 con 0x7f3d1c003aa0 2023-01-24T03:32:43.970 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.953+0000 7f3d40ce2700 1 -- 172.21.15.112:0/1527006381 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d1c003aa0 msgr2=0x7f3d1c003e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:43.971 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.953+0000 7f3d40ce2700 1 --2- 172.21.15.112:0/1527006381 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d1c003aa0 0x7f3d1c003e90 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c00e210 tx=0x7f3d2c00b040 comp rx=0 tx=0).stop 2023-01-24T03:32:43.971 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.953+0000 7f3d3ad9d700 1 -- 172.21.15.112:0/1527006381 reap_dead start 2023-01-24T03:32:43.971 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.953+0000 7f3d40ce2700 1 -- 172.21.15.112:0/1527006381 shutdown_connections 2023-01-24T03:32:43.971 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.953+0000 7f3d40ce2700 1 -- 172.21.15.112:0/1527006381 >> 172.21.15.112:0/1527006381 conn(0x7f3d3c05e2c0 msgr2=0x7f3d3c0cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:32:43.971 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.953+0000 7f3d40ce2700 1 -- 172.21.15.112:0/1527006381 shutdown_connections 2023-01-24T03:32:43.971 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.953+0000 7f3d40ce2700 1 -- 172.21.15.112:0/1527006381 wait complete. 2023-01-24T03:32:43.972 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.954+0000 7f3d40ce2700 1 Processor -- start 2023-01-24T03:32:43.972 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.954+0000 7f3d40ce2700 1 -- start start 2023-01-24T03:32:43.972 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.955+0000 7f3d40ce2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d1c003aa0 0x7f3d3c068280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:43.972 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.955+0000 7f3d40ce2700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d3c151160 0x7f3d3c1515d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:43.972 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.955+0000 7f3d40ce2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d3c14ec00 0x7f3d3c14f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:43.973 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.955+0000 7f3d40ce2700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c0c4460 con 0x7f3d3c151160 2023-01-24T03:32:43.973 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.955+0000 7f3d40ce2700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c151b10 con 0x7f3d3c14ec00 2023-01-24T03:32:43.973 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.955+0000 7f3d40ce2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c0c55b0 con 0x7f3d1c003aa0 2023-01-24T03:32:43.973 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.955+0000 7f3d39d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d3c151160 0x7f3d3c1515d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:43.973 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.955+0000 7f3d39d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d3c151160 0x7f3d3c1515d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51904/0 (socket says 172.21.15.112:51904) 2023-01-24T03:32:43.974 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.955+0000 7f3d39d9b700 1 -- 172.21.15.112:0/4242879516 learned_addr learned my addr 172.21.15.112:0/4242879516 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:32:43.975 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.955+0000 7f3d3ad9d700 1 --2- 172.21.15.112:0/4242879516 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d3c14ec00 0x7f3d3c14f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:43.975 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.955+0000 7f3d3a59c700 1 --2- 172.21.15.112:0/4242879516 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d1c003aa0 0x7f3d3c068280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:43.975 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.955+0000 7f3d39d9b700 1 -- 172.21.15.112:0/4242879516 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d1c003aa0 msgr2=0x7f3d3c068280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:43.976 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.955+0000 7f3d39d9b700 1 --2- 172.21.15.112:0/4242879516 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d1c003aa0 0x7f3d3c068280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:43.976 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.955+0000 7f3d39d9b700 1 -- 172.21.15.112:0/4242879516 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d3c14ec00 msgr2=0x7f3d3c14f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:43.977 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.955+0000 7f3d39d9b700 1 --2- 172.21.15.112:0/4242879516 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d3c14ec00 0x7f3d3c14f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:43.977 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.955+0000 7f3d39d9b700 1 -- 172.21.15.112:0/4242879516 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d3c152770 con 0x7f3d3c151160 2023-01-24T03:32:43.977 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.955+0000 7f3d39d9b700 1 --2- 172.21.15.112:0/4242879516 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d3c151160 0x7f3d3c1515d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f3d2400e840 tx=0x7f3d24009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:43.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.955+0000 7f3d327fc700 1 -- 172.21.15.112:0/4242879516 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d24012590 con 0x7f3d3c151160 2023-01-24T03:32:43.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.955+0000 7f3d327fc700 1 -- 172.21.15.112:0/4242879516 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d240126f0 con 0x7f3d3c151160 2023-01-24T03:32:43.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.955+0000 7f3d40ce2700 1 -- 172.21.15.112:0/4242879516 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d3c0adb60 con 0x7f3d3c151160 2023-01-24T03:32:43.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.955+0000 7f3d327fc700 1 -- 172.21.15.112:0/4242879516 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d24012ab0 con 0x7f3d3c151160 2023-01-24T03:32:43.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.955+0000 7f3d40ce2700 1 -- 172.21.15.112:0/4242879516 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d3c166660 con 0x7f3d3c151160 2023-01-24T03:32:43.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.956+0000 7f3d40ce2700 1 -- 172.21.15.112:0/4242879516 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d3c067290 con 0x7f3d3c151160 2023-01-24T03:32:43.980 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.957+0000 7f3d327fc700 1 -- 172.21.15.112:0/4242879516 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f3d2401d460 con 0x7f3d3c151160 2023-01-24T03:32:43.980 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.957+0000 7f3d327fc700 1 --2- 172.21.15.112:0/4242879516 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3d200611f0 0x7f3d200636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:43.980 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.957+0000 7f3d327fc700 1 -- 172.21.15.112:0/4242879516 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 7009+0+0 (secure 0 0 0) 0x7f3d2407dd40 con 0x7f3d3c151160 2023-01-24T03:32:43.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.957+0000 7f3d3a59c700 1 --2- 172.21.15.112:0/4242879516 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3d200611f0 0x7f3d200636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:43.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.957+0000 7f3d3a59c700 1 --2- 172.21.15.112:0/4242879516 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3d200611f0 0x7f3d200636a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c00afd0 tx=0x7f3d2c01a000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:43.982 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:43.961+0000 7f3d327fc700 1 -- 172.21.15.112:0/4242879516 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f3d2404da50 con 0x7f3d3c151160 2023-01-24T03:32:44.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:44.183+0000 7f3d40ce2700 1 -- 172.21.15.112:0/4242879516 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d3c04bb80 con 0x7f3d3c151160 2023-01-24T03:32:44.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:44.183+0000 7f3d327fc700 1 -- 172.21.15.112:0/4242879516 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+6112 (secure 0 0 0) 0x7f3d24051070 con 0x7f3d3c151160 2023-01-24T03:32:44.187 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:32:44.187 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4897,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/11022091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":11022091},{"type":"v1","addr":"172.21.15.142:6835","nonce":11022091}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:32:09.626128+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4288},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":0,"incarnation":11,"state":"up:active","state_seq":31,"addr":"172.21.15.142:6834/869267413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":869267413},{"type":"v1","addr":"172.21.15.142:6834","nonce":869267413}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:32:44.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:44.186+0000 7f3d40ce2700 1 -- 172.21.15.112:0/4242879516 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3d200611f0 msgr2=0x7f3d200636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:44.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:44.186+0000 7f3d40ce2700 1 --2- 172.21.15.112:0/4242879516 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3d200611f0 0x7f3d200636a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c00afd0 tx=0x7f3d2c01a000 comp rx=0 tx=0).stop 2023-01-24T03:32:44.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:44.186+0000 7f3d40ce2700 1 -- 172.21.15.112:0/4242879516 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d3c151160 msgr2=0x7f3d3c1515d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:44.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:44.186+0000 7f3d40ce2700 1 --2- 172.21.15.112:0/4242879516 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d3c151160 0x7f3d3c1515d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f3d2400e840 tx=0x7f3d24009040 comp rx=0 tx=0).stop 2023-01-24T03:32:44.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:44.186+0000 7f3d40ce2700 1 -- 172.21.15.112:0/4242879516 shutdown_connections 2023-01-24T03:32:44.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:44.186+0000 7f3d40ce2700 1 --2- 172.21.15.112:0/4242879516 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3d200611f0 0x7f3d200636a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:44.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:44.186+0000 7f3d40ce2700 1 --2- 172.21.15.112:0/4242879516 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3d3c14ec00 0x7f3d3c14f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:44.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:44.186+0000 7f3d40ce2700 1 --2- 172.21.15.112:0/4242879516 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3d1c003aa0 0x7f3d3c068280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:44.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:44.186+0000 7f3d40ce2700 1 --2- 172.21.15.112:0/4242879516 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d3c151160 0x7f3d3c1515d0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:44.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:44.186+0000 7f3d40ce2700 1 -- 172.21.15.112:0/4242879516 >> 172.21.15.112:0/4242879516 conn(0x7f3d3c05e2c0 msgr2=0x7f3d3c1571a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:32:44.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:44.186+0000 7f3d40ce2700 1 -- 172.21.15.112:0/4242879516 shutdown_connections 2023-01-24T03:32:44.193 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:44.186+0000 7f3d40ce2700 1 -- 172.21.15.112:0/4242879516 wait complete. 2023-01-24T03:32:44.193 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 15 2023-01-24T03:32:44.200 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-01-24T03:32:46.201 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:32:46.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.362+0000 7fdec6916700 1 Processor -- start 2023-01-24T03:32:46.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.362+0000 7fdec6916700 1 -- start start 2023-01-24T03:32:46.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.363+0000 7fdec6916700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fdec014c660 0x7fdec00c1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:46.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.363+0000 7fdec6916700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdec00c18f0 con 0x7fdec014c290 2023-01-24T03:32:46.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.363+0000 7fdec6916700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdec00c1030 con 0x7fdec00c22c0 2023-01-24T03:32:46.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.363+0000 7fdec6916700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdec006bad0 con 0x7fdec014c660 2023-01-24T03:32:46.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.363+0000 7fdebffff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fdec014c290 0x7fdec006ec80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:32906/0 (socket says 172.21.15.112:32906) 2023-01-24T03:32:46.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.363+0000 7fdebffff700 1 -- 172.21.15.112:0/2196027996 learned_addr learned my addr 172.21.15.112:0/2196027996 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:32:46.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.363+0000 7fdebf7fe700 1 --2- 172.21.15.112:0/2196027996 >> v2:172.21.15.142:3300/0 conn(0x7fdec014c660 0x7fdec00c1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:46.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.363+0000 7fdebeffd700 1 -- 172.21.15.112:0/2196027996 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1488081208 0 0) 0x7fdec00c18f0 con 0x7fdec014c290 2023-01-24T03:32:46.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.363+0000 7fdebeffd700 1 -- 172.21.15.112:0/2196027996 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdeac0033f0 con 0x7fdec014c290 2023-01-24T03:32:46.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.363+0000 7fdebeffd700 1 -- 172.21.15.112:0/2196027996 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1766427359 0 0) 0x7fdec00c1030 con 0x7fdec00c22c0 2023-01-24T03:32:46.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.363+0000 7fdebeffd700 1 -- 172.21.15.112:0/2196027996 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdec00c18f0 con 0x7fdec00c22c0 2023-01-24T03:32:46.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.363+0000 7fdebf7fe700 1 -- 172.21.15.112:0/2196027996 >> v1:172.21.15.142:6789/0 conn(0x7fdec00c22c0 legacy=0x7fdec0155710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:46.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.363+0000 7fdebf7fe700 1 -- 172.21.15.112:0/2196027996 >> v1:172.21.15.112:6789/0 conn(0x7fdec014c290 legacy=0x7fdec006ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:46.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.363+0000 7fdebeffd700 1 -- 172.21.15.112:0/2196027996 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3749825561 0 0) 0x7fdeac0033f0 con 0x7fdec014c290 2023-01-24T03:32:46.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.363+0000 7fdebf7fe700 1 -- 172.21.15.112:0/2196027996 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec006bdb0 con 0x7fdec014c660 2023-01-24T03:32:46.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.363+0000 7fdebf7fe700 1 --2- 172.21.15.112:0/2196027996 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdec014c660 0x7fdec00c1d80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4009760 tx=0x7fdeb4009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7508425c8e5a4487 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:46.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.363+0000 7fdebeffd700 1 -- 172.21.15.112:0/2196027996 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb0001470 con 0x7fdec014c660 2023-01-24T03:32:46.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.364+0000 7fdebeffd700 1 -- 172.21.15.112:0/2196027996 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeb4011440 con 0x7fdec014c660 2023-01-24T03:32:46.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.364+0000 7fdebeffd700 1 -- 172.21.15.112:0/2196027996 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb40117c0 con 0x7fdec014c660 2023-01-24T03:32:46.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.364+0000 7fdec6916700 1 -- 172.21.15.112:0/2196027996 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdec014c660 msgr2=0x7fdec00c1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:46.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.364+0000 7fdec6916700 1 --2- 172.21.15.112:0/2196027996 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdec014c660 0x7fdec00c1d80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4009760 tx=0x7fdeb4009060 comp rx=0 tx=0).stop 2023-01-24T03:32:46.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.364+0000 7fdec6916700 1 -- 172.21.15.112:0/2196027996 shutdown_connections 2023-01-24T03:32:46.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.364+0000 7fdec6916700 1 --2- 172.21.15.112:0/2196027996 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdec014c660 0x7fdec00c1d80 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:46.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.364+0000 7fdec6916700 1 -- 172.21.15.112:0/2196027996 >> 172.21.15.112:0/2196027996 conn(0x7fdec0067f20 msgr2=0x7fdec0068320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:32:46.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.364+0000 7fdec6916700 1 -- 172.21.15.112:0/2196027996 shutdown_connections 2023-01-24T03:32:46.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.364+0000 7fdec6916700 1 -- 172.21.15.112:0/2196027996 wait complete. 2023-01-24T03:32:46.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.365+0000 7fdec6916700 1 Processor -- start 2023-01-24T03:32:46.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.365+0000 7fdec6916700 1 -- start start 2023-01-24T03:32:46.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.366+0000 7fdec6916700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdec00c22c0 0x7fdec014f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:46.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.366+0000 7fdec6916700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdec014c290 0x7fdec01667e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:46.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.366+0000 7fdec6916700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdec014c660 0x7fdec00684b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:46.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.366+0000 7fdec6916700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec0152a80 con 0x7fdec014c290 2023-01-24T03:32:46.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.366+0000 7fdec6916700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec0163c10 con 0x7fdec014c660 2023-01-24T03:32:46.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.366+0000 7fdec6916700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec0152cd0 con 0x7fdec00c22c0 2023-01-24T03:32:46.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.366+0000 7fdebf7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdec014c290 0x7fdec01667e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:46.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.366+0000 7fdebf7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdec014c290 0x7fdec01667e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51926/0 (socket says 172.21.15.112:51926) 2023-01-24T03:32:46.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.366+0000 7fdebf7fe700 1 -- 172.21.15.112:0/3687645260 learned_addr learned my addr 172.21.15.112:0/3687645260 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:32:46.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.366+0000 7fdebffff700 1 --2- 172.21.15.112:0/3687645260 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdec00c22c0 0x7fdec014f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:46.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.366+0000 7fdec4eb3700 1 --2- 172.21.15.112:0/3687645260 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdec014c660 0x7fdec00684b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:46.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.366+0000 7fdebf7fe700 1 -- 172.21.15.112:0/3687645260 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdec00c22c0 msgr2=0x7fdec014f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:46.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.366+0000 7fdebf7fe700 1 --2- 172.21.15.112:0/3687645260 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdec00c22c0 0x7fdec014f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:46.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.366+0000 7fdebf7fe700 1 -- 172.21.15.112:0/3687645260 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdec014c660 msgr2=0x7fdec00684b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:46.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.366+0000 7fdebf7fe700 1 --2- 172.21.15.112:0/3687645260 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdec014c660 0x7fdec00684b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:46.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.366+0000 7fdebf7fe700 1 -- 172.21.15.112:0/3687645260 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb40092a0 con 0x7fdec014c290 2023-01-24T03:32:46.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.366+0000 7fdebf7fe700 1 --2- 172.21.15.112:0/3687645260 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdec014c290 0x7fdec01667e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4009d50 tx=0x7fdeb4015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:46.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.367+0000 7fdea7fff700 1 -- 172.21.15.112:0/3687645260 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb4010970 con 0x7fdec014c290 2023-01-24T03:32:46.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.367+0000 7fdea7fff700 1 -- 172.21.15.112:0/3687645260 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeb4010af0 con 0x7fdec014c290 2023-01-24T03:32:46.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.367+0000 7fdec6916700 1 -- 172.21.15.112:0/3687645260 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdec0152fb0 con 0x7fdec014c290 2023-01-24T03:32:46.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.367+0000 7fdea7fff700 1 -- 172.21.15.112:0/3687645260 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb401c420 con 0x7fdec014c290 2023-01-24T03:32:46.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.367+0000 7fdec6916700 1 -- 172.21.15.112:0/3687645260 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdec01638a0 con 0x7fdec014c290 2023-01-24T03:32:46.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.368+0000 7fdec6916700 1 -- 172.21.15.112:0/3687645260 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdec004bb80 con 0x7fdec014c290 2023-01-24T03:32:46.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.368+0000 7fdea7fff700 1 -- 172.21.15.112:0/3687645260 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fdeb400f070 con 0x7fdec014c290 2023-01-24T03:32:46.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.368+0000 7fdea7fff700 1 --2- 172.21.15.112:0/3687645260 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fdea00611c0 0x7fdea0063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:46.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.368+0000 7fdea7fff700 1 -- 172.21.15.112:0/3687645260 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 7009+0+0 (secure 0 0 0) 0x7fdeb407eda0 con 0x7fdec014c290 2023-01-24T03:32:46.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.368+0000 7fdebffff700 1 --2- 172.21.15.112:0/3687645260 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fdea00611c0 0x7fdea0063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:46.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.369+0000 7fdebffff700 1 --2- 172.21.15.112:0/3687645260 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fdea00611c0 0x7fdea0063670 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fdeb00015a0 tx=0x7fdeb0000bc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:46.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.373+0000 7fdea7fff700 1 -- 172.21.15.112:0/3687645260 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fdeb404ebc0 con 0x7fdec014c290 2023-01-24T03:32:46.593 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.591+0000 7fdec6916700 1 -- 172.21.15.112:0/3687645260 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdec00c12a0 con 0x7fdec014c290 2023-01-24T03:32:46.594 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.592+0000 7fdea7fff700 1 -- 172.21.15.112:0/3687645260 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+6112 (secure 0 0 0) 0x7fdeb40521e0 con 0x7fdec014c290 2023-01-24T03:32:46.594 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:32:46.594 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4897,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/11022091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":11022091},{"type":"v1","addr":"172.21.15.142:6835","nonce":11022091}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:32:09.626128+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4288},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":0,"incarnation":11,"state":"up:active","state_seq":31,"addr":"172.21.15.142:6834/869267413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":869267413},{"type":"v1","addr":"172.21.15.142:6834","nonce":869267413}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:32:46.596 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.593+0000 7fdec6916700 1 -- 172.21.15.112:0/3687645260 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fdea00611c0 msgr2=0x7fdea0063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:46.596 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.593+0000 7fdec6916700 1 --2- 172.21.15.112:0/3687645260 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fdea00611c0 0x7fdea0063670 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fdeb00015a0 tx=0x7fdeb0000bc0 comp rx=0 tx=0).stop 2023-01-24T03:32:46.596 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.593+0000 7fdec6916700 1 -- 172.21.15.112:0/3687645260 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdec014c290 msgr2=0x7fdec01667e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:46.597 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.593+0000 7fdec6916700 1 --2- 172.21.15.112:0/3687645260 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdec014c290 0x7fdec01667e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4009d50 tx=0x7fdeb4015000 comp rx=0 tx=0).stop 2023-01-24T03:32:46.597 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.593+0000 7fdec6916700 1 -- 172.21.15.112:0/3687645260 shutdown_connections 2023-01-24T03:32:46.597 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.593+0000 7fdec6916700 1 --2- 172.21.15.112:0/3687645260 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fdea00611c0 0x7fdea0063670 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:46.597 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.593+0000 7fdec6916700 1 --2- 172.21.15.112:0/3687645260 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdec014c660 0x7fdec00684b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:46.597 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.593+0000 7fdec6916700 1 --2- 172.21.15.112:0/3687645260 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdec00c22c0 0x7fdec014f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:46.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.593+0000 7fdec6916700 1 --2- 172.21.15.112:0/3687645260 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdec014c290 0x7fdec01667e0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:46.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.593+0000 7fdec6916700 1 -- 172.21.15.112:0/3687645260 >> 172.21.15.112:0/3687645260 conn(0x7fdec0067f20 msgr2=0x7fdec0158ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:32:46.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.594+0000 7fdec6916700 1 -- 172.21.15.112:0/3687645260 shutdown_connections 2023-01-24T03:32:46.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:46.594+0000 7fdec6916700 1 -- 172.21.15.112:0/3687645260 wait complete. 2023-01-24T03:32:46.598 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 15 2023-01-24T03:32:46.607 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-01-24T03:32:48.609 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:32:48.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.731+0000 7fbb86206700 1 Processor -- start 2023-01-24T03:32:48.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.731+0000 7fbb86206700 1 -- start start 2023-01-24T03:32:48.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.732+0000 7fbb86206700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbb800c7e40 0x7fbb800b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:48.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.732+0000 7fbb86206700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fbb800ca3e0 0x7fbb800ca820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:48.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.732+0000 7fbb86206700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb80156210 con 0x7fbb80155da0 2023-01-24T03:32:48.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.732+0000 7fbb86206700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb800cad60 con 0x7fbb800ca3e0 2023-01-24T03:32:48.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.732+0000 7fbb86206700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb800b1170 con 0x7fbb800c7e40 2023-01-24T03:32:48.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.732+0000 7fbb7effd700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fbb80155da0 0x7fbb800ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:32926/0 (socket says 172.21.15.112:32926) 2023-01-24T03:32:48.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.732+0000 7fbb7effd700 1 -- 172.21.15.112:0/1298777796 learned_addr learned my addr 172.21.15.112:0/1298777796 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:32:48.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.732+0000 7fbb7ffff700 1 --2- 172.21.15.112:0/1298777796 >> v2:172.21.15.142:3300/0 conn(0x7fbb800ca3e0 0x7fbb800ca820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:48.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.732+0000 7fbb7f7fe700 1 --2- 172.21.15.112:0/1298777796 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbb800c7e40 0x7fbb800b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:48.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.733+0000 7fbb7e7fc700 1 -- 172.21.15.112:0/1298777796 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2131938197 0 0) 0x7fbb80156210 con 0x7fbb80155da0 2023-01-24T03:32:48.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.733+0000 7fbb7e7fc700 1 -- 172.21.15.112:0/1298777796 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb6c0033f0 con 0x7fbb80155da0 2023-01-24T03:32:48.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.733+0000 7fbb7ffff700 1 -- 172.21.15.112:0/1298777796 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbb800c7e40 msgr2=0x7fbb800b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:32:48.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.733+0000 7fbb7ffff700 1 --2- 172.21.15.112:0/1298777796 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbb800c7e40 0x7fbb800b0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:48.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.733+0000 7fbb7ffff700 1 -- 172.21.15.112:0/1298777796 >> v1:172.21.15.112:6789/0 conn(0x7fbb80155da0 legacy=0x7fbb800ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:48.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.733+0000 7fbb7ffff700 1 -- 172.21.15.112:0/1298777796 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb800b1450 con 0x7fbb800ca3e0 2023-01-24T03:32:48.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.733+0000 7fbb7ffff700 1 --2- 172.21.15.112:0/1298777796 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb800ca3e0 0x7fbb800ca820 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbb78009760 tx=0x7fbb78009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3675b901fbb43bbd server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:48.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.733+0000 7fbb7e7fc700 1 -- 172.21.15.112:0/1298777796 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb780105b0 con 0x7fbb800ca3e0 2023-01-24T03:32:48.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.733+0000 7fbb7e7fc700 1 -- 172.21.15.112:0/1298777796 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb78010710 con 0x7fbb800ca3e0 2023-01-24T03:32:48.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.733+0000 7fbb7e7fc700 1 -- 172.21.15.112:0/1298777796 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb78010ab0 con 0x7fbb800ca3e0 2023-01-24T03:32:48.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.734+0000 7fbb86206700 1 -- 172.21.15.112:0/1298777796 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb800ca3e0 msgr2=0x7fbb800ca820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:48.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.734+0000 7fbb86206700 1 --2- 172.21.15.112:0/1298777796 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb800ca3e0 0x7fbb800ca820 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbb78009760 tx=0x7fbb78009060 comp rx=0 tx=0).stop 2023-01-24T03:32:48.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.734+0000 7fbb86206700 1 -- 172.21.15.112:0/1298777796 shutdown_connections 2023-01-24T03:32:48.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.734+0000 7fbb86206700 1 --2- 172.21.15.112:0/1298777796 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb800ca3e0 0x7fbb800ca820 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:48.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.734+0000 7fbb86206700 1 --2- 172.21.15.112:0/1298777796 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbb800c7e40 0x7fbb800b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:48.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.734+0000 7fbb86206700 1 -- 172.21.15.112:0/1298777796 >> 172.21.15.112:0/1298777796 conn(0x7fbb8005e2c0 msgr2=0x7fbb800cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:32:48.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.734+0000 7fbb86206700 1 -- 172.21.15.112:0/1298777796 shutdown_connections 2023-01-24T03:32:48.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.734+0000 7fbb86206700 1 -- 172.21.15.112:0/1298777796 wait complete. 2023-01-24T03:32:48.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.735+0000 7fbb86206700 1 Processor -- start 2023-01-24T03:32:48.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.735+0000 7fbb86206700 1 -- start start 2023-01-24T03:32:48.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.736+0000 7fbb86206700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbb800c7e40 0x7fbb801687b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:48.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.736+0000 7fbb86206700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb800ca3e0 0x7fbb800c1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:48.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.736+0000 7fbb86206700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbb80155da0 0x7fbb8014be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:48.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.736+0000 7fbb86206700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb800c0b70 con 0x7fbb80155da0 2023-01-24T03:32:48.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.736+0000 7fbb86206700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb80151e00 con 0x7fbb800ca3e0 2023-01-24T03:32:48.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.736+0000 7fbb86206700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb800cc180 con 0x7fbb800c7e40 2023-01-24T03:32:48.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.736+0000 7fbb7ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbb80155da0 0x7fbb8014be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:48.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.736+0000 7fbb7ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbb80155da0 0x7fbb8014be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51944/0 (socket says 172.21.15.112:51944) 2023-01-24T03:32:48.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.736+0000 7fbb7ffff700 1 -- 172.21.15.112:0/3162606059 learned_addr learned my addr 172.21.15.112:0/3162606059 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:32:48.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.736+0000 7fbb7effd700 1 --2- 172.21.15.112:0/3162606059 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb800ca3e0 0x7fbb800c1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:48.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.736+0000 7fbb7f7fe700 1 --2- 172.21.15.112:0/3162606059 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbb800c7e40 0x7fbb801687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:48.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.736+0000 7fbb7ffff700 1 -- 172.21.15.112:0/3162606059 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbb800c7e40 msgr2=0x7fbb801687b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:48.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.736+0000 7fbb7ffff700 1 --2- 172.21.15.112:0/3162606059 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbb800c7e40 0x7fbb801687b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:48.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.736+0000 7fbb7ffff700 1 -- 172.21.15.112:0/3162606059 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb800ca3e0 msgr2=0x7fbb800c1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:48.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.736+0000 7fbb7ffff700 1 --2- 172.21.15.112:0/3162606059 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb800ca3e0 0x7fbb800c1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:48.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.736+0000 7fbb7ffff700 1 -- 172.21.15.112:0/3162606059 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb780092a0 con 0x7fbb80155da0 2023-01-24T03:32:48.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.736+0000 7fbb7f7fe700 1 --2- 172.21.15.112:0/3162606059 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbb800c7e40 0x7fbb801687b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:32:48.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.736+0000 7fbb7ffff700 1 --2- 172.21.15.112:0/3162606059 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbb80155da0 0x7fbb8014be60 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fbb78009240 tx=0x7fbb78016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:48.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.736+0000 7fbb677fe700 1 -- 172.21.15.112:0/3162606059 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb78025420 con 0x7fbb80155da0 2023-01-24T03:32:48.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.736+0000 7fbb677fe700 1 -- 172.21.15.112:0/3162606059 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb78025580 con 0x7fbb80155da0 2023-01-24T03:32:48.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.736+0000 7fbb86206700 1 -- 172.21.15.112:0/3162606059 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb80152080 con 0x7fbb80155da0 2023-01-24T03:32:48.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.736+0000 7fbb677fe700 1 -- 172.21.15.112:0/3162606059 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb78025730 con 0x7fbb80155da0 2023-01-24T03:32:48.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.736+0000 7fbb86206700 1 -- 172.21.15.112:0/3162606059 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb80152640 con 0x7fbb80155da0 2023-01-24T03:32:48.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.737+0000 7fbb86206700 1 -- 172.21.15.112:0/3162606059 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb800bf680 con 0x7fbb80155da0 2023-01-24T03:32:48.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.738+0000 7fbb677fe700 1 -- 172.21.15.112:0/3162606059 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fbb7802c070 con 0x7fbb80155da0 2023-01-24T03:32:48.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.738+0000 7fbb677fe700 1 --2- 172.21.15.112:0/3162606059 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fbb600611f0 0x7fbb600636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:48.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.738+0000 7fbb7f7fe700 1 --2- 172.21.15.112:0/3162606059 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fbb600611f0 0x7fbb600636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:48.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.738+0000 7fbb677fe700 1 -- 172.21.15.112:0/3162606059 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 7009+0+0 (secure 0 0 0) 0x7fbb7807f260 con 0x7fbb80155da0 2023-01-24T03:32:48.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.738+0000 7fbb7f7fe700 1 --2- 172.21.15.112:0/3162606059 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fbb600611f0 0x7fbb600636a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbb70007620 tx=0x7fbb7000b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:48.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.742+0000 7fbb677fe700 1 -- 172.21.15.112:0/3162606059 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fbb7804efd0 con 0x7fbb80155da0 2023-01-24T03:32:48.965 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.963+0000 7fbb86206700 1 -- 172.21.15.112:0/3162606059 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb8004bb80 con 0x7fbb80155da0 2023-01-24T03:32:48.966 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.964+0000 7fbb677fe700 1 -- 172.21.15.112:0/3162606059 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+6112 (secure 0 0 0) 0x7fbb780525f0 con 0x7fbb80155da0 2023-01-24T03:32:48.966 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:32:48.966 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4897,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/11022091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":11022091},{"type":"v1","addr":"172.21.15.142:6835","nonce":11022091}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:32:09.626128+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4288},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":0,"incarnation":11,"state":"up:active","state_seq":31,"addr":"172.21.15.142:6834/869267413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":869267413},{"type":"v1","addr":"172.21.15.142:6834","nonce":869267413}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:32:48.968 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.966+0000 7fbb86206700 1 -- 172.21.15.112:0/3162606059 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fbb600611f0 msgr2=0x7fbb600636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:48.968 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.966+0000 7fbb86206700 1 --2- 172.21.15.112:0/3162606059 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fbb600611f0 0x7fbb600636a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbb70007620 tx=0x7fbb7000b040 comp rx=0 tx=0).stop 2023-01-24T03:32:48.968 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.966+0000 7fbb86206700 1 -- 172.21.15.112:0/3162606059 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbb80155da0 msgr2=0x7fbb8014be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:48.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.966+0000 7fbb86206700 1 --2- 172.21.15.112:0/3162606059 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbb80155da0 0x7fbb8014be60 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fbb78009240 tx=0x7fbb78016000 comp rx=0 tx=0).stop 2023-01-24T03:32:48.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.966+0000 7fbb86206700 1 -- 172.21.15.112:0/3162606059 shutdown_connections 2023-01-24T03:32:48.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.966+0000 7fbb86206700 1 --2- 172.21.15.112:0/3162606059 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fbb600611f0 0x7fbb600636a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:48.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.966+0000 7fbb86206700 1 --2- 172.21.15.112:0/3162606059 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbb80155da0 0x7fbb8014be60 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:48.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.966+0000 7fbb86206700 1 --2- 172.21.15.112:0/3162606059 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbb800c7e40 0x7fbb801687b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:48.970 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.966+0000 7fbb86206700 1 --2- 172.21.15.112:0/3162606059 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb800ca3e0 0x7fbb800c1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:48.970 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.966+0000 7fbb86206700 1 -- 172.21.15.112:0/3162606059 >> 172.21.15.112:0/3162606059 conn(0x7fbb8005e2c0 msgr2=0x7fbb80068450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:32:48.970 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.966+0000 7fbb86206700 1 -- 172.21.15.112:0/3162606059 shutdown_connections 2023-01-24T03:32:48.970 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:48.966+0000 7fbb86206700 1 -- 172.21.15.112:0/3162606059 wait complete. 2023-01-24T03:32:48.971 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 15 2023-01-24T03:32:48.980 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-01-24T03:32:50.981 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:32:51.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.105+0000 7f72779ea700 1 Processor -- start 2023-01-24T03:32:51.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.105+0000 7f72779ea700 1 -- start start 2023-01-24T03:32:51.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.106+0000 7f72779ea700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f727014d760 0x7f72700c1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:51.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.106+0000 7f72779ea700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7270066ec0 con 0x7f727006ef60 2023-01-24T03:32:51.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.106+0000 7f72779ea700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72701571c0 con 0x7f727014d390 2023-01-24T03:32:51.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.106+0000 7f72779ea700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72700c3a00 con 0x7f727014d760 2023-01-24T03:32:51.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.106+0000 7f7275f87700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f727006ef60 0x7f727015db90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:32940/0 (socket says 172.21.15.112:32940) 2023-01-24T03:32:51.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.106+0000 7f7275f87700 1 -- 172.21.15.112:0/3433990569 learned_addr learned my addr 172.21.15.112:0/3433990569 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:32:51.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.107+0000 7f7274f85700 1 --2- 172.21.15.112:0/3433990569 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f727014d760 0x7f72700c1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:51.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.107+0000 7f726ffff700 1 -- 172.21.15.112:0/3433990569 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2048307796 0 0) 0x7f7270066ec0 con 0x7f727006ef60 2023-01-24T03:32:51.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.107+0000 7f726ffff700 1 -- 172.21.15.112:0/3433990569 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f725c0033f0 con 0x7f727006ef60 2023-01-24T03:32:51.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.107+0000 7f7274f85700 1 -- 172.21.15.112:0/3433990569 >> v1:172.21.15.142:6789/0 conn(0x7f727014d390 legacy=0x7f7270153220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:51.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.107+0000 7f726ffff700 1 -- 172.21.15.112:0/3433990569 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1349024124 0 0) 0x7f72701571c0 con 0x7f727014d390 2023-01-24T03:32:51.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.107+0000 7f7274f85700 1 -- 172.21.15.112:0/3433990569 >> v1:172.21.15.112:6789/0 conn(0x7f727006ef60 legacy=0x7f727015db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:51.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.107+0000 7f7274f85700 1 -- 172.21.15.112:0/3433990569 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7270157650 con 0x7f727014d760 2023-01-24T03:32:51.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.107+0000 7f7274f85700 1 --2- 172.21.15.112:0/3433990569 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f727014d760 0x7f72700c1d60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7264009790 tx=0x7f7264009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=225de8bf7d3fec68 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:51.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.107+0000 7f726ffff700 1 -- 172.21.15.112:0/3433990569 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7268001470 con 0x7f727014d760 2023-01-24T03:32:51.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.107+0000 7f726ffff700 1 -- 172.21.15.112:0/3433990569 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7264011440 con 0x7f727014d760 2023-01-24T03:32:51.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.107+0000 7f726ffff700 1 -- 172.21.15.112:0/3433990569 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72640117c0 con 0x7f727014d760 2023-01-24T03:32:51.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.108+0000 7f72779ea700 1 -- 172.21.15.112:0/3433990569 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f727014d760 msgr2=0x7f72700c1d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:51.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.108+0000 7f72779ea700 1 --2- 172.21.15.112:0/3433990569 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f727014d760 0x7f72700c1d60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7264009790 tx=0x7f7264009060 comp rx=0 tx=0).stop 2023-01-24T03:32:51.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.108+0000 7f72779ea700 1 -- 172.21.15.112:0/3433990569 shutdown_connections 2023-01-24T03:32:51.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.108+0000 7f72779ea700 1 --2- 172.21.15.112:0/3433990569 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f727014d760 0x7f72700c1d60 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:51.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.108+0000 7f72779ea700 1 -- 172.21.15.112:0/3433990569 >> 172.21.15.112:0/3433990569 conn(0x7f7270067f20 msgr2=0x7f7270068320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:32:51.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.108+0000 7f72779ea700 1 -- 172.21.15.112:0/3433990569 shutdown_connections 2023-01-24T03:32:51.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.108+0000 7f72779ea700 1 -- 172.21.15.112:0/3433990569 wait complete. 2023-01-24T03:32:51.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.109+0000 7f72779ea700 1 Processor -- start 2023-01-24T03:32:51.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.109+0000 7f72779ea700 1 -- start start 2023-01-24T03:32:51.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.110+0000 7f72779ea700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f727006ef60 0x7f7270148b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:51.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.110+0000 7f72779ea700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f727014d390 0x7f7270149f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:51.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.110+0000 7f72779ea700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f727014d760 0x7f727006c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:51.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.110+0000 7f72779ea700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72700bec50 con 0x7f727006ef60 2023-01-24T03:32:51.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.110+0000 7f72779ea700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7270148360 con 0x7f727014d390 2023-01-24T03:32:51.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.110+0000 7f72779ea700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f727014c960 con 0x7f727014d760 2023-01-24T03:32:51.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.110+0000 7f7275786700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f727006ef60 0x7f7270148b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:51.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.110+0000 7f7275786700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f727006ef60 0x7f7270148b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51954/0 (socket says 172.21.15.112:51954) 2023-01-24T03:32:51.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.110+0000 7f7275786700 1 -- 172.21.15.112:0/2655764297 learned_addr learned my addr 172.21.15.112:0/2655764297 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:32:51.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.110+0000 7f7274f85700 1 --2- 172.21.15.112:0/2655764297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f727014d390 0x7f7270149f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:51.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.110+0000 7f7275f87700 1 --2- 172.21.15.112:0/2655764297 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f727014d760 0x7f727006c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:51.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.110+0000 7f7275786700 1 -- 172.21.15.112:0/2655764297 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f727014d760 msgr2=0x7f727006c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:51.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.110+0000 7f7275786700 1 --2- 172.21.15.112:0/2655764297 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f727014d760 0x7f727006c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:51.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.110+0000 7f7275786700 1 -- 172.21.15.112:0/2655764297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f727014d390 msgr2=0x7f7270149f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:51.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.110+0000 7f7275786700 1 --2- 172.21.15.112:0/2655764297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f727014d390 0x7f7270149f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:51.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.110+0000 7f7275786700 1 -- 172.21.15.112:0/2655764297 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72640092a0 con 0x7f727006ef60 2023-01-24T03:32:51.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.110+0000 7f7274f85700 1 --2- 172.21.15.112:0/2655764297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f727014d390 0x7f7270149f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-01-24T03:32:51.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.110+0000 7f7275f87700 1 --2- 172.21.15.112:0/2655764297 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f727014d760 0x7f727006c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-01-24T03:32:51.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.110+0000 7f7275786700 1 --2- 172.21.15.112:0/2655764297 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f727006ef60 0x7f7270148b10 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f726000c990 tx=0x7f7260001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:51.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.110+0000 7f726d7fa700 1 -- 172.21.15.112:0/2655764297 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f726000a700 con 0x7f727006ef60 2023-01-24T03:32:51.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.111+0000 7f726d7fa700 1 -- 172.21.15.112:0/2655764297 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f726000a880 con 0x7f727006ef60 2023-01-24T03:32:51.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.111+0000 7f726d7fa700 1 -- 172.21.15.112:0/2655764297 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f726000ac40 con 0x7f727006ef60 2023-01-24T03:32:51.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.111+0000 7f72779ea700 1 -- 172.21.15.112:0/2655764297 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7270149b00 con 0x7f727006ef60 2023-01-24T03:32:51.124 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.111+0000 7f72779ea700 1 -- 172.21.15.112:0/2655764297 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72700bc520 con 0x7f727006ef60 2023-01-24T03:32:51.124 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.111+0000 7f72779ea700 1 -- 172.21.15.112:0/2655764297 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7270149de0 con 0x7f727006ef60 2023-01-24T03:32:51.124 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.112+0000 7f726d7fa700 1 -- 172.21.15.112:0/2655764297 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f7260018020 con 0x7f727006ef60 2023-01-24T03:32:51.124 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.112+0000 7f726d7fa700 1 --2- 172.21.15.112:0/2655764297 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f72500611c0 0x7f7250063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:51.124 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.112+0000 7f726d7fa700 1 -- 172.21.15.112:0/2655764297 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 7009+0+0 (secure 0 0 0) 0x7f726007daf0 con 0x7f727006ef60 2023-01-24T03:32:51.125 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.112+0000 7f7274f85700 1 --2- 172.21.15.112:0/2655764297 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f72500611c0 0x7f7250063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:51.125 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.112+0000 7f7274f85700 1 --2- 172.21.15.112:0/2655764297 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f72500611c0 0x7f7250063670 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7264009790 tx=0x7f7264016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:51.125 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.116+0000 7f726d7fa700 1 -- 172.21.15.112:0/2655764297 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f7260081030 con 0x7f727006ef60 2023-01-24T03:32:51.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.364+0000 7f72779ea700 1 -- 172.21.15.112:0/2655764297 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72700c7bf0 con 0x7f727006ef60 2023-01-24T03:32:51.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.367+0000 7f726d7fa700 1 -- 172.21.15.112:0/2655764297 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+6112 (secure 0 0 0) 0x7f726004d860 con 0x7f727006ef60 2023-01-24T03:32:51.369 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:32:51.370 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4897,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/11022091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":11022091},{"type":"v1","addr":"172.21.15.142:6835","nonce":11022091}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:32:09.626128+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4288},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4288":{"gid":4288,"name":"b","rank":0,"incarnation":11,"state":"up:active","state_seq":31,"addr":"172.21.15.142:6834/869267413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":869267413},{"type":"v1","addr":"172.21.15.142:6834","nonce":869267413}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:32:51.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.372+0000 7f72779ea700 1 -- 172.21.15.112:0/2655764297 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f72500611c0 msgr2=0x7f7250063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:51.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.372+0000 7f72779ea700 1 --2- 172.21.15.112:0/2655764297 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f72500611c0 0x7f7250063670 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7264009790 tx=0x7f7264016000 comp rx=0 tx=0).stop 2023-01-24T03:32:51.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.372+0000 7f72779ea700 1 -- 172.21.15.112:0/2655764297 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f727006ef60 msgr2=0x7f7270148b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:51.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.372+0000 7f72779ea700 1 --2- 172.21.15.112:0/2655764297 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f727006ef60 0x7f7270148b10 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f726000c990 tx=0x7f7260001630 comp rx=0 tx=0).stop 2023-01-24T03:32:51.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.373+0000 7f72779ea700 1 -- 172.21.15.112:0/2655764297 shutdown_connections 2023-01-24T03:32:51.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.373+0000 7f72779ea700 1 --2- 172.21.15.112:0/2655764297 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f72500611c0 0x7f7250063670 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:51.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.373+0000 7f72779ea700 1 --2- 172.21.15.112:0/2655764297 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f727006ef60 0x7f7270148b10 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:51.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.373+0000 7f72779ea700 1 --2- 172.21.15.112:0/2655764297 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f727014d760 0x7f727006c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:51.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.373+0000 7f72779ea700 1 --2- 172.21.15.112:0/2655764297 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f727014d390 0x7f7270149f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:51.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.373+0000 7f72779ea700 1 -- 172.21.15.112:0/2655764297 >> 172.21.15.112:0/2655764297 conn(0x7f7270067f20 msgr2=0x7f727006b930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:32:51.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.373+0000 7f72779ea700 1 -- 172.21.15.112:0/2655764297 shutdown_connections 2023-01-24T03:32:51.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:51.373+0000 7f72779ea700 1 -- 172.21.15.112:0/2655764297 wait complete. 2023-01-24T03:32:51.380 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 15 2023-01-24T03:32:51.399 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-01-24T03:32:53.399 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:32:53.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.521+0000 7f3295a5a700 1 Processor -- start 2023-01-24T03:32:53.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.521+0000 7f3295a5a700 1 -- start start 2023-01-24T03:32:53.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.522+0000 7f3295a5a700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f32900c7e40 0x7f32900b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:53.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.522+0000 7f3295a5a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3290155da0 0x7f32900ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:53.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.522+0000 7f3295a5a700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3290156240 con 0x7f32900aef20 2023-01-24T03:32:53.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.522+0000 7f3295a5a700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32900b1170 con 0x7f32900c7e40 2023-01-24T03:32:53.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.522+0000 7f3295a5a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32900b1450 con 0x7f3290155da0 2023-01-24T03:32:53.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.522+0000 7f328e7fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3290155da0 0x7f32900ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:53.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.522+0000 7f328effd700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f32900c7e40 0x7f32900b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:53.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.522+0000 7f328f7fe700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f32900aef20 0x7f32900ca3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.112:46750/0 (socket says 172.21.15.112:46750) 2023-01-24T03:32:53.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.522+0000 7f328f7fe700 1 -- 172.21.15.112:0/2883123467 learned_addr learned my addr 172.21.15.112:0/2883123467 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:32:53.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.523+0000 7f328e7fc700 1 -- 172.21.15.112:0/2883123467 >> v2:172.21.15.142:3300/0 conn(0x7f32900c7e40 msgr2=0x7f32900b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:32:53.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.523+0000 7f328e7fc700 1 --2- 172.21.15.112:0/2883123467 >> v2:172.21.15.142:3300/0 conn(0x7f32900c7e40 0x7f32900b0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:53.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.523+0000 7f328e7fc700 1 -- 172.21.15.112:0/2883123467 >> v1:172.21.15.142:6789/0 conn(0x7f32900aef20 legacy=0x7f32900ca3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:53.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.523+0000 7f328e7fc700 1 -- 172.21.15.112:0/2883123467 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32900b1730 con 0x7f3290155da0 2023-01-24T03:32:53.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.523+0000 7f328effd700 1 --2- 172.21.15.112:0/2883123467 >> v2:172.21.15.142:3300/0 conn(0x7f32900c7e40 0x7f32900b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:32:53.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.523+0000 7f328e7fc700 1 --2- 172.21.15.112:0/2883123467 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3290155da0 0x7f32900ae9e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3278009760 tx=0x7f3278009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=737528e653d11bf server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:53.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.523+0000 7f328dffb700 1 -- 172.21.15.112:0/2883123467 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3278011630 con 0x7f3290155da0 2023-01-24T03:32:53.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.523+0000 7f328dffb700 1 -- 172.21.15.112:0/2883123467 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3278011790 con 0x7f3290155da0 2023-01-24T03:32:53.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.523+0000 7f328dffb700 1 -- 172.21.15.112:0/2883123467 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3278011b50 con 0x7f3290155da0 2023-01-24T03:32:53.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.524+0000 7f3295a5a700 1 -- 172.21.15.112:0/2883123467 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3290155da0 msgr2=0x7f32900ae9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:53.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.524+0000 7f3295a5a700 1 --2- 172.21.15.112:0/2883123467 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3290155da0 0x7f32900ae9e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3278009760 tx=0x7f3278009060 comp rx=0 tx=0).stop 2023-01-24T03:32:53.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.524+0000 7f3295a5a700 1 -- 172.21.15.112:0/2883123467 shutdown_connections 2023-01-24T03:32:53.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.524+0000 7f3295a5a700 1 --2- 172.21.15.112:0/2883123467 >> v2:172.21.15.142:3300/0 conn(0x7f32900c7e40 0x7f32900b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:53.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.524+0000 7f3295a5a700 1 --2- 172.21.15.112:0/2883123467 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3290155da0 0x7f32900ae9e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:53.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.524+0000 7f3295a5a700 1 -- 172.21.15.112:0/2883123467 >> 172.21.15.112:0/2883123467 conn(0x7f329005e2c0 msgr2=0x7f32900cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:32:53.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.524+0000 7f3295a5a700 1 -- 172.21.15.112:0/2883123467 shutdown_connections 2023-01-24T03:32:53.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.524+0000 7f3295a5a700 1 -- 172.21.15.112:0/2883123467 wait complete. 2023-01-24T03:32:53.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.525+0000 7f3295a5a700 1 Processor -- start 2023-01-24T03:32:53.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.525+0000 7f3295a5a700 1 -- start start 2023-01-24T03:32:53.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.526+0000 7f3295a5a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32900aef20 0x7f32900bf840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:53.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.526+0000 7f3295a5a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f32900c7e40 0x7f3290165eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:53.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.526+0000 7f3295a5a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3290155da0 0x7f3290154190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:53.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.526+0000 7f3295a5a700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32901663f0 con 0x7f32900c7e40 2023-01-24T03:32:53.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.526+0000 7f3295a5a700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32901552d0 con 0x7f32900aef20 2023-01-24T03:32:53.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.526+0000 7f3295a5a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32900c2c00 con 0x7f3290155da0 2023-01-24T03:32:53.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.526+0000 7f328e7fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f32900c7e40 0x7f3290165eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:53.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.526+0000 7f328e7fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f32900c7e40 0x7f3290165eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51976/0 (socket says 172.21.15.112:51976) 2023-01-24T03:32:53.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.526+0000 7f328e7fc700 1 -- 172.21.15.112:0/404102657 learned_addr learned my addr 172.21.15.112:0/404102657 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:32:53.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.526+0000 7f328effd700 1 --2- 172.21.15.112:0/404102657 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32900aef20 0x7f32900bf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:53.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.526+0000 7f328f7fe700 1 --2- 172.21.15.112:0/404102657 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3290155da0 0x7f3290154190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:53.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.526+0000 7f328e7fc700 1 -- 172.21.15.112:0/404102657 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3290155da0 msgr2=0x7f3290154190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:53.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.526+0000 7f328e7fc700 1 --2- 172.21.15.112:0/404102657 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3290155da0 0x7f3290154190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:53.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.526+0000 7f328e7fc700 1 -- 172.21.15.112:0/404102657 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32900aef20 msgr2=0x7f32900bf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:53.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.526+0000 7f328e7fc700 1 --2- 172.21.15.112:0/404102657 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32900aef20 0x7f32900bf840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:53.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.526+0000 7f328e7fc700 1 -- 172.21.15.112:0/404102657 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3280009a20 con 0x7f32900c7e40 2023-01-24T03:32:53.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.526+0000 7f328e7fc700 1 --2- 172.21.15.112:0/404102657 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f32900c7e40 0x7f3290165eb0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3278009e00 tx=0x7f3278017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:53.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.526+0000 7f3286ffd700 1 -- 172.21.15.112:0/404102657 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3278011900 con 0x7f32900c7e40 2023-01-24T03:32:53.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.527+0000 7f3286ffd700 1 -- 172.21.15.112:0/404102657 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3278011a80 con 0x7f32900c7e40 2023-01-24T03:32:53.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.527+0000 7f3295a5a700 1 -- 172.21.15.112:0/404102657 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32780092a0 con 0x7f32900c7e40 2023-01-24T03:32:53.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.527+0000 7f3286ffd700 1 -- 172.21.15.112:0/404102657 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327801a680 con 0x7f32900c7e40 2023-01-24T03:32:53.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.527+0000 7f3295a5a700 1 -- 172.21.15.112:0/404102657 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3290154c70 con 0x7f32900c7e40 2023-01-24T03:32:53.540 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.528+0000 7f3295a5a700 1 -- 172.21.15.112:0/404102657 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32900bf680 con 0x7f32900c7e40 2023-01-24T03:32:53.540 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.528+0000 7f3286ffd700 1 -- 172.21.15.112:0/404102657 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f327800f050 con 0x7f32900c7e40 2023-01-24T03:32:53.540 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.528+0000 7f3286ffd700 1 --2- 172.21.15.112:0/404102657 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f32740611c0 0x7f3274063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:53.540 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.528+0000 7f3286ffd700 1 -- 172.21.15.112:0/404102657 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 7009+0+0 (secure 0 0 0) 0x7f327807f1e0 con 0x7f32900c7e40 2023-01-24T03:32:53.540 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.528+0000 7f328effd700 1 --2- 172.21.15.112:0/404102657 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f32740611c0 0x7f3274063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:53.540 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.529+0000 7f328effd700 1 --2- 172.21.15.112:0/404102657 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f32740611c0 0x7f3274063670 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3280009fd0 tx=0x7f328000b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:53.541 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.532+0000 7f3286ffd700 1 -- 172.21.15.112:0/404102657 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f327804ef50 con 0x7f32900c7e40 2023-01-24T03:32:53.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.755+0000 7f3295a5a700 1 -- 172.21.15.112:0/404102657 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f329004bb80 con 0x7f32900c7e40 2023-01-24T03:32:53.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.755+0000 7f3286ffd700 1 -- 172.21.15.112:0/404102657 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+5354 (secure 0 0 0) 0x7f3278052570 con 0x7f32900c7e40 2023-01-24T03:32:53.758 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:32:53.758 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:32:53.608119+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4897},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4897":{"gid":4897,"name":"d","rank":0,"incarnation":16,"state":"up:replay","state_seq":1,"addr":"172.21.15.142:6835/11022091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":11022091},{"type":"v1","addr":"172.21.15.142:6835","nonce":11022091}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:32:53.760 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.757+0000 7f3295a5a700 1 -- 172.21.15.112:0/404102657 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f32740611c0 msgr2=0x7f3274063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:53.760 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.757+0000 7f3295a5a700 1 --2- 172.21.15.112:0/404102657 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f32740611c0 0x7f3274063670 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3280009fd0 tx=0x7f328000b040 comp rx=0 tx=0).stop 2023-01-24T03:32:53.760 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.757+0000 7f3295a5a700 1 -- 172.21.15.112:0/404102657 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f32900c7e40 msgr2=0x7f3290165eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:53.760 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.757+0000 7f3295a5a700 1 --2- 172.21.15.112:0/404102657 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f32900c7e40 0x7f3290165eb0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3278009e00 tx=0x7f3278017000 comp rx=0 tx=0).stop 2023-01-24T03:32:53.760 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.757+0000 7f3295a5a700 1 -- 172.21.15.112:0/404102657 shutdown_connections 2023-01-24T03:32:53.761 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.757+0000 7f3295a5a700 1 --2- 172.21.15.112:0/404102657 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f32740611c0 0x7f3274063670 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:53.761 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.757+0000 7f3295a5a700 1 --2- 172.21.15.112:0/404102657 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3290155da0 0x7f3290154190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:53.761 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.757+0000 7f3295a5a700 1 --2- 172.21.15.112:0/404102657 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f32900aef20 0x7f32900bf840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:53.761 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.757+0000 7f3295a5a700 1 --2- 172.21.15.112:0/404102657 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f32900c7e40 0x7f3290165eb0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:53.761 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.757+0000 7f3295a5a700 1 -- 172.21.15.112:0/404102657 >> 172.21.15.112:0/404102657 conn(0x7f329005e2c0 msgr2=0x7f32900cab10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:32:53.762 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.757+0000 7f3295a5a700 1 -- 172.21.15.112:0/404102657 shutdown_connections 2023-01-24T03:32:53.762 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.757+0000 7f3295a5a700 1 -- 172.21.15.112:0/404102657 wait complete. 2023-01-24T03:32:53.763 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 16 2023-01-24T03:32:53.771 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2023-01-24T03:32:53.771 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-01-24T03:32:53.772 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:32:53.885 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.883+0000 7f50f5ce1700 1 Processor -- start 2023-01-24T03:32:53.885 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.883+0000 7f50f5ce1700 1 -- start start 2023-01-24T03:32:53.888 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.884+0000 7f50f5ce1700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f50f00c5680 0x7f50f0156730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:53.888 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.884+0000 7f50f5ce1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f50f0155e20 0x7f50f0156c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:53.889 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.884+0000 7f50f5ce1700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50f00c3d10 con 0x7f50f00c5a50 2023-01-24T03:32:53.889 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.884+0000 7f50f5ce1700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50f00c4eb0 con 0x7f50f00c5680 2023-01-24T03:32:53.889 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.884+0000 7f50f5ce1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50f00c2e20 con 0x7f50f0155e20 2023-01-24T03:32:53.889 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.884+0000 7f50f4cdf700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f50f00c5680 0x7f50f0156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:53.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.884+0000 7f50f4cdf700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f50f00c5680 0x7f50f0156730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51982/0 (socket says 172.21.15.112:51982) 2023-01-24T03:32:53.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.884+0000 7f50f4cdf700 1 -- 172.21.15.112:0/3301740681 learned_addr learned my addr 172.21.15.112:0/3301740681 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:32:53.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.884+0000 7f50f54e0700 1 --1- 172.21.15.112:0/3301740681 >> v1:172.21.15.112:6789/0 conn(0x7f50f00c5a50 0x7f50f00cb870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:32974/0 (socket says 172.21.15.112:32974) 2023-01-24T03:32:53.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.884+0000 7f50effff700 1 --2- 172.21.15.112:0/3301740681 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f50f0155e20 0x7f50f0156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:53.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.884+0000 7f50f4cdf700 1 -- 172.21.15.112:0/3301740681 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f50f0155e20 msgr2=0x7f50f0156c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:32:53.891 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.884+0000 7f50f4cdf700 1 --2- 172.21.15.112:0/3301740681 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f50f0155e20 0x7f50f0156c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:53.891 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.884+0000 7f50f4cdf700 1 -- 172.21.15.112:0/3301740681 >> v1:172.21.15.112:6789/0 conn(0x7f50f00c5a50 legacy=0x7f50f00cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:53.891 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.884+0000 7f50f4cdf700 1 -- 172.21.15.112:0/3301740681 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50f00c3100 con 0x7f50f00c5680 2023-01-24T03:32:53.891 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.884+0000 7f50ef7fe700 1 -- 172.21.15.112:0/3301740681 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4083260311 0 0) 0x7f50f00c3d10 con 0x7f50f00c5a50 2023-01-24T03:32:53.891 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.884+0000 7f50f4cdf700 1 --2- 172.21.15.112:0/3301740681 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f50f00c5680 0x7f50f0156730 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f50e8009f50 tx=0x7f50e8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e27b9a60b7d44cf server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:53.892 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.884+0000 7f50ef7fe700 1 -- 172.21.15.112:0/3301740681 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e800e070 con 0x7f50f00c5680 2023-01-24T03:32:53.892 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.884+0000 7f50ef7fe700 1 -- 172.21.15.112:0/3301740681 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50e800b9c0 con 0x7f50f00c5680 2023-01-24T03:32:53.892 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.885+0000 7f50ef7fe700 1 -- 172.21.15.112:0/3301740681 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e80092e0 con 0x7f50f00c5680 2023-01-24T03:32:53.892 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.885+0000 7f50f5ce1700 1 -- 172.21.15.112:0/3301740681 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f50f00c5680 msgr2=0x7f50f0156730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:53.892 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.885+0000 7f50f5ce1700 1 --2- 172.21.15.112:0/3301740681 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f50f00c5680 0x7f50f0156730 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f50e8009f50 tx=0x7f50e8009840 comp rx=0 tx=0).stop 2023-01-24T03:32:53.893 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.885+0000 7f50f5ce1700 1 -- 172.21.15.112:0/3301740681 shutdown_connections 2023-01-24T03:32:53.893 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.885+0000 7f50f5ce1700 1 --2- 172.21.15.112:0/3301740681 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f50f00c5680 0x7f50f0156730 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:53.893 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.885+0000 7f50f5ce1700 1 --2- 172.21.15.112:0/3301740681 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f50f0155e20 0x7f50f0156c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:53.893 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.885+0000 7f50f5ce1700 1 -- 172.21.15.112:0/3301740681 >> 172.21.15.112:0/3301740681 conn(0x7f50f00685b0 msgr2=0x7f50f00689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:32:53.893 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.885+0000 7f50f5ce1700 1 -- 172.21.15.112:0/3301740681 shutdown_connections 2023-01-24T03:32:53.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.885+0000 7f50f5ce1700 1 -- 172.21.15.112:0/3301740681 wait complete. 2023-01-24T03:32:53.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.886+0000 7f50f5ce1700 1 Processor -- start 2023-01-24T03:32:53.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.886+0000 7f50f5ce1700 1 -- start start 2023-01-24T03:32:53.897 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.887+0000 7f50f5ce1700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f50f00c5680 0x7f50f01658c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:53.897 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.887+0000 7f50f5ce1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f50f00c5a50 0x7f50f0151b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:53.897 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.887+0000 7f50f5ce1700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f50f0155e20 0x7f50f0150ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:53.897 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.887+0000 7f50f5ce1700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50f0165e00 con 0x7f50f00c5680 2023-01-24T03:32:53.897 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.887+0000 7f50f5ce1700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50f0150650 con 0x7f50f0155e20 2023-01-24T03:32:53.898 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.887+0000 7f50f5ce1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50f00bf4c0 con 0x7f50f00c5a50 2023-01-24T03:32:53.898 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.887+0000 7f50f4cdf700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f50f00c5680 0x7f50f01658c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:53.898 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.887+0000 7f50f4cdf700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f50f00c5680 0x7f50f01658c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51988/0 (socket says 172.21.15.112:51988) 2023-01-24T03:32:53.898 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.887+0000 7f50f4cdf700 1 -- 172.21.15.112:0/1154962855 learned_addr learned my addr 172.21.15.112:0/1154962855 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:32:53.898 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.887+0000 7f50f54e0700 1 --2- 172.21.15.112:0/1154962855 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f50f0155e20 0x7f50f0150ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:53.899 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.887+0000 7f50effff700 1 --2- 172.21.15.112:0/1154962855 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f50f00c5a50 0x7f50f0151b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:53.899 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.887+0000 7f50f4cdf700 1 -- 172.21.15.112:0/1154962855 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f50f00c5a50 msgr2=0x7f50f0151b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:53.899 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.887+0000 7f50f4cdf700 1 --2- 172.21.15.112:0/1154962855 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f50f00c5a50 0x7f50f0151b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:53.899 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.887+0000 7f50f4cdf700 1 -- 172.21.15.112:0/1154962855 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f50f0155e20 msgr2=0x7f50f0150ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:53.899 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.887+0000 7f50f4cdf700 1 --2- 172.21.15.112:0/1154962855 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f50f0155e20 0x7f50f0150ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:53.900 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.887+0000 7f50f4cdf700 1 -- 172.21.15.112:0/1154962855 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e8009ae0 con 0x7f50f00c5680 2023-01-24T03:32:53.900 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.887+0000 7f50f4cdf700 1 --2- 172.21.15.112:0/1154962855 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f50f00c5680 0x7f50f01658c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f50f0088fd0 tx=0x7f50e8018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:53.900 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.887+0000 7f50ecff9700 1 -- 172.21.15.112:0/1154962855 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e8008950 con 0x7f50f00c5680 2023-01-24T03:32:53.900 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.888+0000 7f50ecff9700 1 -- 172.21.15.112:0/1154962855 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50e8008ad0 con 0x7f50f00c5680 2023-01-24T03:32:53.900 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.888+0000 7f50ecff9700 1 -- 172.21.15.112:0/1154962855 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e8008e90 con 0x7f50f00c5680 2023-01-24T03:32:53.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.888+0000 7f50f5ce1700 1 -- 172.21.15.112:0/1154962855 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50f0152980 con 0x7f50f00c5680 2023-01-24T03:32:53.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.888+0000 7f50f5ce1700 1 -- 172.21.15.112:0/1154962855 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50f0150000 con 0x7f50f00c5680 2023-01-24T03:32:53.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.889+0000 7f50f5ce1700 1 -- 172.21.15.112:0/1154962855 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50f0150930 con 0x7f50f00c5680 2023-01-24T03:32:53.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.889+0000 7f50ecff9700 1 -- 172.21.15.112:0/1154962855 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f50e802d070 con 0x7f50f00c5680 2023-01-24T03:32:53.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.889+0000 7f50ecff9700 1 --2- 172.21.15.112:0/1154962855 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f50d40611f0 0x7f50d40636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:53.902 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.889+0000 7f50ecff9700 1 -- 172.21.15.112:0/1154962855 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7039+0+0 (secure 0 0 0) 0x7f50e807f1e0 con 0x7f50f00c5680 2023-01-24T03:32:53.902 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.889+0000 7f50effff700 1 --2- 172.21.15.112:0/1154962855 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f50d40611f0 0x7f50d40636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:53.902 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.890+0000 7f50effff700 1 --2- 172.21.15.112:0/1154962855 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f50d40611f0 0x7f50d40636a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f50f00c2d50 tx=0x7f50e0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:53.902 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:53.893+0000 7f50ecff9700 1 -- 172.21.15.112:0/1154962855 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f50e804ef40 con 0x7f50f00c5680 2023-01-24T03:32:54.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:54.113+0000 7f50f5ce1700 1 -- 172.21.15.112:0/1154962855 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f50f005ea20 con 0x7f50f00c5680 2023-01-24T03:32:54.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:54.113+0000 7f50ecff9700 1 -- 172.21.15.112:0/1154962855 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+5354 (secure 0 0 0) 0x7f50e8052560 con 0x7f50f00c5680 2023-01-24T03:32:54.116 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:32:54.116 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:32:53.608119+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4897},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4897":{"gid":4897,"name":"d","rank":0,"incarnation":16,"state":"up:replay","state_seq":1,"addr":"172.21.15.142:6835/11022091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":11022091},{"type":"v1","addr":"172.21.15.142:6835","nonce":11022091}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:32:54.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:54.115+0000 7f50f5ce1700 1 -- 172.21.15.112:0/1154962855 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f50d40611f0 msgr2=0x7f50d40636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:54.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:54.115+0000 7f50f5ce1700 1 --2- 172.21.15.112:0/1154962855 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f50d40611f0 0x7f50d40636a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f50f00c2d50 tx=0x7f50e0008040 comp rx=0 tx=0).stop 2023-01-24T03:32:54.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:54.115+0000 7f50f5ce1700 1 -- 172.21.15.112:0/1154962855 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f50f00c5680 msgr2=0x7f50f01658c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:54.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:54.115+0000 7f50f5ce1700 1 --2- 172.21.15.112:0/1154962855 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f50f00c5680 0x7f50f01658c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f50f0088fd0 tx=0x7f50e8018000 comp rx=0 tx=0).stop 2023-01-24T03:32:54.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:54.115+0000 7f50f5ce1700 1 -- 172.21.15.112:0/1154962855 shutdown_connections 2023-01-24T03:32:54.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:54.115+0000 7f50f5ce1700 1 --2- 172.21.15.112:0/1154962855 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f50d40611f0 0x7f50d40636a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:54.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:54.115+0000 7f50f5ce1700 1 --2- 172.21.15.112:0/1154962855 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f50f00c5680 0x7f50f01658c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:54.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:54.115+0000 7f50f5ce1700 1 --2- 172.21.15.112:0/1154962855 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f50f0155e20 0x7f50f0150ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:54.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:54.115+0000 7f50f5ce1700 1 --2- 172.21.15.112:0/1154962855 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f50f00c5a50 0x7f50f0151b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:54.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:54.115+0000 7f50f5ce1700 1 -- 172.21.15.112:0/1154962855 >> 172.21.15.112:0/1154962855 conn(0x7f50f00685b0 msgr2=0x7f50f0153080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:32:54.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:54.116+0000 7f50f5ce1700 1 -- 172.21.15.112:0/1154962855 shutdown_connections 2023-01-24T03:32:54.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:54.116+0000 7f50f5ce1700 1 -- 172.21.15.112:0/1154962855 wait complete. 2023-01-24T03:32:54.120 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 16 2023-01-24T03:32:54.130 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 16, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-01-24T03:30:20.971710+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 16, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4897": { "addr": "172.21.15.142:6835/11022091", "addrs": { "addrvec": [ { "addr": "172.21.15.142:6833", "nonce": 11022091, "type": "v2" }, { "addr": "172.21.15.142:6835", "nonce": 11022091, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4897, "incarnation": 16, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 31, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-01-24T03:32:53.608119+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4897 } } } ], "standbys": [ { "addr": "172.21.15.142:6837/2181345869", "addrs": { "addrvec": [ { "addr": "172.21.15.142:6836", "nonce": 2181345869, "type": "v2" }, { "addr": "172.21.15.142:6837", "nonce": 2181345869, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6835/330343927", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6834", "nonce": 330343927, "type": "v2" }, { "addr": "172.21.15.112:6835", "nonce": 330343927, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6837/12176817", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6836", "nonce": 12176817, "type": "v2" }, { "addr": "172.21.15.112:6837", "nonce": 12176817, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4276, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6839/1110983841", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6838", "nonce": 1110983841, "type": "v2" }, { "addr": "172.21.15.112:6839", "nonce": 1110983841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4281, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-01-24T03:32:56.131 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:32:56.256 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.254+0000 7f9c212dc700 1 Processor -- start 2023-01-24T03:32:56.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.254+0000 7f9c212dc700 1 -- start start 2023-01-24T03:32:56.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.255+0000 7f9c212dc700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f9c1c0c7e40 0x7f9c1c0b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:56.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.255+0000 7f9c212dc700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f9c1c155da0 0x7f9c1c0ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:56.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.255+0000 7f9c212dc700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c1c156240 con 0x7f9c1c0aef20 2023-01-24T03:32:56.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.255+0000 7f9c212dc700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c1c0af320 con 0x7f9c1c0c7e40 2023-01-24T03:32:56.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.255+0000 7f9c212dc700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c1c0b1170 con 0x7f9c1c155da0 2023-01-24T03:32:56.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.255+0000 7f9c1affd700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f9c1c0c7e40 0x7f9c1c0b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:56.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.255+0000 7f9c1affd700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f9c1c0c7e40 0x7f9c1c0b0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:51996/0 (socket says 172.21.15.112:51996) 2023-01-24T03:32:56.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.255+0000 7f9c1affd700 1 -- 172.21.15.112:0/3393037959 learned_addr learned my addr 172.21.15.112:0/3393037959 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:32:56.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.256+0000 7f9c1a7fc700 1 --2- 172.21.15.112:0/3393037959 >> v2:172.21.15.142:3300/0 conn(0x7f9c1c155da0 0x7f9c1c0ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:56.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.256+0000 7f9c1affd700 1 -- 172.21.15.112:0/3393037959 >> v2:172.21.15.142:3300/0 conn(0x7f9c1c155da0 msgr2=0x7f9c1c0ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:32:56.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.256+0000 7f9c1affd700 1 --2- 172.21.15.112:0/3393037959 >> v2:172.21.15.142:3300/0 conn(0x7f9c1c155da0 0x7f9c1c0ae9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:56.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.256+0000 7f9c1affd700 1 -- 172.21.15.112:0/3393037959 >> v1:172.21.15.112:6789/0 conn(0x7f9c1c0aef20 legacy=0x7f9c1c0ca3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:56.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.256+0000 7f9c1affd700 1 -- 172.21.15.112:0/3393037959 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c1c0b1450 con 0x7f9c1c0c7e40 2023-01-24T03:32:56.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.256+0000 7f9c1a7fc700 1 --2- 172.21.15.112:0/3393037959 >> v2:172.21.15.142:3300/0 conn(0x7f9c1c155da0 0x7f9c1c0ae9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:32:56.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.256+0000 7f9c1affd700 1 --2- 172.21.15.112:0/3393037959 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9c1c0c7e40 0x7f9c1c0b0aa0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c152ea0 tx=0x7f9c0c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89401a787a15a8a9 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:56.263 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.256+0000 7f9c19ffb700 1 -- 172.21.15.112:0/3393037959 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c00e070 con 0x7f9c1c0c7e40 2023-01-24T03:32:56.263 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.256+0000 7f9c19ffb700 1 -- 172.21.15.112:0/3393037959 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c0c00ba50 con 0x7f9c1c0c7e40 2023-01-24T03:32:56.263 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.256+0000 7f9c19ffb700 1 -- 172.21.15.112:0/3393037959 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c0092e0 con 0x7f9c1c0c7e40 2023-01-24T03:32:56.263 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.256+0000 7f9c212dc700 1 -- 172.21.15.112:0/3393037959 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9c1c0c7e40 msgr2=0x7f9c1c0b0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:56.263 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.257+0000 7f9c212dc700 1 --2- 172.21.15.112:0/3393037959 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9c1c0c7e40 0x7f9c1c0b0aa0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c152ea0 tx=0x7f9c0c009840 comp rx=0 tx=0).stop 2023-01-24T03:32:56.264 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.257+0000 7f9c212dc700 1 -- 172.21.15.112:0/3393037959 shutdown_connections 2023-01-24T03:32:56.264 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.257+0000 7f9c212dc700 1 --2- 172.21.15.112:0/3393037959 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9c1c0c7e40 0x7f9c1c0b0aa0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:56.264 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.257+0000 7f9c212dc700 1 --2- 172.21.15.112:0/3393037959 >> v2:172.21.15.142:3300/0 conn(0x7f9c1c155da0 0x7f9c1c0ae9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:56.264 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.257+0000 7f9c212dc700 1 -- 172.21.15.112:0/3393037959 >> 172.21.15.112:0/3393037959 conn(0x7f9c1c05e2c0 msgr2=0x7f9c1c0cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:32:56.264 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.257+0000 7f9c212dc700 1 -- 172.21.15.112:0/3393037959 shutdown_connections 2023-01-24T03:32:56.265 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.257+0000 7f9c212dc700 1 -- 172.21.15.112:0/3393037959 wait complete. 2023-01-24T03:32:56.268 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.258+0000 7f9c212dc700 1 Processor -- start 2023-01-24T03:32:56.268 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.258+0000 7f9c212dc700 1 -- start start 2023-01-24T03:32:56.268 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.259+0000 7f9c212dc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9c1c0aef20 0x7f9c1c14fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:56.269 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.259+0000 7f9c212dc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9c1c0c7e40 0x7f9c1c1507d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:56.269 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.259+0000 7f9c212dc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9c1c155da0 0x7f9c1c153fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:56.269 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.259+0000 7f9c212dc700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1c0c0ee0 con 0x7f9c1c155da0 2023-01-24T03:32:56.269 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.259+0000 7f9c212dc700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1c166310 con 0x7f9c1c0aef20 2023-01-24T03:32:56.270 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.259+0000 7f9c212dc700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1c151e20 con 0x7f9c1c0c7e40 2023-01-24T03:32:56.270 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.259+0000 7f9c1b7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9c1c155da0 0x7f9c1c153fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:56.270 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.259+0000 7f9c1b7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9c1c155da0 0x7f9c1c153fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52006/0 (socket says 172.21.15.112:52006) 2023-01-24T03:32:56.270 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.259+0000 7f9c1b7fe700 1 -- 172.21.15.112:0/4119978298 learned_addr learned my addr 172.21.15.112:0/4119978298 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:32:56.270 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.259+0000 7f9c1affd700 1 --2- 172.21.15.112:0/4119978298 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9c1c0aef20 0x7f9c1c14fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:56.271 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.259+0000 7f9c1a7fc700 1 --2- 172.21.15.112:0/4119978298 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9c1c0c7e40 0x7f9c1c1507d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:56.271 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.259+0000 7f9c1b7fe700 1 -- 172.21.15.112:0/4119978298 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9c1c0c7e40 msgr2=0x7f9c1c1507d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:56.271 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.259+0000 7f9c1b7fe700 1 --2- 172.21.15.112:0/4119978298 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9c1c0c7e40 0x7f9c1c1507d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:56.272 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.259+0000 7f9c1b7fe700 1 -- 172.21.15.112:0/4119978298 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9c1c0aef20 msgr2=0x7f9c1c14fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:56.272 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.259+0000 7f9c1b7fe700 1 --2- 172.21.15.112:0/4119978298 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9c1c0aef20 0x7f9c1c14fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:56.272 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.259+0000 7f9c1b7fe700 1 -- 172.21.15.112:0/4119978298 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c0c009ae0 con 0x7f9c1c155da0 2023-01-24T03:32:56.272 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.259+0000 7f9c1a7fc700 1 --2- 172.21.15.112:0/4119978298 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9c1c0c7e40 0x7f9c1c1507d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-01-24T03:32:56.272 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.259+0000 7f9c1affd700 1 --2- 172.21.15.112:0/4119978298 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9c1c0aef20 0x7f9c1c14fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:32:56.273 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.259+0000 7f9c1b7fe700 1 --2- 172.21.15.112:0/4119978298 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9c1c155da0 0x7f9c1c153fe0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9c1400c440 tx=0x7f9c14007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:56.273 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.259+0000 7f9c12ffd700 1 -- 172.21.15.112:0/4119978298 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c14011590 con 0x7f9c1c155da0 2023-01-24T03:32:56.273 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.259+0000 7f9c12ffd700 1 -- 172.21.15.112:0/4119978298 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c14011710 con 0x7f9c1c155da0 2023-01-24T03:32:56.273 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.259+0000 7f9c212dc700 1 -- 172.21.15.112:0/4119978298 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c1c0c2620 con 0x7f9c1c155da0 2023-01-24T03:32:56.273 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.259+0000 7f9c12ffd700 1 -- 172.21.15.112:0/4119978298 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c14011ad0 con 0x7f9c1c155da0 2023-01-24T03:32:56.274 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.259+0000 7f9c212dc700 1 -- 172.21.15.112:0/4119978298 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c1c165fa0 con 0x7f9c1c155da0 2023-01-24T03:32:56.274 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.260+0000 7f9c212dc700 1 -- 172.21.15.112:0/4119978298 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c1c04bb80 con 0x7f9c1c155da0 2023-01-24T03:32:56.274 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.260+0000 7f9c12ffd700 1 -- 172.21.15.112:0/4119978298 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f9c14019030 con 0x7f9c1c155da0 2023-01-24T03:32:56.274 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.261+0000 7f9c12ffd700 1 --2- 172.21.15.112:0/4119978298 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9c000611c0 0x7f9c00063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:56.274 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.261+0000 7f9c1affd700 1 --2- 172.21.15.112:0/4119978298 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9c000611c0 0x7f9c00063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:56.275 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.261+0000 7f9c12ffd700 1 -- 172.21.15.112:0/4119978298 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7039+0+0 (secure 0 0 0) 0x7f9c1407cac0 con 0x7f9c1c155da0 2023-01-24T03:32:56.275 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.261+0000 7f9c1affd700 1 --2- 172.21.15.112:0/4119978298 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9c000611c0 0x7f9c00063670 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c009880 tx=0x7f9c0c018000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:56.275 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.265+0000 7f9c12ffd700 1 -- 172.21.15.112:0/4119978298 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f9c1404c820 con 0x7f9c1c155da0 2023-01-24T03:32:56.556 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.554+0000 7f9c212dc700 1 -- 172.21.15.112:0/4119978298 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c1c0c9860 con 0x7f9c1c155da0 2023-01-24T03:32:56.558 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.556+0000 7f9c12ffd700 1 -- 172.21.15.112:0/4119978298 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+5354 (secure 0 0 0) 0x7f9c1404fe40 con 0x7f9c1c155da0 2023-01-24T03:32:56.561 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:32:56.562 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:32:53.608119+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4897},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4897":{"gid":4897,"name":"d","rank":0,"incarnation":16,"state":"up:replay","state_seq":1,"addr":"172.21.15.142:6835/11022091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":11022091},{"type":"v1","addr":"172.21.15.142:6835","nonce":11022091}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:32:56.562 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.560+0000 7f9c212dc700 1 -- 172.21.15.112:0/4119978298 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9c000611c0 msgr2=0x7f9c00063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:56.563 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.560+0000 7f9c212dc700 1 --2- 172.21.15.112:0/4119978298 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9c000611c0 0x7f9c00063670 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c009880 tx=0x7f9c0c018000 comp rx=0 tx=0).stop 2023-01-24T03:32:56.563 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.560+0000 7f9c212dc700 1 -- 172.21.15.112:0/4119978298 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9c1c155da0 msgr2=0x7f9c1c153fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:56.563 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.560+0000 7f9c212dc700 1 --2- 172.21.15.112:0/4119978298 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9c1c155da0 0x7f9c1c153fe0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9c1400c440 tx=0x7f9c14007dd0 comp rx=0 tx=0).stop 2023-01-24T03:32:56.563 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.560+0000 7f9c212dc700 1 -- 172.21.15.112:0/4119978298 shutdown_connections 2023-01-24T03:32:56.563 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.561+0000 7f9c212dc700 1 --2- 172.21.15.112:0/4119978298 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9c000611c0 0x7f9c00063670 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:56.564 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.561+0000 7f9c212dc700 1 --2- 172.21.15.112:0/4119978298 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9c1c155da0 0x7f9c1c153fe0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:56.564 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.561+0000 7f9c212dc700 1 --2- 172.21.15.112:0/4119978298 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9c1c0aef20 0x7f9c1c14fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:56.564 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.561+0000 7f9c212dc700 1 --2- 172.21.15.112:0/4119978298 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9c1c0c7e40 0x7f9c1c1507d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:56.564 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.561+0000 7f9c212dc700 1 -- 172.21.15.112:0/4119978298 >> 172.21.15.112:0/4119978298 conn(0x7f9c1c05e2c0 msgr2=0x7f9c1c0680c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:32:56.565 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.561+0000 7f9c212dc700 1 -- 172.21.15.112:0/4119978298 shutdown_connections 2023-01-24T03:32:56.565 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:56.561+0000 7f9c212dc700 1 -- 172.21.15.112:0/4119978298 wait complete. 2023-01-24T03:32:56.568 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 16 2023-01-24T03:32:56.586 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:32:58.586 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:32:58.711 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.709+0000 7f330f59e700 1 Processor -- start 2023-01-24T03:32:58.712 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.709+0000 7f330f59e700 1 -- start start 2023-01-24T03:32:58.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.710+0000 7f330f59e700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f3310067d90 0x7f3310068160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:58.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.710+0000 7f330f59e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f3310155bf0 0x7f33100c5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:58.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.710+0000 7f330f59e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33100686a0 con 0x7f33100c3f30 2023-01-24T03:32:58.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.710+0000 7f330f59e700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3310156070 con 0x7f3310067d90 2023-01-24T03:32:58.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.710+0000 7f330f59e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33100c5300 con 0x7f3310155bf0 2023-01-24T03:32:58.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.710+0000 7f330e59c700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f3310067d90 0x7f3310068160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:58.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.710+0000 7f330e59c700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f3310067d90 0x7f3310068160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52026/0 (socket says 172.21.15.112:52026) 2023-01-24T03:32:58.716 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.710+0000 7f330e59c700 1 -- 172.21.15.112:0/366139450 learned_addr learned my addr 172.21.15.112:0/366139450 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:32:58.716 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.710+0000 7f330dd9b700 1 --2- 172.21.15.112:0/366139450 >> v2:172.21.15.142:3300/0 conn(0x7f3310155bf0 0x7f33100c5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:58.716 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.711+0000 7f330e59c700 1 -- 172.21.15.112:0/366139450 >> v2:172.21.15.142:3300/0 conn(0x7f3310155bf0 msgr2=0x7f33100c5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:32:58.716 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.711+0000 7f330e59c700 1 --2- 172.21.15.112:0/366139450 >> v2:172.21.15.142:3300/0 conn(0x7f3310155bf0 0x7f33100c5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:58.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.711+0000 7f330e59c700 1 -- 172.21.15.112:0/366139450 >> v1:172.21.15.112:6789/0 conn(0x7f33100c3f30 legacy=0x7f33100c2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:58.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.711+0000 7f330e59c700 1 -- 172.21.15.112:0/366139450 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f331006a8a0 con 0x7f3310067d90 2023-01-24T03:32:58.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.711+0000 7f330e59c700 1 --2- 172.21.15.112:0/366139450 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3310067d90 0x7f3310068160 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f331006a100 tx=0x7f3300009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=877b761cd7231040 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:58.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.711+0000 7f330d59a700 1 -- 172.21.15.112:0/366139450 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330000e070 con 0x7f3310067d90 2023-01-24T03:32:58.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.711+0000 7f330d59a700 1 -- 172.21.15.112:0/366139450 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f330000bab0 con 0x7f3310067d90 2023-01-24T03:32:58.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.711+0000 7f330d59a700 1 -- 172.21.15.112:0/366139450 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33000092e0 con 0x7f3310067d90 2023-01-24T03:32:58.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.711+0000 7f330f59e700 1 -- 172.21.15.112:0/366139450 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3310067d90 msgr2=0x7f3310068160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:58.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.711+0000 7f330f59e700 1 --2- 172.21.15.112:0/366139450 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3310067d90 0x7f3310068160 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f331006a100 tx=0x7f3300009840 comp rx=0 tx=0).stop 2023-01-24T03:32:58.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.711+0000 7f330f59e700 1 -- 172.21.15.112:0/366139450 shutdown_connections 2023-01-24T03:32:58.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.711+0000 7f330f59e700 1 --2- 172.21.15.112:0/366139450 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3310067d90 0x7f3310068160 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:58.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.711+0000 7f330f59e700 1 --2- 172.21.15.112:0/366139450 >> v2:172.21.15.142:3300/0 conn(0x7f3310155bf0 0x7f33100c5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:58.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.711+0000 7f330f59e700 1 -- 172.21.15.112:0/366139450 >> 172.21.15.112:0/366139450 conn(0x7f33100ca150 msgr2=0x7f33100c6120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:32:58.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.711+0000 7f330f59e700 1 -- 172.21.15.112:0/366139450 shutdown_connections 2023-01-24T03:32:58.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.711+0000 7f330f59e700 1 -- 172.21.15.112:0/366139450 wait complete. 2023-01-24T03:32:58.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.712+0000 7f330f59e700 1 Processor -- start 2023-01-24T03:32:58.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.713+0000 7f330f59e700 1 -- start start 2023-01-24T03:32:58.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.713+0000 7f330f59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3310067d90 0x7f331014e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:58.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.713+0000 7f330f59e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33100c3f30 0x7f3310165140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:58.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.713+0000 7f330f59e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3310165680 0x7f33101497b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:58.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.713+0000 7f330f59e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3310169060 con 0x7f3310165680 2023-01-24T03:32:58.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.713+0000 7f330f59e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33101694f0 con 0x7f33100c3f30 2023-01-24T03:32:58.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.713+0000 7f330f59e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33100c24b0 con 0x7f3310067d90 2023-01-24T03:32:58.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.713+0000 7f330ed9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3310165680 0x7f33101497b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:58.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.713+0000 7f330ed9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3310165680 0x7f33101497b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52036/0 (socket says 172.21.15.112:52036) 2023-01-24T03:32:58.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.713+0000 7f330ed9d700 1 -- 172.21.15.112:0/1233410706 learned_addr learned my addr 172.21.15.112:0/1233410706 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:32:58.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.713+0000 7f330dd9b700 1 --2- 172.21.15.112:0/1233410706 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33100c3f30 0x7f3310165140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:58.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.713+0000 7f330e59c700 1 --2- 172.21.15.112:0/1233410706 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3310067d90 0x7f331014e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:58.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.714+0000 7f330ed9d700 1 -- 172.21.15.112:0/1233410706 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3310067d90 msgr2=0x7f331014e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:58.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.714+0000 7f330ed9d700 1 --2- 172.21.15.112:0/1233410706 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3310067d90 0x7f331014e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:58.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.714+0000 7f330ed9d700 1 -- 172.21.15.112:0/1233410706 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33100c3f30 msgr2=0x7f3310165140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:58.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.714+0000 7f330ed9d700 1 --2- 172.21.15.112:0/1233410706 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33100c3f30 0x7f3310165140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:58.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.714+0000 7f330ed9d700 1 -- 172.21.15.112:0/1233410706 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3300009ae0 con 0x7f3310165680 2023-01-24T03:32:58.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.714+0000 7f330dd9b700 1 --2- 172.21.15.112:0/1233410706 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33100c3f30 0x7f3310165140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:32:58.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.714+0000 7f330ed9d700 1 --2- 172.21.15.112:0/1233410706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3310165680 0x7f33101497b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f330800c440 tx=0x7f3308007d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:58.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.714+0000 7f33067fc700 1 -- 172.21.15.112:0/1233410706 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3308012420 con 0x7f3310165680 2023-01-24T03:32:58.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.714+0000 7f33067fc700 1 -- 172.21.15.112:0/1233410706 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3308012580 con 0x7f3310165680 2023-01-24T03:32:58.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.714+0000 7f330f59e700 1 -- 172.21.15.112:0/1233410706 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33100c2770 con 0x7f3310165680 2023-01-24T03:32:58.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.714+0000 7f33067fc700 1 -- 172.21.15.112:0/1233410706 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3308012940 con 0x7f3310165680 2023-01-24T03:32:58.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.714+0000 7f330f59e700 1 -- 172.21.15.112:0/1233410706 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3310168a10 con 0x7f3310165680 2023-01-24T03:32:58.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.715+0000 7f330f59e700 1 -- 172.21.15.112:0/1233410706 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f331005ea20 con 0x7f3310165680 2023-01-24T03:32:58.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.715+0000 7f33067fc700 1 -- 172.21.15.112:0/1233410706 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f3308019030 con 0x7f3310165680 2023-01-24T03:32:58.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.716+0000 7f33067fc700 1 --2- 172.21.15.112:0/1233410706 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f32f40611c0 0x7f32f4063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:32:58.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.716+0000 7f33067fc700 1 -- 172.21.15.112:0/1233410706 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7039+0+0 (secure 0 0 0) 0x7f330807cd20 con 0x7f3310165680 2023-01-24T03:32:58.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.716+0000 7f330e59c700 1 --2- 172.21.15.112:0/1233410706 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f32f40611c0 0x7f32f4063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:32:58.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.716+0000 7f330e59c700 1 --2- 172.21.15.112:0/1233410706 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f32f40611c0 0x7f32f4063670 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3300009880 tx=0x7f3300018000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:32:58.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.720+0000 7f33067fc700 1 -- 172.21.15.112:0/1233410706 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f330804ca80 con 0x7f3310165680 2023-01-24T03:32:58.944 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.942+0000 7f330f59e700 1 -- 172.21.15.112:0/1233410706 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33100c46f0 con 0x7f3310165680 2023-01-24T03:32:58.945 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.943+0000 7f33067fc700 1 -- 172.21.15.112:0/1233410706 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+5354 (secure 0 0 0) 0x7f3308017070 con 0x7f3310165680 2023-01-24T03:32:58.945 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:32:58.945 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:32:53.608119+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4897},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4897":{"gid":4897,"name":"d","rank":0,"incarnation":16,"state":"up:replay","state_seq":1,"addr":"172.21.15.142:6835/11022091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":11022091},{"type":"v1","addr":"172.21.15.142:6835","nonce":11022091}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:32:58.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.945+0000 7f330f59e700 1 -- 172.21.15.112:0/1233410706 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f32f40611c0 msgr2=0x7f32f4063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:58.948 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.945+0000 7f330f59e700 1 --2- 172.21.15.112:0/1233410706 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f32f40611c0 0x7f32f4063670 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3300009880 tx=0x7f3300018000 comp rx=0 tx=0).stop 2023-01-24T03:32:58.948 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.945+0000 7f330f59e700 1 -- 172.21.15.112:0/1233410706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3310165680 msgr2=0x7f33101497b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:32:58.948 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.945+0000 7f330f59e700 1 --2- 172.21.15.112:0/1233410706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3310165680 0x7f33101497b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f330800c440 tx=0x7f3308007d40 comp rx=0 tx=0).stop 2023-01-24T03:32:58.948 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.945+0000 7f330f59e700 1 -- 172.21.15.112:0/1233410706 shutdown_connections 2023-01-24T03:32:58.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.945+0000 7f330f59e700 1 --2- 172.21.15.112:0/1233410706 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f32f40611c0 0x7f32f4063670 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:58.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.945+0000 7f330f59e700 1 --2- 172.21.15.112:0/1233410706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3310165680 0x7f33101497b0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:58.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.945+0000 7f330f59e700 1 --2- 172.21.15.112:0/1233410706 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3310067d90 0x7f331014e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:58.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.945+0000 7f330f59e700 1 --2- 172.21.15.112:0/1233410706 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f33100c3f30 0x7f3310165140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:32:58.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.945+0000 7f330f59e700 1 -- 172.21.15.112:0/1233410706 >> 172.21.15.112:0/1233410706 conn(0x7f33100ca150 msgr2=0x7f331014ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:32:58.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.945+0000 7f330f59e700 1 -- 172.21.15.112:0/1233410706 shutdown_connections 2023-01-24T03:32:58.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:32:58.945+0000 7f330f59e700 1 -- 172.21.15.112:0/1233410706 wait complete. 2023-01-24T03:32:58.950 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 16 2023-01-24T03:32:58.959 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:32:59.142 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:32:59.145 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:33:00.960 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:33:01.087 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.085+0000 7fb9ff59e700 1 Processor -- start 2023-01-24T03:33:01.087 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.085+0000 7fb9ff59e700 1 -- start start 2023-01-24T03:33:01.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.086+0000 7fb9ff59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fba00156e40 0x7fba000cb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:01.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.086+0000 7fb9ff59e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba000c3d10 con 0x7fba00155e20 2023-01-24T03:33:01.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.086+0000 7fb9ff59e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba000c2e20 con 0x7fba000c5680 2023-01-24T03:33:01.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.086+0000 7fb9ff59e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba000c4eb0 con 0x7fba00156e40 2023-01-24T03:33:01.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.086+0000 7fb9fed9d700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fba00155e20 0x7fba0014ef60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:33046/0 (socket says 172.21.15.112:33046) 2023-01-24T03:33:01.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.086+0000 7fb9fed9d700 1 -- 172.21.15.112:0/3689949658 learned_addr learned my addr 172.21.15.112:0/3689949658 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:33:01.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.086+0000 7fb9fdd9b700 1 --2- 172.21.15.112:0/3689949658 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fba00156e40 0x7fba000cb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:01.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.086+0000 7fb9fd59a700 1 -- 172.21.15.112:0/3689949658 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 531580060 0 0) 0x7fba000c3d10 con 0x7fba00155e20 2023-01-24T03:33:01.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.086+0000 7fb9fd59a700 1 -- 172.21.15.112:0/3689949658 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9e00033f0 con 0x7fba00155e20 2023-01-24T03:33:01.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.086+0000 7fb9fd59a700 1 -- 172.21.15.112:0/3689949658 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3035350527 0 0) 0x7fba000c2e20 con 0x7fba000c5680 2023-01-24T03:33:01.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.086+0000 7fb9fd59a700 1 -- 172.21.15.112:0/3689949658 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba000c3d10 con 0x7fba000c5680 2023-01-24T03:33:01.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.086+0000 7fb9fdd9b700 1 -- 172.21.15.112:0/3689949658 >> v1:172.21.15.142:6789/0 conn(0x7fba000c5680 legacy=0x7fba00156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:01.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.086+0000 7fb9fdd9b700 1 -- 172.21.15.112:0/3689949658 >> v1:172.21.15.112:6789/0 conn(0x7fba00155e20 legacy=0x7fba0014ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:01.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.087+0000 7fb9fdd9b700 1 -- 172.21.15.112:0/3689949658 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba000c3100 con 0x7fba00156e40 2023-01-24T03:33:01.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.087+0000 7fb9fdd9b700 1 --2- 172.21.15.112:0/3689949658 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fba00156e40 0x7fba000cb870 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb9e8009790 tx=0x7fb9e8009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90aabf8e316368fb server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:33:01.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.087+0000 7fb9fd59a700 1 -- 172.21.15.112:0/3689949658 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e8011630 con 0x7fba00156e40 2023-01-24T03:33:01.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.087+0000 7fb9fd59a700 1 -- 172.21.15.112:0/3689949658 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9e8011790 con 0x7fba00156e40 2023-01-24T03:33:01.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.087+0000 7fb9fd59a700 1 -- 172.21.15.112:0/3689949658 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e8011b10 con 0x7fba00156e40 2023-01-24T03:33:01.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.087+0000 7fb9ff59e700 1 -- 172.21.15.112:0/3689949658 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fba00156e40 msgr2=0x7fba000cb870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:01.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.087+0000 7fb9ff59e700 1 --2- 172.21.15.112:0/3689949658 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fba00156e40 0x7fba000cb870 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb9e8009790 tx=0x7fb9e8009060 comp rx=0 tx=0).stop 2023-01-24T03:33:01.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.087+0000 7fb9ff59e700 1 -- 172.21.15.112:0/3689949658 shutdown_connections 2023-01-24T03:33:01.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.088+0000 7fb9ff59e700 1 --2- 172.21.15.112:0/3689949658 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fba00156e40 0x7fba000cb870 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:01.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.088+0000 7fb9ff59e700 1 -- 172.21.15.112:0/3689949658 >> 172.21.15.112:0/3689949658 conn(0x7fba000685b0 msgr2=0x7fba000689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:33:01.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.088+0000 7fb9ff59e700 1 -- 172.21.15.112:0/3689949658 shutdown_connections 2023-01-24T03:33:01.095 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.088+0000 7fb9ff59e700 1 -- 172.21.15.112:0/3689949658 wait complete. 2023-01-24T03:33:01.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.089+0000 7fb9ff59e700 1 Processor -- start 2023-01-24T03:33:01.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.089+0000 7fb9ff59e700 1 -- start start 2023-01-24T03:33:01.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.089+0000 7fb9ff59e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fba000c5680 0x7fba0014a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:01.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.089+0000 7fb9ff59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fba00155e20 0x7fba0014a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:01.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.089+0000 7fb9ff59e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba00156e40 0x7fba000c20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:01.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.089+0000 7fb9ff59e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba00166100 con 0x7fba000c5680 2023-01-24T03:33:01.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.089+0000 7fb9ff59e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba00151700 con 0x7fba00156e40 2023-01-24T03:33:01.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.089+0000 7fb9ff59e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba001654f0 con 0x7fba00155e20 2023-01-24T03:33:01.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.089+0000 7fb9fe59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fba000c5680 0x7fba0014a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:01.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.089+0000 7fb9fe59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fba000c5680 0x7fba0014a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52060/0 (socket says 172.21.15.112:52060) 2023-01-24T03:33:01.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.089+0000 7fb9fe59c700 1 -- 172.21.15.112:0/1336162202 learned_addr learned my addr 172.21.15.112:0/1336162202 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:33:01.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.090+0000 7fb9fdd9b700 1 --2- 172.21.15.112:0/1336162202 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fba00155e20 0x7fba0014a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:01.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.090+0000 7fb9fed9d700 1 --2- 172.21.15.112:0/1336162202 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba00156e40 0x7fba000c20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:01.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.090+0000 7fb9fe59c700 1 -- 172.21.15.112:0/1336162202 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fba00155e20 msgr2=0x7fba0014a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:01.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.090+0000 7fb9fe59c700 1 --2- 172.21.15.112:0/1336162202 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fba00155e20 0x7fba0014a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:01.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.090+0000 7fb9fe59c700 1 -- 172.21.15.112:0/1336162202 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba00156e40 msgr2=0x7fba000c20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:01.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.090+0000 7fb9fe59c700 1 --2- 172.21.15.112:0/1336162202 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba00156e40 0x7fba000c20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:01.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.090+0000 7fb9fe59c700 1 -- 172.21.15.112:0/1336162202 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e80092a0 con 0x7fba000c5680 2023-01-24T03:33:01.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.090+0000 7fb9fe59c700 1 --2- 172.21.15.112:0/1336162202 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fba000c5680 0x7fba0014a050 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb9f000c7c0 tx=0x7fb9f0002660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:33:01.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.090+0000 7fb9f67fc700 1 -- 172.21.15.112:0/1336162202 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f0012630 con 0x7fba000c5680 2023-01-24T03:33:01.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.090+0000 7fb9f67fc700 1 -- 172.21.15.112:0/1336162202 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9f0012790 con 0x7fba000c5680 2023-01-24T03:33:01.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.090+0000 7fb9ff59e700 1 -- 172.21.15.112:0/1336162202 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba001657d0 con 0x7fba000c5680 2023-01-24T03:33:01.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.090+0000 7fb9f67fc700 1 -- 172.21.15.112:0/1336162202 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f0012b50 con 0x7fba000c5680 2023-01-24T03:33:01.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.090+0000 7fb9ff59e700 1 -- 172.21.15.112:0/1336162202 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba00165d90 con 0x7fba000c5680 2023-01-24T03:33:01.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.091+0000 7fb9ff59e700 1 -- 172.21.15.112:0/1336162202 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba00149ea0 con 0x7fba000c5680 2023-01-24T03:33:01.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.091+0000 7fb9f67fc700 1 -- 172.21.15.112:0/1336162202 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fb9f0022070 con 0x7fba000c5680 2023-01-24T03:33:01.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.092+0000 7fb9f67fc700 1 --2- 172.21.15.112:0/1336162202 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb9e40611f0 0x7fb9e40636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:01.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.092+0000 7fb9f67fc700 1 -- 172.21.15.112:0/1336162202 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7039+0+0 (secure 0 0 0) 0x7fb9f007dbb0 con 0x7fba000c5680 2023-01-24T03:33:01.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.092+0000 7fb9fdd9b700 1 --2- 172.21.15.112:0/1336162202 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb9e40611f0 0x7fb9e40636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:01.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.092+0000 7fb9fdd9b700 1 --2- 172.21.15.112:0/1336162202 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb9e40611f0 0x7fb9e40636a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb9e80096e0 tx=0x7fb9e8017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:33:01.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.096+0000 7fb9f67fc700 1 -- 172.21.15.112:0/1336162202 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fb9f004d820 con 0x7fba000c5680 2023-01-24T03:33:01.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.316+0000 7fb9ff59e700 1 -- 172.21.15.112:0/1336162202 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba0005ea20 con 0x7fba000c5680 2023-01-24T03:33:01.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.316+0000 7fb9f67fc700 1 -- 172.21.15.112:0/1336162202 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+5358 (secure 0 0 0) 0x7fb9f0050e40 con 0x7fba000c5680 2023-01-24T03:33:01.319 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:33:01.319 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:32:59.610673+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4897},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4897":{"gid":4897,"name":"d","rank":0,"incarnation":16,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.142:6835/11022091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":11022091},{"type":"v1","addr":"172.21.15.142:6835","nonce":11022091}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:33:01.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.318+0000 7fb9ff59e700 1 -- 172.21.15.112:0/1336162202 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb9e40611f0 msgr2=0x7fb9e40636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:01.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.318+0000 7fb9ff59e700 1 --2- 172.21.15.112:0/1336162202 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb9e40611f0 0x7fb9e40636a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb9e80096e0 tx=0x7fb9e8017000 comp rx=0 tx=0).stop 2023-01-24T03:33:01.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.318+0000 7fb9ff59e700 1 -- 172.21.15.112:0/1336162202 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fba000c5680 msgr2=0x7fba0014a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:01.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.318+0000 7fb9ff59e700 1 --2- 172.21.15.112:0/1336162202 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fba000c5680 0x7fba0014a050 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb9f000c7c0 tx=0x7fb9f0002660 comp rx=0 tx=0).stop 2023-01-24T03:33:01.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.318+0000 7fb9ff59e700 1 -- 172.21.15.112:0/1336162202 shutdown_connections 2023-01-24T03:33:01.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.318+0000 7fb9ff59e700 1 --2- 172.21.15.112:0/1336162202 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb9e40611f0 0x7fb9e40636a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:01.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.318+0000 7fb9ff59e700 1 --2- 172.21.15.112:0/1336162202 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fba000c5680 0x7fba0014a050 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:01.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.318+0000 7fb9ff59e700 1 --2- 172.21.15.112:0/1336162202 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fba00156e40 0x7fba000c20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:01.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.318+0000 7fb9ff59e700 1 --2- 172.21.15.112:0/1336162202 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fba00155e20 0x7fba0014a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:01.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.318+0000 7fb9ff59e700 1 -- 172.21.15.112:0/1336162202 >> 172.21.15.112:0/1336162202 conn(0x7fba000685b0 msgr2=0x7fba000cbcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:33:01.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.318+0000 7fb9ff59e700 1 -- 172.21.15.112:0/1336162202 shutdown_connections 2023-01-24T03:33:01.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:01.318+0000 7fb9ff59e700 1 -- 172.21.15.112:0/1336162202 wait complete. 2023-01-24T03:33:01.323 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 17 2023-01-24T03:33:01.332 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:33:03.333 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:33:03.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.455+0000 7f255ed31700 1 Processor -- start 2023-01-24T03:33:03.458 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.455+0000 7f255ed31700 1 -- start start 2023-01-24T03:33:03.459 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.456+0000 7f255ed31700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f25580b0aa0 0x7f2558155da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:03.460 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.456+0000 7f255ed31700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25580b0f40 con 0x7f25580ca3e0 2023-01-24T03:33:03.460 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.456+0000 7f255ed31700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25580b1170 con 0x7f25580c7e40 2023-01-24T03:33:03.460 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.456+0000 7f255ed31700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25580b1450 con 0x7f25580b0aa0 2023-01-24T03:33:03.460 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.456+0000 7f2557fff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f25580b0aa0 0x7f2558155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:03.461 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.456+0000 7f2557fff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f25580b0aa0 0x7f2558155da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52100/0 (socket says 172.21.15.112:52100) 2023-01-24T03:33:03.461 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.456+0000 7f2557fff700 1 -- 172.21.15.112:0/4113099401 learned_addr learned my addr 172.21.15.112:0/4113099401 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:33:03.461 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.457+0000 7f2557fff700 1 -- 172.21.15.112:0/4113099401 >> v1:172.21.15.142:6789/0 conn(0x7f25580c7e40 legacy=0x7f25580ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:03.461 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.457+0000 7f2557fff700 1 -- 172.21.15.112:0/4113099401 >> v1:172.21.15.112:6789/0 conn(0x7f25580ca3e0 legacy=0x7f25580ca7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:03.461 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.457+0000 7f2557fff700 1 -- 172.21.15.112:0/4113099401 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25580b1730 con 0x7f25580b0aa0 2023-01-24T03:33:03.462 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.457+0000 7f25577fe700 1 -- 172.21.15.112:0/4113099401 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 787089821 0 0) 0x7f25580b0f40 con 0x7f25580ca3e0 2023-01-24T03:33:03.462 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.457+0000 7f2557fff700 1 --2- 172.21.15.112:0/4113099401 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f25580b0aa0 0x7f2558155da0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f254c006fd0 tx=0x7f254c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=536a5acd0936826b server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:33:03.462 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.457+0000 7f25577fe700 1 -- 172.21.15.112:0/4113099401 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f254c011630 con 0x7f25580b0aa0 2023-01-24T03:33:03.462 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.457+0000 7f25577fe700 1 -- 172.21.15.112:0/4113099401 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f254c0117b0 con 0x7f25580b0aa0 2023-01-24T03:33:03.462 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.457+0000 7f25577fe700 1 -- 172.21.15.112:0/4113099401 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f254c011b70 con 0x7f25580b0aa0 2023-01-24T03:33:03.463 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.457+0000 7f255ed31700 1 -- 172.21.15.112:0/4113099401 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f25580b0aa0 msgr2=0x7f2558155da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:03.463 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.457+0000 7f255ed31700 1 --2- 172.21.15.112:0/4113099401 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f25580b0aa0 0x7f2558155da0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f254c006fd0 tx=0x7f254c0090f0 comp rx=0 tx=0).stop 2023-01-24T03:33:03.463 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.457+0000 7f255ed31700 1 -- 172.21.15.112:0/4113099401 shutdown_connections 2023-01-24T03:33:03.463 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.457+0000 7f255ed31700 1 --2- 172.21.15.112:0/4113099401 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f25580b0aa0 0x7f2558155da0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:03.463 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.457+0000 7f255ed31700 1 -- 172.21.15.112:0/4113099401 >> 172.21.15.112:0/4113099401 conn(0x7f255805e2c0 msgr2=0x7f25580cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:33:03.464 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.457+0000 7f255ed31700 1 -- 172.21.15.112:0/4113099401 shutdown_connections 2023-01-24T03:33:03.464 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.457+0000 7f255ed31700 1 -- 172.21.15.112:0/4113099401 wait complete. 2023-01-24T03:33:03.467 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.459+0000 7f255ed31700 1 Processor -- start 2023-01-24T03:33:03.467 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.459+0000 7f255ed31700 1 -- start start 2023-01-24T03:33:03.467 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.459+0000 7f255ed31700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25580b0aa0 0x7f2558069020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:03.467 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.459+0000 7f255ed31700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f25580c7e40 0x7f25580c1b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:03.468 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.459+0000 7f255ed31700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f25580ca3e0 0x7f255814fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:03.468 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.459+0000 7f255ed31700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2558151450 con 0x7f25580ca3e0 2023-01-24T03:33:03.468 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.459+0000 7f255ed31700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2558165690 con 0x7f25580b0aa0 2023-01-24T03:33:03.468 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.459+0000 7f255ed31700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25580c4240 con 0x7f25580c7e40 2023-01-24T03:33:03.468 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.460+0000 7f255d2ce700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f25580ca3e0 0x7f255814fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:03.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.460+0000 7f255d2ce700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f25580ca3e0 0x7f255814fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52108/0 (socket says 172.21.15.112:52108) 2023-01-24T03:33:03.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.460+0000 7f255d2ce700 1 -- 172.21.15.112:0/1720445002 learned_addr learned my addr 172.21.15.112:0/1720445002 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:33:03.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.460+0000 7f2557fff700 1 --2- 172.21.15.112:0/1720445002 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f25580c7e40 0x7f25580c1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:03.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.460+0000 7f255cacd700 1 --2- 172.21.15.112:0/1720445002 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25580b0aa0 0x7f2558069020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:03.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.460+0000 7f255d2ce700 1 -- 172.21.15.112:0/1720445002 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f25580c7e40 msgr2=0x7f25580c1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:03.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.460+0000 7f255d2ce700 1 --2- 172.21.15.112:0/1720445002 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f25580c7e40 0x7f25580c1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:03.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.460+0000 7f255d2ce700 1 -- 172.21.15.112:0/1720445002 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25580b0aa0 msgr2=0x7f2558069020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:03.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.460+0000 7f255d2ce700 1 --2- 172.21.15.112:0/1720445002 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25580b0aa0 0x7f2558069020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:03.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.460+0000 7f255d2ce700 1 -- 172.21.15.112:0/1720445002 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f254c009370 con 0x7f25580ca3e0 2023-01-24T03:33:03.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.460+0000 7f255cacd700 1 --2- 172.21.15.112:0/1720445002 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25580b0aa0 0x7f2558069020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-01-24T03:33:03.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.460+0000 7f255d2ce700 1 --2- 172.21.15.112:0/1720445002 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f25580ca3e0 0x7f255814fd60 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f255000c810 tx=0x7f25500014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:33:03.471 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.460+0000 7f2554ff9700 1 -- 172.21.15.112:0/1720445002 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2550012630 con 0x7f25580ca3e0 2023-01-24T03:33:03.471 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.460+0000 7f2554ff9700 1 -- 172.21.15.112:0/1720445002 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25500127b0 con 0x7f25580ca3e0 2023-01-24T03:33:03.471 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.460+0000 7f2554ff9700 1 -- 172.21.15.112:0/1720445002 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2550012b70 con 0x7f25580ca3e0 2023-01-24T03:33:03.471 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.460+0000 7f255ed31700 1 -- 172.21.15.112:0/1720445002 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2558166450 con 0x7f25580ca3e0 2023-01-24T03:33:03.471 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.460+0000 7f255ed31700 1 -- 172.21.15.112:0/1720445002 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2558068a20 con 0x7f25580ca3e0 2023-01-24T03:33:03.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.461+0000 7f255ed31700 1 -- 172.21.15.112:0/1720445002 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25581665e0 con 0x7f25580ca3e0 2023-01-24T03:33:03.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.461+0000 7f2554ff9700 1 -- 172.21.15.112:0/1720445002 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f2550018020 con 0x7f25580ca3e0 2023-01-24T03:33:03.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.462+0000 7f2554ff9700 1 --2- 172.21.15.112:0/1720445002 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f25380611f0 0x7f25380636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:03.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.462+0000 7f2554ff9700 1 -- 172.21.15.112:0/1720445002 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7039+0+0 (secure 0 0 0) 0x7f255007d910 con 0x7f25580ca3e0 2023-01-24T03:33:03.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.462+0000 7f255cacd700 1 --2- 172.21.15.112:0/1720445002 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f25380611f0 0x7f25380636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:03.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.462+0000 7f255cacd700 1 --2- 172.21.15.112:0/1720445002 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f25380611f0 0x7f25380636a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2548001670 tx=0x7f2548009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:33:03.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.466+0000 7f2554ff9700 1 -- 172.21.15.112:0/1720445002 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f255004d670 con 0x7f25580ca3e0 2023-01-24T03:33:03.690 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.688+0000 7f255ed31700 1 -- 172.21.15.112:0/1720445002 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25580c8210 con 0x7f25580ca3e0 2023-01-24T03:33:03.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.689+0000 7f2554ff9700 1 -- 172.21.15.112:0/1720445002 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+5358 (secure 0 0 0) 0x7f2550050c90 con 0x7f25580ca3e0 2023-01-24T03:33:03.691 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:33:03.692 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:32:59.610673+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4897},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4897":{"gid":4897,"name":"d","rank":0,"incarnation":16,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.142:6835/11022091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":11022091},{"type":"v1","addr":"172.21.15.142:6835","nonce":11022091}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:33:03.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.691+0000 7f255ed31700 1 -- 172.21.15.112:0/1720445002 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f25380611f0 msgr2=0x7f25380636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:03.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.691+0000 7f255ed31700 1 --2- 172.21.15.112:0/1720445002 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f25380611f0 0x7f25380636a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2548001670 tx=0x7f2548009040 comp rx=0 tx=0).stop 2023-01-24T03:33:03.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.691+0000 7f255ed31700 1 -- 172.21.15.112:0/1720445002 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f25580ca3e0 msgr2=0x7f255814fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:03.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.691+0000 7f255ed31700 1 --2- 172.21.15.112:0/1720445002 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f25580ca3e0 0x7f255814fd60 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f255000c810 tx=0x7f25500014a0 comp rx=0 tx=0).stop 2023-01-24T03:33:03.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.691+0000 7f255ed31700 1 -- 172.21.15.112:0/1720445002 shutdown_connections 2023-01-24T03:33:03.695 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.691+0000 7f255ed31700 1 --2- 172.21.15.112:0/1720445002 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f25380611f0 0x7f25380636a0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:03.695 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.691+0000 7f255ed31700 1 --2- 172.21.15.112:0/1720445002 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f25580ca3e0 0x7f255814fd60 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:03.695 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.691+0000 7f255ed31700 1 --2- 172.21.15.112:0/1720445002 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f25580b0aa0 0x7f2558069020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:03.695 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.691+0000 7f255ed31700 1 --2- 172.21.15.112:0/1720445002 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f25580c7e40 0x7f25580c1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:03.695 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.691+0000 7f255ed31700 1 -- 172.21.15.112:0/1720445002 >> 172.21.15.112:0/1720445002 conn(0x7f255805e2c0 msgr2=0x7f25580ae9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:33:03.696 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.691+0000 7f255ed31700 1 -- 172.21.15.112:0/1720445002 shutdown_connections 2023-01-24T03:33:03.696 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:03.691+0000 7f255ed31700 1 -- 172.21.15.112:0/1720445002 wait complete. 2023-01-24T03:33:03.696 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 17 2023-01-24T03:33:03.705 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:33:05.706 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:33:05.833 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.828+0000 7f842ef0e700 1 Processor -- start 2023-01-24T03:33:05.834 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.828+0000 7f842ef0e700 1 -- start start 2023-01-24T03:33:05.834 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.829+0000 7f842ef0e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f84280c7e40 0x7f84280b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:05.834 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.829+0000 7f842ef0e700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f84280ca3e0 0x7f84280ca820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:05.834 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.829+0000 7f842ef0e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8428156210 con 0x7f8428155da0 2023-01-24T03:33:05.834 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.829+0000 7f842ef0e700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84280cad60 con 0x7f84280ca3e0 2023-01-24T03:33:05.835 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.829+0000 7f842ef0e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84280b1170 con 0x7f84280c7e40 2023-01-24T03:33:05.835 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.829+0000 7f8427fff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f8428155da0 0x7f84280ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:33106/0 (socket says 172.21.15.112:33106) 2023-01-24T03:33:05.835 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.829+0000 7f8427fff700 1 -- 172.21.15.112:0/3296187195 learned_addr learned my addr 172.21.15.112:0/3296187195 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:33:05.835 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.829+0000 7f842d4ab700 1 --2- 172.21.15.112:0/3296187195 >> v2:172.21.15.112:3300/0 conn(0x7f84280ca3e0 0x7f84280ca820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:05.835 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.829+0000 7f842ccaa700 1 --2- 172.21.15.112:0/3296187195 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f84280c7e40 0x7f84280b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:05.836 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.829+0000 7f842d4ab700 1 -- 172.21.15.112:0/3296187195 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f84280c7e40 msgr2=0x7f84280b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:33:05.836 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.829+0000 7f842d4ab700 1 --2- 172.21.15.112:0/3296187195 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f84280c7e40 0x7f84280b0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:05.836 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.829+0000 7f842d4ab700 1 -- 172.21.15.112:0/3296187195 >> v1:172.21.15.112:6789/0 conn(0x7f8428155da0 legacy=0x7f84280ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:05.836 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.829+0000 7f842d4ab700 1 -- 172.21.15.112:0/3296187195 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84280b1450 con 0x7f84280ca3e0 2023-01-24T03:33:05.837 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.829+0000 7f842d4ab700 1 --2- 172.21.15.112:0/3296187195 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f84280ca3e0 0x7f84280ca820 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f8410009760 tx=0x7f8410009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18df68700305db8a server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:33:05.837 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.829+0000 7f84277fe700 1 -- 172.21.15.112:0/3296187195 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8410011630 con 0x7f84280ca3e0 2023-01-24T03:33:05.837 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.829+0000 7f84277fe700 1 -- 172.21.15.112:0/3296187195 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8410011790 con 0x7f84280ca3e0 2023-01-24T03:33:05.837 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.829+0000 7f84277fe700 1 -- 172.21.15.112:0/3296187195 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8410011b50 con 0x7f84280ca3e0 2023-01-24T03:33:05.837 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.830+0000 7f842ef0e700 1 -- 172.21.15.112:0/3296187195 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f84280ca3e0 msgr2=0x7f84280ca820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:05.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.830+0000 7f842ef0e700 1 --2- 172.21.15.112:0/3296187195 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f84280ca3e0 0x7f84280ca820 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f8410009760 tx=0x7f8410009060 comp rx=0 tx=0).stop 2023-01-24T03:33:05.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.830+0000 7f842ef0e700 1 -- 172.21.15.112:0/3296187195 shutdown_connections 2023-01-24T03:33:05.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.830+0000 7f842ef0e700 1 --2- 172.21.15.112:0/3296187195 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f84280ca3e0 0x7f84280ca820 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:05.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.830+0000 7f842ef0e700 1 --2- 172.21.15.112:0/3296187195 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f84280c7e40 0x7f84280b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:05.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.830+0000 7f842ef0e700 1 -- 172.21.15.112:0/3296187195 >> 172.21.15.112:0/3296187195 conn(0x7f842805e2c0 msgr2=0x7f84280cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:33:05.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.830+0000 7f842ef0e700 1 -- 172.21.15.112:0/3296187195 shutdown_connections 2023-01-24T03:33:05.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.830+0000 7f842ef0e700 1 -- 172.21.15.112:0/3296187195 wait complete. 2023-01-24T03:33:05.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.831+0000 7f842ef0e700 1 Processor -- start 2023-01-24T03:33:05.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.831+0000 7f842ef0e700 1 -- start start 2023-01-24T03:33:05.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.832+0000 7f842ef0e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84280c7e40 0x7f8428169530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:05.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.832+0000 7f842ef0e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f84280ca3e0 0x7f842814f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:05.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.832+0000 7f842ef0e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8428155da0 0x7f84280c1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:05.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.832+0000 7f842ef0e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8428152d10 con 0x7f8428155da0 2023-01-24T03:33:05.843 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.832+0000 7f842ef0e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f842814fe30 con 0x7f84280c7e40 2023-01-24T03:33:05.843 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.832+0000 7f842ef0e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f842814f6b0 con 0x7f84280ca3e0 2023-01-24T03:33:05.843 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.832+0000 7f842d4ab700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8428155da0 0x7f84280c1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:05.843 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.832+0000 7f842d4ab700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8428155da0 0x7f84280c1990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52126/0 (socket says 172.21.15.112:52126) 2023-01-24T03:33:05.843 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.832+0000 7f842d4ab700 1 -- 172.21.15.112:0/2489015125 learned_addr learned my addr 172.21.15.112:0/2489015125 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:33:05.843 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.832+0000 7f8427fff700 1 --2- 172.21.15.112:0/2489015125 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f84280ca3e0 0x7f842814f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:05.844 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.832+0000 7f842ccaa700 1 --2- 172.21.15.112:0/2489015125 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84280c7e40 0x7f8428169530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:05.844 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.832+0000 7f842d4ab700 1 -- 172.21.15.112:0/2489015125 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f84280ca3e0 msgr2=0x7f842814f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:05.844 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.832+0000 7f842d4ab700 1 --2- 172.21.15.112:0/2489015125 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f84280ca3e0 0x7f842814f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:05.844 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.832+0000 7f842d4ab700 1 -- 172.21.15.112:0/2489015125 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84280c7e40 msgr2=0x7f8428169530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:05.845 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.832+0000 7f842d4ab700 1 --2- 172.21.15.112:0/2489015125 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84280c7e40 0x7f8428169530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:05.845 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.833+0000 7f842d4ab700 1 -- 172.21.15.112:0/2489015125 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84100092a0 con 0x7f8428155da0 2023-01-24T03:33:05.845 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.833+0000 7f842d4ab700 1 --2- 172.21.15.112:0/2489015125 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8428155da0 0x7f84280c1990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f8410009e00 tx=0x7f8410017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:33:05.845 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.833+0000 7f8424ff9700 1 -- 172.21.15.112:0/2489015125 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8410011900 con 0x7f8428155da0 2023-01-24T03:33:05.845 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.833+0000 7f8424ff9700 1 -- 172.21.15.112:0/2489015125 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8410011a60 con 0x7f8428155da0 2023-01-24T03:33:05.846 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.833+0000 7f842ef0e700 1 -- 172.21.15.112:0/2489015125 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8428152470 con 0x7f8428155da0 2023-01-24T03:33:05.846 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.833+0000 7f8424ff9700 1 -- 172.21.15.112:0/2489015125 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8410025670 con 0x7f8428155da0 2023-01-24T03:33:05.846 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.833+0000 7f842ef0e700 1 -- 172.21.15.112:0/2489015125 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8428152a30 con 0x7f8428155da0 2023-01-24T03:33:05.846 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.834+0000 7f842ef0e700 1 -- 172.21.15.112:0/2489015125 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84280c0800 con 0x7f8428155da0 2023-01-24T03:33:05.846 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.834+0000 7f8424ff9700 1 -- 172.21.15.112:0/2489015125 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f841002c030 con 0x7f8428155da0 2023-01-24T03:33:05.846 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.835+0000 7f8424ff9700 1 --2- 172.21.15.112:0/2489015125 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f840c0611c0 0x7f840c063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:05.847 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.835+0000 7f8424ff9700 1 -- 172.21.15.112:0/2489015125 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7039+0+0 (secure 0 0 0) 0x7f841007f110 con 0x7f8428155da0 2023-01-24T03:33:05.847 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.835+0000 7f842ccaa700 1 --2- 172.21.15.112:0/2489015125 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f840c0611c0 0x7f840c063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:05.847 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.835+0000 7f842ccaa700 1 --2- 172.21.15.112:0/2489015125 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f840c0611c0 0x7f840c063670 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8420009b20 tx=0x7f8420008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:33:05.847 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:05.839+0000 7f8424ff9700 1 -- 172.21.15.112:0/2489015125 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f841004ee70 con 0x7f8428155da0 2023-01-24T03:33:06.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:06.057+0000 7f842ef0e700 1 -- 172.21.15.112:0/2489015125 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f842804bb80 con 0x7f8428155da0 2023-01-24T03:33:06.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:06.058+0000 7f8424ff9700 1 -- 172.21.15.112:0/2489015125 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+5355 (secure 0 0 0) 0x7f8410052490 con 0x7f8428155da0 2023-01-24T03:33:06.062 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:33:06.062 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:33:05.613093+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4897},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4897":{"gid":4897,"name":"d","rank":0,"incarnation":16,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.142:6835/11022091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":11022091},{"type":"v1","addr":"172.21.15.142:6835","nonce":11022091}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:33:06.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:06.060+0000 7f842ef0e700 1 -- 172.21.15.112:0/2489015125 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f840c0611c0 msgr2=0x7f840c063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:06.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:06.060+0000 7f842ef0e700 1 --2- 172.21.15.112:0/2489015125 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f840c0611c0 0x7f840c063670 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8420009b20 tx=0x7f8420008040 comp rx=0 tx=0).stop 2023-01-24T03:33:06.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:06.060+0000 7f842ef0e700 1 -- 172.21.15.112:0/2489015125 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8428155da0 msgr2=0x7f84280c1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:06.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:06.060+0000 7f842ef0e700 1 --2- 172.21.15.112:0/2489015125 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8428155da0 0x7f84280c1990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f8410009e00 tx=0x7f8410017000 comp rx=0 tx=0).stop 2023-01-24T03:33:06.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:06.060+0000 7f842ef0e700 1 -- 172.21.15.112:0/2489015125 shutdown_connections 2023-01-24T03:33:06.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:06.060+0000 7f842ef0e700 1 --2- 172.21.15.112:0/2489015125 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f840c0611c0 0x7f840c063670 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:06.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:06.060+0000 7f842ef0e700 1 --2- 172.21.15.112:0/2489015125 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8428155da0 0x7f84280c1990 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:06.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:06.060+0000 7f842ef0e700 1 --2- 172.21.15.112:0/2489015125 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f84280c7e40 0x7f8428169530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:06.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:06.060+0000 7f842ef0e700 1 --2- 172.21.15.112:0/2489015125 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f84280ca3e0 0x7f842814f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:06.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:06.060+0000 7f842ef0e700 1 -- 172.21.15.112:0/2489015125 >> 172.21.15.112:0/2489015125 conn(0x7f842805e2c0 msgr2=0x7f8428154f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:33:06.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:06.060+0000 7f842ef0e700 1 -- 172.21.15.112:0/2489015125 shutdown_connections 2023-01-24T03:33:06.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:06.060+0000 7f842ef0e700 1 -- 172.21.15.112:0/2489015125 wait complete. 2023-01-24T03:33:06.068 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 18 2023-01-24T03:33:06.075 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 18, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-01-24T03:30:20.971710+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 18, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4897": { "addr": "172.21.15.142:6835/11022091", "addrs": { "addrvec": [ { "addr": "172.21.15.142:6833", "nonce": 11022091, "type": "v2" }, { "addr": "172.21.15.142:6835", "nonce": 11022091, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4897, "incarnation": 16, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:rejoin", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 31, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-01-24T03:33:05.613093+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4897 } } } ], "standbys": [ { "addr": "172.21.15.142:6837/2181345869", "addrs": { "addrvec": [ { "addr": "172.21.15.142:6836", "nonce": 2181345869, "type": "v2" }, { "addr": "172.21.15.142:6837", "nonce": 2181345869, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6835/330343927", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6834", "nonce": 330343927, "type": "v2" }, { "addr": "172.21.15.112:6835", "nonce": 330343927, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6837/12176817", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6836", "nonce": 12176817, "type": "v2" }, { "addr": "172.21.15.112:6837", "nonce": 12176817, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4276, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6839/1110983841", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6838", "nonce": 1110983841, "type": "v2" }, { "addr": "172.21.15.112:6839", "nonce": 1110983841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4281, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-01-24T03:33:08.077 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:33:08.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.199+0000 7f48cca0a700 1 Processor -- start 2023-01-24T03:33:08.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.199+0000 7f48cca0a700 1 -- start start 2023-01-24T03:33:08.206 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.200+0000 7f48cca0a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48c80c15b0 0x7f48c814cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:08.206 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.200+0000 7f48cca0a700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f48c80c3450 0x7f48c814d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:08.206 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.200+0000 7f48cca0a700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48c81566c0 con 0x7f48c80c3080 2023-01-24T03:33:08.207 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.200+0000 7f48cca0a700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48c805e650 con 0x7f48c80c3450 2023-01-24T03:33:08.207 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.200+0000 7f48cca0a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48c81569a0 con 0x7f48c80c15b0 2023-01-24T03:33:08.207 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.200+0000 7f48c6d9d700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f48c80c3450 0x7f48c814d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:08.207 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.200+0000 7f48c6d9d700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f48c80c3450 0x7f48c814d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52138/0 (socket says 172.21.15.112:52138) 2023-01-24T03:33:08.207 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.200+0000 7f48c659c700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f48c80c3080 0x7f48c805df40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:33124/0 (socket says 172.21.15.112:33124) 2023-01-24T03:33:08.208 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.200+0000 7f48c6d9d700 1 -- 172.21.15.112:0/612739907 learned_addr learned my addr 172.21.15.112:0/612739907 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:33:08.208 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.201+0000 7f48c5d9b700 1 --2- 172.21.15.112:0/612739907 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48c80c15b0 0x7f48c814cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:08.208 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.201+0000 7f48c6d9d700 1 -- 172.21.15.112:0/612739907 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48c80c15b0 msgr2=0x7f48c814cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:33:08.208 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.201+0000 7f48c6d9d700 1 --2- 172.21.15.112:0/612739907 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48c80c15b0 0x7f48c814cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:08.208 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.201+0000 7f48c6d9d700 1 -- 172.21.15.112:0/612739907 >> v1:172.21.15.112:6789/0 conn(0x7f48c80c3080 legacy=0x7f48c805df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:08.209 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.201+0000 7f48c559a700 1 -- 172.21.15.112:0/612739907 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 432676629 0 0) 0x7f48c81566c0 con 0x7f48c80c3080 2023-01-24T03:33:08.209 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.201+0000 7f48c6d9d700 1 -- 172.21.15.112:0/612739907 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c8156c80 con 0x7f48c80c3450 2023-01-24T03:33:08.209 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.201+0000 7f48c6d9d700 1 --2- 172.21.15.112:0/612739907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f48c80c3450 0x7f48c814d400 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f48c0006fd0 tx=0x7f48c00090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfcd216029637d73 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:33:08.209 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.201+0000 7f48c559a700 1 -- 172.21.15.112:0/612739907 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c0011630 con 0x7f48c80c3450 2023-01-24T03:33:08.209 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.201+0000 7f48c559a700 1 -- 172.21.15.112:0/612739907 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48c0011790 con 0x7f48c80c3450 2023-01-24T03:33:08.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.201+0000 7f48c559a700 1 -- 172.21.15.112:0/612739907 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48c0011b50 con 0x7f48c80c3450 2023-01-24T03:33:08.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.202+0000 7f48cca0a700 1 -- 172.21.15.112:0/612739907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f48c80c3450 msgr2=0x7f48c814d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:08.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.202+0000 7f48cca0a700 1 --2- 172.21.15.112:0/612739907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f48c80c3450 0x7f48c814d400 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f48c0006fd0 tx=0x7f48c00090f0 comp rx=0 tx=0).stop 2023-01-24T03:33:08.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.202+0000 7f48cca0a700 1 -- 172.21.15.112:0/612739907 shutdown_connections 2023-01-24T03:33:08.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.202+0000 7f48cca0a700 1 --2- 172.21.15.112:0/612739907 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f48c80c3450 0x7f48c814d400 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:08.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.202+0000 7f48cca0a700 1 --2- 172.21.15.112:0/612739907 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48c80c15b0 0x7f48c814cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:08.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.202+0000 7f48cca0a700 1 -- 172.21.15.112:0/612739907 >> 172.21.15.112:0/612739907 conn(0x7f48c806daf0 msgr2=0x7f48c805d830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:33:08.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.202+0000 7f48cca0a700 1 -- 172.21.15.112:0/612739907 shutdown_connections 2023-01-24T03:33:08.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.202+0000 7f48cca0a700 1 -- 172.21.15.112:0/612739907 wait complete. 2023-01-24T03:33:08.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.203+0000 7f48cca0a700 1 Processor -- start 2023-01-24T03:33:08.212 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.203+0000 7f48cca0a700 1 -- start start 2023-01-24T03:33:08.214 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.204+0000 7f48cca0a700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f48c80c15b0 0x7f48c81662e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:08.214 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.204+0000 7f48cca0a700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48c80c3080 0x7f48c80c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:08.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.204+0000 7f48cca0a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48c80c3450 0x7f48c8158600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:08.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.204+0000 7f48cca0a700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c81528c0 con 0x7f48c80c15b0 2023-01-24T03:33:08.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.204+0000 7f48cca0a700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48c8163430 con 0x7f48c80c3450 2023-01-24T03:33:08.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.204+0000 7f48cca0a700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48c80c6990 con 0x7f48c80c3080 2023-01-24T03:33:08.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.204+0000 7f48c659c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f48c80c15b0 0x7f48c81662e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:08.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.204+0000 7f48c659c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f48c80c15b0 0x7f48c81662e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52142/0 (socket says 172.21.15.112:52142) 2023-01-24T03:33:08.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.204+0000 7f48c659c700 1 -- 172.21.15.112:0/632297522 learned_addr learned my addr 172.21.15.112:0/632297522 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:33:08.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.204+0000 7f48c6d9d700 1 --2- 172.21.15.112:0/632297522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48c80c3450 0x7f48c8158600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:08.217 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.204+0000 7f48c5d9b700 1 --2- 172.21.15.112:0/632297522 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48c80c3080 0x7f48c80c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:08.217 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.204+0000 7f48c659c700 1 -- 172.21.15.112:0/632297522 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48c80c3080 msgr2=0x7f48c80c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:08.217 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.204+0000 7f48c659c700 1 --2- 172.21.15.112:0/632297522 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48c80c3080 0x7f48c80c5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:08.217 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.204+0000 7f48c659c700 1 -- 172.21.15.112:0/632297522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48c80c3450 msgr2=0x7f48c8158600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:08.217 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.204+0000 7f48c659c700 1 --2- 172.21.15.112:0/632297522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48c80c3450 0x7f48c8158600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:08.218 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.204+0000 7f48c659c700 1 -- 172.21.15.112:0/632297522 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48c0009370 con 0x7f48c80c15b0 2023-01-24T03:33:08.218 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.204+0000 7f48c659c700 1 --2- 172.21.15.112:0/632297522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f48c80c15b0 0x7f48c81662e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f48b800c7f0 tx=0x7f48b8001580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:33:08.218 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.204+0000 7f48be7fc700 1 -- 172.21.15.112:0/632297522 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b8005c70 con 0x7f48c80c15b0 2023-01-24T03:33:08.218 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.204+0000 7f48be7fc700 1 -- 172.21.15.112:0/632297522 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48b8005df0 con 0x7f48c80c15b0 2023-01-24T03:33:08.218 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.204+0000 7f48be7fc700 1 -- 172.21.15.112:0/632297522 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b8007340 con 0x7f48c80c15b0 2023-01-24T03:33:08.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.204+0000 7f48cca0a700 1 -- 172.21.15.112:0/632297522 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48c8162e20 con 0x7f48c80c15b0 2023-01-24T03:33:08.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.204+0000 7f48cca0a700 1 -- 172.21.15.112:0/632297522 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48c80ca220 con 0x7f48c80c15b0 2023-01-24T03:33:08.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.205+0000 7f48cca0a700 1 -- 172.21.15.112:0/632297522 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48c804bb80 con 0x7f48c80c15b0 2023-01-24T03:33:08.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.206+0000 7f48be7fc700 1 -- 172.21.15.112:0/632297522 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f48b8010070 con 0x7f48c80c15b0 2023-01-24T03:33:08.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.206+0000 7f48be7fc700 1 --2- 172.21.15.112:0/632297522 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f48ac0611c0 0x7f48ac063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:08.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.206+0000 7f48be7fc700 1 -- 172.21.15.112:0/632297522 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7039+0+0 (secure 0 0 0) 0x7f48b807d340 con 0x7f48c80c15b0 2023-01-24T03:33:08.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.206+0000 7f48c5d9b700 1 --2- 172.21.15.112:0/632297522 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f48ac0611c0 0x7f48ac063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:08.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.206+0000 7f48c5d9b700 1 --2- 172.21.15.112:0/632297522 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f48ac0611c0 0x7f48ac063670 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f48c80c6860 tx=0x7f48b0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:33:08.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.210+0000 7f48be7fc700 1 -- 172.21.15.112:0/632297522 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f48b804d120 con 0x7f48c80c15b0 2023-01-24T03:33:08.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.430+0000 7f48cca0a700 1 -- 172.21.15.112:0/632297522 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48c8156f60 con 0x7f48c80c15b0 2023-01-24T03:33:08.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.430+0000 7f48be7fc700 1 -- 172.21.15.112:0/632297522 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+5355 (secure 0 0 0) 0x7f48b8050740 con 0x7f48c80c15b0 2023-01-24T03:33:08.434 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:33:08.435 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:33:05.613093+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4897},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4897":{"gid":4897,"name":"d","rank":0,"incarnation":16,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.142:6835/11022091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":11022091},{"type":"v1","addr":"172.21.15.142:6835","nonce":11022091}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:33:08.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.432+0000 7f48cca0a700 1 -- 172.21.15.112:0/632297522 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f48ac0611c0 msgr2=0x7f48ac063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:08.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.432+0000 7f48cca0a700 1 --2- 172.21.15.112:0/632297522 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f48ac0611c0 0x7f48ac063670 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f48c80c6860 tx=0x7f48b0008040 comp rx=0 tx=0).stop 2023-01-24T03:33:08.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.432+0000 7f48cca0a700 1 -- 172.21.15.112:0/632297522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f48c80c15b0 msgr2=0x7f48c81662e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:08.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.432+0000 7f48cca0a700 1 --2- 172.21.15.112:0/632297522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f48c80c15b0 0x7f48c81662e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f48b800c7f0 tx=0x7f48b8001580 comp rx=0 tx=0).stop 2023-01-24T03:33:08.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.432+0000 7f48cca0a700 1 -- 172.21.15.112:0/632297522 shutdown_connections 2023-01-24T03:33:08.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.432+0000 7f48cca0a700 1 --2- 172.21.15.112:0/632297522 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f48ac0611c0 0x7f48ac063670 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:08.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.432+0000 7f48cca0a700 1 --2- 172.21.15.112:0/632297522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f48c80c15b0 0x7f48c81662e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:08.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.432+0000 7f48cca0a700 1 --2- 172.21.15.112:0/632297522 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f48c80c3450 0x7f48c8158600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:08.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.432+0000 7f48cca0a700 1 --2- 172.21.15.112:0/632297522 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f48c80c3080 0x7f48c80c5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:08.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.432+0000 7f48cca0a700 1 -- 172.21.15.112:0/632297522 >> 172.21.15.112:0/632297522 conn(0x7f48c806daf0 msgr2=0x7f48c80c8e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:33:08.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.432+0000 7f48cca0a700 1 -- 172.21.15.112:0/632297522 shutdown_connections 2023-01-24T03:33:08.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:08.432+0000 7f48cca0a700 1 -- 172.21.15.112:0/632297522 wait complete. 2023-01-24T03:33:08.441 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 18 2023-01-24T03:33:08.447 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:33:10.447 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:33:10.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.564+0000 7f220509f700 1 Processor -- start 2023-01-24T03:33:10.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.564+0000 7f220509f700 1 -- start start 2023-01-24T03:33:10.572 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.564+0000 7f220509f700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f22000c4ea0 0x7f22000c1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:10.573 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.565+0000 7f220509f700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2200153270 con 0x7f22000c3560 2023-01-24T03:33:10.573 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.565+0000 7f220509f700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22001534e0 con 0x7f220006cf50 2023-01-24T03:33:10.573 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.565+0000 7f220509f700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2200153e80 con 0x7f22000c4ea0 2023-01-24T03:33:10.573 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.565+0000 7f21fe59c700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f22000c3560 0x7f22000c4790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:33140/0 (socket says 172.21.15.112:33140) 2023-01-24T03:33:10.573 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.565+0000 7f21fe59c700 1 -- 172.21.15.112:0/3899624874 learned_addr learned my addr 172.21.15.112:0/3899624874 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:33:10.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.565+0000 7f21fdd9b700 1 -- 172.21.15.112:0/3899624874 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 126460934 0 0) 0x7f2200153270 con 0x7f22000c3560 2023-01-24T03:33:10.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.565+0000 7f21fdd9b700 1 -- 172.21.15.112:0/3899624874 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21ec0033f0 con 0x7f22000c3560 2023-01-24T03:33:10.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.565+0000 7f21ff59e700 1 --2- 172.21.15.112:0/3899624874 >> v2:172.21.15.142:3300/0 conn(0x7f22000c4ea0 0x7f22000c1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:10.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.565+0000 7f21fdd9b700 1 -- 172.21.15.112:0/3899624874 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 498372286 0 0) 0x7f21ec0033f0 con 0x7f22000c3560 2023-01-24T03:33:10.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.565+0000 7f21fdd9b700 1 -- 172.21.15.112:0/3899624874 >> v2:172.21.15.142:3300/0 conn(0x7f22000c4ea0 msgr2=0x7f22000c1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:33:10.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.565+0000 7f21fdd9b700 1 --2- 172.21.15.112:0/3899624874 >> v2:172.21.15.142:3300/0 conn(0x7f22000c4ea0 0x7f22000c1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:10.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.565+0000 7f21fdd9b700 1 -- 172.21.15.112:0/3899624874 >> v1:172.21.15.142:6789/0 conn(0x7f220006cf50 legacy=0x7f22000c2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:10.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.565+0000 7f21fdd9b700 1 -- 172.21.15.112:0/3899624874 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22001537c0 con 0x7f22000c3560 2023-01-24T03:33:10.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.565+0000 7f21fdd9b700 1 -- 172.21.15.112:0/3899624874 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7f21f40017f0 con 0x7f22000c3560 2023-01-24T03:33:10.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.565+0000 7f21fdd9b700 1 -- 172.21.15.112:0/3899624874 >> v1:172.21.15.112:6789/0 conn(0x7f22000c3560 legacy=0x7f22000c4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:10.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.565+0000 7f21fdd9b700 1 --2- 172.21.15.112:0/3899624874 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21ec0038e0 0x7f21ec003cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:10.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.565+0000 7f21fdd9b700 1 --2- 172.21.15.112:0/3899624874 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f21ec004690 0x7f21ec004b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:10.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.565+0000 7f21fdd9b700 1 --2- 172.21.15.112:0/3899624874 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21ec005040 0x7f21ec00b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:10.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.565+0000 7f21fdd9b700 1 -- 172.21.15.112:0/3899624874 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2200153e80 con 0x7f21ec0038e0 2023-01-24T03:33:10.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.565+0000 7f21fdd9b700 1 -- 172.21.15.112:0/3899624874 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21ec00bf50 con 0x7f21ec005040 2023-01-24T03:33:10.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.565+0000 7f21fdd9b700 1 -- 172.21.15.112:0/3899624874 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21ec00c110 con 0x7f21ec004690 2023-01-24T03:33:10.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.565+0000 7f21fed9d700 1 --2- 172.21.15.112:0/3899624874 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21ec0038e0 0x7f21ec003cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:10.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.565+0000 7f21fe59c700 1 --2- 172.21.15.112:0/3899624874 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f21ec004690 0x7f21ec004b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:10.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.565+0000 7f21ff59e700 1 --2- 172.21.15.112:0/3899624874 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21ec005040 0x7f21ec00b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:10.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.566+0000 7f21fed9d700 1 -- 172.21.15.112:0/3899624874 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f21ec004690 msgr2=0x7f21ec004b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:33:10.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.566+0000 7f21fed9d700 1 --2- 172.21.15.112:0/3899624874 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f21ec004690 0x7f21ec004b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:10.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.566+0000 7f21fed9d700 1 -- 172.21.15.112:0/3899624874 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21ec005040 msgr2=0x7f21ec00b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:33:10.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.566+0000 7f21fed9d700 1 --2- 172.21.15.112:0/3899624874 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21ec005040 0x7f21ec00b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:10.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.566+0000 7f21fed9d700 1 -- 172.21.15.112:0/3899624874 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21ec0033f0 con 0x7f21ec0038e0 2023-01-24T03:33:10.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.566+0000 7f21ff59e700 1 --2- 172.21.15.112:0/3899624874 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21ec005040 0x7f21ec00b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:33:10.579 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.566+0000 7f21fe59c700 1 --2- 172.21.15.112:0/3899624874 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f21ec004690 0x7f21ec004b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:33:10.579 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.566+0000 7f21ff59e700 1 -- 172.21.15.112:0/3899624874 reap_dead start 2023-01-24T03:33:10.579 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.566+0000 7f21fed9d700 1 --2- 172.21.15.112:0/3899624874 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21ec0038e0 0x7f21ec003cf0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f21f8001730 tx=0x7f21f8007bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaf3ba10a2acf6d0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:33:10.579 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.566+0000 7f21fdd9b700 1 -- 172.21.15.112:0/3899624874 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f8012440 con 0x7f21ec0038e0 2023-01-24T03:33:10.579 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.566+0000 7f21fdd9b700 1 -- 172.21.15.112:0/3899624874 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21f80125a0 con 0x7f21ec0038e0 2023-01-24T03:33:10.580 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.566+0000 7f21fdd9b700 1 -- 172.21.15.112:0/3899624874 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f8012960 con 0x7f21ec0038e0 2023-01-24T03:33:10.580 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.566+0000 7f220509f700 1 -- 172.21.15.112:0/3899624874 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21ec0038e0 msgr2=0x7f21ec003cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:10.580 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.566+0000 7f220509f700 1 --2- 172.21.15.112:0/3899624874 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21ec0038e0 0x7f21ec003cf0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f21f8001730 tx=0x7f21f8007bb0 comp rx=0 tx=0).stop 2023-01-24T03:33:10.580 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.566+0000 7f220509f700 1 -- 172.21.15.112:0/3899624874 shutdown_connections 2023-01-24T03:33:10.580 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.566+0000 7f220509f700 1 --2- 172.21.15.112:0/3899624874 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21ec0038e0 0x7f21ec003cf0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:10.581 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.566+0000 7f220509f700 1 -- 172.21.15.112:0/3899624874 >> 172.21.15.112:0/3899624874 conn(0x7f22000c9820 msgr2=0x7f22000c9c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:33:10.581 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.566+0000 7f220509f700 1 -- 172.21.15.112:0/3899624874 shutdown_connections 2023-01-24T03:33:10.581 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.566+0000 7f220509f700 1 -- 172.21.15.112:0/3899624874 wait complete. 2023-01-24T03:33:10.581 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.568+0000 7f220509f700 1 Processor -- start 2023-01-24T03:33:10.581 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.568+0000 7f220509f700 1 -- start start 2023-01-24T03:33:10.582 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.568+0000 7f220509f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21ec0038e0 0x7f2200154910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:10.582 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.568+0000 7f220509f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2200166f50 0x7f220014ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:10.582 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.568+0000 7f220509f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f220014b070 0x7f22001561a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:10.582 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.568+0000 7f220509f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2200155d10 con 0x7f2200166f50 2023-01-24T03:33:10.582 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.568+0000 7f220509f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220014b4e0 con 0x7f21ec0038e0 2023-01-24T03:33:10.583 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.568+0000 7f220509f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22001550b0 con 0x7f220014b070 2023-01-24T03:33:10.583 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.568+0000 7f21fe59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2200166f50 0x7f220014ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:10.583 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.568+0000 7f21fe59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2200166f50 0x7f220014ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52166/0 (socket says 172.21.15.112:52166) 2023-01-24T03:33:10.583 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.568+0000 7f21fe59c700 1 -- 172.21.15.112:0/721800534 learned_addr learned my addr 172.21.15.112:0/721800534 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:33:10.583 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.568+0000 7f21fed9d700 1 --2- 172.21.15.112:0/721800534 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21ec0038e0 0x7f2200154910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:10.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.568+0000 7f21ff59e700 1 --2- 172.21.15.112:0/721800534 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f220014b070 0x7f22001561a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:10.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.569+0000 7f21fe59c700 1 -- 172.21.15.112:0/721800534 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f220014b070 msgr2=0x7f22001561a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:10.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.569+0000 7f21fe59c700 1 --2- 172.21.15.112:0/721800534 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f220014b070 0x7f22001561a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:10.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.569+0000 7f21fe59c700 1 -- 172.21.15.112:0/721800534 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21ec0038e0 msgr2=0x7f2200154910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:10.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.569+0000 7f21fe59c700 1 --2- 172.21.15.112:0/721800534 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21ec0038e0 0x7f2200154910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:10.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.569+0000 7f21fe59c700 1 -- 172.21.15.112:0/721800534 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2200155370 con 0x7f2200166f50 2023-01-24T03:33:10.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.569+0000 7f21fe59c700 1 --2- 172.21.15.112:0/721800534 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2200166f50 0x7f220014ab30 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f21f40074d0 tx=0x7f21f40017b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:33:10.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.569+0000 7f21e6ffd700 1 -- 172.21.15.112:0/721800534 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f400e860 con 0x7f2200166f50 2023-01-24T03:33:10.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.569+0000 7f21e6ffd700 1 -- 172.21.15.112:0/721800534 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21f400e9c0 con 0x7f2200166f50 2023-01-24T03:33:10.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.569+0000 7f220509f700 1 -- 172.21.15.112:0/721800534 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2200155650 con 0x7f2200166f50 2023-01-24T03:33:10.586 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.569+0000 7f21e6ffd700 1 -- 172.21.15.112:0/721800534 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f400ed80 con 0x7f2200166f50 2023-01-24T03:33:10.586 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.569+0000 7f220509f700 1 -- 172.21.15.112:0/721800534 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22001566e0 con 0x7f2200166f50 2023-01-24T03:33:10.586 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.570+0000 7f220509f700 1 -- 172.21.15.112:0/721800534 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f220004bb80 con 0x7f2200166f50 2023-01-24T03:33:10.587 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.570+0000 7f21e6ffd700 1 -- 172.21.15.112:0/721800534 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f21f4013630 con 0x7f2200166f50 2023-01-24T03:33:10.587 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.570+0000 7f21e6ffd700 1 --2- 172.21.15.112:0/721800534 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f21e00611c0 0x7f21e0063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:10.588 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.571+0000 7f21e6ffd700 1 -- 172.21.15.112:0/721800534 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7039+0+0 (secure 0 0 0) 0x7f21f407dc30 con 0x7f2200166f50 2023-01-24T03:33:10.588 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.571+0000 7f21fed9d700 1 --2- 172.21.15.112:0/721800534 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f21e00611c0 0x7f21e0063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:10.588 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.571+0000 7f21fed9d700 1 --2- 172.21.15.112:0/721800534 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f21e00611c0 0x7f21e0063670 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f21f8001010 tx=0x7f21f8015000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:33:10.588 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.575+0000 7f21e6ffd700 1 -- 172.21.15.112:0/721800534 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f21f404d990 con 0x7f2200166f50 2023-01-24T03:33:10.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.797+0000 7f220509f700 1 -- 172.21.15.112:0/721800534 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22000c4270 con 0x7f2200166f50 2023-01-24T03:33:10.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.797+0000 7f21e6ffd700 1 -- 172.21.15.112:0/721800534 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+5355 (secure 0 0 0) 0x7f21f4050fb0 con 0x7f2200166f50 2023-01-24T03:33:10.801 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:33:10.801 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:33:05.613093+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4897},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4897":{"gid":4897,"name":"d","rank":0,"incarnation":16,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.142:6835/11022091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":11022091},{"type":"v1","addr":"172.21.15.142:6835","nonce":11022091}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:33:10.802 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.799+0000 7f220509f700 1 -- 172.21.15.112:0/721800534 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f21e00611c0 msgr2=0x7f21e0063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:10.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.799+0000 7f220509f700 1 --2- 172.21.15.112:0/721800534 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f21e00611c0 0x7f21e0063670 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f21f8001010 tx=0x7f21f8015000 comp rx=0 tx=0).stop 2023-01-24T03:33:10.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.799+0000 7f220509f700 1 -- 172.21.15.112:0/721800534 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2200166f50 msgr2=0x7f220014ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:10.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.799+0000 7f220509f700 1 --2- 172.21.15.112:0/721800534 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2200166f50 0x7f220014ab30 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f21f40074d0 tx=0x7f21f40017b0 comp rx=0 tx=0).stop 2023-01-24T03:33:10.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.799+0000 7f220509f700 1 -- 172.21.15.112:0/721800534 shutdown_connections 2023-01-24T03:33:10.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.799+0000 7f220509f700 1 --2- 172.21.15.112:0/721800534 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f21e00611c0 0x7f21e0063670 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:10.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.799+0000 7f220509f700 1 --2- 172.21.15.112:0/721800534 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f220014b070 0x7f22001561a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:10.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.799+0000 7f220509f700 1 --2- 172.21.15.112:0/721800534 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f21ec0038e0 0x7f2200154910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:10.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.799+0000 7f220509f700 1 --2- 172.21.15.112:0/721800534 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2200166f50 0x7f220014ab30 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:10.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.799+0000 7f220509f700 1 -- 172.21.15.112:0/721800534 >> 172.21.15.112:0/721800534 conn(0x7f22000c9820 msgr2=0x7f220014ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:33:10.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.799+0000 7f220509f700 1 -- 172.21.15.112:0/721800534 shutdown_connections 2023-01-24T03:33:10.805 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:10.799+0000 7f220509f700 1 -- 172.21.15.112:0/721800534 wait complete. 2023-01-24T03:33:10.805 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 18 2023-01-24T03:33:10.814 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:33:12.815 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:33:12.939 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.936+0000 7fd21e81e700 1 Processor -- start 2023-01-24T03:33:12.939 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.936+0000 7fd21e81e700 1 -- start start 2023-01-24T03:33:12.946 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.936+0000 7fd21e81e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd2180caaf0 0x7fd218155da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:12.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.936+0000 7fd21e81e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2180b0f40 con 0x7fd2180b0aa0 2023-01-24T03:33:12.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.936+0000 7fd21e81e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2180b1170 con 0x7fd2180c7e40 2023-01-24T03:33:12.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.936+0000 7fd21e81e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2180b1450 con 0x7fd2180caaf0 2023-01-24T03:33:12.948 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.937+0000 7fd2177fe700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fd2180b0aa0 0x7fd2180ca3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:33172/0 (socket says 172.21.15.112:33172) 2023-01-24T03:33:12.948 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.937+0000 7fd2177fe700 1 -- 172.21.15.112:0/1842932096 learned_addr learned my addr 172.21.15.112:0/1842932096 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:33:12.948 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.937+0000 7fd216ffd700 1 -- 172.21.15.112:0/1842932096 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1539350597 0 0) 0x7fd2180b0f40 con 0x7fd2180b0aa0 2023-01-24T03:33:12.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.937+0000 7fd21cdbb700 1 --2- 172.21.15.112:0/1842932096 >> v2:172.21.15.142:3300/0 conn(0x7fd2180caaf0 0x7fd218155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:12.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.937+0000 7fd216ffd700 1 -- 172.21.15.112:0/1842932096 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2040033f0 con 0x7fd2180b0aa0 2023-01-24T03:33:12.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.937+0000 7fd216ffd700 1 -- 172.21.15.112:0/1842932096 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2179203643 0 0) 0x7fd2040033f0 con 0x7fd2180b0aa0 2023-01-24T03:33:12.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.937+0000 7fd216ffd700 1 -- 172.21.15.112:0/1842932096 >> v2:172.21.15.142:3300/0 conn(0x7fd2180caaf0 msgr2=0x7fd218155da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:33:12.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.937+0000 7fd216ffd700 1 --2- 172.21.15.112:0/1842932096 >> v2:172.21.15.142:3300/0 conn(0x7fd2180caaf0 0x7fd218155da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:12.951 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.937+0000 7fd216ffd700 1 -- 172.21.15.112:0/1842932096 >> v1:172.21.15.142:6789/0 conn(0x7fd2180c7e40 legacy=0x7fd2180ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:12.951 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.937+0000 7fd216ffd700 1 -- 172.21.15.112:0/1842932096 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2180b1730 con 0x7fd2180b0aa0 2023-01-24T03:33:12.951 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.937+0000 7fd216ffd700 1 -- 172.21.15.112:0/1842932096 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7fd20c001470 con 0x7fd2180b0aa0 2023-01-24T03:33:12.951 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.937+0000 7fd216ffd700 1 -- 172.21.15.112:0/1842932096 >> v1:172.21.15.112:6789/0 conn(0x7fd2180b0aa0 legacy=0x7fd2180ca3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:12.952 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.937+0000 7fd216ffd700 1 --2- 172.21.15.112:0/1842932096 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2040038e0 0x7fd204003cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:12.952 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.937+0000 7fd216ffd700 1 --2- 172.21.15.112:0/1842932096 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd204004690 0x7fd204004b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:12.953 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.937+0000 7fd216ffd700 1 --2- 172.21.15.112:0/1842932096 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd204005040 0x7fd20400b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:12.953 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.937+0000 7fd216ffd700 1 -- 172.21.15.112:0/1842932096 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2180b1450 con 0x7fd204004690 2023-01-24T03:33:12.953 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.937+0000 7fd216ffd700 1 -- 172.21.15.112:0/1842932096 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd20400bfe0 con 0x7fd2040038e0 2023-01-24T03:33:12.953 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.937+0000 7fd216ffd700 1 -- 172.21.15.112:0/1842932096 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd20400c1a0 con 0x7fd204005040 2023-01-24T03:33:12.954 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.937+0000 7fd2177fe700 1 --2- 172.21.15.112:0/1842932096 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd204004690 0x7fd204004b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:12.954 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.937+0000 7fd217fff700 1 --2- 172.21.15.112:0/1842932096 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2040038e0 0x7fd204003cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:12.954 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.937+0000 7fd21cdbb700 1 --2- 172.21.15.112:0/1842932096 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd204005040 0x7fd20400b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:12.954 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.938+0000 7fd2177fe700 1 -- 172.21.15.112:0/1842932096 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd204005040 msgr2=0x7fd20400b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:33:12.954 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.938+0000 7fd2177fe700 1 --2- 172.21.15.112:0/1842932096 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd204005040 0x7fd20400b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:12.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.938+0000 7fd2177fe700 1 -- 172.21.15.112:0/1842932096 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2040038e0 msgr2=0x7fd204003cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:33:12.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.938+0000 7fd2177fe700 1 --2- 172.21.15.112:0/1842932096 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2040038e0 0x7fd204003cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:12.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.938+0000 7fd2177fe700 1 -- 172.21.15.112:0/1842932096 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2040033f0 con 0x7fd204004690 2023-01-24T03:33:12.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.938+0000 7fd21cdbb700 1 --2- 172.21.15.112:0/1842932096 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd204005040 0x7fd20400b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:33:12.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.938+0000 7fd21cdbb700 1 -- 172.21.15.112:0/1842932096 reap_dead start 2023-01-24T03:33:12.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.938+0000 7fd2177fe700 1 --2- 172.21.15.112:0/1842932096 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd204004690 0x7fd204004b00 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd20c00c190 tx=0x7fd20c001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c53e099dc2f824a server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:33:12.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.938+0000 7fd216ffd700 1 -- 172.21.15.112:0/1842932096 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd20c00cac0 con 0x7fd204004690 2023-01-24T03:33:12.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.938+0000 7fd216ffd700 1 -- 172.21.15.112:0/1842932096 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd20c00cc20 con 0x7fd204004690 2023-01-24T03:33:12.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.938+0000 7fd216ffd700 1 -- 172.21.15.112:0/1842932096 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd20c0070d0 con 0x7fd204004690 2023-01-24T03:33:12.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.938+0000 7fd21e81e700 1 -- 172.21.15.112:0/1842932096 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd204004690 msgr2=0x7fd204004b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:12.957 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.938+0000 7fd21e81e700 1 --2- 172.21.15.112:0/1842932096 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd204004690 0x7fd204004b00 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd20c00c190 tx=0x7fd20c001e50 comp rx=0 tx=0).stop 2023-01-24T03:33:12.957 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.938+0000 7fd21e81e700 1 -- 172.21.15.112:0/1842932096 shutdown_connections 2023-01-24T03:33:12.957 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.938+0000 7fd21e81e700 1 --2- 172.21.15.112:0/1842932096 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd204004690 0x7fd204004b00 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:12.957 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.938+0000 7fd21e81e700 1 -- 172.21.15.112:0/1842932096 >> 172.21.15.112:0/1842932096 conn(0x7fd21805e2c0 msgr2=0x7fd2180cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:33:12.957 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.938+0000 7fd21e81e700 1 -- 172.21.15.112:0/1842932096 shutdown_connections 2023-01-24T03:33:12.958 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.938+0000 7fd21e81e700 1 -- 172.21.15.112:0/1842932096 wait complete. 2023-01-24T03:33:12.958 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.939+0000 7fd21e81e700 1 Processor -- start 2023-01-24T03:33:12.958 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.940+0000 7fd21e81e700 1 -- start start 2023-01-24T03:33:12.958 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.940+0000 7fd21e81e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd204004690 0x7fd218165db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:12.959 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.940+0000 7fd21e81e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd21814fc80 0x7fd218166840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:12.959 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.940+0000 7fd21e81e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2180c4720 0x7fd218157f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:12.959 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.940+0000 7fd21e81e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2180bfe80 con 0x7fd204004690 2023-01-24T03:33:12.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.940+0000 7fd21e81e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2180c4290 con 0x7fd2180c4720 2023-01-24T03:33:12.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.940+0000 7fd21e81e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd218150cb0 con 0x7fd21814fc80 2023-01-24T03:33:12.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.940+0000 7fd217fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd204004690 0x7fd218165db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:12.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.940+0000 7fd217fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd204004690 0x7fd218165db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52196/0 (socket says 172.21.15.112:52196) 2023-01-24T03:33:12.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.940+0000 7fd217fff700 1 -- 172.21.15.112:0/3135346018 learned_addr learned my addr 172.21.15.112:0/3135346018 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:33:12.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.941+0000 7fd21cdbb700 1 --2- 172.21.15.112:0/3135346018 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2180c4720 0x7fd218157f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:12.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.941+0000 7fd2177fe700 1 --2- 172.21.15.112:0/3135346018 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd21814fc80 0x7fd218166840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:12.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.941+0000 7fd217fff700 1 -- 172.21.15.112:0/3135346018 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd21814fc80 msgr2=0x7fd218166840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:12.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.941+0000 7fd217fff700 1 --2- 172.21.15.112:0/3135346018 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd21814fc80 0x7fd218166840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:12.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.941+0000 7fd217fff700 1 -- 172.21.15.112:0/3135346018 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2180c4720 msgr2=0x7fd218157f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:12.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.941+0000 7fd217fff700 1 --2- 172.21.15.112:0/3135346018 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2180c4720 0x7fd218157f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:12.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.941+0000 7fd217fff700 1 -- 172.21.15.112:0/3135346018 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2180bf8e0 con 0x7fd204004690 2023-01-24T03:33:12.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.941+0000 7fd2177fe700 1 --2- 172.21.15.112:0/3135346018 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd21814fc80 0x7fd218166840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:33:12.965 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.941+0000 7fd217fff700 1 --2- 172.21.15.112:0/3135346018 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd204004690 0x7fd218165db0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fd208002920 tx=0x7fd208007bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:33:12.965 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.941+0000 7fd1fffff700 1 -- 172.21.15.112:0/3135346018 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd208013600 con 0x7fd204004690 2023-01-24T03:33:12.965 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.941+0000 7fd1fffff700 1 -- 172.21.15.112:0/3135346018 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd208013760 con 0x7fd204004690 2023-01-24T03:33:12.966 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.941+0000 7fd21e81e700 1 -- 172.21.15.112:0/3135346018 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2180bfba0 con 0x7fd204004690 2023-01-24T03:33:12.966 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.941+0000 7fd1fffff700 1 -- 172.21.15.112:0/3135346018 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd208013b00 con 0x7fd204004690 2023-01-24T03:33:12.967 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.941+0000 7fd21e81e700 1 -- 172.21.15.112:0/3135346018 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2180c3f20 con 0x7fd204004690 2023-01-24T03:33:12.967 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.942+0000 7fd21e81e700 1 -- 172.21.15.112:0/3135346018 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2180b0ec0 con 0x7fd204004690 2023-01-24T03:33:12.967 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.942+0000 7fd1fffff700 1 -- 172.21.15.112:0/3135346018 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fd20801c400 con 0x7fd204004690 2023-01-24T03:33:12.968 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.943+0000 7fd1fffff700 1 --2- 172.21.15.112:0/3135346018 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd1f80611f0 0x7fd1f80636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:12.968 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.943+0000 7fd1fffff700 1 -- 172.21.15.112:0/3135346018 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7039+0+0 (secure 0 0 0) 0x7fd20807dad0 con 0x7fd204004690 2023-01-24T03:33:12.968 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.943+0000 7fd2177fe700 1 --2- 172.21.15.112:0/3135346018 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd1f80611f0 0x7fd1f80636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:12.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.943+0000 7fd2177fe700 1 --2- 172.21.15.112:0/3135346018 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd1f80611f0 0x7fd1f80636a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd2180693e0 tx=0x7fd20c015000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:33:12.970 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:12.947+0000 7fd1fffff700 1 -- 172.21.15.112:0/3135346018 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fd20804d7d0 con 0x7fd204004690 2023-01-24T03:33:13.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:13.173+0000 7fd21e81e700 1 -- 172.21.15.112:0/3135346018 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2180c5df0 con 0x7fd204004690 2023-01-24T03:33:13.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:13.174+0000 7fd1fffff700 1 -- 172.21.15.112:0/3135346018 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+5361 (secure 0 0 0) 0x7fd208050df0 con 0x7fd204004690 2023-01-24T03:33:13.177 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:33:13.177 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:33:11.627835+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4897},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4897":{"gid":4897,"name":"d","rank":0,"incarnation":16,"state":"up:clientreplay","state_seq":17,"addr":"172.21.15.142:6835/11022091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":11022091},{"type":"v1","addr":"172.21.15.142:6835","nonce":11022091}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:33:13.179 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:13.176+0000 7fd21e81e700 1 -- 172.21.15.112:0/3135346018 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd1f80611f0 msgr2=0x7fd1f80636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:13.179 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:13.176+0000 7fd21e81e700 1 --2- 172.21.15.112:0/3135346018 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd1f80611f0 0x7fd1f80636a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd2180693e0 tx=0x7fd20c015000 comp rx=0 tx=0).stop 2023-01-24T03:33:13.180 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:13.176+0000 7fd21e81e700 1 -- 172.21.15.112:0/3135346018 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd204004690 msgr2=0x7fd218165db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:13.180 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:13.176+0000 7fd21e81e700 1 --2- 172.21.15.112:0/3135346018 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd204004690 0x7fd218165db0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fd208002920 tx=0x7fd208007bb0 comp rx=0 tx=0).stop 2023-01-24T03:33:13.180 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:13.176+0000 7fd21e81e700 1 -- 172.21.15.112:0/3135346018 shutdown_connections 2023-01-24T03:33:13.180 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:13.176+0000 7fd21e81e700 1 --2- 172.21.15.112:0/3135346018 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd1f80611f0 0x7fd1f80636a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:13.180 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:13.176+0000 7fd21e81e700 1 --2- 172.21.15.112:0/3135346018 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd204004690 0x7fd218165db0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:13.181 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:13.176+0000 7fd21e81e700 1 --2- 172.21.15.112:0/3135346018 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd2180c4720 0x7fd218157f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:13.181 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:13.176+0000 7fd21e81e700 1 --2- 172.21.15.112:0/3135346018 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd21814fc80 0x7fd218166840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:13.181 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:13.176+0000 7fd21e81e700 1 -- 172.21.15.112:0/3135346018 >> 172.21.15.112:0/3135346018 conn(0x7fd21805e2c0 msgr2=0x7fd218068aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:33:13.181 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:13.176+0000 7fd21e81e700 1 -- 172.21.15.112:0/3135346018 shutdown_connections 2023-01-24T03:33:13.181 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:13.176+0000 7fd21e81e700 1 -- 172.21.15.112:0/3135346018 wait complete. 2023-01-24T03:33:13.182 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 19 2023-01-24T03:33:13.191 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:33:15.192 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:33:15.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.313+0000 7fb20d7fa700 1 Processor -- start 2023-01-24T03:33:15.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.313+0000 7fb20d7fa700 1 -- start start 2023-01-24T03:33:15.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.314+0000 7fb20d7fa700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb2080b0aa0 0x7fb208155da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:15.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.314+0000 7fb20d7fa700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb2080ca3e0 0x7fb2080ca840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:15.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.314+0000 7fb20d7fa700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2080b0f40 con 0x7fb2080c7e40 2023-01-24T03:33:15.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.314+0000 7fb20d7fa700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2080cad80 con 0x7fb2080ca3e0 2023-01-24T03:33:15.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.314+0000 7fb20d7fa700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2080b1170 con 0x7fb2080b0aa0 2023-01-24T03:33:15.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.314+0000 7fb2067fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb2080b0aa0 0x7fb208155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:15.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.314+0000 7fb2077fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb2080ca3e0 0x7fb2080ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:15.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.314+0000 7fb2067fc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb2080b0aa0 0x7fb208155da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.112:39510/0 (socket says 172.21.15.112:39510) 2023-01-24T03:33:15.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.314+0000 7fb2067fc700 1 -- 172.21.15.112:0/1774381589 learned_addr learned my addr 172.21.15.112:0/1774381589 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:33:15.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.315+0000 7fb205ffb700 1 -- 172.21.15.112:0/1774381589 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2267877485 0 0) 0x7fb2080b0f40 con 0x7fb2080c7e40 2023-01-24T03:33:15.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.315+0000 7fb205ffb700 1 -- 172.21.15.112:0/1774381589 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1e80033f0 con 0x7fb2080c7e40 2023-01-24T03:33:15.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.315+0000 7fb2067fc700 1 -- 172.21.15.112:0/1774381589 >> v2:172.21.15.142:3300/0 conn(0x7fb2080ca3e0 msgr2=0x7fb2080ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:33:15.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.315+0000 7fb2067fc700 1 --2- 172.21.15.112:0/1774381589 >> v2:172.21.15.142:3300/0 conn(0x7fb2080ca3e0 0x7fb2080ca840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:15.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.315+0000 7fb2067fc700 1 -- 172.21.15.112:0/1774381589 >> v1:172.21.15.142:6789/0 conn(0x7fb2080c7e40 legacy=0x7fb2080ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:15.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.315+0000 7fb2067fc700 1 -- 172.21.15.112:0/1774381589 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2080b1450 con 0x7fb2080b0aa0 2023-01-24T03:33:15.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.315+0000 7fb2077fe700 1 --2- 172.21.15.112:0/1774381589 >> v2:172.21.15.142:3300/0 conn(0x7fb2080ca3e0 0x7fb2080ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:33:15.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.315+0000 7fb2067fc700 1 --2- 172.21.15.112:0/1774381589 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb2080b0aa0 0x7fb208155da0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0006fd0 tx=0x7fb1f00090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e18b2bc569a42f6e server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:33:15.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.315+0000 7fb205ffb700 1 -- 172.21.15.112:0/1774381589 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f0011630 con 0x7fb2080b0aa0 2023-01-24T03:33:15.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.315+0000 7fb205ffb700 1 -- 172.21.15.112:0/1774381589 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1f0011790 con 0x7fb2080b0aa0 2023-01-24T03:33:15.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.315+0000 7fb205ffb700 1 -- 172.21.15.112:0/1774381589 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f0011b50 con 0x7fb2080b0aa0 2023-01-24T03:33:15.324 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.316+0000 7fb20d7fa700 1 -- 172.21.15.112:0/1774381589 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb2080b0aa0 msgr2=0x7fb208155da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:15.324 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.316+0000 7fb20d7fa700 1 --2- 172.21.15.112:0/1774381589 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb2080b0aa0 0x7fb208155da0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0006fd0 tx=0x7fb1f00090f0 comp rx=0 tx=0).stop 2023-01-24T03:33:15.324 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.316+0000 7fb20d7fa700 1 -- 172.21.15.112:0/1774381589 shutdown_connections 2023-01-24T03:33:15.324 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.316+0000 7fb20d7fa700 1 --2- 172.21.15.112:0/1774381589 >> v2:172.21.15.142:3300/0 conn(0x7fb2080ca3e0 0x7fb2080ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:15.324 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.316+0000 7fb20d7fa700 1 --2- 172.21.15.112:0/1774381589 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb2080b0aa0 0x7fb208155da0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:15.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.316+0000 7fb20d7fa700 1 -- 172.21.15.112:0/1774381589 >> 172.21.15.112:0/1774381589 conn(0x7fb20805e2c0 msgr2=0x7fb2080cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:33:15.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.316+0000 7fb20d7fa700 1 -- 172.21.15.112:0/1774381589 shutdown_connections 2023-01-24T03:33:15.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.316+0000 7fb20d7fa700 1 -- 172.21.15.112:0/1774381589 wait complete. 2023-01-24T03:33:15.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.317+0000 7fb20d7fa700 1 Processor -- start 2023-01-24T03:33:15.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.317+0000 7fb20d7fa700 1 -- start start 2023-01-24T03:33:15.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.318+0000 7fb20d7fa700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2080b0aa0 0x7fb208164f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:15.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.318+0000 7fb20d7fa700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb2080c7e40 0x7fb2080c2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:15.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.318+0000 7fb20d7fa700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2080ca3e0 0x7fb20814bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:15.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.318+0000 7fb20d7fa700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2080c9d10 con 0x7fb2080ca3e0 2023-01-24T03:33:15.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.318+0000 7fb20d7fa700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb208155160 con 0x7fb2080b0aa0 2023-01-24T03:33:15.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.318+0000 7fb20d7fa700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2080c0a60 con 0x7fb2080c7e40 2023-01-24T03:33:15.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.318+0000 7fb2077fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2080ca3e0 0x7fb20814bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:15.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.318+0000 7fb2077fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2080ca3e0 0x7fb20814bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52214/0 (socket says 172.21.15.112:52214) 2023-01-24T03:33:15.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.318+0000 7fb2077fe700 1 -- 172.21.15.112:0/3741065519 learned_addr learned my addr 172.21.15.112:0/3741065519 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:33:15.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.318+0000 7fb2067fc700 1 --2- 172.21.15.112:0/3741065519 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb2080c7e40 0x7fb2080c2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:15.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.318+0000 7fb206ffd700 1 --2- 172.21.15.112:0/3741065519 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2080b0aa0 0x7fb208164f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:15.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.318+0000 7fb2077fe700 1 -- 172.21.15.112:0/3741065519 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb2080c7e40 msgr2=0x7fb2080c2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:15.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.318+0000 7fb2077fe700 1 --2- 172.21.15.112:0/3741065519 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb2080c7e40 0x7fb2080c2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:15.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.318+0000 7fb2077fe700 1 -- 172.21.15.112:0/3741065519 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2080b0aa0 msgr2=0x7fb208164f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:15.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.318+0000 7fb2077fe700 1 --2- 172.21.15.112:0/3741065519 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2080b0aa0 0x7fb208164f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:15.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.318+0000 7fb2077fe700 1 -- 172.21.15.112:0/3741065519 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2000092a0 con 0x7fb2080ca3e0 2023-01-24T03:33:15.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.318+0000 7fb2067fc700 1 --2- 172.21.15.112:0/3741065519 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb2080c7e40 0x7fb2080c2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:33:15.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.318+0000 7fb206ffd700 1 --2- 172.21.15.112:0/3741065519 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2080b0aa0 0x7fb208164f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:33:15.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.318+0000 7fb2077fe700 1 --2- 172.21.15.112:0/3741065519 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2080ca3e0 0x7fb20814bd30 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb20000e9b0 tx=0x7fb200009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:33:15.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.318+0000 7fb1feffd700 1 -- 172.21.15.112:0/3741065519 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20000c850 con 0x7fb2080ca3e0 2023-01-24T03:33:15.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.318+0000 7fb1feffd700 1 -- 172.21.15.112:0/3741065519 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb20000c9b0 con 0x7fb2080ca3e0 2023-01-24T03:33:15.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.318+0000 7fb1feffd700 1 -- 172.21.15.112:0/3741065519 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20000cd50 con 0x7fb2080ca3e0 2023-01-24T03:33:15.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.318+0000 7fb20d7fa700 1 -- 172.21.15.112:0/3741065519 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1f0009370 con 0x7fb2080ca3e0 2023-01-24T03:33:15.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.318+0000 7fb20d7fa700 1 -- 172.21.15.112:0/3741065519 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2080c1020 con 0x7fb2080ca3e0 2023-01-24T03:33:15.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.319+0000 7fb20d7fa700 1 -- 172.21.15.112:0/3741065519 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb20804bb80 con 0x7fb2080ca3e0 2023-01-24T03:33:15.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.320+0000 7fb1feffd700 1 -- 172.21.15.112:0/3741065519 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fb200013680 con 0x7fb2080ca3e0 2023-01-24T03:33:15.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.320+0000 7fb1feffd700 1 --2- 172.21.15.112:0/3741065519 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb1ec0611c0 0x7fb1ec063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:15.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.320+0000 7fb1feffd700 1 -- 172.21.15.112:0/3741065519 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7039+0+0 (secure 0 0 0) 0x7fb20007daa0 con 0x7fb2080ca3e0 2023-01-24T03:33:15.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.320+0000 7fb206ffd700 1 --2- 172.21.15.112:0/3741065519 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb1ec0611c0 0x7fb1ec063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:15.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.320+0000 7fb206ffd700 1 --2- 172.21.15.112:0/3741065519 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb1ec0611c0 0x7fb1ec063670 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb1f8003eb0 tx=0x7fb1f8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:33:15.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.324+0000 7fb1feffd700 1 -- 172.21.15.112:0/3741065519 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fb20004d800 con 0x7fb2080ca3e0 2023-01-24T03:33:15.546 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.543+0000 7fb20d7fa700 1 -- 172.21.15.112:0/3741065519 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2080c5df0 con 0x7fb2080ca3e0 2023-01-24T03:33:15.547 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.544+0000 7fb1feffd700 1 -- 172.21.15.112:0/3741065519 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+5361 (secure 0 0 0) 0x7fb200050e20 con 0x7fb2080ca3e0 2023-01-24T03:33:15.547 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:33:15.547 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:33:11.627835+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4897},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4897":{"gid":4897,"name":"d","rank":0,"incarnation":16,"state":"up:clientreplay","state_seq":17,"addr":"172.21.15.142:6835/11022091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":11022091},{"type":"v1","addr":"172.21.15.142:6835","nonce":11022091}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:33:15.549 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.545+0000 7fb20d7fa700 1 -- 172.21.15.112:0/3741065519 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb1ec0611c0 msgr2=0x7fb1ec063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:15.549 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.545+0000 7fb20d7fa700 1 --2- 172.21.15.112:0/3741065519 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb1ec0611c0 0x7fb1ec063670 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb1f8003eb0 tx=0x7fb1f8009040 comp rx=0 tx=0).stop 2023-01-24T03:33:15.549 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.545+0000 7fb20d7fa700 1 -- 172.21.15.112:0/3741065519 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2080ca3e0 msgr2=0x7fb20814bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:15.550 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.545+0000 7fb20d7fa700 1 --2- 172.21.15.112:0/3741065519 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2080ca3e0 0x7fb20814bd30 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb20000e9b0 tx=0x7fb200009060 comp rx=0 tx=0).stop 2023-01-24T03:33:15.550 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.546+0000 7fb20d7fa700 1 -- 172.21.15.112:0/3741065519 shutdown_connections 2023-01-24T03:33:15.550 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.546+0000 7fb20d7fa700 1 --2- 172.21.15.112:0/3741065519 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb1ec0611c0 0x7fb1ec063670 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:15.550 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.546+0000 7fb20d7fa700 1 --2- 172.21.15.112:0/3741065519 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb2080ca3e0 0x7fb20814bd30 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:15.550 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.546+0000 7fb20d7fa700 1 --2- 172.21.15.112:0/3741065519 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2080b0aa0 0x7fb208164f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:15.551 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.546+0000 7fb20d7fa700 1 --2- 172.21.15.112:0/3741065519 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb2080c7e40 0x7fb2080c2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:15.551 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.546+0000 7fb20d7fa700 1 -- 172.21.15.112:0/3741065519 >> 172.21.15.112:0/3741065519 conn(0x7fb20805e2c0 msgr2=0x7fb2080ae9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:33:15.551 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.546+0000 7fb20d7fa700 1 -- 172.21.15.112:0/3741065519 shutdown_connections 2023-01-24T03:33:15.551 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:15.546+0000 7fb20d7fa700 1 -- 172.21.15.112:0/3741065519 wait complete. 2023-01-24T03:33:15.552 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 19 2023-01-24T03:33:15.560 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:33:17.562 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:33:17.709 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.706+0000 7f22646d4700 1 Processor -- start 2023-01-24T03:33:17.710 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.706+0000 7f22646d4700 1 -- start start 2023-01-24T03:33:17.712 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.707+0000 7f22646d4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f225c0b0aa0 0x7f225c155da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:17.712 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.707+0000 7f22646d4700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f225c0b0f40 con 0x7f225c0ca3e0 2023-01-24T03:33:17.712 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.707+0000 7f22646d4700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f225c0b1170 con 0x7f225c0c7e40 2023-01-24T03:33:17.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.707+0000 7f22646d4700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f225c0b1450 con 0x7f225c0b0aa0 2023-01-24T03:33:17.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.707+0000 7f2262c71700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f225c0ca3e0 0x7f225c0ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:33228/0 (socket says 172.21.15.112:33228) 2023-01-24T03:33:17.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.707+0000 7f2262c71700 1 -- 172.21.15.112:0/2049401384 learned_addr learned my addr 172.21.15.112:0/2049401384 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:33:17.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.707+0000 7f2261c6f700 1 --2- 172.21.15.112:0/2049401384 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f225c0b0aa0 0x7f225c155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:17.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.708+0000 7f226146e700 1 -- 172.21.15.112:0/2049401384 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4073221749 0 0) 0x7f225c0b0f40 con 0x7f225c0ca3e0 2023-01-24T03:33:17.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.708+0000 7f226146e700 1 -- 172.21.15.112:0/2049401384 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22440033f0 con 0x7f225c0ca3e0 2023-01-24T03:33:17.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.708+0000 7f226146e700 1 -- 172.21.15.112:0/2049401384 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3242711559 0 0) 0x7f225c0b1170 con 0x7f225c0c7e40 2023-01-24T03:33:17.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.708+0000 7f226146e700 1 -- 172.21.15.112:0/2049401384 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f225c0b0f40 con 0x7f225c0c7e40 2023-01-24T03:33:17.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.708+0000 7f2261c6f700 1 -- 172.21.15.112:0/2049401384 >> v1:172.21.15.142:6789/0 conn(0x7f225c0c7e40 legacy=0x7f225c0ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:17.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.708+0000 7f2261c6f700 1 -- 172.21.15.112:0/2049401384 >> v1:172.21.15.112:6789/0 conn(0x7f225c0ca3e0 legacy=0x7f225c0ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:17.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.708+0000 7f2261c6f700 1 -- 172.21.15.112:0/2049401384 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f225c0b1730 con 0x7f225c0b0aa0 2023-01-24T03:33:17.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.708+0000 7f2261c6f700 1 --2- 172.21.15.112:0/2049401384 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f225c0b0aa0 0x7f225c155da0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f224c009790 tx=0x7f224c009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10aa212660249b60 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:33:17.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.708+0000 7f226146e700 1 -- 172.21.15.112:0/2049401384 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f224c011630 con 0x7f225c0b0aa0 2023-01-24T03:33:17.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.708+0000 7f226146e700 1 -- 172.21.15.112:0/2049401384 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f224c011790 con 0x7f225c0b0aa0 2023-01-24T03:33:17.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.708+0000 7f226146e700 1 -- 172.21.15.112:0/2049401384 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f224c011b50 con 0x7f225c0b0aa0 2023-01-24T03:33:17.716 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.708+0000 7f22646d4700 1 -- 172.21.15.112:0/2049401384 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f225c0b0aa0 msgr2=0x7f225c155da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:17.716 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.708+0000 7f22646d4700 1 --2- 172.21.15.112:0/2049401384 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f225c0b0aa0 0x7f225c155da0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f224c009790 tx=0x7f224c009060 comp rx=0 tx=0).stop 2023-01-24T03:33:17.716 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.709+0000 7f22646d4700 1 -- 172.21.15.112:0/2049401384 shutdown_connections 2023-01-24T03:33:17.716 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.709+0000 7f22646d4700 1 --2- 172.21.15.112:0/2049401384 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f225c0b0aa0 0x7f225c155da0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:17.716 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.709+0000 7f22646d4700 1 -- 172.21.15.112:0/2049401384 >> 172.21.15.112:0/2049401384 conn(0x7f225c05e2c0 msgr2=0x7f225c0cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:33:17.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.709+0000 7f22646d4700 1 -- 172.21.15.112:0/2049401384 shutdown_connections 2023-01-24T03:33:17.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.709+0000 7f22646d4700 1 -- 172.21.15.112:0/2049401384 wait complete. 2023-01-24T03:33:17.720 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.710+0000 7f22646d4700 1 Processor -- start 2023-01-24T03:33:17.720 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.710+0000 7f22646d4700 1 -- start start 2023-01-24T03:33:17.720 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.710+0000 7f22646d4700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f225c0b0aa0 0x7f225c0c19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:17.720 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.710+0000 7f22646d4700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f225c0c7e40 0x7f225c0c2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:17.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.710+0000 7f22646d4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f225c0ca3e0 0x7f225c0c34a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:17.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.710+0000 7f22646d4700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f225c0c2ad0 con 0x7f225c0c7e40 2023-01-24T03:33:17.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.710+0000 7f22646d4700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f225c0c9bd0 con 0x7f225c0b0aa0 2023-01-24T03:33:17.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.710+0000 7f22646d4700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f225c168dd0 con 0x7f225c0ca3e0 2023-01-24T03:33:17.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.710+0000 7f2261c6f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f225c0c7e40 0x7f225c0c2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:17.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.710+0000 7f2261c6f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f225c0c7e40 0x7f225c0c2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52246/0 (socket says 172.21.15.112:52246) 2023-01-24T03:33:17.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.710+0000 7f2261c6f700 1 -- 172.21.15.112:0/1042877075 learned_addr learned my addr 172.21.15.112:0/1042877075 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:33:17.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.710+0000 7f2262470700 1 --2- 172.21.15.112:0/1042877075 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f225c0b0aa0 0x7f225c0c19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:17.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.710+0000 7f2262c71700 1 --2- 172.21.15.112:0/1042877075 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f225c0ca3e0 0x7f225c0c34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:17.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.711+0000 7f2261c6f700 1 -- 172.21.15.112:0/1042877075 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f225c0ca3e0 msgr2=0x7f225c0c34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:17.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.711+0000 7f2261c6f700 1 --2- 172.21.15.112:0/1042877075 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f225c0ca3e0 0x7f225c0c34a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:17.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.711+0000 7f2261c6f700 1 -- 172.21.15.112:0/1042877075 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f225c0b0aa0 msgr2=0x7f225c0c19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:17.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.711+0000 7f2261c6f700 1 --2- 172.21.15.112:0/1042877075 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f225c0b0aa0 0x7f225c0c19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:17.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.711+0000 7f2261c6f700 1 -- 172.21.15.112:0/1042877075 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f224c0092a0 con 0x7f225c0c7e40 2023-01-24T03:33:17.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.711+0000 7f2262470700 1 --2- 172.21.15.112:0/1042877075 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f225c0b0aa0 0x7f225c0c19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:33:17.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.711+0000 7f2261c6f700 1 --2- 172.21.15.112:0/1042877075 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f225c0c7e40 0x7f225c0c2f60 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f225c089340 tx=0x7f224c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:33:17.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.711+0000 7f224a7fc700 1 -- 172.21.15.112:0/1042877075 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f224c002f70 con 0x7f225c0c7e40 2023-01-24T03:33:17.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.711+0000 7f224a7fc700 1 -- 172.21.15.112:0/1042877075 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f224c000bc0 con 0x7f225c0c7e40 2023-01-24T03:33:17.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.711+0000 7f224a7fc700 1 -- 172.21.15.112:0/1042877075 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f224c011b50 con 0x7f225c0c7e40 2023-01-24T03:33:17.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.711+0000 7f22646d4700 1 -- 172.21.15.112:0/1042877075 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f225c1690b0 con 0x7f225c0c7e40 2023-01-24T03:33:17.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.711+0000 7f22646d4700 1 -- 172.21.15.112:0/1042877075 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f225c169670 con 0x7f225c0c7e40 2023-01-24T03:33:17.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.712+0000 7f22646d4700 1 -- 172.21.15.112:0/1042877075 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f225c0c2c40 con 0x7f225c0c7e40 2023-01-24T03:33:17.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.712+0000 7f224a7fc700 1 -- 172.21.15.112:0/1042877075 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f224c02c030 con 0x7f225c0c7e40 2023-01-24T03:33:17.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.713+0000 7f224a7fc700 1 --2- 172.21.15.112:0/1042877075 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f223c0611c0 0x7f223c063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:17.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.713+0000 7f2262470700 1 --2- 172.21.15.112:0/1042877075 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f223c0611c0 0x7f223c063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:17.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.713+0000 7f224a7fc700 1 -- 172.21.15.112:0/1042877075 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7039+0+0 (secure 0 0 0) 0x7f224c07ec50 con 0x7f225c0c7e40 2023-01-24T03:33:17.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.713+0000 7f2262470700 1 --2- 172.21.15.112:0/1042877075 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f223c0611c0 0x7f223c063670 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2258007fd0 tx=0x7f2258007bd0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:33:17.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.717+0000 7f224a7fc700 1 -- 172.21.15.112:0/1042877075 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f224c04e910 con 0x7f225c0c7e40 2023-01-24T03:33:17.943 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.940+0000 7f22646d4700 1 -- 172.21.15.112:0/1042877075 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f225c04bb80 con 0x7f225c0c7e40 2023-01-24T03:33:17.944 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.941+0000 7f224a7fc700 1 -- 172.21.15.112:0/1042877075 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+5361 (secure 0 0 0) 0x7f224c051f30 con 0x7f225c0c7e40 2023-01-24T03:33:17.944 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:33:17.944 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:33:11.627835+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4897},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4897":{"gid":4897,"name":"d","rank":0,"incarnation":16,"state":"up:clientreplay","state_seq":17,"addr":"172.21.15.142:6835/11022091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":11022091},{"type":"v1","addr":"172.21.15.142:6835","nonce":11022091}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:33:17.946 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.943+0000 7f22646d4700 1 -- 172.21.15.112:0/1042877075 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f223c0611c0 msgr2=0x7f223c063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:17.946 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.943+0000 7f22646d4700 1 --2- 172.21.15.112:0/1042877075 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f223c0611c0 0x7f223c063670 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2258007fd0 tx=0x7f2258007bd0 comp rx=0 tx=0).stop 2023-01-24T03:33:17.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.943+0000 7f22646d4700 1 -- 172.21.15.112:0/1042877075 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f225c0c7e40 msgr2=0x7f225c0c2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:17.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.943+0000 7f22646d4700 1 --2- 172.21.15.112:0/1042877075 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f225c0c7e40 0x7f225c0c2f60 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f225c089340 tx=0x7f224c017000 comp rx=0 tx=0).stop 2023-01-24T03:33:17.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.943+0000 7f22646d4700 1 -- 172.21.15.112:0/1042877075 shutdown_connections 2023-01-24T03:33:17.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.943+0000 7f22646d4700 1 --2- 172.21.15.112:0/1042877075 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f223c0611c0 0x7f223c063670 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:17.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.943+0000 7f22646d4700 1 --2- 172.21.15.112:0/1042877075 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f225c0ca3e0 0x7f225c0c34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:17.948 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.943+0000 7f22646d4700 1 --2- 172.21.15.112:0/1042877075 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f225c0b0aa0 0x7f225c0c19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:17.948 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.943+0000 7f22646d4700 1 --2- 172.21.15.112:0/1042877075 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f225c0c7e40 0x7f225c0c2f60 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:17.948 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.943+0000 7f22646d4700 1 -- 172.21.15.112:0/1042877075 >> 172.21.15.112:0/1042877075 conn(0x7f225c05e2c0 msgr2=0x7f225c068c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:33:17.948 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.943+0000 7f22646d4700 1 -- 172.21.15.112:0/1042877075 shutdown_connections 2023-01-24T03:33:17.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:17.943+0000 7f22646d4700 1 -- 172.21.15.112:0/1042877075 wait complete. 2023-01-24T03:33:17.949 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 19 2023-01-24T03:33:17.958 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 19, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-01-24T03:30:20.971710+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 19, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4897": { "addr": "172.21.15.142:6835/11022091", "addrs": { "addrvec": [ { "addr": "172.21.15.142:6833", "nonce": 11022091, "type": "v2" }, { "addr": "172.21.15.142:6835", "nonce": 11022091, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4897, "incarnation": 16, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:clientreplay", "state_seq": 17 } }, "last_failure": 0, "last_failure_osd_epoch": 31, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-01-24T03:33:11.627835+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4897 } } } ], "standbys": [ { "addr": "172.21.15.142:6837/2181345869", "addrs": { "addrvec": [ { "addr": "172.21.15.142:6836", "nonce": 2181345869, "type": "v2" }, { "addr": "172.21.15.142:6837", "nonce": 2181345869, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6835/330343927", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6834", "nonce": 330343927, "type": "v2" }, { "addr": "172.21.15.112:6835", "nonce": 330343927, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6837/12176817", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6836", "nonce": 12176817, "type": "v2" }, { "addr": "172.21.15.112:6837", "nonce": 12176817, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4276, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6839/1110983841", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6838", "nonce": 1110983841, "type": "v2" }, { "addr": "172.21.15.112:6839", "nonce": 1110983841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4281, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-01-24T03:33:19.959 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:33:20.084 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.081+0000 7fef5ad3d700 1 Processor -- start 2023-01-24T03:33:20.085 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.081+0000 7fef5ad3d700 1 -- start start 2023-01-24T03:33:20.087 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.082+0000 7fef5ad3d700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fef540c7e40 0x7fef540b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:20.088 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.082+0000 7fef5ad3d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fef54155da0 0x7fef540ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:20.088 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.082+0000 7fef5ad3d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fef540aef20 0x7fef540ca3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:20.088 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.082+0000 7fef5ad3d700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef540af320 con 0x7fef540c7e40 2023-01-24T03:33:20.088 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.082+0000 7fef5ad3d700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef54156240 con 0x7fef54155da0 2023-01-24T03:33:20.089 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.082+0000 7fef5ad3d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef540b1170 con 0x7fef540aef20 2023-01-24T03:33:20.089 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.082+0000 7fef58ad9700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fef540c7e40 0x7fef540b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:20.089 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.082+0000 7fef58ad9700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fef540c7e40 0x7fef540b0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52254/0 (socket says 172.21.15.112:52254) 2023-01-24T03:33:20.089 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.082+0000 7fef58ad9700 1 -- 172.21.15.112:0/2408103106 learned_addr learned my addr 172.21.15.112:0/2408103106 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:33:20.089 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.082+0000 7fef53fff700 1 --2- 172.21.15.112:0/2408103106 >> v2:172.21.15.142:3300/0 conn(0x7fef54155da0 0x7fef540ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:20.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.083+0000 7fef592da700 1 --2- 172.21.15.112:0/2408103106 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fef540aef20 0x7fef540ca3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:20.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.083+0000 7fef58ad9700 1 -- 172.21.15.112:0/2408103106 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fef540aef20 msgr2=0x7fef540ca3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:33:20.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.083+0000 7fef58ad9700 1 --2- 172.21.15.112:0/2408103106 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fef540aef20 0x7fef540ca3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:20.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.083+0000 7fef58ad9700 1 -- 172.21.15.112:0/2408103106 >> v2:172.21.15.142:3300/0 conn(0x7fef54155da0 msgr2=0x7fef540ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:33:20.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.083+0000 7fef58ad9700 1 --2- 172.21.15.112:0/2408103106 >> v2:172.21.15.142:3300/0 conn(0x7fef54155da0 0x7fef540ae9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:20.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.083+0000 7fef58ad9700 1 -- 172.21.15.112:0/2408103106 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef540b1450 con 0x7fef540c7e40 2023-01-24T03:33:20.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.083+0000 7fef58ad9700 1 --2- 172.21.15.112:0/2408103106 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef540c7e40 0x7fef540b0aa0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fef54152ea0 tx=0x7fef440090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a770d656244eafcd server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:33:20.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.083+0000 7fef537fe700 1 -- 172.21.15.112:0/2408103106 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef44011630 con 0x7fef540c7e40 2023-01-24T03:33:20.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.083+0000 7fef537fe700 1 -- 172.21.15.112:0/2408103106 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef44011790 con 0x7fef540c7e40 2023-01-24T03:33:20.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.083+0000 7fef537fe700 1 -- 172.21.15.112:0/2408103106 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef44011b50 con 0x7fef540c7e40 2023-01-24T03:33:20.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.083+0000 7fef5ad3d700 1 -- 172.21.15.112:0/2408103106 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef540c7e40 msgr2=0x7fef540b0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:20.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.083+0000 7fef5ad3d700 1 --2- 172.21.15.112:0/2408103106 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef540c7e40 0x7fef540b0aa0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fef54152ea0 tx=0x7fef440090f0 comp rx=0 tx=0).stop 2023-01-24T03:33:20.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.083+0000 7fef5ad3d700 1 -- 172.21.15.112:0/2408103106 shutdown_connections 2023-01-24T03:33:20.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.083+0000 7fef5ad3d700 1 --2- 172.21.15.112:0/2408103106 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fef540aef20 0x7fef540ca3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:20.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.083+0000 7fef5ad3d700 1 --2- 172.21.15.112:0/2408103106 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef540c7e40 0x7fef540b0aa0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:20.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.083+0000 7fef5ad3d700 1 --2- 172.21.15.112:0/2408103106 >> v2:172.21.15.142:3300/0 conn(0x7fef54155da0 0x7fef540ae9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:20.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.083+0000 7fef5ad3d700 1 -- 172.21.15.112:0/2408103106 >> 172.21.15.112:0/2408103106 conn(0x7fef5405e2c0 msgr2=0x7fef540cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:33:20.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.083+0000 7fef5ad3d700 1 -- 172.21.15.112:0/2408103106 shutdown_connections 2023-01-24T03:33:20.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.084+0000 7fef5ad3d700 1 -- 172.21.15.112:0/2408103106 wait complete. 2023-01-24T03:33:20.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.084+0000 7fef5ad3d700 1 Processor -- start 2023-01-24T03:33:20.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.085+0000 7fef5ad3d700 1 -- start start 2023-01-24T03:33:20.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.085+0000 7fef5ad3d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fef540aef20 0x7fef540c4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:20.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.085+0000 7fef5ad3d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fef540c7e40 0x7fef540cb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:20.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.085+0000 7fef5ad3d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef54155da0 0x7fef541543b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:20.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.085+0000 7fef5ad3d700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef541660c0 con 0x7fef54155da0 2023-01-24T03:33:20.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.085+0000 7fef5ad3d700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef540c52d0 con 0x7fef540c7e40 2023-01-24T03:33:20.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.085+0000 7fef5ad3d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef541651f0 con 0x7fef540aef20 2023-01-24T03:33:20.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.085+0000 7fef592da700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef54155da0 0x7fef541543b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:20.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.085+0000 7fef592da700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef54155da0 0x7fef541543b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52262/0 (socket says 172.21.15.112:52262) 2023-01-24T03:33:20.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.085+0000 7fef592da700 1 -- 172.21.15.112:0/3719306454 learned_addr learned my addr 172.21.15.112:0/3719306454 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:33:20.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.086+0000 7fef58ad9700 1 --2- 172.21.15.112:0/3719306454 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fef540aef20 0x7fef540c4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:20.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.086+0000 7fef53fff700 1 --2- 172.21.15.112:0/3719306454 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fef540c7e40 0x7fef540cb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:20.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.086+0000 7fef592da700 1 -- 172.21.15.112:0/3719306454 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fef540aef20 msgr2=0x7fef540c4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:20.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.086+0000 7fef592da700 1 --2- 172.21.15.112:0/3719306454 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fef540aef20 0x7fef540c4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:20.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.086+0000 7fef592da700 1 -- 172.21.15.112:0/3719306454 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fef540c7e40 msgr2=0x7fef540cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:20.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.086+0000 7fef592da700 1 --2- 172.21.15.112:0/3719306454 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fef540c7e40 0x7fef540cb4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:20.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.086+0000 7fef592da700 1 -- 172.21.15.112:0/3719306454 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef44009370 con 0x7fef54155da0 2023-01-24T03:33:20.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.086+0000 7fef592da700 1 --2- 172.21.15.112:0/3719306454 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef54155da0 0x7fef541543b0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fef4c00cdb0 tx=0x7fef4c0077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:33:20.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.086+0000 7fef50ff9700 1 -- 172.21.15.112:0/3719306454 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef4c00aa60 con 0x7fef54155da0 2023-01-24T03:33:20.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.086+0000 7fef50ff9700 1 -- 172.21.15.112:0/3719306454 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef4c00abc0 con 0x7fef54155da0 2023-01-24T03:33:20.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.086+0000 7fef50ff9700 1 -- 172.21.15.112:0/3719306454 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef4c017430 con 0x7fef54155da0 2023-01-24T03:33:20.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.086+0000 7fef5ad3d700 1 -- 172.21.15.112:0/3719306454 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef541654b0 con 0x7fef54155da0 2023-01-24T03:33:20.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.086+0000 7fef5ad3d700 1 -- 172.21.15.112:0/3719306454 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef54165a70 con 0x7fef54155da0 2023-01-24T03:33:20.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.087+0000 7fef5ad3d700 1 -- 172.21.15.112:0/3719306454 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef541555b0 con 0x7fef54155da0 2023-01-24T03:33:20.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.087+0000 7fef50ff9700 1 -- 172.21.15.112:0/3719306454 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fef4c00f400 con 0x7fef54155da0 2023-01-24T03:33:20.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.088+0000 7fef50ff9700 1 --2- 172.21.15.112:0/3719306454 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fef340611c0 0x7fef34063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:20.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.088+0000 7fef58ad9700 1 --2- 172.21.15.112:0/3719306454 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fef340611c0 0x7fef34063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:20.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.088+0000 7fef50ff9700 1 -- 172.21.15.112:0/3719306454 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7039+0+0 (secure 0 0 0) 0x7fef4c013070 con 0x7fef54155da0 2023-01-24T03:33:20.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.088+0000 7fef58ad9700 1 --2- 172.21.15.112:0/3719306454 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fef340611c0 0x7fef34063670 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fef54152ea0 tx=0x7fef44016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:33:20.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.092+0000 7fef50ff9700 1 -- 172.21.15.112:0/3719306454 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fef4c04dd90 con 0x7fef54155da0 2023-01-24T03:33:20.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.314+0000 7fef5ad3d700 1 -- 172.21.15.112:0/3719306454 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef540c5df0 con 0x7fef54155da0 2023-01-24T03:33:20.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.315+0000 7fef50ff9700 1 -- 172.21.15.112:0/3719306454 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+5355 (secure 0 0 0) 0x7fef4c0513b0 con 0x7fef54155da0 2023-01-24T03:33:20.318 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:33:20.319 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:33:18.651832+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4897},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4897":{"gid":4897,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":19,"addr":"172.21.15.142:6835/11022091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":11022091},{"type":"v1","addr":"172.21.15.142:6835","nonce":11022091}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:33:20.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.317+0000 7fef5ad3d700 1 -- 172.21.15.112:0/3719306454 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fef340611c0 msgr2=0x7fef34063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:20.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.317+0000 7fef5ad3d700 1 --2- 172.21.15.112:0/3719306454 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fef340611c0 0x7fef34063670 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fef54152ea0 tx=0x7fef44016000 comp rx=0 tx=0).stop 2023-01-24T03:33:20.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.317+0000 7fef5ad3d700 1 -- 172.21.15.112:0/3719306454 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef54155da0 msgr2=0x7fef541543b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:20.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.317+0000 7fef5ad3d700 1 --2- 172.21.15.112:0/3719306454 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef54155da0 0x7fef541543b0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fef4c00cdb0 tx=0x7fef4c0077d0 comp rx=0 tx=0).stop 2023-01-24T03:33:20.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.317+0000 7fef5ad3d700 1 -- 172.21.15.112:0/3719306454 shutdown_connections 2023-01-24T03:33:20.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.317+0000 7fef5ad3d700 1 --2- 172.21.15.112:0/3719306454 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fef340611c0 0x7fef34063670 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:20.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.317+0000 7fef5ad3d700 1 --2- 172.21.15.112:0/3719306454 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fef54155da0 0x7fef541543b0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:20.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.317+0000 7fef5ad3d700 1 --2- 172.21.15.112:0/3719306454 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fef540aef20 0x7fef540c4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:20.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.317+0000 7fef5ad3d700 1 --2- 172.21.15.112:0/3719306454 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fef540c7e40 0x7fef540cb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:20.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.317+0000 7fef5ad3d700 1 -- 172.21.15.112:0/3719306454 >> 172.21.15.112:0/3719306454 conn(0x7fef5405e2c0 msgr2=0x7fef54157ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:33:20.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.317+0000 7fef5ad3d700 1 -- 172.21.15.112:0/3719306454 shutdown_connections 2023-01-24T03:33:20.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:20.317+0000 7fef5ad3d700 1 -- 172.21.15.112:0/3719306454 wait complete. 2023-01-24T03:33:20.323 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 20 2023-01-24T03:33:20.331 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=0, replacing gid=4288 2023-01-24T03:33:20.332 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before reviving mds.b 2023-01-24T03:33:23.676 INFO:tasks.workunit.client.0.smithi112.stdout: create 103/33 file after the snapshot 2023-01-24T03:33:23.865 INFO:tasks.workunit.client.0.smithi112.stdout: create 103/34 file after the snapshot 2023-01-24T03:33:23.866 INFO:tasks.workunit.client.0.smithi112.stdout: create 103/35 file after the snapshot 2023-01-24T03:33:23.867 INFO:tasks.workunit.client.0.smithi112.stdout: create 103/36 file after the snapshot 2023-01-24T03:33:23.869 INFO:tasks.workunit.client.0.smithi112.stdout: create 103/37 file after the snapshot 2023-01-24T03:33:23.870 INFO:tasks.workunit.client.0.smithi112.stdout: create 103/38 file after the snapshot 2023-01-24T03:33:23.871 INFO:tasks.workunit.client.0.smithi112.stdout: create 103/39 file after the snapshot 2023-01-24T03:33:23.872 INFO:tasks.workunit.client.0.smithi112.stdout: create 104/.snap/snap-subdir-test 2023-01-24T03:33:24.749 INFO:tasks.ceph.mds.d.smithi142.stderr:2023-01-24T03:33:24.747+0000 7f29e663e700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-01-24T03:33:24.775 INFO:tasks.workunit.client.0.smithi112.stdout: create 104/30 file after the snapshot 2023-01-24T03:33:24.776 INFO:tasks.workunit.client.0.smithi112.stdout: create 104/31 file after the snapshot 2023-01-24T03:33:24.777 INFO:tasks.workunit.client.0.smithi112.stdout: create 104/32 file after the snapshot 2023-01-24T03:33:24.778 INFO:tasks.workunit.client.0.smithi112.stdout: create 104/33 file after the snapshot 2023-01-24T03:33:24.779 INFO:tasks.workunit.client.0.smithi112.stdout: create 104/34 file after the snapshot 2023-01-24T03:33:25.270 INFO:tasks.workunit.client.0.smithi112.stdout: create 104/35 file after the snapshot 2023-01-24T03:33:25.271 INFO:tasks.workunit.client.0.smithi112.stdout: create 104/36 file after the snapshot 2023-01-24T03:33:25.273 INFO:tasks.workunit.client.0.smithi112.stdout: create 104/37 file after the snapshot 2023-01-24T03:33:25.273 INFO:tasks.workunit.client.0.smithi112.stdout: create 104/38 file after the snapshot 2023-01-24T03:33:25.275 INFO:tasks.workunit.client.0.smithi112.stdout: create 104/39 file after the snapshot 2023-01-24T03:33:25.276 INFO:tasks.workunit.client.0.smithi112.stdout: create 105/.snap/snap-subdir-test 2023-01-24T03:33:25.304 INFO:tasks.workunit.client.0.smithi112.stdout: create 105/30 file after the snapshot 2023-01-24T03:33:25.305 INFO:tasks.workunit.client.0.smithi112.stdout: create 105/31 file after the snapshot 2023-01-24T03:33:25.306 INFO:tasks.workunit.client.0.smithi112.stdout: create 105/32 file after the snapshot 2023-01-24T03:33:25.307 INFO:tasks.workunit.client.0.smithi112.stdout: create 105/33 file after the snapshot 2023-01-24T03:33:25.308 INFO:tasks.workunit.client.0.smithi112.stdout: create 105/34 file after the snapshot 2023-01-24T03:33:25.309 INFO:tasks.workunit.client.0.smithi112.stdout: create 105/35 file after the snapshot 2023-01-24T03:33:25.309 INFO:tasks.workunit.client.0.smithi112.stdout: create 105/36 file after the snapshot 2023-01-24T03:33:25.310 INFO:tasks.workunit.client.0.smithi112.stdout: create 105/37 file after the snapshot 2023-01-24T03:33:25.311 INFO:tasks.workunit.client.0.smithi112.stdout: create 105/38 file after the snapshot 2023-01-24T03:33:25.312 INFO:tasks.workunit.client.0.smithi112.stdout: create 105/39 file after the snapshot 2023-01-24T03:33:25.313 INFO:tasks.workunit.client.0.smithi112.stdout: create 106/.snap/snap-subdir-test 2023-01-24T03:33:25.338 INFO:tasks.workunit.client.0.smithi112.stdout: create 106/30 file after the snapshot 2023-01-24T03:33:25.339 INFO:tasks.workunit.client.0.smithi112.stdout: create 106/31 file after the snapshot 2023-01-24T03:33:25.340 INFO:tasks.workunit.client.0.smithi112.stdout: create 106/32 file after the snapshot 2023-01-24T03:33:25.341 INFO:tasks.workunit.client.0.smithi112.stdout: create 106/33 file after the snapshot 2023-01-24T03:33:25.342 INFO:tasks.workunit.client.0.smithi112.stdout: create 106/34 file after the snapshot 2023-01-24T03:33:25.343 INFO:tasks.workunit.client.0.smithi112.stdout: create 106/35 file after the snapshot 2023-01-24T03:33:25.343 INFO:tasks.workunit.client.0.smithi112.stdout: create 106/36 file after the snapshot 2023-01-24T03:33:25.344 INFO:tasks.workunit.client.0.smithi112.stdout: create 106/37 file after the snapshot 2023-01-24T03:33:25.345 INFO:tasks.workunit.client.0.smithi112.stdout: create 106/38 file after the snapshot 2023-01-24T03:33:25.346 INFO:tasks.workunit.client.0.smithi112.stdout: create 106/39 file after the snapshot 2023-01-24T03:33:25.346 INFO:tasks.workunit.client.0.smithi112.stdout: create 107/.snap/snap-subdir-test 2023-01-24T03:33:25.372 INFO:tasks.workunit.client.0.smithi112.stdout: create 107/30 file after the snapshot 2023-01-24T03:33:25.373 INFO:tasks.workunit.client.0.smithi112.stdout: create 107/31 file after the snapshot 2023-01-24T03:33:25.374 INFO:tasks.workunit.client.0.smithi112.stdout: create 107/32 file after the snapshot 2023-01-24T03:33:25.375 INFO:tasks.workunit.client.0.smithi112.stdout: create 107/33 file after the snapshot 2023-01-24T03:33:25.376 INFO:tasks.workunit.client.0.smithi112.stdout: create 107/34 file after the snapshot 2023-01-24T03:33:25.377 INFO:tasks.workunit.client.0.smithi112.stdout: create 107/35 file after the snapshot 2023-01-24T03:33:25.378 INFO:tasks.workunit.client.0.smithi112.stdout: create 107/36 file after the snapshot 2023-01-24T03:33:25.379 INFO:tasks.workunit.client.0.smithi112.stdout: create 107/37 file after the snapshot 2023-01-24T03:33:25.379 INFO:tasks.workunit.client.0.smithi112.stdout: create 107/38 file after the snapshot 2023-01-24T03:33:25.380 INFO:tasks.workunit.client.0.smithi112.stdout: create 107/39 file after the snapshot 2023-01-24T03:33:25.381 INFO:tasks.workunit.client.0.smithi112.stdout: create 108/.snap/snap-subdir-test 2023-01-24T03:33:25.405 INFO:tasks.workunit.client.0.smithi112.stdout: create 108/30 file after the snapshot 2023-01-24T03:33:25.406 INFO:tasks.workunit.client.0.smithi112.stdout: create 108/31 file after the snapshot 2023-01-24T03:33:25.407 INFO:tasks.workunit.client.0.smithi112.stdout: create 108/32 file after the snapshot 2023-01-24T03:33:25.408 INFO:tasks.workunit.client.0.smithi112.stdout: create 108/33 file after the snapshot 2023-01-24T03:33:25.409 INFO:tasks.workunit.client.0.smithi112.stdout: create 108/34 file after the snapshot 2023-01-24T03:33:25.409 INFO:tasks.workunit.client.0.smithi112.stdout: create 108/35 file after the snapshot 2023-01-24T03:33:25.410 INFO:tasks.workunit.client.0.smithi112.stdout: create 108/36 file after the snapshot 2023-01-24T03:33:25.411 INFO:tasks.workunit.client.0.smithi112.stdout: create 108/37 file after the snapshot 2023-01-24T03:33:25.412 INFO:tasks.workunit.client.0.smithi112.stdout: create 108/38 file after the snapshot 2023-01-24T03:33:25.412 INFO:tasks.workunit.client.0.smithi112.stdout: create 108/39 file after the snapshot 2023-01-24T03:33:25.413 INFO:tasks.workunit.client.0.smithi112.stdout: create 109/.snap/snap-subdir-test 2023-01-24T03:33:25.438 INFO:tasks.workunit.client.0.smithi112.stdout: create 109/30 file after the snapshot 2023-01-24T03:33:25.439 INFO:tasks.workunit.client.0.smithi112.stdout: create 109/31 file after the snapshot 2023-01-24T03:33:25.440 INFO:tasks.workunit.client.0.smithi112.stdout: create 109/32 file after the snapshot 2023-01-24T03:33:25.441 INFO:tasks.workunit.client.0.smithi112.stdout: create 109/33 file after the snapshot 2023-01-24T03:33:25.442 INFO:tasks.workunit.client.0.smithi112.stdout: create 109/34 file after the snapshot 2023-01-24T03:33:25.443 INFO:tasks.workunit.client.0.smithi112.stdout: create 109/35 file after the snapshot 2023-01-24T03:33:25.444 INFO:tasks.workunit.client.0.smithi112.stdout: create 109/36 file after the snapshot 2023-01-24T03:33:25.445 INFO:tasks.workunit.client.0.smithi112.stdout: create 109/37 file after the snapshot 2023-01-24T03:33:25.446 INFO:tasks.workunit.client.0.smithi112.stdout: create 109/38 file after the snapshot 2023-01-24T03:33:25.447 INFO:tasks.workunit.client.0.smithi112.stdout: create 109/39 file after the snapshot 2023-01-24T03:33:25.448 INFO:tasks.workunit.client.0.smithi112.stdout: create 110/.snap/snap-subdir-test 2023-01-24T03:33:25.472 INFO:tasks.workunit.client.0.smithi112.stdout: create 110/30 file after the snapshot 2023-01-24T03:33:25.473 INFO:tasks.workunit.client.0.smithi112.stdout: create 110/31 file after the snapshot 2023-01-24T03:33:25.474 INFO:tasks.workunit.client.0.smithi112.stdout: create 110/32 file after the snapshot 2023-01-24T03:33:25.475 INFO:tasks.workunit.client.0.smithi112.stdout: create 110/33 file after the snapshot 2023-01-24T03:33:25.476 INFO:tasks.workunit.client.0.smithi112.stdout: create 110/34 file after the snapshot 2023-01-24T03:33:25.477 INFO:tasks.workunit.client.0.smithi112.stdout: create 110/35 file after the snapshot 2023-01-24T03:33:25.478 INFO:tasks.workunit.client.0.smithi112.stdout: create 110/36 file after the snapshot 2023-01-24T03:33:25.479 INFO:tasks.workunit.client.0.smithi112.stdout: create 110/37 file after the snapshot 2023-01-24T03:33:25.480 INFO:tasks.workunit.client.0.smithi112.stdout: create 110/38 file after the snapshot 2023-01-24T03:33:25.481 INFO:tasks.workunit.client.0.smithi112.stdout: create 110/39 file after the snapshot 2023-01-24T03:33:25.482 INFO:tasks.workunit.client.0.smithi112.stdout: create 111/.snap/snap-subdir-test 2023-01-24T03:33:25.506 INFO:tasks.workunit.client.0.smithi112.stdout: create 111/30 file after the snapshot 2023-01-24T03:33:25.507 INFO:tasks.workunit.client.0.smithi112.stdout: create 111/31 file after the snapshot 2023-01-24T03:33:25.508 INFO:tasks.workunit.client.0.smithi112.stdout: create 111/32 file after the snapshot 2023-01-24T03:33:25.509 INFO:tasks.workunit.client.0.smithi112.stdout: create 111/33 file after the snapshot 2023-01-24T03:33:25.510 INFO:tasks.workunit.client.0.smithi112.stdout: create 111/34 file after the snapshot 2023-01-24T03:33:25.511 INFO:tasks.workunit.client.0.smithi112.stdout: create 111/35 file after the snapshot 2023-01-24T03:33:25.512 INFO:tasks.workunit.client.0.smithi112.stdout: create 111/36 file after the snapshot 2023-01-24T03:33:25.513 INFO:tasks.workunit.client.0.smithi112.stdout: create 111/37 file after the snapshot 2023-01-24T03:33:25.514 INFO:tasks.workunit.client.0.smithi112.stdout: create 111/38 file after the snapshot 2023-01-24T03:33:25.515 INFO:tasks.workunit.client.0.smithi112.stdout: create 111/39 file after the snapshot 2023-01-24T03:33:25.516 INFO:tasks.workunit.client.0.smithi112.stdout: create 112/.snap/snap-subdir-test 2023-01-24T03:33:25.541 INFO:tasks.workunit.client.0.smithi112.stdout: create 112/30 file after the snapshot 2023-01-24T03:33:25.542 INFO:tasks.workunit.client.0.smithi112.stdout: create 112/31 file after the snapshot 2023-01-24T03:33:25.542 INFO:tasks.workunit.client.0.smithi112.stdout: create 112/32 file after the snapshot 2023-01-24T03:33:25.543 INFO:tasks.workunit.client.0.smithi112.stdout: create 112/33 file after the snapshot 2023-01-24T03:33:25.544 INFO:tasks.workunit.client.0.smithi112.stdout: create 112/34 file after the snapshot 2023-01-24T03:33:25.545 INFO:tasks.workunit.client.0.smithi112.stdout: create 112/35 file after the snapshot 2023-01-24T03:33:25.546 INFO:tasks.workunit.client.0.smithi112.stdout: create 112/36 file after the snapshot 2023-01-24T03:33:25.546 INFO:tasks.workunit.client.0.smithi112.stdout: create 112/37 file after the snapshot 2023-01-24T03:33:25.547 INFO:tasks.workunit.client.0.smithi112.stdout: create 112/38 file after the snapshot 2023-01-24T03:33:25.548 INFO:tasks.workunit.client.0.smithi112.stdout: create 112/39 file after the snapshot 2023-01-24T03:33:25.549 INFO:tasks.workunit.client.0.smithi112.stdout: create 113/.snap/snap-subdir-test 2023-01-24T03:33:25.574 INFO:tasks.workunit.client.0.smithi112.stdout: create 113/30 file after the snapshot 2023-01-24T03:33:25.575 INFO:tasks.workunit.client.0.smithi112.stdout: create 113/31 file after the snapshot 2023-01-24T03:33:25.576 INFO:tasks.workunit.client.0.smithi112.stdout: create 113/32 file after the snapshot 2023-01-24T03:33:25.577 INFO:tasks.workunit.client.0.smithi112.stdout: create 113/33 file after the snapshot 2023-01-24T03:33:25.578 INFO:tasks.workunit.client.0.smithi112.stdout: create 113/34 file after the snapshot 2023-01-24T03:33:25.578 INFO:tasks.workunit.client.0.smithi112.stdout: create 113/35 file after the snapshot 2023-01-24T03:33:25.579 INFO:tasks.workunit.client.0.smithi112.stdout: create 113/36 file after the snapshot 2023-01-24T03:33:25.580 INFO:tasks.workunit.client.0.smithi112.stdout: create 113/37 file after the snapshot 2023-01-24T03:33:25.580 INFO:tasks.workunit.client.0.smithi112.stdout: create 113/38 file after the snapshot 2023-01-24T03:33:25.581 INFO:tasks.workunit.client.0.smithi112.stdout: create 113/39 file after the snapshot 2023-01-24T03:33:25.582 INFO:tasks.workunit.client.0.smithi112.stdout: create 114/.snap/snap-subdir-test 2023-01-24T03:33:25.600 INFO:tasks.workunit.client.0.smithi112.stdout: create 114/30 file after the snapshot 2023-01-24T03:33:25.601 INFO:tasks.workunit.client.0.smithi112.stdout: create 114/31 file after the snapshot 2023-01-24T03:33:25.602 INFO:tasks.workunit.client.0.smithi112.stdout: create 114/32 file after the snapshot 2023-01-24T03:33:25.603 INFO:tasks.workunit.client.0.smithi112.stdout: create 114/33 file after the snapshot 2023-01-24T03:33:25.604 INFO:tasks.workunit.client.0.smithi112.stdout: create 114/34 file after the snapshot 2023-01-24T03:33:25.605 INFO:tasks.workunit.client.0.smithi112.stdout: create 114/35 file after the snapshot 2023-01-24T03:33:25.606 INFO:tasks.workunit.client.0.smithi112.stdout: create 114/36 file after the snapshot 2023-01-24T03:33:25.606 INFO:tasks.workunit.client.0.smithi112.stdout: create 114/37 file after the snapshot 2023-01-24T03:33:25.607 INFO:tasks.workunit.client.0.smithi112.stdout: create 114/38 file after the snapshot 2023-01-24T03:33:25.608 INFO:tasks.workunit.client.0.smithi112.stdout: create 114/39 file after the snapshot 2023-01-24T03:33:25.608 INFO:tasks.workunit.client.0.smithi112.stdout: create 115/.snap/snap-subdir-test 2023-01-24T03:33:25.626 INFO:tasks.workunit.client.0.smithi112.stdout: create 115/30 file after the snapshot 2023-01-24T03:33:25.627 INFO:tasks.workunit.client.0.smithi112.stdout: create 115/31 file after the snapshot 2023-01-24T03:33:25.628 INFO:tasks.workunit.client.0.smithi112.stdout: create 115/32 file after the snapshot 2023-01-24T03:33:25.629 INFO:tasks.workunit.client.0.smithi112.stdout: create 115/33 file after the snapshot 2023-01-24T03:33:25.630 INFO:tasks.workunit.client.0.smithi112.stdout: create 115/34 file after the snapshot 2023-01-24T03:33:25.631 INFO:tasks.workunit.client.0.smithi112.stdout: create 115/35 file after the snapshot 2023-01-24T03:33:25.631 INFO:tasks.workunit.client.0.smithi112.stdout: create 115/36 file after the snapshot 2023-01-24T03:33:25.632 INFO:tasks.workunit.client.0.smithi112.stdout: create 115/37 file after the snapshot 2023-01-24T03:33:25.633 INFO:tasks.workunit.client.0.smithi112.stdout: create 115/38 file after the snapshot 2023-01-24T03:33:25.633 INFO:tasks.workunit.client.0.smithi112.stdout: create 115/39 file after the snapshot 2023-01-24T03:33:25.634 INFO:tasks.workunit.client.0.smithi112.stdout: create 116/.snap/snap-subdir-test 2023-01-24T03:33:25.652 INFO:tasks.workunit.client.0.smithi112.stdout: create 116/30 file after the snapshot 2023-01-24T03:33:25.653 INFO:tasks.workunit.client.0.smithi112.stdout: create 116/31 file after the snapshot 2023-01-24T03:33:25.654 INFO:tasks.workunit.client.0.smithi112.stdout: create 116/32 file after the snapshot 2023-01-24T03:33:25.655 INFO:tasks.workunit.client.0.smithi112.stdout: create 116/33 file after the snapshot 2023-01-24T03:33:25.655 INFO:tasks.workunit.client.0.smithi112.stdout: create 116/34 file after the snapshot 2023-01-24T03:33:25.656 INFO:tasks.workunit.client.0.smithi112.stdout: create 116/35 file after the snapshot 2023-01-24T03:33:25.657 INFO:tasks.workunit.client.0.smithi112.stdout: create 116/36 file after the snapshot 2023-01-24T03:33:25.658 INFO:tasks.workunit.client.0.smithi112.stdout: create 116/37 file after the snapshot 2023-01-24T03:33:25.659 INFO:tasks.workunit.client.0.smithi112.stdout: create 116/38 file after the snapshot 2023-01-24T03:33:26.313 INFO:tasks.workunit.client.0.smithi112.stdout: create 116/39 file after the snapshot 2023-01-24T03:33:26.313 INFO:tasks.workunit.client.0.smithi112.stdout: create 117/.snap/snap-subdir-test 2023-01-24T03:33:26.333 INFO:tasks.workunit.client.0.smithi112.stdout: create 117/30 file after the snapshot 2023-01-24T03:33:26.334 INFO:tasks.workunit.client.0.smithi112.stdout: create 117/31 file after the snapshot 2023-01-24T03:33:26.335 INFO:tasks.workunit.client.0.smithi112.stdout: create 117/32 file after the snapshot 2023-01-24T03:33:26.336 INFO:tasks.workunit.client.0.smithi112.stdout: create 117/33 file after the snapshot 2023-01-24T03:33:26.337 INFO:tasks.workunit.client.0.smithi112.stdout: create 117/34 file after the snapshot 2023-01-24T03:33:26.338 INFO:tasks.workunit.client.0.smithi112.stdout: create 117/35 file after the snapshot 2023-01-24T03:33:26.338 INFO:tasks.workunit.client.0.smithi112.stdout: create 117/36 file after the snapshot 2023-01-24T03:33:26.339 INFO:tasks.workunit.client.0.smithi112.stdout: create 117/37 file after the snapshot 2023-01-24T03:33:26.340 INFO:tasks.workunit.client.0.smithi112.stdout: create 117/38 file after the snapshot 2023-01-24T03:33:26.341 INFO:tasks.workunit.client.0.smithi112.stdout: create 117/39 file after the snapshot 2023-01-24T03:33:26.341 INFO:tasks.workunit.client.0.smithi112.stdout: create 118/.snap/snap-subdir-test 2023-01-24T03:33:26.361 INFO:tasks.workunit.client.0.smithi112.stdout: create 118/30 file after the snapshot 2023-01-24T03:33:26.361 INFO:tasks.workunit.client.0.smithi112.stdout: create 118/31 file after the snapshot 2023-01-24T03:33:26.362 INFO:tasks.workunit.client.0.smithi112.stdout: create 118/32 file after the snapshot 2023-01-24T03:33:26.363 INFO:tasks.workunit.client.0.smithi112.stdout: create 118/33 file after the snapshot 2023-01-24T03:33:26.364 INFO:tasks.workunit.client.0.smithi112.stdout: create 118/34 file after the snapshot 2023-01-24T03:33:26.365 INFO:tasks.workunit.client.0.smithi112.stdout: create 118/35 file after the snapshot 2023-01-24T03:33:26.365 INFO:tasks.workunit.client.0.smithi112.stdout: create 118/36 file after the snapshot 2023-01-24T03:33:26.366 INFO:tasks.workunit.client.0.smithi112.stdout: create 118/37 file after the snapshot 2023-01-24T03:33:26.367 INFO:tasks.workunit.client.0.smithi112.stdout: create 118/38 file after the snapshot 2023-01-24T03:33:26.367 INFO:tasks.workunit.client.0.smithi112.stdout: create 118/39 file after the snapshot 2023-01-24T03:33:26.368 INFO:tasks.workunit.client.0.smithi112.stdout: create 119/.snap/snap-subdir-test 2023-01-24T03:33:26.387 INFO:tasks.workunit.client.0.smithi112.stdout: create 119/30 file after the snapshot 2023-01-24T03:33:26.388 INFO:tasks.workunit.client.0.smithi112.stdout: create 119/31 file after the snapshot 2023-01-24T03:33:26.388 INFO:tasks.workunit.client.0.smithi112.stdout: create 119/32 file after the snapshot 2023-01-24T03:33:26.389 INFO:tasks.workunit.client.0.smithi112.stdout: create 119/33 file after the snapshot 2023-01-24T03:33:26.390 INFO:tasks.workunit.client.0.smithi112.stdout: create 119/34 file after the snapshot 2023-01-24T03:33:26.391 INFO:tasks.workunit.client.0.smithi112.stdout: create 119/35 file after the snapshot 2023-01-24T03:33:26.392 INFO:tasks.workunit.client.0.smithi112.stdout: create 119/36 file after the snapshot 2023-01-24T03:33:26.393 INFO:tasks.workunit.client.0.smithi112.stdout: create 119/37 file after the snapshot 2023-01-24T03:33:26.394 INFO:tasks.workunit.client.0.smithi112.stdout: create 119/38 file after the snapshot 2023-01-24T03:33:26.395 INFO:tasks.workunit.client.0.smithi112.stdout: create 119/39 file after the snapshot 2023-01-24T03:33:26.396 INFO:tasks.workunit.client.0.smithi112.stdout: create 120/.snap/snap-subdir-test 2023-01-24T03:33:26.414 INFO:tasks.workunit.client.0.smithi112.stdout: create 120/30 file after the snapshot 2023-01-24T03:33:26.416 INFO:tasks.workunit.client.0.smithi112.stdout: create 120/31 file after the snapshot 2023-01-24T03:33:26.416 INFO:tasks.workunit.client.0.smithi112.stdout: create 120/32 file after the snapshot 2023-01-24T03:33:26.417 INFO:tasks.workunit.client.0.smithi112.stdout: create 120/33 file after the snapshot 2023-01-24T03:33:26.418 INFO:tasks.workunit.client.0.smithi112.stdout: create 120/34 file after the snapshot 2023-01-24T03:33:26.419 INFO:tasks.workunit.client.0.smithi112.stdout: create 120/35 file after the snapshot 2023-01-24T03:33:26.420 INFO:tasks.workunit.client.0.smithi112.stdout: create 120/36 file after the snapshot 2023-01-24T03:33:26.420 INFO:tasks.workunit.client.0.smithi112.stdout: create 120/37 file after the snapshot 2023-01-24T03:33:26.421 INFO:tasks.workunit.client.0.smithi112.stdout: create 120/38 file after the snapshot 2023-01-24T03:33:26.422 INFO:tasks.workunit.client.0.smithi112.stdout: create 120/39 file after the snapshot 2023-01-24T03:33:26.423 INFO:tasks.workunit.client.0.smithi112.stdout: create 121/.snap/snap-subdir-test 2023-01-24T03:33:26.441 INFO:tasks.workunit.client.0.smithi112.stdout: create 121/30 file after the snapshot 2023-01-24T03:33:26.442 INFO:tasks.workunit.client.0.smithi112.stdout: create 121/31 file after the snapshot 2023-01-24T03:33:26.443 INFO:tasks.workunit.client.0.smithi112.stdout: create 121/32 file after the snapshot 2023-01-24T03:33:26.444 INFO:tasks.workunit.client.0.smithi112.stdout: create 121/33 file after the snapshot 2023-01-24T03:33:26.445 INFO:tasks.workunit.client.0.smithi112.stdout: create 121/34 file after the snapshot 2023-01-24T03:33:26.446 INFO:tasks.workunit.client.0.smithi112.stdout: create 121/35 file after the snapshot 2023-01-24T03:33:26.447 INFO:tasks.workunit.client.0.smithi112.stdout: create 121/36 file after the snapshot 2023-01-24T03:33:26.448 INFO:tasks.workunit.client.0.smithi112.stdout: create 121/37 file after the snapshot 2023-01-24T03:33:26.449 INFO:tasks.workunit.client.0.smithi112.stdout: create 121/38 file after the snapshot 2023-01-24T03:33:26.450 INFO:tasks.workunit.client.0.smithi112.stdout: create 121/39 file after the snapshot 2023-01-24T03:33:26.450 INFO:tasks.workunit.client.0.smithi112.stdout: create 122/.snap/snap-subdir-test 2023-01-24T03:33:26.468 INFO:tasks.workunit.client.0.smithi112.stdout: create 122/30 file after the snapshot 2023-01-24T03:33:26.469 INFO:tasks.workunit.client.0.smithi112.stdout: create 122/31 file after the snapshot 2023-01-24T03:33:26.470 INFO:tasks.workunit.client.0.smithi112.stdout: create 122/32 file after the snapshot 2023-01-24T03:33:26.471 INFO:tasks.workunit.client.0.smithi112.stdout: create 122/33 file after the snapshot 2023-01-24T03:33:26.472 INFO:tasks.workunit.client.0.smithi112.stdout: create 122/34 file after the snapshot 2023-01-24T03:33:26.473 INFO:tasks.workunit.client.0.smithi112.stdout: create 122/35 file after the snapshot 2023-01-24T03:33:26.474 INFO:tasks.workunit.client.0.smithi112.stdout: create 122/36 file after the snapshot 2023-01-24T03:33:26.475 INFO:tasks.workunit.client.0.smithi112.stdout: create 122/37 file after the snapshot 2023-01-24T03:33:26.476 INFO:tasks.workunit.client.0.smithi112.stdout: create 122/38 file after the snapshot 2023-01-24T03:33:26.476 INFO:tasks.workunit.client.0.smithi112.stdout: create 122/39 file after the snapshot 2023-01-24T03:33:26.477 INFO:tasks.workunit.client.0.smithi112.stdout: create 123/.snap/snap-subdir-test 2023-01-24T03:33:26.496 INFO:tasks.workunit.client.0.smithi112.stdout: create 123/30 file after the snapshot 2023-01-24T03:33:26.497 INFO:tasks.workunit.client.0.smithi112.stdout: create 123/31 file after the snapshot 2023-01-24T03:33:26.498 INFO:tasks.workunit.client.0.smithi112.stdout: create 123/32 file after the snapshot 2023-01-24T03:33:26.499 INFO:tasks.workunit.client.0.smithi112.stdout: create 123/33 file after the snapshot 2023-01-24T03:33:26.500 INFO:tasks.workunit.client.0.smithi112.stdout: create 123/34 file after the snapshot 2023-01-24T03:33:26.501 INFO:tasks.workunit.client.0.smithi112.stdout: create 123/35 file after the snapshot 2023-01-24T03:33:26.501 INFO:tasks.workunit.client.0.smithi112.stdout: create 123/36 file after the snapshot 2023-01-24T03:33:26.502 INFO:tasks.workunit.client.0.smithi112.stdout: create 123/37 file after the snapshot 2023-01-24T03:33:26.503 INFO:tasks.workunit.client.0.smithi112.stdout: create 123/38 file after the snapshot 2023-01-24T03:33:26.504 INFO:tasks.workunit.client.0.smithi112.stdout: create 123/39 file after the snapshot 2023-01-24T03:33:26.505 INFO:tasks.workunit.client.0.smithi112.stdout: create 124/.snap/snap-subdir-test 2023-01-24T03:33:26.523 INFO:tasks.workunit.client.0.smithi112.stdout: create 124/30 file after the snapshot 2023-01-24T03:33:26.525 INFO:tasks.workunit.client.0.smithi112.stdout: create 124/31 file after the snapshot 2023-01-24T03:33:26.525 INFO:tasks.workunit.client.0.smithi112.stdout: create 124/32 file after the snapshot 2023-01-24T03:33:26.526 INFO:tasks.workunit.client.0.smithi112.stdout: create 124/33 file after the snapshot 2023-01-24T03:33:26.527 INFO:tasks.workunit.client.0.smithi112.stdout: create 124/34 file after the snapshot 2023-01-24T03:33:26.528 INFO:tasks.workunit.client.0.smithi112.stdout: create 124/35 file after the snapshot 2023-01-24T03:33:26.529 INFO:tasks.workunit.client.0.smithi112.stdout: create 124/36 file after the snapshot 2023-01-24T03:33:26.529 INFO:tasks.workunit.client.0.smithi112.stdout: create 124/37 file after the snapshot 2023-01-24T03:33:26.530 INFO:tasks.workunit.client.0.smithi112.stdout: create 124/38 file after the snapshot 2023-01-24T03:33:26.531 INFO:tasks.workunit.client.0.smithi112.stdout: create 124/39 file after the snapshot 2023-01-24T03:33:26.531 INFO:tasks.workunit.client.0.smithi112.stdout: create 125/.snap/snap-subdir-test 2023-01-24T03:33:27.366 INFO:tasks.workunit.client.0.smithi112.stdout: create 125/30 file after the snapshot 2023-01-24T03:33:27.450 INFO:tasks.workunit.client.0.smithi112.stdout: create 125/31 file after the snapshot 2023-01-24T03:33:27.451 INFO:tasks.workunit.client.0.smithi112.stdout: create 125/32 file after the snapshot 2023-01-24T03:33:27.452 INFO:tasks.workunit.client.0.smithi112.stdout: create 125/33 file after the snapshot 2023-01-24T03:33:27.453 INFO:tasks.workunit.client.0.smithi112.stdout: create 125/34 file after the snapshot 2023-01-24T03:33:27.454 INFO:tasks.workunit.client.0.smithi112.stdout: create 125/35 file after the snapshot 2023-01-24T03:33:27.455 INFO:tasks.workunit.client.0.smithi112.stdout: create 125/36 file after the snapshot 2023-01-24T03:33:27.456 INFO:tasks.workunit.client.0.smithi112.stdout: create 125/37 file after the snapshot 2023-01-24T03:33:27.457 INFO:tasks.workunit.client.0.smithi112.stdout: create 125/38 file after the snapshot 2023-01-24T03:33:27.458 INFO:tasks.workunit.client.0.smithi112.stdout: create 125/39 file after the snapshot 2023-01-24T03:33:27.459 INFO:tasks.workunit.client.0.smithi112.stdout: create 126/.snap/snap-subdir-test 2023-01-24T03:33:27.480 INFO:tasks.workunit.client.0.smithi112.stdout: create 126/30 file after the snapshot 2023-01-24T03:33:27.481 INFO:tasks.workunit.client.0.smithi112.stdout: create 126/31 file after the snapshot 2023-01-24T03:33:27.482 INFO:tasks.workunit.client.0.smithi112.stdout: create 126/32 file after the snapshot 2023-01-24T03:33:27.483 INFO:tasks.workunit.client.0.smithi112.stdout: create 126/33 file after the snapshot 2023-01-24T03:33:27.484 INFO:tasks.workunit.client.0.smithi112.stdout: create 126/34 file after the snapshot 2023-01-24T03:33:27.485 INFO:tasks.workunit.client.0.smithi112.stdout: create 126/35 file after the snapshot 2023-01-24T03:33:27.486 INFO:tasks.workunit.client.0.smithi112.stdout: create 126/36 file after the snapshot 2023-01-24T03:33:27.487 INFO:tasks.workunit.client.0.smithi112.stdout: create 126/37 file after the snapshot 2023-01-24T03:33:27.487 INFO:tasks.workunit.client.0.smithi112.stdout: create 126/38 file after the snapshot 2023-01-24T03:33:27.488 INFO:tasks.workunit.client.0.smithi112.stdout: create 126/39 file after the snapshot 2023-01-24T03:33:27.489 INFO:tasks.workunit.client.0.smithi112.stdout: create 127/.snap/snap-subdir-test 2023-01-24T03:33:27.509 INFO:tasks.workunit.client.0.smithi112.stdout: create 127/30 file after the snapshot 2023-01-24T03:33:27.511 INFO:tasks.workunit.client.0.smithi112.stdout: create 127/31 file after the snapshot 2023-01-24T03:33:27.512 INFO:tasks.workunit.client.0.smithi112.stdout: create 127/32 file after the snapshot 2023-01-24T03:33:27.513 INFO:tasks.workunit.client.0.smithi112.stdout: create 127/33 file after the snapshot 2023-01-24T03:33:27.514 INFO:tasks.workunit.client.0.smithi112.stdout: create 127/34 file after the snapshot 2023-01-24T03:33:27.515 INFO:tasks.workunit.client.0.smithi112.stdout: create 127/35 file after the snapshot 2023-01-24T03:33:27.516 INFO:tasks.workunit.client.0.smithi112.stdout: create 127/36 file after the snapshot 2023-01-24T03:33:27.517 INFO:tasks.workunit.client.0.smithi112.stdout: create 127/37 file after the snapshot 2023-01-24T03:33:27.518 INFO:tasks.workunit.client.0.smithi112.stdout: create 127/38 file after the snapshot 2023-01-24T03:33:27.519 INFO:tasks.workunit.client.0.smithi112.stdout: create 127/39 file after the snapshot 2023-01-24T03:33:27.520 INFO:tasks.workunit.client.0.smithi112.stdout: create 128/.snap/snap-subdir-test 2023-01-24T03:33:27.540 INFO:tasks.workunit.client.0.smithi112.stdout: create 128/30 file after the snapshot 2023-01-24T03:33:27.541 INFO:tasks.workunit.client.0.smithi112.stdout: create 128/31 file after the snapshot 2023-01-24T03:33:27.542 INFO:tasks.workunit.client.0.smithi112.stdout: create 128/32 file after the snapshot 2023-01-24T03:33:27.543 INFO:tasks.workunit.client.0.smithi112.stdout: create 128/33 file after the snapshot 2023-01-24T03:33:27.544 INFO:tasks.workunit.client.0.smithi112.stdout: create 128/34 file after the snapshot 2023-01-24T03:33:27.546 INFO:tasks.workunit.client.0.smithi112.stdout: create 128/35 file after the snapshot 2023-01-24T03:33:27.547 INFO:tasks.workunit.client.0.smithi112.stdout: create 128/36 file after the snapshot 2023-01-24T03:33:27.547 INFO:tasks.workunit.client.0.smithi112.stdout: create 128/37 file after the snapshot 2023-01-24T03:33:27.548 INFO:tasks.workunit.client.0.smithi112.stdout: create 128/38 file after the snapshot 2023-01-24T03:33:27.549 INFO:tasks.workunit.client.0.smithi112.stdout: create 128/39 file after the snapshot 2023-01-24T03:33:27.550 INFO:tasks.workunit.client.0.smithi112.stdout: create 129/.snap/snap-subdir-test 2023-01-24T03:33:27.570 INFO:tasks.workunit.client.0.smithi112.stdout: create 129/30 file after the snapshot 2023-01-24T03:33:27.571 INFO:tasks.workunit.client.0.smithi112.stdout: create 129/31 file after the snapshot 2023-01-24T03:33:27.571 INFO:tasks.workunit.client.0.smithi112.stdout: create 129/32 file after the snapshot 2023-01-24T03:33:27.572 INFO:tasks.workunit.client.0.smithi112.stdout: create 129/33 file after the snapshot 2023-01-24T03:33:27.573 INFO:tasks.workunit.client.0.smithi112.stdout: create 129/34 file after the snapshot 2023-01-24T03:33:27.574 INFO:tasks.workunit.client.0.smithi112.stdout: create 129/35 file after the snapshot 2023-01-24T03:33:27.575 INFO:tasks.workunit.client.0.smithi112.stdout: create 129/36 file after the snapshot 2023-01-24T03:33:27.575 INFO:tasks.workunit.client.0.smithi112.stdout: create 129/37 file after the snapshot 2023-01-24T03:33:27.576 INFO:tasks.workunit.client.0.smithi112.stdout: create 129/38 file after the snapshot 2023-01-24T03:33:27.577 INFO:tasks.workunit.client.0.smithi112.stdout: create 129/39 file after the snapshot 2023-01-24T03:33:27.578 INFO:tasks.workunit.client.0.smithi112.stdout: create 130/.snap/snap-subdir-test 2023-01-24T03:33:27.599 INFO:tasks.workunit.client.0.smithi112.stdout: create 130/30 file after the snapshot 2023-01-24T03:33:27.599 INFO:tasks.workunit.client.0.smithi112.stdout: create 130/31 file after the snapshot 2023-01-24T03:33:27.600 INFO:tasks.workunit.client.0.smithi112.stdout: create 130/32 file after the snapshot 2023-01-24T03:33:27.601 INFO:tasks.workunit.client.0.smithi112.stdout: create 130/33 file after the snapshot 2023-01-24T03:33:27.602 INFO:tasks.workunit.client.0.smithi112.stdout: create 130/34 file after the snapshot 2023-01-24T03:33:27.603 INFO:tasks.workunit.client.0.smithi112.stdout: create 130/35 file after the snapshot 2023-01-24T03:33:27.604 INFO:tasks.workunit.client.0.smithi112.stdout: create 130/36 file after the snapshot 2023-01-24T03:33:27.605 INFO:tasks.workunit.client.0.smithi112.stdout: create 130/37 file after the snapshot 2023-01-24T03:33:27.605 INFO:tasks.workunit.client.0.smithi112.stdout: create 130/38 file after the snapshot 2023-01-24T03:33:27.606 INFO:tasks.workunit.client.0.smithi112.stdout: create 130/39 file after the snapshot 2023-01-24T03:33:27.607 INFO:tasks.workunit.client.0.smithi112.stdout: create 131/.snap/snap-subdir-test 2023-01-24T03:33:27.626 INFO:tasks.workunit.client.0.smithi112.stdout: create 131/30 file after the snapshot 2023-01-24T03:33:27.628 INFO:tasks.workunit.client.0.smithi112.stdout: create 131/31 file after the snapshot 2023-01-24T03:33:27.629 INFO:tasks.workunit.client.0.smithi112.stdout: create 131/32 file after the snapshot 2023-01-24T03:33:27.629 INFO:tasks.workunit.client.0.smithi112.stdout: create 131/33 file after the snapshot 2023-01-24T03:33:27.630 INFO:tasks.workunit.client.0.smithi112.stdout: create 131/34 file after the snapshot 2023-01-24T03:33:27.631 INFO:tasks.workunit.client.0.smithi112.stdout: create 131/35 file after the snapshot 2023-01-24T03:33:27.632 INFO:tasks.workunit.client.0.smithi112.stdout: create 131/36 file after the snapshot 2023-01-24T03:33:27.632 INFO:tasks.workunit.client.0.smithi112.stdout: create 131/37 file after the snapshot 2023-01-24T03:33:27.633 INFO:tasks.workunit.client.0.smithi112.stdout: create 131/38 file after the snapshot 2023-01-24T03:33:27.634 INFO:tasks.workunit.client.0.smithi112.stdout: create 131/39 file after the snapshot 2023-01-24T03:33:27.635 INFO:tasks.workunit.client.0.smithi112.stdout: create 132/.snap/snap-subdir-test 2023-01-24T03:33:27.654 INFO:tasks.workunit.client.0.smithi112.stdout: create 132/30 file after the snapshot 2023-01-24T03:33:27.916 INFO:tasks.workunit.client.0.smithi112.stdout: create 132/31 file after the snapshot 2023-01-24T03:33:27.916 INFO:tasks.workunit.client.0.smithi112.stdout: create 132/32 file after the snapshot 2023-01-24T03:33:27.917 INFO:tasks.workunit.client.0.smithi112.stdout: create 132/33 file after the snapshot 2023-01-24T03:33:27.918 INFO:tasks.workunit.client.0.smithi112.stdout: create 132/34 file after the snapshot 2023-01-24T03:33:27.919 INFO:tasks.workunit.client.0.smithi112.stdout: create 132/35 file after the snapshot 2023-01-24T03:33:27.920 INFO:tasks.workunit.client.0.smithi112.stdout: create 132/36 file after the snapshot 2023-01-24T03:33:27.921 INFO:tasks.workunit.client.0.smithi112.stdout: create 132/37 file after the snapshot 2023-01-24T03:33:27.922 INFO:tasks.workunit.client.0.smithi112.stdout: create 132/38 file after the snapshot 2023-01-24T03:33:27.923 INFO:tasks.workunit.client.0.smithi112.stdout: create 132/39 file after the snapshot 2023-01-24T03:33:27.924 INFO:tasks.workunit.client.0.smithi112.stdout: create 133/.snap/snap-subdir-test 2023-01-24T03:33:27.948 INFO:tasks.workunit.client.0.smithi112.stdout: create 133/30 file after the snapshot 2023-01-24T03:33:27.949 INFO:tasks.workunit.client.0.smithi112.stdout: create 133/31 file after the snapshot 2023-01-24T03:33:27.950 INFO:tasks.workunit.client.0.smithi112.stdout: create 133/32 file after the snapshot 2023-01-24T03:33:27.951 INFO:tasks.workunit.client.0.smithi112.stdout: create 133/33 file after the snapshot 2023-01-24T03:33:27.952 INFO:tasks.workunit.client.0.smithi112.stdout: create 133/34 file after the snapshot 2023-01-24T03:33:27.953 INFO:tasks.workunit.client.0.smithi112.stdout: create 133/35 file after the snapshot 2023-01-24T03:33:27.954 INFO:tasks.workunit.client.0.smithi112.stdout: create 133/36 file after the snapshot 2023-01-24T03:33:27.955 INFO:tasks.workunit.client.0.smithi112.stdout: create 133/37 file after the snapshot 2023-01-24T03:33:27.955 INFO:tasks.workunit.client.0.smithi112.stdout: create 133/38 file after the snapshot 2023-01-24T03:33:27.956 INFO:tasks.workunit.client.0.smithi112.stdout: create 133/39 file after the snapshot 2023-01-24T03:33:27.957 INFO:tasks.workunit.client.0.smithi112.stdout: create 134/.snap/snap-subdir-test 2023-01-24T03:33:27.979 INFO:tasks.workunit.client.0.smithi112.stdout: create 134/30 file after the snapshot 2023-01-24T03:33:27.980 INFO:tasks.workunit.client.0.smithi112.stdout: create 134/31 file after the snapshot 2023-01-24T03:33:27.981 INFO:tasks.workunit.client.0.smithi112.stdout: create 134/32 file after the snapshot 2023-01-24T03:33:27.982 INFO:tasks.workunit.client.0.smithi112.stdout: create 134/33 file after the snapshot 2023-01-24T03:33:27.983 INFO:tasks.workunit.client.0.smithi112.stdout: create 134/34 file after the snapshot 2023-01-24T03:33:27.984 INFO:tasks.workunit.client.0.smithi112.stdout: create 134/35 file after the snapshot 2023-01-24T03:33:27.985 INFO:tasks.workunit.client.0.smithi112.stdout: create 134/36 file after the snapshot 2023-01-24T03:33:27.986 INFO:tasks.workunit.client.0.smithi112.stdout: create 134/37 file after the snapshot 2023-01-24T03:33:27.987 INFO:tasks.workunit.client.0.smithi112.stdout: create 134/38 file after the snapshot 2023-01-24T03:33:27.988 INFO:tasks.workunit.client.0.smithi112.stdout: create 134/39 file after the snapshot 2023-01-24T03:33:27.989 INFO:tasks.workunit.client.0.smithi112.stdout: create 135/.snap/snap-subdir-test 2023-01-24T03:33:28.010 INFO:tasks.workunit.client.0.smithi112.stdout: create 135/30 file after the snapshot 2023-01-24T03:33:28.011 INFO:tasks.workunit.client.0.smithi112.stdout: create 135/31 file after the snapshot 2023-01-24T03:33:28.012 INFO:tasks.workunit.client.0.smithi112.stdout: create 135/32 file after the snapshot 2023-01-24T03:33:28.013 INFO:tasks.workunit.client.0.smithi112.stdout: create 135/33 file after the snapshot 2023-01-24T03:33:28.014 INFO:tasks.workunit.client.0.smithi112.stdout: create 135/34 file after the snapshot 2023-01-24T03:33:28.015 INFO:tasks.workunit.client.0.smithi112.stdout: create 135/35 file after the snapshot 2023-01-24T03:33:28.016 INFO:tasks.workunit.client.0.smithi112.stdout: create 135/36 file after the snapshot 2023-01-24T03:33:28.017 INFO:tasks.workunit.client.0.smithi112.stdout: create 135/37 file after the snapshot 2023-01-24T03:33:28.018 INFO:tasks.workunit.client.0.smithi112.stdout: create 135/38 file after the snapshot 2023-01-24T03:33:28.018 INFO:tasks.workunit.client.0.smithi112.stdout: create 135/39 file after the snapshot 2023-01-24T03:33:28.019 INFO:tasks.workunit.client.0.smithi112.stdout: create 136/.snap/snap-subdir-test 2023-01-24T03:33:28.039 INFO:tasks.workunit.client.0.smithi112.stdout: create 136/30 file after the snapshot 2023-01-24T03:33:28.040 INFO:tasks.workunit.client.0.smithi112.stdout: create 136/31 file after the snapshot 2023-01-24T03:33:28.041 INFO:tasks.workunit.client.0.smithi112.stdout: create 136/32 file after the snapshot 2023-01-24T03:33:28.042 INFO:tasks.workunit.client.0.smithi112.stdout: create 136/33 file after the snapshot 2023-01-24T03:33:28.043 INFO:tasks.workunit.client.0.smithi112.stdout: create 136/34 file after the snapshot 2023-01-24T03:33:28.043 INFO:tasks.workunit.client.0.smithi112.stdout: create 136/35 file after the snapshot 2023-01-24T03:33:28.044 INFO:tasks.workunit.client.0.smithi112.stdout: create 136/36 file after the snapshot 2023-01-24T03:33:28.045 INFO:tasks.workunit.client.0.smithi112.stdout: create 136/37 file after the snapshot 2023-01-24T03:33:28.046 INFO:tasks.workunit.client.0.smithi112.stdout: create 136/38 file after the snapshot 2023-01-24T03:33:28.047 INFO:tasks.workunit.client.0.smithi112.stdout: create 136/39 file after the snapshot 2023-01-24T03:33:28.048 INFO:tasks.workunit.client.0.smithi112.stdout: create 137/.snap/snap-subdir-test 2023-01-24T03:33:28.068 INFO:tasks.workunit.client.0.smithi112.stdout: create 137/30 file after the snapshot 2023-01-24T03:33:28.069 INFO:tasks.workunit.client.0.smithi112.stdout: create 137/31 file after the snapshot 2023-01-24T03:33:28.070 INFO:tasks.workunit.client.0.smithi112.stdout: create 137/32 file after the snapshot 2023-01-24T03:33:28.071 INFO:tasks.workunit.client.0.smithi112.stdout: create 137/33 file after the snapshot 2023-01-24T03:33:28.072 INFO:tasks.workunit.client.0.smithi112.stdout: create 137/34 file after the snapshot 2023-01-24T03:33:28.072 INFO:tasks.workunit.client.0.smithi112.stdout: create 137/35 file after the snapshot 2023-01-24T03:33:28.073 INFO:tasks.workunit.client.0.smithi112.stdout: create 137/36 file after the snapshot 2023-01-24T03:33:28.074 INFO:tasks.workunit.client.0.smithi112.stdout: create 137/37 file after the snapshot 2023-01-24T03:33:28.074 INFO:tasks.workunit.client.0.smithi112.stdout: create 137/38 file after the snapshot 2023-01-24T03:33:28.075 INFO:tasks.workunit.client.0.smithi112.stdout: create 137/39 file after the snapshot 2023-01-24T03:33:28.076 INFO:tasks.workunit.client.0.smithi112.stdout: create 138/.snap/snap-subdir-test 2023-01-24T03:33:28.096 INFO:tasks.workunit.client.0.smithi112.stdout: create 138/30 file after the snapshot 2023-01-24T03:33:28.097 INFO:tasks.workunit.client.0.smithi112.stdout: create 138/31 file after the snapshot 2023-01-24T03:33:28.098 INFO:tasks.workunit.client.0.smithi112.stdout: create 138/32 file after the snapshot 2023-01-24T03:33:28.099 INFO:tasks.workunit.client.0.smithi112.stdout: create 138/33 file after the snapshot 2023-01-24T03:33:28.100 INFO:tasks.workunit.client.0.smithi112.stdout: create 138/34 file after the snapshot 2023-01-24T03:33:28.101 INFO:tasks.workunit.client.0.smithi112.stdout: create 138/35 file after the snapshot 2023-01-24T03:33:28.102 INFO:tasks.workunit.client.0.smithi112.stdout: create 138/36 file after the snapshot 2023-01-24T03:33:28.103 INFO:tasks.workunit.client.0.smithi112.stdout: create 138/37 file after the snapshot 2023-01-24T03:33:28.103 INFO:tasks.workunit.client.0.smithi112.stdout: create 138/38 file after the snapshot 2023-01-24T03:33:28.104 INFO:tasks.workunit.client.0.smithi112.stdout: create 138/39 file after the snapshot 2023-01-24T03:33:28.105 INFO:tasks.workunit.client.0.smithi112.stdout: create 139/.snap/snap-subdir-test 2023-01-24T03:33:28.124 INFO:tasks.workunit.client.0.smithi112.stdout: create 139/30 file after the snapshot 2023-01-24T03:33:28.125 INFO:tasks.workunit.client.0.smithi112.stdout: create 139/31 file after the snapshot 2023-01-24T03:33:28.126 INFO:tasks.workunit.client.0.smithi112.stdout: create 139/32 file after the snapshot 2023-01-24T03:33:28.127 INFO:tasks.workunit.client.0.smithi112.stdout: create 139/33 file after the snapshot 2023-01-24T03:33:28.127 INFO:tasks.workunit.client.0.smithi112.stdout: create 139/34 file after the snapshot 2023-01-24T03:33:28.129 INFO:tasks.workunit.client.0.smithi112.stdout: create 139/35 file after the snapshot 2023-01-24T03:33:28.129 INFO:tasks.workunit.client.0.smithi112.stdout: create 139/36 file after the snapshot 2023-01-24T03:33:28.130 INFO:tasks.workunit.client.0.smithi112.stdout: create 139/37 file after the snapshot 2023-01-24T03:33:28.131 INFO:tasks.workunit.client.0.smithi112.stdout: create 139/38 file after the snapshot 2023-01-24T03:33:28.131 INFO:tasks.workunit.client.0.smithi112.stdout: create 139/39 file after the snapshot 2023-01-24T03:33:28.132 INFO:tasks.workunit.client.0.smithi112.stdout: create 140/.snap/snap-subdir-test 2023-01-24T03:33:28.150 INFO:tasks.workunit.client.0.smithi112.stdout: create 140/30 file after the snapshot 2023-01-24T03:33:28.151 INFO:tasks.workunit.client.0.smithi112.stdout: create 140/31 file after the snapshot 2023-01-24T03:33:28.152 INFO:tasks.workunit.client.0.smithi112.stdout: create 140/32 file after the snapshot 2023-01-24T03:33:28.153 INFO:tasks.workunit.client.0.smithi112.stdout: create 140/33 file after the snapshot 2023-01-24T03:33:28.154 INFO:tasks.workunit.client.0.smithi112.stdout: create 140/34 file after the snapshot 2023-01-24T03:33:28.154 INFO:tasks.workunit.client.0.smithi112.stdout: create 140/35 file after the snapshot 2023-01-24T03:33:28.155 INFO:tasks.workunit.client.0.smithi112.stdout: create 140/36 file after the snapshot 2023-01-24T03:33:28.156 INFO:tasks.workunit.client.0.smithi112.stdout: create 140/37 file after the snapshot 2023-01-24T03:33:28.157 INFO:tasks.workunit.client.0.smithi112.stdout: create 140/38 file after the snapshot 2023-01-24T03:33:28.157 INFO:tasks.workunit.client.0.smithi112.stdout: create 140/39 file after the snapshot 2023-01-24T03:33:28.158 INFO:tasks.workunit.client.0.smithi112.stdout: create 141/.snap/snap-subdir-test 2023-01-24T03:33:28.178 INFO:tasks.workunit.client.0.smithi112.stdout: create 141/30 file after the snapshot 2023-01-24T03:33:28.179 INFO:tasks.workunit.client.0.smithi112.stdout: create 141/31 file after the snapshot 2023-01-24T03:33:28.180 INFO:tasks.workunit.client.0.smithi112.stdout: create 141/32 file after the snapshot 2023-01-24T03:33:28.181 INFO:tasks.workunit.client.0.smithi112.stdout: create 141/33 file after the snapshot 2023-01-24T03:33:28.182 INFO:tasks.workunit.client.0.smithi112.stdout: create 141/34 file after the snapshot 2023-01-24T03:33:28.183 INFO:tasks.workunit.client.0.smithi112.stdout: create 141/35 file after the snapshot 2023-01-24T03:33:28.184 INFO:tasks.workunit.client.0.smithi112.stdout: create 141/36 file after the snapshot 2023-01-24T03:33:28.184 INFO:tasks.workunit.client.0.smithi112.stdout: create 141/37 file after the snapshot 2023-01-24T03:33:28.185 INFO:tasks.workunit.client.0.smithi112.stdout: create 141/38 file after the snapshot 2023-01-24T03:33:28.186 INFO:tasks.workunit.client.0.smithi112.stdout: create 141/39 file after the snapshot 2023-01-24T03:33:28.187 INFO:tasks.workunit.client.0.smithi112.stdout: create 142/.snap/snap-subdir-test 2023-01-24T03:33:28.289 INFO:tasks.workunit.client.0.smithi112.stdout: create 142/30 file after the snapshot 2023-01-24T03:33:28.290 INFO:tasks.workunit.client.0.smithi112.stdout: create 142/31 file after the snapshot 2023-01-24T03:33:28.291 INFO:tasks.workunit.client.0.smithi112.stdout: create 142/32 file after the snapshot 2023-01-24T03:33:28.292 INFO:tasks.workunit.client.0.smithi112.stdout: create 142/33 file after the snapshot 2023-01-24T03:33:28.292 INFO:tasks.workunit.client.0.smithi112.stdout: create 142/34 file after the snapshot 2023-01-24T03:33:28.293 INFO:tasks.workunit.client.0.smithi112.stdout: create 142/35 file after the snapshot 2023-01-24T03:33:28.294 INFO:tasks.workunit.client.0.smithi112.stdout: create 142/36 file after the snapshot 2023-01-24T03:33:28.294 INFO:tasks.workunit.client.0.smithi112.stdout: create 142/37 file after the snapshot 2023-01-24T03:33:28.295 INFO:tasks.workunit.client.0.smithi112.stdout: create 142/38 file after the snapshot 2023-01-24T03:33:28.296 INFO:tasks.workunit.client.0.smithi112.stdout: create 142/39 file after the snapshot 2023-01-24T03:33:28.297 INFO:tasks.workunit.client.0.smithi112.stdout: create 143/.snap/snap-subdir-test 2023-01-24T03:33:28.316 INFO:tasks.workunit.client.0.smithi112.stdout: create 143/30 file after the snapshot 2023-01-24T03:33:28.317 INFO:tasks.workunit.client.0.smithi112.stdout: create 143/31 file after the snapshot 2023-01-24T03:33:28.318 INFO:tasks.workunit.client.0.smithi112.stdout: create 143/32 file after the snapshot 2023-01-24T03:33:28.319 INFO:tasks.workunit.client.0.smithi112.stdout: create 143/33 file after the snapshot 2023-01-24T03:33:28.320 INFO:tasks.workunit.client.0.smithi112.stdout: create 143/34 file after the snapshot 2023-01-24T03:33:28.321 INFO:tasks.workunit.client.0.smithi112.stdout: create 143/35 file after the snapshot 2023-01-24T03:33:28.322 INFO:tasks.workunit.client.0.smithi112.stdout: create 143/36 file after the snapshot 2023-01-24T03:33:28.323 INFO:tasks.workunit.client.0.smithi112.stdout: create 143/37 file after the snapshot 2023-01-24T03:33:28.324 INFO:tasks.workunit.client.0.smithi112.stdout: create 143/38 file after the snapshot 2023-01-24T03:33:28.325 INFO:tasks.workunit.client.0.smithi112.stdout: create 143/39 file after the snapshot 2023-01-24T03:33:28.326 INFO:tasks.workunit.client.0.smithi112.stdout: create 144/.snap/snap-subdir-test 2023-01-24T03:33:28.333 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2023-01-24T03:33:28.333 INFO:tasks.ceph.mds.b:Restarting daemon 2023-01-24T03:33:28.333 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-01-24T03:33:28.336 INFO:tasks.ceph.mds.b:Started 2023-01-24T03:33:28.336 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:33:28.345 INFO:tasks.workunit.client.0.smithi112.stdout: create 144/30 file after the snapshot 2023-01-24T03:33:28.404 INFO:tasks.ceph.mds.b.smithi142.stdout:starting mds.b at 2023-01-24T03:33:28.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.447+0000 7ff08a6bf700 1 Processor -- start 2023-01-24T03:33:28.450 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.447+0000 7ff08a6bf700 1 -- start start 2023-01-24T03:33:28.454 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.448+0000 7ff08a6bf700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7ff0840c5680 0x7ff084156730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:28.455 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.448+0000 7ff08a6bf700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7ff084155e20 0x7ff084156c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:28.455 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.448+0000 7ff08a6bf700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff0840c5a50 0x7ff0840cb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:28.455 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.448+0000 7ff08a6bf700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0840c4eb0 con 0x7ff084155e20 2023-01-24T03:33:28.455 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.448+0000 7ff08a6bf700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0840c3d10 con 0x7ff0840c5680 2023-01-24T03:33:28.456 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.448+0000 7ff08a6bf700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0840c2e20 con 0x7ff0840c5a50 2023-01-24T03:33:28.456 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.448+0000 7ff088ebc700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7ff084155e20 0x7ff084156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:28.456 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.448+0000 7ff088ebc700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7ff084155e20 0x7ff084156c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52292/0 (socket says 172.21.15.112:52292) 2023-01-24T03:33:28.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.448+0000 7ff088ebc700 1 -- 172.21.15.112:0/2459808624 learned_addr learned my addr 172.21.15.112:0/2459808624 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:33:28.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.448+0000 7ff089ebe700 1 --2- 172.21.15.112:0/2459808624 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff0840c5a50 0x7ff0840cb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:28.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.448+0000 7ff0896bd700 1 --2- 172.21.15.112:0/2459808624 >> v2:172.21.15.142:3300/0 conn(0x7ff0840c5680 0x7ff084156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:28.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.448+0000 7ff088ebc700 1 -- 172.21.15.112:0/2459808624 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff0840c5a50 msgr2=0x7ff0840cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:33:28.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.448+0000 7ff088ebc700 1 --2- 172.21.15.112:0/2459808624 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff0840c5a50 0x7ff0840cb870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:28.458 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.448+0000 7ff088ebc700 1 -- 172.21.15.112:0/2459808624 >> v2:172.21.15.142:3300/0 conn(0x7ff0840c5680 msgr2=0x7ff084156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:33:28.458 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.448+0000 7ff088ebc700 1 --2- 172.21.15.112:0/2459808624 >> v2:172.21.15.142:3300/0 conn(0x7ff0840c5680 0x7ff084156730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:28.458 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.448+0000 7ff088ebc700 1 -- 172.21.15.112:0/2459808624 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0840c3100 con 0x7ff084155e20 2023-01-24T03:33:28.458 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.448+0000 7ff0896bd700 1 --2- 172.21.15.112:0/2459808624 >> v2:172.21.15.142:3300/0 conn(0x7ff0840c5680 0x7ff084156730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:33:28.458 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.448+0000 7ff088ebc700 1 --2- 172.21.15.112:0/2459808624 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff084155e20 0x7ff084156c70 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff084070980 tx=0x7ff0780090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af4f1d5928e5c7bd server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:33:28.459 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.449+0000 7ff083fff700 1 -- 172.21.15.112:0/2459808624 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff078011630 con 0x7ff084155e20 2023-01-24T03:33:28.459 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.449+0000 7ff083fff700 1 -- 172.21.15.112:0/2459808624 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff078011790 con 0x7ff084155e20 2023-01-24T03:33:28.459 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.449+0000 7ff083fff700 1 -- 172.21.15.112:0/2459808624 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff078011b50 con 0x7ff084155e20 2023-01-24T03:33:28.459 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.449+0000 7ff08a6bf700 1 -- 172.21.15.112:0/2459808624 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff084155e20 msgr2=0x7ff084156c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:28.459 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.449+0000 7ff08a6bf700 1 --2- 172.21.15.112:0/2459808624 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff084155e20 0x7ff084156c70 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff084070980 tx=0x7ff0780090f0 comp rx=0 tx=0).stop 2023-01-24T03:33:28.460 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.449+0000 7ff08a6bf700 1 -- 172.21.15.112:0/2459808624 shutdown_connections 2023-01-24T03:33:28.461 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.449+0000 7ff08a6bf700 1 --2- 172.21.15.112:0/2459808624 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff0840c5a50 0x7ff0840cb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:28.461 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.449+0000 7ff08a6bf700 1 --2- 172.21.15.112:0/2459808624 >> v2:172.21.15.142:3300/0 conn(0x7ff0840c5680 0x7ff084156730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:28.461 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.449+0000 7ff08a6bf700 1 --2- 172.21.15.112:0/2459808624 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff084155e20 0x7ff084156c70 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:28.461 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.449+0000 7ff08a6bf700 1 -- 172.21.15.112:0/2459808624 >> 172.21.15.112:0/2459808624 conn(0x7ff0840685b0 msgr2=0x7ff0840689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:33:28.462 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.449+0000 7ff08a6bf700 1 -- 172.21.15.112:0/2459808624 shutdown_connections 2023-01-24T03:33:28.462 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.449+0000 7ff08a6bf700 1 -- 172.21.15.112:0/2459808624 wait complete. 2023-01-24T03:33:28.462 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.450+0000 7ff08a6bf700 1 Processor -- start 2023-01-24T03:33:28.462 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.450+0000 7ff08a6bf700 1 -- start start 2023-01-24T03:33:28.462 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.451+0000 7ff08a6bf700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff0840c5680 0x7ff08414fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:28.463 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.451+0000 7ff08a6bf700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff0840c5a50 0x7ff084166630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:28.463 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.451+0000 7ff08a6bf700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff084155e20 0x7ff0840b1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:28.463 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.451+0000 7ff08a6bf700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff084151840 con 0x7ff0840c5680 2023-01-24T03:33:28.463 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.451+0000 7ff08a6bf700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0841513b0 con 0x7ff084155e20 2023-01-24T03:33:28.463 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.451+0000 7ff08a6bf700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0841504c0 con 0x7ff0840c5a50 2023-01-24T03:33:28.464 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.451+0000 7ff0896bd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff0840c5680 0x7ff08414fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:28.464 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.451+0000 7ff0896bd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff0840c5680 0x7ff08414fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52296/0 (socket says 172.21.15.112:52296) 2023-01-24T03:33:28.464 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.451+0000 7ff0896bd700 1 -- 172.21.15.112:0/638160917 learned_addr learned my addr 172.21.15.112:0/638160917 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:33:28.464 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.451+0000 7ff089ebe700 1 --2- 172.21.15.112:0/638160917 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff084155e20 0x7ff0840b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:28.464 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.451+0000 7ff088ebc700 1 --2- 172.21.15.112:0/638160917 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff0840c5a50 0x7ff084166630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:28.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.451+0000 7ff0896bd700 1 -- 172.21.15.112:0/638160917 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff0840c5a50 msgr2=0x7ff084166630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:28.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.451+0000 7ff0896bd700 1 --2- 172.21.15.112:0/638160917 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff0840c5a50 0x7ff084166630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:28.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.451+0000 7ff0896bd700 1 -- 172.21.15.112:0/638160917 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff084155e20 msgr2=0x7ff0840b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:28.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.451+0000 7ff0896bd700 1 --2- 172.21.15.112:0/638160917 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff084155e20 0x7ff0840b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:28.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.451+0000 7ff0896bd700 1 -- 172.21.15.112:0/638160917 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff078009370 con 0x7ff0840c5680 2023-01-24T03:33:28.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.452+0000 7ff0896bd700 1 --2- 172.21.15.112:0/638160917 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff0840c5680 0x7ff08414fba0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff07400d180 tx=0x7ff0740099f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:33:28.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.452+0000 7ff0817fa700 1 -- 172.21.15.112:0/638160917 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07400dcc0 con 0x7ff0840c5680 2023-01-24T03:33:28.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.452+0000 7ff0817fa700 1 -- 172.21.15.112:0/638160917 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff07400f040 con 0x7ff0840c5680 2023-01-24T03:33:28.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.452+0000 7ff08a6bf700 1 -- 172.21.15.112:0/638160917 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0841507a0 con 0x7ff0840c5680 2023-01-24T03:33:28.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.452+0000 7ff0817fa700 1 -- 172.21.15.112:0/638160917 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff074015670 con 0x7ff0840c5680 2023-01-24T03:33:28.467 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.452+0000 7ff08a6bf700 1 -- 172.21.15.112:0/638160917 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff084150d60 con 0x7ff0840c5680 2023-01-24T03:33:28.467 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.453+0000 7ff08a6bf700 1 -- 172.21.15.112:0/638160917 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff084151690 con 0x7ff0840c5680 2023-01-24T03:33:28.468 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.453+0000 7ff0817fa700 1 -- 172.21.15.112:0/638160917 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7ff07401c030 con 0x7ff0840c5680 2023-01-24T03:33:28.468 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.453+0000 7ff0817fa700 1 --2- 172.21.15.112:0/638160917 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff0640611c0 0x7ff064063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:33:28.468 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.453+0000 7ff0817fa700 1 -- 172.21.15.112:0/638160917 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7039+0+0 (secure 0 0 0) 0x7ff07407daa0 con 0x7ff0840c5680 2023-01-24T03:33:28.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.453+0000 7ff088ebc700 1 --2- 172.21.15.112:0/638160917 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff0640611c0 0x7ff064063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:33:28.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.454+0000 7ff088ebc700 1 --2- 172.21.15.112:0/638160917 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff0640611c0 0x7ff064063670 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff084070980 tx=0x7ff078016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:33:28.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.458+0000 7ff0817fa700 1 -- 172.21.15.112:0/638160917 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7ff07404d8b0 con 0x7ff0840c5680 2023-01-24T03:33:28.616 INFO:tasks.workunit.client.0.smithi112.stdout: create 144/31 file after the snapshot 2023-01-24T03:33:28.617 INFO:tasks.workunit.client.0.smithi112.stdout: create 144/32 file after the snapshot 2023-01-24T03:33:28.617 INFO:tasks.workunit.client.0.smithi112.stdout: create 144/33 file after the snapshot 2023-01-24T03:33:28.618 INFO:tasks.workunit.client.0.smithi112.stdout: create 144/34 file after the snapshot 2023-01-24T03:33:28.619 INFO:tasks.workunit.client.0.smithi112.stdout: create 144/35 file after the snapshot 2023-01-24T03:33:28.619 INFO:tasks.workunit.client.0.smithi112.stdout: create 144/36 file after the snapshot 2023-01-24T03:33:28.620 INFO:tasks.workunit.client.0.smithi112.stdout: create 144/37 file after the snapshot 2023-01-24T03:33:28.621 INFO:tasks.workunit.client.0.smithi112.stdout: create 144/38 file after the snapshot 2023-01-24T03:33:28.621 INFO:tasks.workunit.client.0.smithi112.stdout: create 144/39 file after the snapshot 2023-01-24T03:33:28.622 INFO:tasks.workunit.client.0.smithi112.stdout: create 145/.snap/snap-subdir-test 2023-01-24T03:33:28.684 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.681+0000 7ff08a6bf700 1 -- 172.21.15.112:0/638160917 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff08405ea20 con 0x7ff0840c5680 2023-01-24T03:33:28.684 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.682+0000 7ff0817fa700 1 -- 172.21.15.112:0/638160917 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+6112 (secure 0 0 0) 0x7ff074050ed0 con 0x7ff0840c5680 2023-01-24T03:33:28.685 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:33:28.685 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5041,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/612058861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":612058861},{"type":"v1","addr":"172.21.15.142:6834","nonce":612058861}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:33:18.651832+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4897},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4897":{"gid":4897,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":19,"addr":"172.21.15.142:6835/11022091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":11022091},{"type":"v1","addr":"172.21.15.142:6835","nonce":11022091}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:33:28.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.684+0000 7ff08a6bf700 1 -- 172.21.15.112:0/638160917 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff0640611c0 msgr2=0x7ff064063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:28.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.684+0000 7ff08a6bf700 1 --2- 172.21.15.112:0/638160917 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff0640611c0 0x7ff064063670 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff084070980 tx=0x7ff078016000 comp rx=0 tx=0).stop 2023-01-24T03:33:28.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.684+0000 7ff08a6bf700 1 -- 172.21.15.112:0/638160917 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff0840c5680 msgr2=0x7ff08414fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:33:28.687 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.684+0000 7ff08a6bf700 1 --2- 172.21.15.112:0/638160917 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff0840c5680 0x7ff08414fba0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff07400d180 tx=0x7ff0740099f0 comp rx=0 tx=0).stop 2023-01-24T03:33:28.687 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.684+0000 7ff08a6bf700 1 -- 172.21.15.112:0/638160917 shutdown_connections 2023-01-24T03:33:28.687 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.684+0000 7ff08a6bf700 1 --2- 172.21.15.112:0/638160917 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff0640611c0 0x7ff064063670 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:28.687 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.684+0000 7ff08a6bf700 1 --2- 172.21.15.112:0/638160917 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff0840c5680 0x7ff08414fba0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:28.687 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.684+0000 7ff08a6bf700 1 --2- 172.21.15.112:0/638160917 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff084155e20 0x7ff0840b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:28.688 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.684+0000 7ff08a6bf700 1 --2- 172.21.15.112:0/638160917 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff0840c5a50 0x7ff084166630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:33:28.688 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.684+0000 7ff08a6bf700 1 -- 172.21.15.112:0/638160917 >> 172.21.15.112:0/638160917 conn(0x7ff0840685b0 msgr2=0x7ff0841555b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:33:28.688 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.684+0000 7ff08a6bf700 1 -- 172.21.15.112:0/638160917 shutdown_connections 2023-01-24T03:33:28.688 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:33:28.684+0000 7ff08a6bf700 1 -- 172.21.15.112:0/638160917 wait complete. 2023-01-24T03:33:28.689 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 21 2023-01-24T03:33:28.698 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2023-01-24T03:33:28.698 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 42 secs before thrashing 2023-01-24T03:33:28.898 INFO:tasks.workunit.client.0.smithi112.stdout: create 145/30 file after the snapshot 2023-01-24T03:33:28.899 INFO:tasks.workunit.client.0.smithi112.stdout: create 145/31 file after the snapshot 2023-01-24T03:33:28.900 INFO:tasks.workunit.client.0.smithi112.stdout: create 145/32 file after the snapshot 2023-01-24T03:33:28.901 INFO:tasks.workunit.client.0.smithi112.stdout: create 145/33 file after the snapshot 2023-01-24T03:33:28.902 INFO:tasks.workunit.client.0.smithi112.stdout: create 145/34 file after the snapshot 2023-01-24T03:33:28.902 INFO:tasks.workunit.client.0.smithi112.stdout: create 145/35 file after the snapshot 2023-01-24T03:33:28.903 INFO:tasks.workunit.client.0.smithi112.stdout: create 145/36 file after the snapshot 2023-01-24T03:33:28.904 INFO:tasks.workunit.client.0.smithi112.stdout: create 145/37 file after the snapshot 2023-01-24T03:33:28.905 INFO:tasks.workunit.client.0.smithi112.stdout: create 145/38 file after the snapshot 2023-01-24T03:33:28.905 INFO:tasks.workunit.client.0.smithi112.stdout: create 145/39 file after the snapshot 2023-01-24T03:33:28.906 INFO:tasks.workunit.client.0.smithi112.stdout: create 146/.snap/snap-subdir-test 2023-01-24T03:33:28.925 INFO:tasks.workunit.client.0.smithi112.stdout: create 146/30 file after the snapshot 2023-01-24T03:33:28.926 INFO:tasks.workunit.client.0.smithi112.stdout: create 146/31 file after the snapshot 2023-01-24T03:33:28.927 INFO:tasks.workunit.client.0.smithi112.stdout: create 146/32 file after the snapshot 2023-01-24T03:33:28.928 INFO:tasks.workunit.client.0.smithi112.stdout: create 146/33 file after the snapshot 2023-01-24T03:33:28.929 INFO:tasks.workunit.client.0.smithi112.stdout: create 146/34 file after the snapshot 2023-01-24T03:33:28.929 INFO:tasks.workunit.client.0.smithi112.stdout: create 146/35 file after the snapshot 2023-01-24T03:33:28.930 INFO:tasks.workunit.client.0.smithi112.stdout: create 146/36 file after the snapshot 2023-01-24T03:33:28.931 INFO:tasks.workunit.client.0.smithi112.stdout: create 146/37 file after the snapshot 2023-01-24T03:33:28.932 INFO:tasks.workunit.client.0.smithi112.stdout: create 146/38 file after the snapshot 2023-01-24T03:33:28.932 INFO:tasks.workunit.client.0.smithi112.stdout: create 146/39 file after the snapshot 2023-01-24T03:33:28.933 INFO:tasks.workunit.client.0.smithi112.stdout: create 147/.snap/snap-subdir-test 2023-01-24T03:33:28.954 INFO:tasks.workunit.client.0.smithi112.stdout: create 147/30 file after the snapshot 2023-01-24T03:33:28.955 INFO:tasks.workunit.client.0.smithi112.stdout: create 147/31 file after the snapshot 2023-01-24T03:33:28.956 INFO:tasks.workunit.client.0.smithi112.stdout: create 147/32 file after the snapshot 2023-01-24T03:33:28.957 INFO:tasks.workunit.client.0.smithi112.stdout: create 147/33 file after the snapshot 2023-01-24T03:33:28.958 INFO:tasks.workunit.client.0.smithi112.stdout: create 147/34 file after the snapshot 2023-01-24T03:33:28.959 INFO:tasks.workunit.client.0.smithi112.stdout: create 147/35 file after the snapshot 2023-01-24T03:33:28.960 INFO:tasks.workunit.client.0.smithi112.stdout: create 147/36 file after the snapshot 2023-01-24T03:33:28.961 INFO:tasks.workunit.client.0.smithi112.stdout: create 147/37 file after the snapshot 2023-01-24T03:33:28.962 INFO:tasks.workunit.client.0.smithi112.stdout: create 147/38 file after the snapshot 2023-01-24T03:33:28.963 INFO:tasks.workunit.client.0.smithi112.stdout: create 147/39 file after the snapshot 2023-01-24T03:33:28.964 INFO:tasks.workunit.client.0.smithi112.stdout: create 148/.snap/snap-subdir-test 2023-01-24T03:33:28.985 INFO:tasks.workunit.client.0.smithi112.stdout: create 148/30 file after the snapshot 2023-01-24T03:33:28.986 INFO:tasks.workunit.client.0.smithi112.stdout: create 148/31 file after the snapshot 2023-01-24T03:33:28.987 INFO:tasks.workunit.client.0.smithi112.stdout: create 148/32 file after the snapshot 2023-01-24T03:33:28.988 INFO:tasks.workunit.client.0.smithi112.stdout: create 148/33 file after the snapshot 2023-01-24T03:33:28.988 INFO:tasks.workunit.client.0.smithi112.stdout: create 148/34 file after the snapshot 2023-01-24T03:33:28.989 INFO:tasks.workunit.client.0.smithi112.stdout: create 148/35 file after the snapshot 2023-01-24T03:33:28.990 INFO:tasks.workunit.client.0.smithi112.stdout: create 148/36 file after the snapshot 2023-01-24T03:33:28.991 INFO:tasks.workunit.client.0.smithi112.stdout: create 148/37 file after the snapshot 2023-01-24T03:33:28.992 INFO:tasks.workunit.client.0.smithi112.stdout: create 148/38 file after the snapshot 2023-01-24T03:33:28.993 INFO:tasks.workunit.client.0.smithi112.stdout: create 148/39 file after the snapshot 2023-01-24T03:33:28.993 INFO:tasks.workunit.client.0.smithi112.stdout: create 149/.snap/snap-subdir-test 2023-01-24T03:33:29.213 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:33:29.216 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:33:29.351 INFO:tasks.workunit.client.0.smithi112.stdout: create 149/30 file after the snapshot 2023-01-24T03:33:29.351 INFO:tasks.workunit.client.0.smithi112.stdout: create 149/31 file after the snapshot 2023-01-24T03:33:29.352 INFO:tasks.workunit.client.0.smithi112.stdout: create 149/32 file after the snapshot 2023-01-24T03:33:29.353 INFO:tasks.workunit.client.0.smithi112.stdout: create 149/33 file after the snapshot 2023-01-24T03:33:29.354 INFO:tasks.workunit.client.0.smithi112.stdout: create 149/34 file after the snapshot 2023-01-24T03:33:29.355 INFO:tasks.workunit.client.0.smithi112.stdout: create 149/35 file after the snapshot 2023-01-24T03:33:29.355 INFO:tasks.workunit.client.0.smithi112.stdout: create 149/36 file after the snapshot 2023-01-24T03:33:29.356 INFO:tasks.workunit.client.0.smithi112.stdout: create 149/37 file after the snapshot 2023-01-24T03:33:29.357 INFO:tasks.workunit.client.0.smithi112.stdout: create 149/38 file after the snapshot 2023-01-24T03:33:29.358 INFO:tasks.workunit.client.0.smithi112.stdout: create 149/39 file after the snapshot 2023-01-24T03:33:29.358 INFO:tasks.workunit.client.0.smithi112.stdout: create 150/.snap/snap-subdir-test 2023-01-24T03:33:29.379 INFO:tasks.workunit.client.0.smithi112.stdout: create 150/30 file after the snapshot 2023-01-24T03:33:29.381 INFO:tasks.workunit.client.0.smithi112.stdout: create 150/31 file after the snapshot 2023-01-24T03:33:29.381 INFO:tasks.workunit.client.0.smithi112.stdout: create 150/32 file after the snapshot 2023-01-24T03:33:29.382 INFO:tasks.workunit.client.0.smithi112.stdout: create 150/33 file after the snapshot 2023-01-24T03:33:29.383 INFO:tasks.workunit.client.0.smithi112.stdout: create 150/34 file after the snapshot 2023-01-24T03:33:29.384 INFO:tasks.workunit.client.0.smithi112.stdout: create 150/35 file after the snapshot 2023-01-24T03:33:29.385 INFO:tasks.workunit.client.0.smithi112.stdout: create 150/36 file after the snapshot 2023-01-24T03:33:29.386 INFO:tasks.workunit.client.0.smithi112.stdout: create 150/37 file after the snapshot 2023-01-24T03:33:29.387 INFO:tasks.workunit.client.0.smithi112.stdout: create 150/38 file after the snapshot 2023-01-24T03:33:29.387 INFO:tasks.workunit.client.0.smithi112.stdout: create 150/39 file after the snapshot 2023-01-24T03:33:29.388 INFO:tasks.workunit.client.0.smithi112.stdout: create 151/.snap/snap-subdir-test 2023-01-24T03:33:29.408 INFO:tasks.workunit.client.0.smithi112.stdout: create 151/30 file after the snapshot 2023-01-24T03:33:29.409 INFO:tasks.workunit.client.0.smithi112.stdout: create 151/31 file after the snapshot 2023-01-24T03:33:29.410 INFO:tasks.workunit.client.0.smithi112.stdout: create 151/32 file after the snapshot 2023-01-24T03:33:29.411 INFO:tasks.workunit.client.0.smithi112.stdout: create 151/33 file after the snapshot 2023-01-24T03:33:29.412 INFO:tasks.workunit.client.0.smithi112.stdout: create 151/34 file after the snapshot 2023-01-24T03:33:29.413 INFO:tasks.workunit.client.0.smithi112.stdout: create 151/35 file after the snapshot 2023-01-24T03:33:29.414 INFO:tasks.workunit.client.0.smithi112.stdout: create 151/36 file after the snapshot 2023-01-24T03:33:29.414 INFO:tasks.workunit.client.0.smithi112.stdout: create 151/37 file after the snapshot 2023-01-24T03:33:29.415 INFO:tasks.workunit.client.0.smithi112.stdout: create 151/38 file after the snapshot 2023-01-24T03:33:29.416 INFO:tasks.workunit.client.0.smithi112.stdout: create 151/39 file after the snapshot 2023-01-24T03:33:29.417 INFO:tasks.workunit.client.0.smithi112.stdout: create 152/.snap/snap-subdir-test 2023-01-24T03:33:29.436 INFO:tasks.workunit.client.0.smithi112.stdout: create 152/30 file after the snapshot 2023-01-24T03:33:29.437 INFO:tasks.workunit.client.0.smithi112.stdout: create 152/31 file after the snapshot 2023-01-24T03:33:29.438 INFO:tasks.workunit.client.0.smithi112.stdout: create 152/32 file after the snapshot 2023-01-24T03:33:29.438 INFO:tasks.workunit.client.0.smithi112.stdout: create 152/33 file after the snapshot 2023-01-24T03:33:29.439 INFO:tasks.workunit.client.0.smithi112.stdout: create 152/34 file after the snapshot 2023-01-24T03:33:29.440 INFO:tasks.workunit.client.0.smithi112.stdout: create 152/35 file after the snapshot 2023-01-24T03:33:29.441 INFO:tasks.workunit.client.0.smithi112.stdout: create 152/36 file after the snapshot 2023-01-24T03:33:29.442 INFO:tasks.workunit.client.0.smithi112.stdout: create 152/37 file after the snapshot 2023-01-24T03:33:29.443 INFO:tasks.workunit.client.0.smithi112.stdout: create 152/38 file after the snapshot 2023-01-24T03:33:29.444 INFO:tasks.workunit.client.0.smithi112.stdout: create 152/39 file after the snapshot 2023-01-24T03:33:29.445 INFO:tasks.workunit.client.0.smithi112.stdout: create 153/.snap/snap-subdir-test 2023-01-24T03:33:29.465 INFO:tasks.workunit.client.0.smithi112.stdout: create 153/30 file after the snapshot 2023-01-24T03:33:29.465 INFO:tasks.workunit.client.0.smithi112.stdout: create 153/31 file after the snapshot 2023-01-24T03:33:29.466 INFO:tasks.workunit.client.0.smithi112.stdout: create 153/32 file after the snapshot 2023-01-24T03:33:29.467 INFO:tasks.workunit.client.0.smithi112.stdout: create 153/33 file after the snapshot 2023-01-24T03:33:29.468 INFO:tasks.workunit.client.0.smithi112.stdout: create 153/34 file after the snapshot 2023-01-24T03:33:29.469 INFO:tasks.workunit.client.0.smithi112.stdout: create 153/35 file after the snapshot 2023-01-24T03:33:29.469 INFO:tasks.workunit.client.0.smithi112.stdout: create 153/36 file after the snapshot 2023-01-24T03:33:29.470 INFO:tasks.workunit.client.0.smithi112.stdout: create 153/37 file after the snapshot 2023-01-24T03:33:29.471 INFO:tasks.workunit.client.0.smithi112.stdout: create 153/38 file after the snapshot 2023-01-24T03:33:29.472 INFO:tasks.workunit.client.0.smithi112.stdout: create 153/39 file after the snapshot 2023-01-24T03:33:29.473 INFO:tasks.workunit.client.0.smithi112.stdout: create 154/.snap/snap-subdir-test 2023-01-24T03:33:29.492 INFO:tasks.workunit.client.0.smithi112.stdout: create 154/30 file after the snapshot 2023-01-24T03:33:29.493 INFO:tasks.workunit.client.0.smithi112.stdout: create 154/31 file after the snapshot 2023-01-24T03:33:29.494 INFO:tasks.workunit.client.0.smithi112.stdout: create 154/32 file after the snapshot 2023-01-24T03:33:29.495 INFO:tasks.workunit.client.0.smithi112.stdout: create 154/33 file after the snapshot 2023-01-24T03:33:29.495 INFO:tasks.workunit.client.0.smithi112.stdout: create 154/34 file after the snapshot 2023-01-24T03:33:29.496 INFO:tasks.workunit.client.0.smithi112.stdout: create 154/35 file after the snapshot 2023-01-24T03:33:29.497 INFO:tasks.workunit.client.0.smithi112.stdout: create 154/36 file after the snapshot 2023-01-24T03:33:29.498 INFO:tasks.workunit.client.0.smithi112.stdout: create 154/37 file after the snapshot 2023-01-24T03:33:29.499 INFO:tasks.workunit.client.0.smithi112.stdout: create 154/38 file after the snapshot 2023-01-24T03:33:29.500 INFO:tasks.workunit.client.0.smithi112.stdout: create 154/39 file after the snapshot 2023-01-24T03:33:29.501 INFO:tasks.workunit.client.0.smithi112.stdout: create 155/.snap/snap-subdir-test 2023-01-24T03:33:29.520 INFO:tasks.workunit.client.0.smithi112.stdout: create 155/30 file after the snapshot 2023-01-24T03:33:29.522 INFO:tasks.workunit.client.0.smithi112.stdout: create 155/31 file after the snapshot 2023-01-24T03:33:29.522 INFO:tasks.workunit.client.0.smithi112.stdout: create 155/32 file after the snapshot 2023-01-24T03:33:29.523 INFO:tasks.workunit.client.0.smithi112.stdout: create 155/33 file after the snapshot 2023-01-24T03:33:29.524 INFO:tasks.workunit.client.0.smithi112.stdout: create 155/34 file after the snapshot 2023-01-24T03:33:29.525 INFO:tasks.workunit.client.0.smithi112.stdout: create 155/35 file after the snapshot 2023-01-24T03:33:29.526 INFO:tasks.workunit.client.0.smithi112.stdout: create 155/36 file after the snapshot 2023-01-24T03:33:29.527 INFO:tasks.workunit.client.0.smithi112.stdout: create 155/37 file after the snapshot 2023-01-24T03:33:29.528 INFO:tasks.workunit.client.0.smithi112.stdout: create 155/38 file after the snapshot 2023-01-24T03:33:29.529 INFO:tasks.workunit.client.0.smithi112.stdout: create 155/39 file after the snapshot 2023-01-24T03:33:29.530 INFO:tasks.workunit.client.0.smithi112.stdout: create 156/.snap/snap-subdir-test 2023-01-24T03:33:29.549 INFO:tasks.workunit.client.0.smithi112.stdout: create 156/30 file after the snapshot 2023-01-24T03:33:29.550 INFO:tasks.workunit.client.0.smithi112.stdout: create 156/31 file after the snapshot 2023-01-24T03:33:29.551 INFO:tasks.workunit.client.0.smithi112.stdout: create 156/32 file after the snapshot 2023-01-24T03:33:29.552 INFO:tasks.workunit.client.0.smithi112.stdout: create 156/33 file after the snapshot 2023-01-24T03:33:29.553 INFO:tasks.workunit.client.0.smithi112.stdout: create 156/34 file after the snapshot 2023-01-24T03:33:29.554 INFO:tasks.workunit.client.0.smithi112.stdout: create 156/35 file after the snapshot 2023-01-24T03:33:29.555 INFO:tasks.workunit.client.0.smithi112.stdout: create 156/36 file after the snapshot 2023-01-24T03:33:29.556 INFO:tasks.workunit.client.0.smithi112.stdout: create 156/37 file after the snapshot 2023-01-24T03:33:29.557 INFO:tasks.workunit.client.0.smithi112.stdout: create 156/38 file after the snapshot 2023-01-24T03:33:29.558 INFO:tasks.workunit.client.0.smithi112.stdout: create 156/39 file after the snapshot 2023-01-24T03:33:29.558 INFO:tasks.workunit.client.0.smithi112.stdout: create 157/.snap/snap-subdir-test 2023-01-24T03:33:29.577 INFO:tasks.workunit.client.0.smithi112.stdout: create 157/30 file after the snapshot 2023-01-24T03:33:29.578 INFO:tasks.workunit.client.0.smithi112.stdout: create 157/31 file after the snapshot 2023-01-24T03:33:29.579 INFO:tasks.workunit.client.0.smithi112.stdout: create 157/32 file after the snapshot 2023-01-24T03:33:29.580 INFO:tasks.workunit.client.0.smithi112.stdout: create 157/33 file after the snapshot 2023-01-24T03:33:29.584 INFO:tasks.workunit.client.0.smithi112.stdout: create 157/34 file after the snapshot 2023-01-24T03:33:29.585 INFO:tasks.workunit.client.0.smithi112.stdout: create 157/35 file after the snapshot 2023-01-24T03:33:29.585 INFO:tasks.workunit.client.0.smithi112.stdout: create 157/36 file after the snapshot 2023-01-24T03:33:29.585 INFO:tasks.workunit.client.0.smithi112.stdout: create 157/37 file after the snapshot 2023-01-24T03:33:29.585 INFO:tasks.workunit.client.0.smithi112.stdout: create 157/38 file after the snapshot 2023-01-24T03:33:29.586 INFO:tasks.workunit.client.0.smithi112.stdout: create 157/39 file after the snapshot 2023-01-24T03:33:29.587 INFO:tasks.workunit.client.0.smithi112.stdout: create 158/.snap/snap-subdir-test 2023-01-24T03:33:29.607 INFO:tasks.workunit.client.0.smithi112.stdout: create 158/30 file after the snapshot 2023-01-24T03:33:29.609 INFO:tasks.workunit.client.0.smithi112.stdout: create 158/31 file after the snapshot 2023-01-24T03:33:29.610 INFO:tasks.workunit.client.0.smithi112.stdout: create 158/32 file after the snapshot 2023-01-24T03:33:29.611 INFO:tasks.workunit.client.0.smithi112.stdout: create 158/33 file after the snapshot 2023-01-24T03:33:29.612 INFO:tasks.workunit.client.0.smithi112.stdout: create 158/34 file after the snapshot 2023-01-24T03:33:29.613 INFO:tasks.workunit.client.0.smithi112.stdout: create 158/35 file after the snapshot 2023-01-24T03:33:29.613 INFO:tasks.workunit.client.0.smithi112.stdout: create 158/36 file after the snapshot 2023-01-24T03:33:29.614 INFO:tasks.workunit.client.0.smithi112.stdout: create 158/37 file after the snapshot 2023-01-24T03:33:29.615 INFO:tasks.workunit.client.0.smithi112.stdout: create 158/38 file after the snapshot 2023-01-24T03:33:29.616 INFO:tasks.workunit.client.0.smithi112.stdout: create 158/39 file after the snapshot 2023-01-24T03:33:29.617 INFO:tasks.workunit.client.0.smithi112.stdout: create 159/.snap/snap-subdir-test 2023-01-24T03:33:29.637 INFO:tasks.workunit.client.0.smithi112.stdout: create 159/30 file after the snapshot 2023-01-24T03:33:29.638 INFO:tasks.workunit.client.0.smithi112.stdout: create 159/31 file after the snapshot 2023-01-24T03:33:29.639 INFO:tasks.workunit.client.0.smithi112.stdout: create 159/32 file after the snapshot 2023-01-24T03:33:29.640 INFO:tasks.workunit.client.0.smithi112.stdout: create 159/33 file after the snapshot 2023-01-24T03:33:29.641 INFO:tasks.workunit.client.0.smithi112.stdout: create 159/34 file after the snapshot 2023-01-24T03:33:29.642 INFO:tasks.workunit.client.0.smithi112.stdout: create 159/35 file after the snapshot 2023-01-24T03:33:29.643 INFO:tasks.workunit.client.0.smithi112.stdout: create 159/36 file after the snapshot 2023-01-24T03:33:29.644 INFO:tasks.workunit.client.0.smithi112.stdout: create 159/37 file after the snapshot 2023-01-24T03:33:29.645 INFO:tasks.workunit.client.0.smithi112.stdout: create 159/38 file after the snapshot 2023-01-24T03:33:29.646 INFO:tasks.workunit.client.0.smithi112.stdout: create 159/39 file after the snapshot 2023-01-24T03:33:29.648 INFO:tasks.workunit.client.0.smithi112.stdout: create 160/.snap/snap-subdir-test 2023-01-24T03:33:29.667 INFO:tasks.workunit.client.0.smithi112.stdout: create 160/30 file after the snapshot 2023-01-24T03:33:29.668 INFO:tasks.workunit.client.0.smithi112.stdout: create 160/31 file after the snapshot 2023-01-24T03:33:29.669 INFO:tasks.workunit.client.0.smithi112.stdout: create 160/32 file after the snapshot 2023-01-24T03:33:29.669 INFO:tasks.workunit.client.0.smithi112.stdout: create 160/33 file after the snapshot 2023-01-24T03:33:29.670 INFO:tasks.workunit.client.0.smithi112.stdout: create 160/34 file after the snapshot 2023-01-24T03:33:29.671 INFO:tasks.workunit.client.0.smithi112.stdout: create 160/35 file after the snapshot 2023-01-24T03:33:29.672 INFO:tasks.workunit.client.0.smithi112.stdout: create 160/36 file after the snapshot 2023-01-24T03:33:29.673 INFO:tasks.workunit.client.0.smithi112.stdout: create 160/37 file after the snapshot 2023-01-24T03:33:29.674 INFO:tasks.workunit.client.0.smithi112.stdout: create 160/38 file after the snapshot 2023-01-24T03:33:29.674 INFO:tasks.workunit.client.0.smithi112.stdout: create 160/39 file after the snapshot 2023-01-24T03:33:29.675 INFO:tasks.workunit.client.0.smithi112.stdout: create 161/.snap/snap-subdir-test 2023-01-24T03:33:29.694 INFO:tasks.workunit.client.0.smithi112.stdout: create 161/30 file after the snapshot 2023-01-24T03:33:29.696 INFO:tasks.workunit.client.0.smithi112.stdout: create 161/31 file after the snapshot 2023-01-24T03:33:29.697 INFO:tasks.workunit.client.0.smithi112.stdout: create 161/32 file after the snapshot 2023-01-24T03:33:29.698 INFO:tasks.workunit.client.0.smithi112.stdout: create 161/33 file after the snapshot 2023-01-24T03:33:29.698 INFO:tasks.workunit.client.0.smithi112.stdout: create 161/34 file after the snapshot 2023-01-24T03:33:29.699 INFO:tasks.workunit.client.0.smithi112.stdout: create 161/35 file after the snapshot 2023-01-24T03:33:29.700 INFO:tasks.workunit.client.0.smithi112.stdout: create 161/36 file after the snapshot 2023-01-24T03:33:29.701 INFO:tasks.workunit.client.0.smithi112.stdout: create 161/37 file after the snapshot 2023-01-24T03:33:29.701 INFO:tasks.workunit.client.0.smithi112.stdout: create 161/38 file after the snapshot 2023-01-24T03:33:29.702 INFO:tasks.workunit.client.0.smithi112.stdout: create 161/39 file after the snapshot 2023-01-24T03:33:29.703 INFO:tasks.workunit.client.0.smithi112.stdout: create 162/.snap/snap-subdir-test 2023-01-24T03:33:29.724 INFO:tasks.workunit.client.0.smithi112.stdout: create 162/30 file after the snapshot 2023-01-24T03:33:29.724 INFO:tasks.workunit.client.0.smithi112.stdout: create 162/31 file after the snapshot 2023-01-24T03:33:29.725 INFO:tasks.workunit.client.0.smithi112.stdout: create 162/32 file after the snapshot 2023-01-24T03:33:29.726 INFO:tasks.workunit.client.0.smithi112.stdout: create 162/33 file after the snapshot 2023-01-24T03:33:29.727 INFO:tasks.workunit.client.0.smithi112.stdout: create 162/34 file after the snapshot 2023-01-24T03:33:29.728 INFO:tasks.workunit.client.0.smithi112.stdout: create 162/35 file after the snapshot 2023-01-24T03:33:29.729 INFO:tasks.workunit.client.0.smithi112.stdout: create 162/36 file after the snapshot 2023-01-24T03:33:29.730 INFO:tasks.workunit.client.0.smithi112.stdout: create 162/37 file after the snapshot 2023-01-24T03:33:29.731 INFO:tasks.workunit.client.0.smithi112.stdout: create 162/38 file after the snapshot 2023-01-24T03:33:29.732 INFO:tasks.workunit.client.0.smithi112.stdout: create 162/39 file after the snapshot 2023-01-24T03:33:29.732 INFO:tasks.workunit.client.0.smithi112.stdout: create 163/.snap/snap-subdir-test 2023-01-24T03:33:29.753 INFO:tasks.workunit.client.0.smithi112.stdout: create 163/30 file after the snapshot 2023-01-24T03:33:29.754 INFO:tasks.workunit.client.0.smithi112.stdout: create 163/31 file after the snapshot 2023-01-24T03:33:29.755 INFO:tasks.workunit.client.0.smithi112.stdout: create 163/32 file after the snapshot 2023-01-24T03:33:29.756 INFO:tasks.workunit.client.0.smithi112.stdout: create 163/33 file after the snapshot 2023-01-24T03:33:29.757 INFO:tasks.workunit.client.0.smithi112.stdout: create 163/34 file after the snapshot 2023-01-24T03:33:29.758 INFO:tasks.workunit.client.0.smithi112.stdout: create 163/35 file after the snapshot 2023-01-24T03:33:29.759 INFO:tasks.workunit.client.0.smithi112.stdout: create 163/36 file after the snapshot 2023-01-24T03:33:29.760 INFO:tasks.workunit.client.0.smithi112.stdout: create 163/37 file after the snapshot 2023-01-24T03:33:29.760 INFO:tasks.workunit.client.0.smithi112.stdout: create 163/38 file after the snapshot 2023-01-24T03:33:29.761 INFO:tasks.workunit.client.0.smithi112.stdout: create 163/39 file after the snapshot 2023-01-24T03:33:30.355 INFO:tasks.workunit.client.0.smithi112.stdout: create 164/.snap/snap-subdir-test 2023-01-24T03:33:30.388 INFO:tasks.workunit.client.0.smithi112.stdout: create 164/30 file after the snapshot 2023-01-24T03:33:30.389 INFO:tasks.workunit.client.0.smithi112.stdout: create 164/31 file after the snapshot 2023-01-24T03:33:30.390 INFO:tasks.workunit.client.0.smithi112.stdout: create 164/32 file after the snapshot 2023-01-24T03:33:30.391 INFO:tasks.workunit.client.0.smithi112.stdout: create 164/33 file after the snapshot 2023-01-24T03:33:30.392 INFO:tasks.workunit.client.0.smithi112.stdout: create 164/34 file after the snapshot 2023-01-24T03:33:30.393 INFO:tasks.workunit.client.0.smithi112.stdout: create 164/35 file after the snapshot 2023-01-24T03:33:30.394 INFO:tasks.workunit.client.0.smithi112.stdout: create 164/36 file after the snapshot 2023-01-24T03:33:30.395 INFO:tasks.workunit.client.0.smithi112.stdout: create 164/37 file after the snapshot 2023-01-24T03:33:30.396 INFO:tasks.workunit.client.0.smithi112.stdout: create 164/38 file after the snapshot 2023-01-24T03:33:30.397 INFO:tasks.workunit.client.0.smithi112.stdout: create 164/39 file after the snapshot 2023-01-24T03:33:30.398 INFO:tasks.workunit.client.0.smithi112.stdout: create 165/.snap/snap-subdir-test 2023-01-24T03:33:30.424 INFO:tasks.workunit.client.0.smithi112.stdout: create 165/30 file after the snapshot 2023-01-24T03:33:30.425 INFO:tasks.workunit.client.0.smithi112.stdout: create 165/31 file after the snapshot 2023-01-24T03:33:30.426 INFO:tasks.workunit.client.0.smithi112.stdout: create 165/32 file after the snapshot 2023-01-24T03:33:30.427 INFO:tasks.workunit.client.0.smithi112.stdout: create 165/33 file after the snapshot 2023-01-24T03:33:30.428 INFO:tasks.workunit.client.0.smithi112.stdout: create 165/34 file after the snapshot 2023-01-24T03:33:30.429 INFO:tasks.workunit.client.0.smithi112.stdout: create 165/35 file after the snapshot 2023-01-24T03:33:30.430 INFO:tasks.workunit.client.0.smithi112.stdout: create 165/36 file after the snapshot 2023-01-24T03:33:30.431 INFO:tasks.workunit.client.0.smithi112.stdout: create 165/37 file after the snapshot 2023-01-24T03:33:30.432 INFO:tasks.workunit.client.0.smithi112.stdout: create 165/38 file after the snapshot 2023-01-24T03:33:30.433 INFO:tasks.workunit.client.0.smithi112.stdout: create 165/39 file after the snapshot 2023-01-24T03:33:30.434 INFO:tasks.workunit.client.0.smithi112.stdout: create 166/.snap/snap-subdir-test 2023-01-24T03:33:30.460 INFO:tasks.workunit.client.0.smithi112.stdout: create 166/30 file after the snapshot 2023-01-24T03:33:30.461 INFO:tasks.workunit.client.0.smithi112.stdout: create 166/31 file after the snapshot 2023-01-24T03:33:30.462 INFO:tasks.workunit.client.0.smithi112.stdout: create 166/32 file after the snapshot 2023-01-24T03:33:30.463 INFO:tasks.workunit.client.0.smithi112.stdout: create 166/33 file after the snapshot 2023-01-24T03:33:30.464 INFO:tasks.workunit.client.0.smithi112.stdout: create 166/34 file after the snapshot 2023-01-24T03:33:30.465 INFO:tasks.workunit.client.0.smithi112.stdout: create 166/35 file after the snapshot 2023-01-24T03:33:30.465 INFO:tasks.workunit.client.0.smithi112.stdout: create 166/36 file after the snapshot 2023-01-24T03:33:30.466 INFO:tasks.workunit.client.0.smithi112.stdout: create 166/37 file after the snapshot 2023-01-24T03:33:30.467 INFO:tasks.workunit.client.0.smithi112.stdout: create 166/38 file after the snapshot 2023-01-24T03:33:30.468 INFO:tasks.workunit.client.0.smithi112.stdout: create 166/39 file after the snapshot 2023-01-24T03:33:30.469 INFO:tasks.workunit.client.0.smithi112.stdout: create 167/.snap/snap-subdir-test 2023-01-24T03:33:30.495 INFO:tasks.workunit.client.0.smithi112.stdout: create 167/30 file after the snapshot 2023-01-24T03:33:30.496 INFO:tasks.workunit.client.0.smithi112.stdout: create 167/31 file after the snapshot 2023-01-24T03:33:30.497 INFO:tasks.workunit.client.0.smithi112.stdout: create 167/32 file after the snapshot 2023-01-24T03:33:30.498 INFO:tasks.workunit.client.0.smithi112.stdout: create 167/33 file after the snapshot 2023-01-24T03:33:30.498 INFO:tasks.workunit.client.0.smithi112.stdout: create 167/34 file after the snapshot 2023-01-24T03:33:30.499 INFO:tasks.workunit.client.0.smithi112.stdout: create 167/35 file after the snapshot 2023-01-24T03:33:30.500 INFO:tasks.workunit.client.0.smithi112.stdout: create 167/36 file after the snapshot 2023-01-24T03:33:30.501 INFO:tasks.workunit.client.0.smithi112.stdout: create 167/37 file after the snapshot 2023-01-24T03:33:30.502 INFO:tasks.workunit.client.0.smithi112.stdout: create 167/38 file after the snapshot 2023-01-24T03:33:30.502 INFO:tasks.workunit.client.0.smithi112.stdout: create 167/39 file after the snapshot 2023-01-24T03:33:30.503 INFO:tasks.workunit.client.0.smithi112.stdout: create 168/.snap/snap-subdir-test 2023-01-24T03:33:30.529 INFO:tasks.workunit.client.0.smithi112.stdout: create 168/30 file after the snapshot 2023-01-24T03:33:30.530 INFO:tasks.workunit.client.0.smithi112.stdout: create 168/31 file after the snapshot 2023-01-24T03:33:30.531 INFO:tasks.workunit.client.0.smithi112.stdout: create 168/32 file after the snapshot 2023-01-24T03:33:30.532 INFO:tasks.workunit.client.0.smithi112.stdout: create 168/33 file after the snapshot 2023-01-24T03:33:30.533 INFO:tasks.workunit.client.0.smithi112.stdout: create 168/34 file after the snapshot 2023-01-24T03:33:30.534 INFO:tasks.workunit.client.0.smithi112.stdout: create 168/35 file after the snapshot 2023-01-24T03:33:30.535 INFO:tasks.workunit.client.0.smithi112.stdout: create 168/36 file after the snapshot 2023-01-24T03:33:30.536 INFO:tasks.workunit.client.0.smithi112.stdout: create 168/37 file after the snapshot 2023-01-24T03:33:30.537 INFO:tasks.workunit.client.0.smithi112.stdout: create 168/38 file after the snapshot 2023-01-24T03:33:30.538 INFO:tasks.workunit.client.0.smithi112.stdout: create 168/39 file after the snapshot 2023-01-24T03:33:30.539 INFO:tasks.workunit.client.0.smithi112.stdout: create 169/.snap/snap-subdir-test 2023-01-24T03:33:30.564 INFO:tasks.workunit.client.0.smithi112.stdout: create 169/30 file after the snapshot 2023-01-24T03:33:30.565 INFO:tasks.workunit.client.0.smithi112.stdout: create 169/31 file after the snapshot 2023-01-24T03:33:30.566 INFO:tasks.workunit.client.0.smithi112.stdout: create 169/32 file after the snapshot 2023-01-24T03:33:30.567 INFO:tasks.workunit.client.0.smithi112.stdout: create 169/33 file after the snapshot 2023-01-24T03:33:30.568 INFO:tasks.workunit.client.0.smithi112.stdout: create 169/34 file after the snapshot 2023-01-24T03:33:30.569 INFO:tasks.workunit.client.0.smithi112.stdout: create 169/35 file after the snapshot 2023-01-24T03:33:30.570 INFO:tasks.workunit.client.0.smithi112.stdout: create 169/36 file after the snapshot 2023-01-24T03:33:30.571 INFO:tasks.workunit.client.0.smithi112.stdout: create 169/37 file after the snapshot 2023-01-24T03:33:30.572 INFO:tasks.workunit.client.0.smithi112.stdout: create 169/38 file after the snapshot 2023-01-24T03:33:30.573 INFO:tasks.workunit.client.0.smithi112.stdout: create 169/39 file after the snapshot 2023-01-24T03:33:30.574 INFO:tasks.workunit.client.0.smithi112.stdout: create 170/.snap/snap-subdir-test 2023-01-24T03:33:30.600 INFO:tasks.workunit.client.0.smithi112.stdout: create 170/30 file after the snapshot 2023-01-24T03:33:30.601 INFO:tasks.workunit.client.0.smithi112.stdout: create 170/31 file after the snapshot 2023-01-24T03:33:30.602 INFO:tasks.workunit.client.0.smithi112.stdout: create 170/32 file after the snapshot 2023-01-24T03:33:30.603 INFO:tasks.workunit.client.0.smithi112.stdout: create 170/33 file after the snapshot 2023-01-24T03:33:30.603 INFO:tasks.workunit.client.0.smithi112.stdout: create 170/34 file after the snapshot 2023-01-24T03:33:30.604 INFO:tasks.workunit.client.0.smithi112.stdout: create 170/35 file after the snapshot 2023-01-24T03:33:30.605 INFO:tasks.workunit.client.0.smithi112.stdout: create 170/36 file after the snapshot 2023-01-24T03:33:30.606 INFO:tasks.workunit.client.0.smithi112.stdout: create 170/37 file after the snapshot 2023-01-24T03:33:30.607 INFO:tasks.workunit.client.0.smithi112.stdout: create 170/38 file after the snapshot 2023-01-24T03:33:30.607 INFO:tasks.workunit.client.0.smithi112.stdout: create 170/39 file after the snapshot 2023-01-24T03:33:30.608 INFO:tasks.workunit.client.0.smithi112.stdout: create 171/.snap/snap-subdir-test 2023-01-24T03:33:30.634 INFO:tasks.workunit.client.0.smithi112.stdout: create 171/30 file after the snapshot 2023-01-24T03:33:30.635 INFO:tasks.workunit.client.0.smithi112.stdout: create 171/31 file after the snapshot 2023-01-24T03:33:30.636 INFO:tasks.workunit.client.0.smithi112.stdout: create 171/32 file after the snapshot 2023-01-24T03:33:30.637 INFO:tasks.workunit.client.0.smithi112.stdout: create 171/33 file after the snapshot 2023-01-24T03:33:30.637 INFO:tasks.workunit.client.0.smithi112.stdout: create 171/34 file after the snapshot 2023-01-24T03:33:30.638 INFO:tasks.workunit.client.0.smithi112.stdout: create 171/35 file after the snapshot 2023-01-24T03:33:30.638 INFO:tasks.workunit.client.0.smithi112.stdout: create 171/36 file after the snapshot 2023-01-24T03:33:30.639 INFO:tasks.workunit.client.0.smithi112.stdout: create 171/37 file after the snapshot 2023-01-24T03:33:30.640 INFO:tasks.workunit.client.0.smithi112.stdout: create 171/38 file after the snapshot 2023-01-24T03:33:30.641 INFO:tasks.workunit.client.0.smithi112.stdout: create 171/39 file after the snapshot 2023-01-24T03:33:30.641 INFO:tasks.workunit.client.0.smithi112.stdout: create 172/.snap/snap-subdir-test 2023-01-24T03:33:30.667 INFO:tasks.workunit.client.0.smithi112.stdout: create 172/30 file after the snapshot 2023-01-24T03:33:30.668 INFO:tasks.workunit.client.0.smithi112.stdout: create 172/31 file after the snapshot 2023-01-24T03:33:30.668 INFO:tasks.workunit.client.0.smithi112.stdout: create 172/32 file after the snapshot 2023-01-24T03:33:30.669 INFO:tasks.workunit.client.0.smithi112.stdout: create 172/33 file after the snapshot 2023-01-24T03:33:30.670 INFO:tasks.workunit.client.0.smithi112.stdout: create 172/34 file after the snapshot 2023-01-24T03:33:30.671 INFO:tasks.workunit.client.0.smithi112.stdout: create 172/35 file after the snapshot 2023-01-24T03:33:30.672 INFO:tasks.workunit.client.0.smithi112.stdout: create 172/36 file after the snapshot 2023-01-24T03:33:30.673 INFO:tasks.workunit.client.0.smithi112.stdout: create 172/37 file after the snapshot 2023-01-24T03:33:30.674 INFO:tasks.workunit.client.0.smithi112.stdout: create 172/38 file after the snapshot 2023-01-24T03:33:30.675 INFO:tasks.workunit.client.0.smithi112.stdout: create 172/39 file after the snapshot 2023-01-24T03:33:30.675 INFO:tasks.workunit.client.0.smithi112.stdout: create 173/.snap/snap-subdir-test 2023-01-24T03:33:30.701 INFO:tasks.workunit.client.0.smithi112.stdout: create 173/30 file after the snapshot 2023-01-24T03:33:30.702 INFO:tasks.workunit.client.0.smithi112.stdout: create 173/31 file after the snapshot 2023-01-24T03:33:30.703 INFO:tasks.workunit.client.0.smithi112.stdout: create 173/32 file after the snapshot 2023-01-24T03:33:30.704 INFO:tasks.workunit.client.0.smithi112.stdout: create 173/33 file after the snapshot 2023-01-24T03:33:30.805 INFO:tasks.workunit.client.0.smithi112.stdout: create 173/34 file after the snapshot 2023-01-24T03:33:30.806 INFO:tasks.workunit.client.0.smithi112.stdout: create 173/35 file after the snapshot 2023-01-24T03:33:30.807 INFO:tasks.workunit.client.0.smithi112.stdout: create 173/36 file after the snapshot 2023-01-24T03:33:30.808 INFO:tasks.workunit.client.0.smithi112.stdout: create 173/37 file after the snapshot 2023-01-24T03:33:30.809 INFO:tasks.workunit.client.0.smithi112.stdout: create 173/38 file after the snapshot 2023-01-24T03:33:30.810 INFO:tasks.workunit.client.0.smithi112.stdout: create 173/39 file after the snapshot 2023-01-24T03:33:30.811 INFO:tasks.workunit.client.0.smithi112.stdout: create 174/.snap/snap-subdir-test 2023-01-24T03:33:30.831 INFO:tasks.workunit.client.0.smithi112.stdout: create 174/30 file after the snapshot 2023-01-24T03:33:30.832 INFO:tasks.workunit.client.0.smithi112.stdout: create 174/31 file after the snapshot 2023-01-24T03:33:30.833 INFO:tasks.workunit.client.0.smithi112.stdout: create 174/32 file after the snapshot 2023-01-24T03:33:30.834 INFO:tasks.workunit.client.0.smithi112.stdout: create 174/33 file after the snapshot 2023-01-24T03:33:30.834 INFO:tasks.workunit.client.0.smithi112.stdout: create 174/34 file after the snapshot 2023-01-24T03:33:30.836 INFO:tasks.workunit.client.0.smithi112.stdout: create 174/35 file after the snapshot 2023-01-24T03:33:30.836 INFO:tasks.workunit.client.0.smithi112.stdout: create 174/36 file after the snapshot 2023-01-24T03:33:30.837 INFO:tasks.workunit.client.0.smithi112.stdout: create 174/37 file after the snapshot 2023-01-24T03:33:30.838 INFO:tasks.workunit.client.0.smithi112.stdout: create 174/38 file after the snapshot 2023-01-24T03:33:30.838 INFO:tasks.workunit.client.0.smithi112.stdout: create 174/39 file after the snapshot 2023-01-24T03:33:30.839 INFO:tasks.workunit.client.0.smithi112.stdout: create 175/.snap/snap-subdir-test 2023-01-24T03:33:30.858 INFO:tasks.workunit.client.0.smithi112.stdout: create 175/30 file after the snapshot 2023-01-24T03:33:30.859 INFO:tasks.workunit.client.0.smithi112.stdout: create 175/31 file after the snapshot 2023-01-24T03:33:30.860 INFO:tasks.workunit.client.0.smithi112.stdout: create 175/32 file after the snapshot 2023-01-24T03:33:30.861 INFO:tasks.workunit.client.0.smithi112.stdout: create 175/33 file after the snapshot 2023-01-24T03:33:30.862 INFO:tasks.workunit.client.0.smithi112.stdout: create 175/34 file after the snapshot 2023-01-24T03:33:30.863 INFO:tasks.workunit.client.0.smithi112.stdout: create 175/35 file after the snapshot 2023-01-24T03:33:30.864 INFO:tasks.workunit.client.0.smithi112.stdout: create 175/36 file after the snapshot 2023-01-24T03:33:30.865 INFO:tasks.workunit.client.0.smithi112.stdout: create 175/37 file after the snapshot 2023-01-24T03:33:30.866 INFO:tasks.workunit.client.0.smithi112.stdout: create 175/38 file after the snapshot 2023-01-24T03:33:30.867 INFO:tasks.workunit.client.0.smithi112.stdout: create 175/39 file after the snapshot 2023-01-24T03:33:30.868 INFO:tasks.workunit.client.0.smithi112.stdout: create 176/.snap/snap-subdir-test 2023-01-24T03:33:30.888 INFO:tasks.workunit.client.0.smithi112.stdout: create 176/30 file after the snapshot 2023-01-24T03:33:30.890 INFO:tasks.workunit.client.0.smithi112.stdout: create 176/31 file after the snapshot 2023-01-24T03:33:30.891 INFO:tasks.workunit.client.0.smithi112.stdout: create 176/32 file after the snapshot 2023-01-24T03:33:30.892 INFO:tasks.workunit.client.0.smithi112.stdout: create 176/33 file after the snapshot 2023-01-24T03:33:30.893 INFO:tasks.workunit.client.0.smithi112.stdout: create 176/34 file after the snapshot 2023-01-24T03:33:30.894 INFO:tasks.workunit.client.0.smithi112.stdout: create 176/35 file after the snapshot 2023-01-24T03:33:30.894 INFO:tasks.workunit.client.0.smithi112.stdout: create 176/36 file after the snapshot 2023-01-24T03:33:30.895 INFO:tasks.workunit.client.0.smithi112.stdout: create 176/37 file after the snapshot 2023-01-24T03:33:30.896 INFO:tasks.workunit.client.0.smithi112.stdout: create 176/38 file after the snapshot 2023-01-24T03:33:30.896 INFO:tasks.workunit.client.0.smithi112.stdout: create 176/39 file after the snapshot 2023-01-24T03:33:30.897 INFO:tasks.workunit.client.0.smithi112.stdout: create 177/.snap/snap-subdir-test 2023-01-24T03:33:30.916 INFO:tasks.workunit.client.0.smithi112.stdout: create 177/30 file after the snapshot 2023-01-24T03:33:30.918 INFO:tasks.workunit.client.0.smithi112.stdout: create 177/31 file after the snapshot 2023-01-24T03:33:30.918 INFO:tasks.workunit.client.0.smithi112.stdout: create 177/32 file after the snapshot 2023-01-24T03:33:30.919 INFO:tasks.workunit.client.0.smithi112.stdout: create 177/33 file after the snapshot 2023-01-24T03:33:30.919 INFO:tasks.workunit.client.0.smithi112.stdout: create 177/34 file after the snapshot 2023-01-24T03:33:30.920 INFO:tasks.workunit.client.0.smithi112.stdout: create 177/35 file after the snapshot 2023-01-24T03:33:30.921 INFO:tasks.workunit.client.0.smithi112.stdout: create 177/36 file after the snapshot 2023-01-24T03:33:30.921 INFO:tasks.workunit.client.0.smithi112.stdout: create 177/37 file after the snapshot 2023-01-24T03:33:30.922 INFO:tasks.workunit.client.0.smithi112.stdout: create 177/38 file after the snapshot 2023-01-24T03:33:30.923 INFO:tasks.workunit.client.0.smithi112.stdout: create 177/39 file after the snapshot 2023-01-24T03:33:30.924 INFO:tasks.workunit.client.0.smithi112.stdout: create 178/.snap/snap-subdir-test 2023-01-24T03:33:30.943 INFO:tasks.workunit.client.0.smithi112.stdout: create 178/30 file after the snapshot 2023-01-24T03:33:30.943 INFO:tasks.workunit.client.0.smithi112.stdout: create 178/31 file after the snapshot 2023-01-24T03:33:30.944 INFO:tasks.workunit.client.0.smithi112.stdout: create 178/32 file after the snapshot 2023-01-24T03:33:30.945 INFO:tasks.workunit.client.0.smithi112.stdout: create 178/33 file after the snapshot 2023-01-24T03:33:30.946 INFO:tasks.workunit.client.0.smithi112.stdout: create 178/34 file after the snapshot 2023-01-24T03:33:30.947 INFO:tasks.workunit.client.0.smithi112.stdout: create 178/35 file after the snapshot 2023-01-24T03:33:30.948 INFO:tasks.workunit.client.0.smithi112.stdout: create 178/36 file after the snapshot 2023-01-24T03:33:30.949 INFO:tasks.workunit.client.0.smithi112.stdout: create 178/37 file after the snapshot 2023-01-24T03:33:30.950 INFO:tasks.workunit.client.0.smithi112.stdout: create 178/38 file after the snapshot 2023-01-24T03:33:30.950 INFO:tasks.workunit.client.0.smithi112.stdout: create 178/39 file after the snapshot 2023-01-24T03:33:30.951 INFO:tasks.workunit.client.0.smithi112.stdout: create 179/.snap/snap-subdir-test 2023-01-24T03:33:30.970 INFO:tasks.workunit.client.0.smithi112.stdout: create 179/30 file after the snapshot 2023-01-24T03:33:30.971 INFO:tasks.workunit.client.0.smithi112.stdout: create 179/31 file after the snapshot 2023-01-24T03:33:30.972 INFO:tasks.workunit.client.0.smithi112.stdout: create 179/32 file after the snapshot 2023-01-24T03:33:30.973 INFO:tasks.workunit.client.0.smithi112.stdout: create 179/33 file after the snapshot 2023-01-24T03:33:30.973 INFO:tasks.workunit.client.0.smithi112.stdout: create 179/34 file after the snapshot 2023-01-24T03:33:30.974 INFO:tasks.workunit.client.0.smithi112.stdout: create 179/35 file after the snapshot 2023-01-24T03:33:30.975 INFO:tasks.workunit.client.0.smithi112.stdout: create 179/36 file after the snapshot 2023-01-24T03:33:30.975 INFO:tasks.workunit.client.0.smithi112.stdout: create 179/37 file after the snapshot 2023-01-24T03:33:30.976 INFO:tasks.workunit.client.0.smithi112.stdout: create 179/38 file after the snapshot 2023-01-24T03:33:30.977 INFO:tasks.workunit.client.0.smithi112.stdout: create 179/39 file after the snapshot 2023-01-24T03:33:30.977 INFO:tasks.workunit.client.0.smithi112.stdout: create 180/.snap/snap-subdir-test 2023-01-24T03:33:30.996 INFO:tasks.workunit.client.0.smithi112.stdout: create 180/30 file after the snapshot 2023-01-24T03:33:30.997 INFO:tasks.workunit.client.0.smithi112.stdout: create 180/31 file after the snapshot 2023-01-24T03:33:30.998 INFO:tasks.workunit.client.0.smithi112.stdout: create 180/32 file after the snapshot 2023-01-24T03:33:30.999 INFO:tasks.workunit.client.0.smithi112.stdout: create 180/33 file after the snapshot 2023-01-24T03:33:31.000 INFO:tasks.workunit.client.0.smithi112.stdout: create 180/34 file after the snapshot 2023-01-24T03:33:31.000 INFO:tasks.workunit.client.0.smithi112.stdout: create 180/35 file after the snapshot 2023-01-24T03:33:31.001 INFO:tasks.workunit.client.0.smithi112.stdout: create 180/36 file after the snapshot 2023-01-24T03:33:31.002 INFO:tasks.workunit.client.0.smithi112.stdout: create 180/37 file after the snapshot 2023-01-24T03:33:31.002 INFO:tasks.workunit.client.0.smithi112.stdout: create 180/38 file after the snapshot 2023-01-24T03:33:31.003 INFO:tasks.workunit.client.0.smithi112.stdout: create 180/39 file after the snapshot 2023-01-24T03:33:31.004 INFO:tasks.workunit.client.0.smithi112.stdout: create 181/.snap/snap-subdir-test 2023-01-24T03:33:31.023 INFO:tasks.workunit.client.0.smithi112.stdout: create 181/30 file after the snapshot 2023-01-24T03:33:31.024 INFO:tasks.workunit.client.0.smithi112.stdout: create 181/31 file after the snapshot 2023-01-24T03:33:31.025 INFO:tasks.workunit.client.0.smithi112.stdout: create 181/32 file after the snapshot 2023-01-24T03:33:31.026 INFO:tasks.workunit.client.0.smithi112.stdout: create 181/33 file after the snapshot 2023-01-24T03:33:31.027 INFO:tasks.workunit.client.0.smithi112.stdout: create 181/34 file after the snapshot 2023-01-24T03:33:31.028 INFO:tasks.workunit.client.0.smithi112.stdout: create 181/35 file after the snapshot 2023-01-24T03:33:31.029 INFO:tasks.workunit.client.0.smithi112.stdout: create 181/36 file after the snapshot 2023-01-24T03:33:31.030 INFO:tasks.workunit.client.0.smithi112.stdout: create 181/37 file after the snapshot 2023-01-24T03:33:31.031 INFO:tasks.workunit.client.0.smithi112.stdout: create 181/38 file after the snapshot 2023-01-24T03:33:31.032 INFO:tasks.workunit.client.0.smithi112.stdout: create 181/39 file after the snapshot 2023-01-24T03:33:31.033 INFO:tasks.workunit.client.0.smithi112.stdout: create 182/.snap/snap-subdir-test 2023-01-24T03:33:31.052 INFO:tasks.workunit.client.0.smithi112.stdout: create 182/30 file after the snapshot 2023-01-24T03:33:31.053 INFO:tasks.workunit.client.0.smithi112.stdout: create 182/31 file after the snapshot 2023-01-24T03:33:31.055 INFO:tasks.workunit.client.0.smithi112.stdout: create 182/32 file after the snapshot 2023-01-24T03:33:31.055 INFO:tasks.workunit.client.0.smithi112.stdout: create 182/33 file after the snapshot 2023-01-24T03:33:31.056 INFO:tasks.workunit.client.0.smithi112.stdout: create 182/34 file after the snapshot 2023-01-24T03:33:31.057 INFO:tasks.workunit.client.0.smithi112.stdout: create 182/35 file after the snapshot 2023-01-24T03:33:31.058 INFO:tasks.workunit.client.0.smithi112.stdout: create 182/36 file after the snapshot 2023-01-24T03:33:31.059 INFO:tasks.workunit.client.0.smithi112.stdout: create 182/37 file after the snapshot 2023-01-24T03:33:31.060 INFO:tasks.workunit.client.0.smithi112.stdout: create 182/38 file after the snapshot 2023-01-24T03:33:31.061 INFO:tasks.workunit.client.0.smithi112.stdout: create 182/39 file after the snapshot 2023-01-24T03:33:31.062 INFO:tasks.workunit.client.0.smithi112.stdout: create 183/.snap/snap-subdir-test 2023-01-24T03:33:31.082 INFO:tasks.workunit.client.0.smithi112.stdout: create 183/30 file after the snapshot 2023-01-24T03:33:31.083 INFO:tasks.workunit.client.0.smithi112.stdout: create 183/31 file after the snapshot 2023-01-24T03:33:31.084 INFO:tasks.workunit.client.0.smithi112.stdout: create 183/32 file after the snapshot 2023-01-24T03:33:31.084 INFO:tasks.workunit.client.0.smithi112.stdout: create 183/33 file after the snapshot 2023-01-24T03:33:31.085 INFO:tasks.workunit.client.0.smithi112.stdout: create 183/34 file after the snapshot 2023-01-24T03:33:31.086 INFO:tasks.workunit.client.0.smithi112.stdout: create 183/35 file after the snapshot 2023-01-24T03:33:31.086 INFO:tasks.workunit.client.0.smithi112.stdout: create 183/36 file after the snapshot 2023-01-24T03:33:31.087 INFO:tasks.workunit.client.0.smithi112.stdout: create 183/37 file after the snapshot 2023-01-24T03:33:31.088 INFO:tasks.workunit.client.0.smithi112.stdout: create 183/38 file after the snapshot 2023-01-24T03:33:31.089 INFO:tasks.workunit.client.0.smithi112.stdout: create 183/39 file after the snapshot 2023-01-24T03:33:31.089 INFO:tasks.workunit.client.0.smithi112.stdout: create 184/.snap/snap-subdir-test 2023-01-24T03:33:31.108 INFO:tasks.workunit.client.0.smithi112.stdout: create 184/30 file after the snapshot 2023-01-24T03:33:31.109 INFO:tasks.workunit.client.0.smithi112.stdout: create 184/31 file after the snapshot 2023-01-24T03:33:31.110 INFO:tasks.workunit.client.0.smithi112.stdout: create 184/32 file after the snapshot 2023-01-24T03:33:31.111 INFO:tasks.workunit.client.0.smithi112.stdout: create 184/33 file after the snapshot 2023-01-24T03:33:31.112 INFO:tasks.workunit.client.0.smithi112.stdout: create 184/34 file after the snapshot 2023-01-24T03:33:31.113 INFO:tasks.workunit.client.0.smithi112.stdout: create 184/35 file after the snapshot 2023-01-24T03:33:31.114 INFO:tasks.workunit.client.0.smithi112.stdout: create 184/36 file after the snapshot 2023-01-24T03:33:31.114 INFO:tasks.workunit.client.0.smithi112.stdout: create 184/37 file after the snapshot 2023-01-24T03:33:31.115 INFO:tasks.workunit.client.0.smithi112.stdout: create 184/38 file after the snapshot 2023-01-24T03:33:31.116 INFO:tasks.workunit.client.0.smithi112.stdout: create 184/39 file after the snapshot 2023-01-24T03:33:31.117 INFO:tasks.workunit.client.0.smithi112.stdout: create 185/.snap/snap-subdir-test 2023-01-24T03:33:31.137 INFO:tasks.workunit.client.0.smithi112.stdout: create 185/30 file after the snapshot 2023-01-24T03:33:31.138 INFO:tasks.workunit.client.0.smithi112.stdout: create 185/31 file after the snapshot 2023-01-24T03:33:31.139 INFO:tasks.workunit.client.0.smithi112.stdout: create 185/32 file after the snapshot 2023-01-24T03:33:31.140 INFO:tasks.workunit.client.0.smithi112.stdout: create 185/33 file after the snapshot 2023-01-24T03:33:31.141 INFO:tasks.workunit.client.0.smithi112.stdout: create 185/34 file after the snapshot 2023-01-24T03:33:31.142 INFO:tasks.workunit.client.0.smithi112.stdout: create 185/35 file after the snapshot 2023-01-24T03:33:31.143 INFO:tasks.workunit.client.0.smithi112.stdout: create 185/36 file after the snapshot 2023-01-24T03:33:31.143 INFO:tasks.workunit.client.0.smithi112.stdout: create 185/37 file after the snapshot 2023-01-24T03:33:31.144 INFO:tasks.workunit.client.0.smithi112.stdout: create 185/38 file after the snapshot 2023-01-24T03:33:31.145 INFO:tasks.workunit.client.0.smithi112.stdout: create 185/39 file after the snapshot 2023-01-24T03:33:31.146 INFO:tasks.workunit.client.0.smithi112.stdout: create 186/.snap/snap-subdir-test 2023-01-24T03:33:31.165 INFO:tasks.workunit.client.0.smithi112.stdout: create 186/30 file after the snapshot 2023-01-24T03:33:31.166 INFO:tasks.workunit.client.0.smithi112.stdout: create 186/31 file after the snapshot 2023-01-24T03:33:31.167 INFO:tasks.workunit.client.0.smithi112.stdout: create 186/32 file after the snapshot 2023-01-24T03:33:31.168 INFO:tasks.workunit.client.0.smithi112.stdout: create 186/33 file after the snapshot 2023-01-24T03:33:31.169 INFO:tasks.workunit.client.0.smithi112.stdout: create 186/34 file after the snapshot 2023-01-24T03:33:31.170 INFO:tasks.workunit.client.0.smithi112.stdout: create 186/35 file after the snapshot 2023-01-24T03:33:31.171 INFO:tasks.workunit.client.0.smithi112.stdout: create 186/36 file after the snapshot 2023-01-24T03:33:31.172 INFO:tasks.workunit.client.0.smithi112.stdout: create 186/37 file after the snapshot 2023-01-24T03:33:31.173 INFO:tasks.workunit.client.0.smithi112.stdout: create 186/38 file after the snapshot 2023-01-24T03:33:31.174 INFO:tasks.workunit.client.0.smithi112.stdout: create 186/39 file after the snapshot 2023-01-24T03:33:31.175 INFO:tasks.workunit.client.0.smithi112.stdout: create 187/.snap/snap-subdir-test 2023-01-24T03:33:31.195 INFO:tasks.workunit.client.0.smithi112.stdout: create 187/30 file after the snapshot 2023-01-24T03:33:31.196 INFO:tasks.workunit.client.0.smithi112.stdout: create 187/31 file after the snapshot 2023-01-24T03:33:31.197 INFO:tasks.workunit.client.0.smithi112.stdout: create 187/32 file after the snapshot 2023-01-24T03:33:31.198 INFO:tasks.workunit.client.0.smithi112.stdout: create 187/33 file after the snapshot 2023-01-24T03:33:31.199 INFO:tasks.workunit.client.0.smithi112.stdout: create 187/34 file after the snapshot 2023-01-24T03:33:31.200 INFO:tasks.workunit.client.0.smithi112.stdout: create 187/35 file after the snapshot 2023-01-24T03:33:31.200 INFO:tasks.workunit.client.0.smithi112.stdout: create 187/36 file after the snapshot 2023-01-24T03:33:31.201 INFO:tasks.workunit.client.0.smithi112.stdout: create 187/37 file after the snapshot 2023-01-24T03:33:31.202 INFO:tasks.workunit.client.0.smithi112.stdout: create 187/38 file after the snapshot 2023-01-24T03:33:31.203 INFO:tasks.workunit.client.0.smithi112.stdout: create 187/39 file after the snapshot 2023-01-24T03:33:31.203 INFO:tasks.workunit.client.0.smithi112.stdout: create 188/.snap/snap-subdir-test 2023-01-24T03:33:31.222 INFO:tasks.workunit.client.0.smithi112.stdout: create 188/30 file after the snapshot 2023-01-24T03:33:31.223 INFO:tasks.workunit.client.0.smithi112.stdout: create 188/31 file after the snapshot 2023-01-24T03:33:31.224 INFO:tasks.workunit.client.0.smithi112.stdout: create 188/32 file after the snapshot 2023-01-24T03:33:31.225 INFO:tasks.workunit.client.0.smithi112.stdout: create 188/33 file after the snapshot 2023-01-24T03:33:31.226 INFO:tasks.workunit.client.0.smithi112.stdout: create 188/34 file after the snapshot 2023-01-24T03:33:31.227 INFO:tasks.workunit.client.0.smithi112.stdout: create 188/35 file after the snapshot 2023-01-24T03:33:31.227 INFO:tasks.workunit.client.0.smithi112.stdout: create 188/36 file after the snapshot 2023-01-24T03:33:31.228 INFO:tasks.workunit.client.0.smithi112.stdout: create 188/37 file after the snapshot 2023-01-24T03:33:31.229 INFO:tasks.workunit.client.0.smithi112.stdout: create 188/38 file after the snapshot 2023-01-24T03:33:31.230 INFO:tasks.workunit.client.0.smithi112.stdout: create 188/39 file after the snapshot 2023-01-24T03:33:31.231 INFO:tasks.workunit.client.0.smithi112.stdout: create 189/.snap/snap-subdir-test 2023-01-24T03:33:31.250 INFO:tasks.workunit.client.0.smithi112.stdout: create 189/30 file after the snapshot 2023-01-24T03:33:31.251 INFO:tasks.workunit.client.0.smithi112.stdout: create 189/31 file after the snapshot 2023-01-24T03:33:31.252 INFO:tasks.workunit.client.0.smithi112.stdout: create 189/32 file after the snapshot 2023-01-24T03:33:31.253 INFO:tasks.workunit.client.0.smithi112.stdout: create 189/33 file after the snapshot 2023-01-24T03:33:31.254 INFO:tasks.workunit.client.0.smithi112.stdout: create 189/34 file after the snapshot 2023-01-24T03:33:31.255 INFO:tasks.workunit.client.0.smithi112.stdout: create 189/35 file after the snapshot 2023-01-24T03:33:31.256 INFO:tasks.workunit.client.0.smithi112.stdout: create 189/36 file after the snapshot 2023-01-24T03:33:31.257 INFO:tasks.workunit.client.0.smithi112.stdout: create 189/37 file after the snapshot 2023-01-24T03:33:31.258 INFO:tasks.workunit.client.0.smithi112.stdout: create 189/38 file after the snapshot 2023-01-24T03:33:31.259 INFO:tasks.workunit.client.0.smithi112.stdout: create 189/39 file after the snapshot 2023-01-24T03:33:31.259 INFO:tasks.workunit.client.0.smithi112.stdout: create 190/.snap/snap-subdir-test 2023-01-24T03:33:31.279 INFO:tasks.workunit.client.0.smithi112.stdout: create 190/30 file after the snapshot 2023-01-24T03:33:31.280 INFO:tasks.workunit.client.0.smithi112.stdout: create 190/31 file after the snapshot 2023-01-24T03:33:31.281 INFO:tasks.workunit.client.0.smithi112.stdout: create 190/32 file after the snapshot 2023-01-24T03:33:31.281 INFO:tasks.workunit.client.0.smithi112.stdout: create 190/33 file after the snapshot 2023-01-24T03:33:31.282 INFO:tasks.workunit.client.0.smithi112.stdout: create 190/34 file after the snapshot 2023-01-24T03:33:31.283 INFO:tasks.workunit.client.0.smithi112.stdout: create 190/35 file after the snapshot 2023-01-24T03:33:31.284 INFO:tasks.workunit.client.0.smithi112.stdout: create 190/36 file after the snapshot 2023-01-24T03:33:31.285 INFO:tasks.workunit.client.0.smithi112.stdout: create 190/37 file after the snapshot 2023-01-24T03:33:31.286 INFO:tasks.workunit.client.0.smithi112.stdout: create 190/38 file after the snapshot 2023-01-24T03:33:31.286 INFO:tasks.workunit.client.0.smithi112.stdout: create 190/39 file after the snapshot 2023-01-24T03:33:31.287 INFO:tasks.workunit.client.0.smithi112.stdout: create 191/.snap/snap-subdir-test 2023-01-24T03:33:31.306 INFO:tasks.workunit.client.0.smithi112.stdout: create 191/30 file after the snapshot 2023-01-24T03:33:31.307 INFO:tasks.workunit.client.0.smithi112.stdout: create 191/31 file after the snapshot 2023-01-24T03:33:31.308 INFO:tasks.workunit.client.0.smithi112.stdout: create 191/32 file after the snapshot 2023-01-24T03:33:31.309 INFO:tasks.workunit.client.0.smithi112.stdout: create 191/33 file after the snapshot 2023-01-24T03:33:31.310 INFO:tasks.workunit.client.0.smithi112.stdout: create 191/34 file after the snapshot 2023-01-24T03:33:31.311 INFO:tasks.workunit.client.0.smithi112.stdout: create 191/35 file after the snapshot 2023-01-24T03:33:31.312 INFO:tasks.workunit.client.0.smithi112.stdout: create 191/36 file after the snapshot 2023-01-24T03:33:31.313 INFO:tasks.workunit.client.0.smithi112.stdout: create 191/37 file after the snapshot 2023-01-24T03:33:31.314 INFO:tasks.workunit.client.0.smithi112.stdout: create 191/38 file after the snapshot 2023-01-24T03:33:31.314 INFO:tasks.workunit.client.0.smithi112.stdout: create 191/39 file after the snapshot 2023-01-24T03:33:31.315 INFO:tasks.workunit.client.0.smithi112.stdout: create 192/.snap/snap-subdir-test 2023-01-24T03:33:31.334 INFO:tasks.workunit.client.0.smithi112.stdout: create 192/30 file after the snapshot 2023-01-24T03:33:31.336 INFO:tasks.workunit.client.0.smithi112.stdout: create 192/31 file after the snapshot 2023-01-24T03:33:31.337 INFO:tasks.workunit.client.0.smithi112.stdout: create 192/32 file after the snapshot 2023-01-24T03:33:31.338 INFO:tasks.workunit.client.0.smithi112.stdout: create 192/33 file after the snapshot 2023-01-24T03:33:31.339 INFO:tasks.workunit.client.0.smithi112.stdout: create 192/34 file after the snapshot 2023-01-24T03:33:31.340 INFO:tasks.workunit.client.0.smithi112.stdout: create 192/35 file after the snapshot 2023-01-24T03:33:31.342 INFO:tasks.workunit.client.0.smithi112.stdout: create 192/36 file after the snapshot 2023-01-24T03:33:31.343 INFO:tasks.workunit.client.0.smithi112.stdout: create 192/37 file after the snapshot 2023-01-24T03:33:31.344 INFO:tasks.workunit.client.0.smithi112.stdout: create 192/38 file after the snapshot 2023-01-24T03:33:31.345 INFO:tasks.workunit.client.0.smithi112.stdout: create 192/39 file after the snapshot 2023-01-24T03:33:31.346 INFO:tasks.workunit.client.0.smithi112.stdout: create 193/.snap/snap-subdir-test 2023-01-24T03:33:31.366 INFO:tasks.workunit.client.0.smithi112.stdout: create 193/30 file after the snapshot 2023-01-24T03:33:31.367 INFO:tasks.workunit.client.0.smithi112.stdout: create 193/31 file after the snapshot 2023-01-24T03:33:31.368 INFO:tasks.workunit.client.0.smithi112.stdout: create 193/32 file after the snapshot 2023-01-24T03:33:31.369 INFO:tasks.workunit.client.0.smithi112.stdout: create 193/33 file after the snapshot 2023-01-24T03:33:31.369 INFO:tasks.workunit.client.0.smithi112.stdout: create 193/34 file after the snapshot 2023-01-24T03:33:31.370 INFO:tasks.workunit.client.0.smithi112.stdout: create 193/35 file after the snapshot 2023-01-24T03:33:31.371 INFO:tasks.workunit.client.0.smithi112.stdout: create 193/36 file after the snapshot 2023-01-24T03:33:31.372 INFO:tasks.workunit.client.0.smithi112.stdout: create 193/37 file after the snapshot 2023-01-24T03:33:31.372 INFO:tasks.workunit.client.0.smithi112.stdout: create 193/38 file after the snapshot 2023-01-24T03:33:31.373 INFO:tasks.workunit.client.0.smithi112.stdout: create 193/39 file after the snapshot 2023-01-24T03:33:31.374 INFO:tasks.workunit.client.0.smithi112.stdout: create 194/.snap/snap-subdir-test 2023-01-24T03:33:31.393 INFO:tasks.workunit.client.0.smithi112.stdout: create 194/30 file after the snapshot 2023-01-24T03:33:31.394 INFO:tasks.workunit.client.0.smithi112.stdout: create 194/31 file after the snapshot 2023-01-24T03:33:31.395 INFO:tasks.workunit.client.0.smithi112.stdout: create 194/32 file after the snapshot 2023-01-24T03:33:31.396 INFO:tasks.workunit.client.0.smithi112.stdout: create 194/33 file after the snapshot 2023-01-24T03:33:31.397 INFO:tasks.workunit.client.0.smithi112.stdout: create 194/34 file after the snapshot 2023-01-24T03:33:31.397 INFO:tasks.workunit.client.0.smithi112.stdout: create 194/35 file after the snapshot 2023-01-24T03:33:31.398 INFO:tasks.workunit.client.0.smithi112.stdout: create 194/36 file after the snapshot 2023-01-24T03:33:31.399 INFO:tasks.workunit.client.0.smithi112.stdout: create 194/37 file after the snapshot 2023-01-24T03:33:31.400 INFO:tasks.workunit.client.0.smithi112.stdout: create 194/38 file after the snapshot 2023-01-24T03:33:31.400 INFO:tasks.workunit.client.0.smithi112.stdout: create 194/39 file after the snapshot 2023-01-24T03:33:31.401 INFO:tasks.workunit.client.0.smithi112.stdout: create 195/.snap/snap-subdir-test 2023-01-24T03:33:31.420 INFO:tasks.workunit.client.0.smithi112.stdout: create 195/30 file after the snapshot 2023-01-24T03:33:31.421 INFO:tasks.workunit.client.0.smithi112.stdout: create 195/31 file after the snapshot 2023-01-24T03:33:31.422 INFO:tasks.workunit.client.0.smithi112.stdout: create 195/32 file after the snapshot 2023-01-24T03:33:31.423 INFO:tasks.workunit.client.0.smithi112.stdout: create 195/33 file after the snapshot 2023-01-24T03:33:31.424 INFO:tasks.workunit.client.0.smithi112.stdout: create 195/34 file after the snapshot 2023-01-24T03:33:31.425 INFO:tasks.workunit.client.0.smithi112.stdout: create 195/35 file after the snapshot 2023-01-24T03:33:31.425 INFO:tasks.workunit.client.0.smithi112.stdout: create 195/36 file after the snapshot 2023-01-24T03:33:31.426 INFO:tasks.workunit.client.0.smithi112.stdout: create 195/37 file after the snapshot 2023-01-24T03:33:31.427 INFO:tasks.workunit.client.0.smithi112.stdout: create 195/38 file after the snapshot 2023-01-24T03:33:31.427 INFO:tasks.workunit.client.0.smithi112.stdout: create 195/39 file after the snapshot 2023-01-24T03:33:31.428 INFO:tasks.workunit.client.0.smithi112.stdout: create 196/.snap/snap-subdir-test 2023-01-24T03:33:31.448 INFO:tasks.workunit.client.0.smithi112.stdout: create 196/30 file after the snapshot 2023-01-24T03:33:31.449 INFO:tasks.workunit.client.0.smithi112.stdout: create 196/31 file after the snapshot 2023-01-24T03:33:31.450 INFO:tasks.workunit.client.0.smithi112.stdout: create 196/32 file after the snapshot 2023-01-24T03:33:31.450 INFO:tasks.workunit.client.0.smithi112.stdout: create 196/33 file after the snapshot 2023-01-24T03:33:31.451 INFO:tasks.workunit.client.0.smithi112.stdout: create 196/34 file after the snapshot 2023-01-24T03:33:31.452 INFO:tasks.workunit.client.0.smithi112.stdout: create 196/35 file after the snapshot 2023-01-24T03:33:31.453 INFO:tasks.workunit.client.0.smithi112.stdout: create 196/36 file after the snapshot 2023-01-24T03:33:31.454 INFO:tasks.workunit.client.0.smithi112.stdout: create 196/37 file after the snapshot 2023-01-24T03:33:31.454 INFO:tasks.workunit.client.0.smithi112.stdout: create 196/38 file after the snapshot 2023-01-24T03:33:31.455 INFO:tasks.workunit.client.0.smithi112.stdout: create 196/39 file after the snapshot 2023-01-24T03:33:31.456 INFO:tasks.workunit.client.0.smithi112.stdout: create 197/.snap/snap-subdir-test 2023-01-24T03:33:31.475 INFO:tasks.workunit.client.0.smithi112.stdout: create 197/30 file after the snapshot 2023-01-24T03:33:31.476 INFO:tasks.workunit.client.0.smithi112.stdout: create 197/31 file after the snapshot 2023-01-24T03:33:31.476 INFO:tasks.workunit.client.0.smithi112.stdout: create 197/32 file after the snapshot 2023-01-24T03:33:31.477 INFO:tasks.workunit.client.0.smithi112.stdout: create 197/33 file after the snapshot 2023-01-24T03:33:31.478 INFO:tasks.workunit.client.0.smithi112.stdout: create 197/34 file after the snapshot 2023-01-24T03:33:31.480 INFO:tasks.workunit.client.0.smithi112.stdout: create 197/35 file after the snapshot 2023-01-24T03:33:31.481 INFO:tasks.workunit.client.0.smithi112.stdout: create 197/36 file after the snapshot 2023-01-24T03:33:31.481 INFO:tasks.workunit.client.0.smithi112.stdout: create 197/37 file after the snapshot 2023-01-24T03:33:31.483 INFO:tasks.workunit.client.0.smithi112.stdout: create 197/38 file after the snapshot 2023-01-24T03:33:31.484 INFO:tasks.workunit.client.0.smithi112.stdout: create 197/39 file after the snapshot 2023-01-24T03:33:31.485 INFO:tasks.workunit.client.0.smithi112.stdout: create 198/.snap/snap-subdir-test 2023-01-24T03:33:31.505 INFO:tasks.workunit.client.0.smithi112.stdout: create 198/30 file after the snapshot 2023-01-24T03:33:31.507 INFO:tasks.workunit.client.0.smithi112.stdout: create 198/31 file after the snapshot 2023-01-24T03:33:31.508 INFO:tasks.workunit.client.0.smithi112.stdout: create 198/32 file after the snapshot 2023-01-24T03:33:31.509 INFO:tasks.workunit.client.0.smithi112.stdout: create 198/33 file after the snapshot 2023-01-24T03:33:31.510 INFO:tasks.workunit.client.0.smithi112.stdout: create 198/34 file after the snapshot 2023-01-24T03:33:31.511 INFO:tasks.workunit.client.0.smithi112.stdout: create 198/35 file after the snapshot 2023-01-24T03:33:31.511 INFO:tasks.workunit.client.0.smithi112.stdout: create 198/36 file after the snapshot 2023-01-24T03:33:31.512 INFO:tasks.workunit.client.0.smithi112.stdout: create 198/37 file after the snapshot 2023-01-24T03:33:31.513 INFO:tasks.workunit.client.0.smithi112.stdout: create 198/38 file after the snapshot 2023-01-24T03:33:31.514 INFO:tasks.workunit.client.0.smithi112.stdout: create 198/39 file after the snapshot 2023-01-24T03:33:31.515 INFO:tasks.workunit.client.0.smithi112.stdout: create 199/.snap/snap-subdir-test 2023-01-24T03:33:31.535 INFO:tasks.workunit.client.0.smithi112.stdout: create 199/30 file after the snapshot 2023-01-24T03:33:31.536 INFO:tasks.workunit.client.0.smithi112.stdout: create 199/31 file after the snapshot 2023-01-24T03:33:31.537 INFO:tasks.workunit.client.0.smithi112.stdout: create 199/32 file after the snapshot 2023-01-24T03:33:31.538 INFO:tasks.workunit.client.0.smithi112.stdout: create 199/33 file after the snapshot 2023-01-24T03:33:31.539 INFO:tasks.workunit.client.0.smithi112.stdout: create 199/34 file after the snapshot 2023-01-24T03:33:31.540 INFO:tasks.workunit.client.0.smithi112.stdout: create 199/35 file after the snapshot 2023-01-24T03:33:31.541 INFO:tasks.workunit.client.0.smithi112.stdout: create 199/36 file after the snapshot 2023-01-24T03:33:31.542 INFO:tasks.workunit.client.0.smithi112.stdout: create 199/37 file after the snapshot 2023-01-24T03:33:31.543 INFO:tasks.workunit.client.0.smithi112.stdout: create 199/38 file after the snapshot 2023-01-24T03:33:31.544 INFO:tasks.workunit.client.0.smithi112.stdout: create 199/39 file after the snapshot 2023-01-24T03:33:31.545 INFO:tasks.workunit.client.0.smithi112.stdout: create 200/.snap/snap-subdir-test 2023-01-24T03:33:31.565 INFO:tasks.workunit.client.0.smithi112.stdout: create 200/30 file after the snapshot 2023-01-24T03:33:31.567 INFO:tasks.workunit.client.0.smithi112.stdout: create 200/31 file after the snapshot 2023-01-24T03:33:31.568 INFO:tasks.workunit.client.0.smithi112.stdout: create 200/32 file after the snapshot 2023-01-24T03:33:32.340 INFO:tasks.workunit.client.0.smithi112.stdout: create 200/33 file after the snapshot 2023-01-24T03:33:32.506 INFO:tasks.workunit.client.0.smithi112.stdout: create 200/34 file after the snapshot 2023-01-24T03:33:32.507 INFO:tasks.workunit.client.0.smithi112.stdout: create 200/35 file after the snapshot 2023-01-24T03:33:32.508 INFO:tasks.workunit.client.0.smithi112.stdout: create 200/36 file after the snapshot 2023-01-24T03:33:32.509 INFO:tasks.workunit.client.0.smithi112.stdout: create 200/37 file after the snapshot 2023-01-24T03:33:32.511 INFO:tasks.workunit.client.0.smithi112.stdout: create 200/38 file after the snapshot 2023-01-24T03:33:32.512 INFO:tasks.workunit.client.0.smithi112.stdout: create 200/39 file after the snapshot 2023-01-24T03:33:32.513 INFO:tasks.workunit.client.0.smithi112.stdout: create 201/.snap/snap-subdir-test 2023-01-24T03:33:32.537 INFO:tasks.workunit.client.0.smithi112.stdout: create 201/30 file after the snapshot 2023-01-24T03:33:32.539 INFO:tasks.workunit.client.0.smithi112.stdout: create 201/31 file after the snapshot 2023-01-24T03:33:32.540 INFO:tasks.workunit.client.0.smithi112.stdout: create 201/32 file after the snapshot 2023-01-24T03:33:32.541 INFO:tasks.workunit.client.0.smithi112.stdout: create 201/33 file after the snapshot 2023-01-24T03:33:32.542 INFO:tasks.workunit.client.0.smithi112.stdout: create 201/34 file after the snapshot 2023-01-24T03:33:32.544 INFO:tasks.workunit.client.0.smithi112.stdout: create 201/35 file after the snapshot 2023-01-24T03:33:32.545 INFO:tasks.workunit.client.0.smithi112.stdout: create 201/36 file after the snapshot 2023-01-24T03:33:32.546 INFO:tasks.workunit.client.0.smithi112.stdout: create 201/37 file after the snapshot 2023-01-24T03:33:32.547 INFO:tasks.workunit.client.0.smithi112.stdout: create 201/38 file after the snapshot 2023-01-24T03:33:32.548 INFO:tasks.workunit.client.0.smithi112.stdout: create 201/39 file after the snapshot 2023-01-24T03:33:32.550 INFO:tasks.workunit.client.0.smithi112.stdout: create 202/.snap/snap-subdir-test 2023-01-24T03:33:32.571 INFO:tasks.workunit.client.0.smithi112.stdout: create 202/30 file after the snapshot 2023-01-24T03:33:32.573 INFO:tasks.workunit.client.0.smithi112.stdout: create 202/31 file after the snapshot 2023-01-24T03:33:32.574 INFO:tasks.workunit.client.0.smithi112.stdout: create 202/32 file after the snapshot 2023-01-24T03:33:32.575 INFO:tasks.workunit.client.0.smithi112.stdout: create 202/33 file after the snapshot 2023-01-24T03:33:32.575 INFO:tasks.workunit.client.0.smithi112.stdout: create 202/34 file after the snapshot 2023-01-24T03:33:32.576 INFO:tasks.workunit.client.0.smithi112.stdout: create 202/35 file after the snapshot 2023-01-24T03:33:32.577 INFO:tasks.workunit.client.0.smithi112.stdout: create 202/36 file after the snapshot 2023-01-24T03:33:32.578 INFO:tasks.workunit.client.0.smithi112.stdout: create 202/37 file after the snapshot 2023-01-24T03:33:32.579 INFO:tasks.workunit.client.0.smithi112.stdout: create 202/38 file after the snapshot 2023-01-24T03:33:32.580 INFO:tasks.workunit.client.0.smithi112.stdout: create 202/39 file after the snapshot 2023-01-24T03:33:32.581 INFO:tasks.workunit.client.0.smithi112.stdout: create 203/.snap/snap-subdir-test 2023-01-24T03:33:32.602 INFO:tasks.workunit.client.0.smithi112.stdout: create 203/30 file after the snapshot 2023-01-24T03:33:32.603 INFO:tasks.workunit.client.0.smithi112.stdout: create 203/31 file after the snapshot 2023-01-24T03:33:32.604 INFO:tasks.workunit.client.0.smithi112.stdout: create 203/32 file after the snapshot 2023-01-24T03:33:32.605 INFO:tasks.workunit.client.0.smithi112.stdout: create 203/33 file after the snapshot 2023-01-24T03:33:32.606 INFO:tasks.workunit.client.0.smithi112.stdout: create 203/34 file after the snapshot 2023-01-24T03:33:32.607 INFO:tasks.workunit.client.0.smithi112.stdout: create 203/35 file after the snapshot 2023-01-24T03:33:32.608 INFO:tasks.workunit.client.0.smithi112.stdout: create 203/36 file after the snapshot 2023-01-24T03:33:32.609 INFO:tasks.workunit.client.0.smithi112.stdout: create 203/37 file after the snapshot 2023-01-24T03:33:32.610 INFO:tasks.workunit.client.0.smithi112.stdout: create 203/38 file after the snapshot 2023-01-24T03:33:32.611 INFO:tasks.workunit.client.0.smithi112.stdout: create 203/39 file after the snapshot 2023-01-24T03:33:32.612 INFO:tasks.workunit.client.0.smithi112.stdout: create 204/.snap/snap-subdir-test 2023-01-24T03:33:32.632 INFO:tasks.workunit.client.0.smithi112.stdout: create 204/30 file after the snapshot 2023-01-24T03:33:32.892 INFO:tasks.workunit.client.0.smithi112.stdout: create 204/31 file after the snapshot 2023-01-24T03:33:32.892 INFO:tasks.workunit.client.0.smithi112.stdout: create 204/32 file after the snapshot 2023-01-24T03:33:32.893 INFO:tasks.workunit.client.0.smithi112.stdout: create 204/33 file after the snapshot 2023-01-24T03:33:32.894 INFO:tasks.workunit.client.0.smithi112.stdout: create 204/34 file after the snapshot 2023-01-24T03:33:32.895 INFO:tasks.workunit.client.0.smithi112.stdout: create 204/35 file after the snapshot 2023-01-24T03:33:32.896 INFO:tasks.workunit.client.0.smithi112.stdout: create 204/36 file after the snapshot 2023-01-24T03:33:32.897 INFO:tasks.workunit.client.0.smithi112.stdout: create 204/37 file after the snapshot 2023-01-24T03:33:32.898 INFO:tasks.workunit.client.0.smithi112.stdout: create 204/38 file after the snapshot 2023-01-24T03:33:32.899 INFO:tasks.workunit.client.0.smithi112.stdout: create 204/39 file after the snapshot 2023-01-24T03:33:32.900 INFO:tasks.workunit.client.0.smithi112.stdout: create 205/.snap/snap-subdir-test 2023-01-24T03:33:32.922 INFO:tasks.workunit.client.0.smithi112.stdout: create 205/30 file after the snapshot 2023-01-24T03:33:32.923 INFO:tasks.workunit.client.0.smithi112.stdout: create 205/31 file after the snapshot 2023-01-24T03:33:32.924 INFO:tasks.workunit.client.0.smithi112.stdout: create 205/32 file after the snapshot 2023-01-24T03:33:32.925 INFO:tasks.workunit.client.0.smithi112.stdout: create 205/33 file after the snapshot 2023-01-24T03:33:32.926 INFO:tasks.workunit.client.0.smithi112.stdout: create 205/34 file after the snapshot 2023-01-24T03:33:32.927 INFO:tasks.workunit.client.0.smithi112.stdout: create 205/35 file after the snapshot 2023-01-24T03:33:32.927 INFO:tasks.workunit.client.0.smithi112.stdout: create 205/36 file after the snapshot 2023-01-24T03:33:32.928 INFO:tasks.workunit.client.0.smithi112.stdout: create 205/37 file after the snapshot 2023-01-24T03:33:32.929 INFO:tasks.workunit.client.0.smithi112.stdout: create 205/38 file after the snapshot 2023-01-24T03:33:32.930 INFO:tasks.workunit.client.0.smithi112.stdout: create 205/39 file after the snapshot 2023-01-24T03:33:32.931 INFO:tasks.workunit.client.0.smithi112.stdout: create 206/.snap/snap-subdir-test 2023-01-24T03:33:32.952 INFO:tasks.workunit.client.0.smithi112.stdout: create 206/30 file after the snapshot 2023-01-24T03:33:32.953 INFO:tasks.workunit.client.0.smithi112.stdout: create 206/31 file after the snapshot 2023-01-24T03:33:32.954 INFO:tasks.workunit.client.0.smithi112.stdout: create 206/32 file after the snapshot 2023-01-24T03:33:32.955 INFO:tasks.workunit.client.0.smithi112.stdout: create 206/33 file after the snapshot 2023-01-24T03:33:32.956 INFO:tasks.workunit.client.0.smithi112.stdout: create 206/34 file after the snapshot 2023-01-24T03:33:32.957 INFO:tasks.workunit.client.0.smithi112.stdout: create 206/35 file after the snapshot 2023-01-24T03:33:32.958 INFO:tasks.workunit.client.0.smithi112.stdout: create 206/36 file after the snapshot 2023-01-24T03:33:32.959 INFO:tasks.workunit.client.0.smithi112.stdout: create 206/37 file after the snapshot 2023-01-24T03:33:32.960 INFO:tasks.workunit.client.0.smithi112.stdout: create 206/38 file after the snapshot 2023-01-24T03:33:32.961 INFO:tasks.workunit.client.0.smithi112.stdout: create 206/39 file after the snapshot 2023-01-24T03:33:32.962 INFO:tasks.workunit.client.0.smithi112.stdout: create 207/.snap/snap-subdir-test 2023-01-24T03:33:32.984 INFO:tasks.workunit.client.0.smithi112.stdout: create 207/30 file after the snapshot 2023-01-24T03:33:32.985 INFO:tasks.workunit.client.0.smithi112.stdout: create 207/31 file after the snapshot 2023-01-24T03:33:33.364 INFO:tasks.workunit.client.0.smithi112.stdout: create 207/32 file after the snapshot 2023-01-24T03:33:33.698 INFO:tasks.workunit.client.0.smithi112.stdout: create 207/33 file after the snapshot 2023-01-24T03:33:33.698 INFO:tasks.workunit.client.0.smithi112.stdout: create 207/34 file after the snapshot 2023-01-24T03:33:33.699 INFO:tasks.workunit.client.0.smithi112.stdout: create 207/35 file after the snapshot 2023-01-24T03:33:33.700 INFO:tasks.workunit.client.0.smithi112.stdout: create 207/36 file after the snapshot 2023-01-24T03:33:33.702 INFO:tasks.workunit.client.0.smithi112.stdout: create 207/37 file after the snapshot 2023-01-24T03:33:33.703 INFO:tasks.workunit.client.0.smithi112.stdout: create 207/38 file after the snapshot 2023-01-24T03:33:33.704 INFO:tasks.workunit.client.0.smithi112.stdout: create 207/39 file after the snapshot 2023-01-24T03:33:33.705 INFO:tasks.workunit.client.0.smithi112.stdout: create 208/.snap/snap-subdir-test 2023-01-24T03:33:33.726 INFO:tasks.workunit.client.0.smithi112.stdout: create 208/30 file after the snapshot 2023-01-24T03:33:33.728 INFO:tasks.workunit.client.0.smithi112.stdout: create 208/31 file after the snapshot 2023-01-24T03:33:33.729 INFO:tasks.workunit.client.0.smithi112.stdout: create 208/32 file after the snapshot 2023-01-24T03:33:33.730 INFO:tasks.workunit.client.0.smithi112.stdout: create 208/33 file after the snapshot 2023-01-24T03:33:33.731 INFO:tasks.workunit.client.0.smithi112.stdout: create 208/34 file after the snapshot 2023-01-24T03:33:33.732 INFO:tasks.workunit.client.0.smithi112.stdout: create 208/35 file after the snapshot 2023-01-24T03:33:33.733 INFO:tasks.workunit.client.0.smithi112.stdout: create 208/36 file after the snapshot 2023-01-24T03:33:33.734 INFO:tasks.workunit.client.0.smithi112.stdout: create 208/37 file after the snapshot 2023-01-24T03:33:33.735 INFO:tasks.workunit.client.0.smithi112.stdout: create 208/38 file after the snapshot 2023-01-24T03:33:33.736 INFO:tasks.workunit.client.0.smithi112.stdout: create 208/39 file after the snapshot 2023-01-24T03:33:33.737 INFO:tasks.workunit.client.0.smithi112.stdout: create 209/.snap/snap-subdir-test 2023-01-24T03:33:33.759 INFO:tasks.workunit.client.0.smithi112.stdout: create 209/30 file after the snapshot 2023-01-24T03:33:33.760 INFO:tasks.workunit.client.0.smithi112.stdout: create 209/31 file after the snapshot 2023-01-24T03:33:33.761 INFO:tasks.workunit.client.0.smithi112.stdout: create 209/32 file after the snapshot 2023-01-24T03:33:33.761 INFO:tasks.workunit.client.0.smithi112.stdout: create 209/33 file after the snapshot 2023-01-24T03:33:33.762 INFO:tasks.workunit.client.0.smithi112.stdout: create 209/34 file after the snapshot 2023-01-24T03:33:33.763 INFO:tasks.workunit.client.0.smithi112.stdout: create 209/35 file after the snapshot 2023-01-24T03:33:33.764 INFO:tasks.workunit.client.0.smithi112.stdout: create 209/36 file after the snapshot 2023-01-24T03:33:33.765 INFO:tasks.workunit.client.0.smithi112.stdout: create 209/37 file after the snapshot 2023-01-24T03:33:33.766 INFO:tasks.workunit.client.0.smithi112.stdout: create 209/38 file after the snapshot 2023-01-24T03:33:33.767 INFO:tasks.workunit.client.0.smithi112.stdout: create 209/39 file after the snapshot 2023-01-24T03:33:33.768 INFO:tasks.workunit.client.0.smithi112.stdout: create 210/.snap/snap-subdir-test 2023-01-24T03:33:33.789 INFO:tasks.workunit.client.0.smithi112.stdout: create 210/30 file after the snapshot 2023-01-24T03:33:33.790 INFO:tasks.workunit.client.0.smithi112.stdout: create 210/31 file after the snapshot 2023-01-24T03:33:33.791 INFO:tasks.workunit.client.0.smithi112.stdout: create 210/32 file after the snapshot 2023-01-24T03:33:33.792 INFO:tasks.workunit.client.0.smithi112.stdout: create 210/33 file after the snapshot 2023-01-24T03:33:33.793 INFO:tasks.workunit.client.0.smithi112.stdout: create 210/34 file after the snapshot 2023-01-24T03:33:33.794 INFO:tasks.workunit.client.0.smithi112.stdout: create 210/35 file after the snapshot 2023-01-24T03:33:33.795 INFO:tasks.workunit.client.0.smithi112.stdout: create 210/36 file after the snapshot 2023-01-24T03:33:33.796 INFO:tasks.workunit.client.0.smithi112.stdout: create 210/37 file after the snapshot 2023-01-24T03:33:33.797 INFO:tasks.workunit.client.0.smithi112.stdout: create 210/38 file after the snapshot 2023-01-24T03:33:33.798 INFO:tasks.workunit.client.0.smithi112.stdout: create 210/39 file after the snapshot 2023-01-24T03:33:33.799 INFO:tasks.workunit.client.0.smithi112.stdout: create 211/.snap/snap-subdir-test 2023-01-24T03:33:33.823 INFO:tasks.workunit.client.0.smithi112.stdout: create 211/30 file after the snapshot 2023-01-24T03:33:33.834 INFO:tasks.workunit.client.0.smithi112.stdout: create 211/31 file after the snapshot 2023-01-24T03:33:33.840 INFO:tasks.workunit.client.0.smithi112.stdout: create 211/32 file after the snapshot 2023-01-24T03:33:33.844 INFO:tasks.workunit.client.0.smithi112.stdout: create 211/33 file after the snapshot 2023-01-24T03:33:33.848 INFO:tasks.workunit.client.0.smithi112.stdout: create 211/34 file after the snapshot 2023-01-24T03:33:33.852 INFO:tasks.workunit.client.0.smithi112.stdout: create 211/35 file after the snapshot 2023-01-24T03:33:33.861 INFO:tasks.workunit.client.0.smithi112.stdout: create 211/36 file after the snapshot 2023-01-24T03:33:33.873 INFO:tasks.workunit.client.0.smithi112.stdout: create 211/37 file after the snapshot 2023-01-24T03:33:33.876 INFO:tasks.workunit.client.0.smithi112.stdout: create 211/38 file after the snapshot 2023-01-24T03:33:33.885 INFO:tasks.workunit.client.0.smithi112.stdout: create 211/39 file after the snapshot 2023-01-24T03:33:33.911 INFO:tasks.workunit.client.0.smithi112.stdout: create 212/.snap/snap-subdir-test 2023-01-24T03:33:34.867 INFO:tasks.workunit.client.0.smithi112.stdout: create 212/30 file after the snapshot 2023-01-24T03:33:34.868 INFO:tasks.workunit.client.0.smithi112.stdout: create 212/31 file after the snapshot 2023-01-24T03:33:34.869 INFO:tasks.workunit.client.0.smithi112.stdout: create 212/32 file after the snapshot 2023-01-24T03:33:34.870 INFO:tasks.workunit.client.0.smithi112.stdout: create 212/33 file after the snapshot 2023-01-24T03:33:34.871 INFO:tasks.workunit.client.0.smithi112.stdout: create 212/34 file after the snapshot 2023-01-24T03:33:34.872 INFO:tasks.workunit.client.0.smithi112.stdout: create 212/35 file after the snapshot 2023-01-24T03:33:34.873 INFO:tasks.workunit.client.0.smithi112.stdout: create 212/36 file after the snapshot 2023-01-24T03:33:34.873 INFO:tasks.workunit.client.0.smithi112.stdout: create 212/37 file after the snapshot 2023-01-24T03:33:34.874 INFO:tasks.workunit.client.0.smithi112.stdout: create 212/38 file after the snapshot 2023-01-24T03:33:34.875 INFO:tasks.workunit.client.0.smithi112.stdout: create 212/39 file after the snapshot 2023-01-24T03:33:34.876 INFO:tasks.workunit.client.0.smithi112.stdout: create 213/.snap/snap-subdir-test 2023-01-24T03:33:34.895 INFO:tasks.workunit.client.0.smithi112.stdout: create 213/30 file after the snapshot 2023-01-24T03:33:34.896 INFO:tasks.workunit.client.0.smithi112.stdout: create 213/31 file after the snapshot 2023-01-24T03:33:34.897 INFO:tasks.workunit.client.0.smithi112.stdout: create 213/32 file after the snapshot 2023-01-24T03:33:34.898 INFO:tasks.workunit.client.0.smithi112.stdout: create 213/33 file after the snapshot 2023-01-24T03:33:34.899 INFO:tasks.workunit.client.0.smithi112.stdout: create 213/34 file after the snapshot 2023-01-24T03:33:34.900 INFO:tasks.workunit.client.0.smithi112.stdout: create 213/35 file after the snapshot 2023-01-24T03:33:34.901 INFO:tasks.workunit.client.0.smithi112.stdout: create 213/36 file after the snapshot 2023-01-24T03:33:34.902 INFO:tasks.workunit.client.0.smithi112.stdout: create 213/37 file after the snapshot 2023-01-24T03:33:34.903 INFO:tasks.workunit.client.0.smithi112.stdout: create 213/38 file after the snapshot 2023-01-24T03:33:34.904 INFO:tasks.workunit.client.0.smithi112.stdout: create 213/39 file after the snapshot 2023-01-24T03:33:34.905 INFO:tasks.workunit.client.0.smithi112.stdout: create 214/.snap/snap-subdir-test 2023-01-24T03:33:34.924 INFO:tasks.workunit.client.0.smithi112.stdout: create 214/30 file after the snapshot 2023-01-24T03:33:34.925 INFO:tasks.workunit.client.0.smithi112.stdout: create 214/31 file after the snapshot 2023-01-24T03:33:34.926 INFO:tasks.workunit.client.0.smithi112.stdout: create 214/32 file after the snapshot 2023-01-24T03:33:34.927 INFO:tasks.workunit.client.0.smithi112.stdout: create 214/33 file after the snapshot 2023-01-24T03:33:34.928 INFO:tasks.workunit.client.0.smithi112.stdout: create 214/34 file after the snapshot 2023-01-24T03:33:34.928 INFO:tasks.workunit.client.0.smithi112.stdout: create 214/35 file after the snapshot 2023-01-24T03:33:34.929 INFO:tasks.workunit.client.0.smithi112.stdout: create 214/36 file after the snapshot 2023-01-24T03:33:34.930 INFO:tasks.workunit.client.0.smithi112.stdout: create 214/37 file after the snapshot 2023-01-24T03:33:34.931 INFO:tasks.workunit.client.0.smithi112.stdout: create 214/38 file after the snapshot 2023-01-24T03:33:34.932 INFO:tasks.workunit.client.0.smithi112.stdout: create 214/39 file after the snapshot 2023-01-24T03:33:34.933 INFO:tasks.workunit.client.0.smithi112.stdout: create 215/.snap/snap-subdir-test 2023-01-24T03:33:34.953 INFO:tasks.workunit.client.0.smithi112.stdout: create 215/30 file after the snapshot 2023-01-24T03:33:34.954 INFO:tasks.workunit.client.0.smithi112.stdout: create 215/31 file after the snapshot 2023-01-24T03:33:34.955 INFO:tasks.workunit.client.0.smithi112.stdout: create 215/32 file after the snapshot 2023-01-24T03:33:34.955 INFO:tasks.workunit.client.0.smithi112.stdout: create 215/33 file after the snapshot 2023-01-24T03:33:34.956 INFO:tasks.workunit.client.0.smithi112.stdout: create 215/34 file after the snapshot 2023-01-24T03:33:34.957 INFO:tasks.workunit.client.0.smithi112.stdout: create 215/35 file after the snapshot 2023-01-24T03:33:34.958 INFO:tasks.workunit.client.0.smithi112.stdout: create 215/36 file after the snapshot 2023-01-24T03:33:34.959 INFO:tasks.workunit.client.0.smithi112.stdout: create 215/37 file after the snapshot 2023-01-24T03:33:34.960 INFO:tasks.workunit.client.0.smithi112.stdout: create 215/38 file after the snapshot 2023-01-24T03:33:34.961 INFO:tasks.workunit.client.0.smithi112.stdout: create 215/39 file after the snapshot 2023-01-24T03:33:34.962 INFO:tasks.workunit.client.0.smithi112.stdout: create 216/.snap/snap-subdir-test 2023-01-24T03:33:34.981 INFO:tasks.workunit.client.0.smithi112.stdout: create 216/30 file after the snapshot 2023-01-24T03:33:34.982 INFO:tasks.workunit.client.0.smithi112.stdout: create 216/31 file after the snapshot 2023-01-24T03:33:34.983 INFO:tasks.workunit.client.0.smithi112.stdout: create 216/32 file after the snapshot 2023-01-24T03:33:34.983 INFO:tasks.workunit.client.0.smithi112.stdout: create 216/33 file after the snapshot 2023-01-24T03:33:34.984 INFO:tasks.workunit.client.0.smithi112.stdout: create 216/34 file after the snapshot 2023-01-24T03:33:34.985 INFO:tasks.workunit.client.0.smithi112.stdout: create 216/35 file after the snapshot 2023-01-24T03:33:34.986 INFO:tasks.workunit.client.0.smithi112.stdout: create 216/36 file after the snapshot 2023-01-24T03:33:34.986 INFO:tasks.workunit.client.0.smithi112.stdout: create 216/37 file after the snapshot 2023-01-24T03:33:34.987 INFO:tasks.workunit.client.0.smithi112.stdout: create 216/38 file after the snapshot 2023-01-24T03:33:34.988 INFO:tasks.workunit.client.0.smithi112.stdout: create 216/39 file after the snapshot 2023-01-24T03:33:34.989 INFO:tasks.workunit.client.0.smithi112.stdout: create 217/.snap/snap-subdir-test 2023-01-24T03:33:35.009 INFO:tasks.workunit.client.0.smithi112.stdout: create 217/30 file after the snapshot 2023-01-24T03:33:35.009 INFO:tasks.workunit.client.0.smithi112.stdout: create 217/31 file after the snapshot 2023-01-24T03:33:35.010 INFO:tasks.workunit.client.0.smithi112.stdout: create 217/32 file after the snapshot 2023-01-24T03:33:35.011 INFO:tasks.workunit.client.0.smithi112.stdout: create 217/33 file after the snapshot 2023-01-24T03:33:35.011 INFO:tasks.workunit.client.0.smithi112.stdout: create 217/34 file after the snapshot 2023-01-24T03:33:35.012 INFO:tasks.workunit.client.0.smithi112.stdout: create 217/35 file after the snapshot 2023-01-24T03:33:35.013 INFO:tasks.workunit.client.0.smithi112.stdout: create 217/36 file after the snapshot 2023-01-24T03:33:35.013 INFO:tasks.workunit.client.0.smithi112.stdout: create 217/37 file after the snapshot 2023-01-24T03:33:35.014 INFO:tasks.workunit.client.0.smithi112.stdout: create 217/38 file after the snapshot 2023-01-24T03:33:35.015 INFO:tasks.workunit.client.0.smithi112.stdout: create 217/39 file after the snapshot 2023-01-24T03:33:35.016 INFO:tasks.workunit.client.0.smithi112.stdout: create 218/.snap/snap-subdir-test 2023-01-24T03:33:35.037 INFO:tasks.workunit.client.0.smithi112.stdout: create 218/30 file after the snapshot 2023-01-24T03:33:35.038 INFO:tasks.workunit.client.0.smithi112.stdout: create 218/31 file after the snapshot 2023-01-24T03:33:35.039 INFO:tasks.workunit.client.0.smithi112.stdout: create 218/32 file after the snapshot 2023-01-24T03:33:35.040 INFO:tasks.workunit.client.0.smithi112.stdout: create 218/33 file after the snapshot 2023-01-24T03:33:35.041 INFO:tasks.workunit.client.0.smithi112.stdout: create 218/34 file after the snapshot 2023-01-24T03:33:35.041 INFO:tasks.workunit.client.0.smithi112.stdout: create 218/35 file after the snapshot 2023-01-24T03:33:35.042 INFO:tasks.workunit.client.0.smithi112.stdout: create 218/36 file after the snapshot 2023-01-24T03:33:35.043 INFO:tasks.workunit.client.0.smithi112.stdout: create 218/37 file after the snapshot 2023-01-24T03:33:35.044 INFO:tasks.workunit.client.0.smithi112.stdout: create 218/38 file after the snapshot 2023-01-24T03:33:35.045 INFO:tasks.workunit.client.0.smithi112.stdout: create 218/39 file after the snapshot 2023-01-24T03:33:35.046 INFO:tasks.workunit.client.0.smithi112.stdout: create 219/.snap/snap-subdir-test 2023-01-24T03:33:35.325 INFO:tasks.workunit.client.0.smithi112.stdout: create 219/30 file after the snapshot 2023-01-24T03:33:35.327 INFO:tasks.workunit.client.0.smithi112.stdout: create 219/31 file after the snapshot 2023-01-24T03:33:35.327 INFO:tasks.workunit.client.0.smithi112.stdout: create 219/32 file after the snapshot 2023-01-24T03:33:35.328 INFO:tasks.workunit.client.0.smithi112.stdout: create 219/33 file after the snapshot 2023-01-24T03:33:35.329 INFO:tasks.workunit.client.0.smithi112.stdout: create 219/34 file after the snapshot 2023-01-24T03:33:35.330 INFO:tasks.workunit.client.0.smithi112.stdout: create 219/35 file after the snapshot 2023-01-24T03:33:35.331 INFO:tasks.workunit.client.0.smithi112.stdout: create 219/36 file after the snapshot 2023-01-24T03:33:35.332 INFO:tasks.workunit.client.0.smithi112.stdout: create 219/37 file after the snapshot 2023-01-24T03:33:35.333 INFO:tasks.workunit.client.0.smithi112.stdout: create 219/38 file after the snapshot 2023-01-24T03:33:35.334 INFO:tasks.workunit.client.0.smithi112.stdout: create 219/39 file after the snapshot 2023-01-24T03:33:35.335 INFO:tasks.workunit.client.0.smithi112.stdout: create 220/.snap/snap-subdir-test 2023-01-24T03:33:35.357 INFO:tasks.workunit.client.0.smithi112.stdout: create 220/30 file after the snapshot 2023-01-24T03:33:35.357 INFO:tasks.workunit.client.0.smithi112.stdout: create 220/31 file after the snapshot 2023-01-24T03:33:35.358 INFO:tasks.workunit.client.0.smithi112.stdout: create 220/32 file after the snapshot 2023-01-24T03:33:35.359 INFO:tasks.workunit.client.0.smithi112.stdout: create 220/33 file after the snapshot 2023-01-24T03:33:35.360 INFO:tasks.workunit.client.0.smithi112.stdout: create 220/34 file after the snapshot 2023-01-24T03:33:35.361 INFO:tasks.workunit.client.0.smithi112.stdout: create 220/35 file after the snapshot 2023-01-24T03:33:35.362 INFO:tasks.workunit.client.0.smithi112.stdout: create 220/36 file after the snapshot 2023-01-24T03:33:35.363 INFO:tasks.workunit.client.0.smithi112.stdout: create 220/37 file after the snapshot 2023-01-24T03:33:35.363 INFO:tasks.workunit.client.0.smithi112.stdout: create 220/38 file after the snapshot 2023-01-24T03:33:35.364 INFO:tasks.workunit.client.0.smithi112.stdout: create 220/39 file after the snapshot 2023-01-24T03:33:35.365 INFO:tasks.workunit.client.0.smithi112.stdout: create 221/.snap/snap-subdir-test 2023-01-24T03:33:35.397 INFO:tasks.workunit.client.0.smithi112.stdout: create 221/30 file after the snapshot 2023-01-24T03:33:35.398 INFO:tasks.workunit.client.0.smithi112.stdout: create 221/31 file after the snapshot 2023-01-24T03:33:35.399 INFO:tasks.workunit.client.0.smithi112.stdout: create 221/32 file after the snapshot 2023-01-24T03:33:35.400 INFO:tasks.workunit.client.0.smithi112.stdout: create 221/33 file after the snapshot 2023-01-24T03:33:35.401 INFO:tasks.workunit.client.0.smithi112.stdout: create 221/34 file after the snapshot 2023-01-24T03:33:35.402 INFO:tasks.workunit.client.0.smithi112.stdout: create 221/35 file after the snapshot 2023-01-24T03:33:35.403 INFO:tasks.workunit.client.0.smithi112.stdout: create 221/36 file after the snapshot 2023-01-24T03:33:35.404 INFO:tasks.workunit.client.0.smithi112.stdout: create 221/37 file after the snapshot 2023-01-24T03:33:35.404 INFO:tasks.workunit.client.0.smithi112.stdout: create 221/38 file after the snapshot 2023-01-24T03:33:35.405 INFO:tasks.workunit.client.0.smithi112.stdout: create 221/39 file after the snapshot 2023-01-24T03:33:35.406 INFO:tasks.workunit.client.0.smithi112.stdout: create 222/.snap/snap-subdir-test 2023-01-24T03:33:35.432 INFO:tasks.workunit.client.0.smithi112.stdout: create 222/30 file after the snapshot 2023-01-24T03:33:35.433 INFO:tasks.workunit.client.0.smithi112.stdout: create 222/31 file after the snapshot 2023-01-24T03:33:35.434 INFO:tasks.workunit.client.0.smithi112.stdout: create 222/32 file after the snapshot 2023-01-24T03:33:35.435 INFO:tasks.workunit.client.0.smithi112.stdout: create 222/33 file after the snapshot 2023-01-24T03:33:35.435 INFO:tasks.workunit.client.0.smithi112.stdout: create 222/34 file after the snapshot 2023-01-24T03:33:35.436 INFO:tasks.workunit.client.0.smithi112.stdout: create 222/35 file after the snapshot 2023-01-24T03:33:35.437 INFO:tasks.workunit.client.0.smithi112.stdout: create 222/36 file after the snapshot 2023-01-24T03:33:35.438 INFO:tasks.workunit.client.0.smithi112.stdout: create 222/37 file after the snapshot 2023-01-24T03:33:35.439 INFO:tasks.workunit.client.0.smithi112.stdout: create 222/38 file after the snapshot 2023-01-24T03:33:35.439 INFO:tasks.workunit.client.0.smithi112.stdout: create 222/39 file after the snapshot 2023-01-24T03:33:35.440 INFO:tasks.workunit.client.0.smithi112.stdout: create 223/.snap/snap-subdir-test 2023-01-24T03:33:35.466 INFO:tasks.workunit.client.0.smithi112.stdout: create 223/30 file after the snapshot 2023-01-24T03:33:35.467 INFO:tasks.workunit.client.0.smithi112.stdout: create 223/31 file after the snapshot 2023-01-24T03:33:35.468 INFO:tasks.workunit.client.0.smithi112.stdout: create 223/32 file after the snapshot 2023-01-24T03:33:35.469 INFO:tasks.workunit.client.0.smithi112.stdout: create 223/33 file after the snapshot 2023-01-24T03:33:35.469 INFO:tasks.workunit.client.0.smithi112.stdout: create 223/34 file after the snapshot 2023-01-24T03:33:35.470 INFO:tasks.workunit.client.0.smithi112.stdout: create 223/35 file after the snapshot 2023-01-24T03:33:35.471 INFO:tasks.workunit.client.0.smithi112.stdout: create 223/36 file after the snapshot 2023-01-24T03:33:35.472 INFO:tasks.workunit.client.0.smithi112.stdout: create 223/37 file after the snapshot 2023-01-24T03:33:35.473 INFO:tasks.workunit.client.0.smithi112.stdout: create 223/38 file after the snapshot 2023-01-24T03:33:35.474 INFO:tasks.workunit.client.0.smithi112.stdout: create 223/39 file after the snapshot 2023-01-24T03:33:35.475 INFO:tasks.workunit.client.0.smithi112.stdout: create 224/.snap/snap-subdir-test 2023-01-24T03:33:35.501 INFO:tasks.workunit.client.0.smithi112.stdout: create 224/30 file after the snapshot 2023-01-24T03:33:35.502 INFO:tasks.workunit.client.0.smithi112.stdout: create 224/31 file after the snapshot 2023-01-24T03:33:35.503 INFO:tasks.workunit.client.0.smithi112.stdout: create 224/32 file after the snapshot 2023-01-24T03:33:35.503 INFO:tasks.workunit.client.0.smithi112.stdout: create 224/33 file after the snapshot 2023-01-24T03:33:35.504 INFO:tasks.workunit.client.0.smithi112.stdout: create 224/34 file after the snapshot 2023-01-24T03:33:35.505 INFO:tasks.workunit.client.0.smithi112.stdout: create 224/35 file after the snapshot 2023-01-24T03:33:35.506 INFO:tasks.workunit.client.0.smithi112.stdout: create 224/36 file after the snapshot 2023-01-24T03:33:35.507 INFO:tasks.workunit.client.0.smithi112.stdout: create 224/37 file after the snapshot 2023-01-24T03:33:35.508 INFO:tasks.workunit.client.0.smithi112.stdout: create 224/38 file after the snapshot 2023-01-24T03:33:35.509 INFO:tasks.workunit.client.0.smithi112.stdout: create 224/39 file after the snapshot 2023-01-24T03:33:35.510 INFO:tasks.workunit.client.0.smithi112.stdout: create 225/.snap/snap-subdir-test 2023-01-24T03:33:35.536 INFO:tasks.workunit.client.0.smithi112.stdout: create 225/30 file after the snapshot 2023-01-24T03:33:35.537 INFO:tasks.workunit.client.0.smithi112.stdout: create 225/31 file after the snapshot 2023-01-24T03:33:35.538 INFO:tasks.workunit.client.0.smithi112.stdout: create 225/32 file after the snapshot 2023-01-24T03:33:35.539 INFO:tasks.workunit.client.0.smithi112.stdout: create 225/33 file after the snapshot 2023-01-24T03:33:35.540 INFO:tasks.workunit.client.0.smithi112.stdout: create 225/34 file after the snapshot 2023-01-24T03:33:35.541 INFO:tasks.workunit.client.0.smithi112.stdout: create 225/35 file after the snapshot 2023-01-24T03:33:35.542 INFO:tasks.workunit.client.0.smithi112.stdout: create 225/36 file after the snapshot 2023-01-24T03:33:35.542 INFO:tasks.workunit.client.0.smithi112.stdout: create 225/37 file after the snapshot 2023-01-24T03:33:35.543 INFO:tasks.workunit.client.0.smithi112.stdout: create 225/38 file after the snapshot 2023-01-24T03:33:35.544 INFO:tasks.workunit.client.0.smithi112.stdout: create 225/39 file after the snapshot 2023-01-24T03:33:35.545 INFO:tasks.workunit.client.0.smithi112.stdout: create 226/.snap/snap-subdir-test 2023-01-24T03:33:35.570 INFO:tasks.workunit.client.0.smithi112.stdout: create 226/30 file after the snapshot 2023-01-24T03:33:35.572 INFO:tasks.workunit.client.0.smithi112.stdout: create 226/31 file after the snapshot 2023-01-24T03:33:35.572 INFO:tasks.workunit.client.0.smithi112.stdout: create 226/32 file after the snapshot 2023-01-24T03:33:35.573 INFO:tasks.workunit.client.0.smithi112.stdout: create 226/33 file after the snapshot 2023-01-24T03:33:35.574 INFO:tasks.workunit.client.0.smithi112.stdout: create 226/34 file after the snapshot 2023-01-24T03:33:35.576 INFO:tasks.workunit.client.0.smithi112.stdout: create 226/35 file after the snapshot 2023-01-24T03:33:35.576 INFO:tasks.workunit.client.0.smithi112.stdout: create 226/36 file after the snapshot 2023-01-24T03:33:35.577 INFO:tasks.workunit.client.0.smithi112.stdout: create 226/37 file after the snapshot 2023-01-24T03:33:35.577 INFO:tasks.workunit.client.0.smithi112.stdout: create 226/38 file after the snapshot 2023-01-24T03:33:35.578 INFO:tasks.workunit.client.0.smithi112.stdout: create 226/39 file after the snapshot 2023-01-24T03:33:35.579 INFO:tasks.workunit.client.0.smithi112.stdout: create 227/.snap/snap-subdir-test 2023-01-24T03:33:35.859 INFO:tasks.workunit.client.0.smithi112.stdout: create 227/30 file after the snapshot 2023-01-24T03:33:35.860 INFO:tasks.workunit.client.0.smithi112.stdout: create 227/31 file after the snapshot 2023-01-24T03:33:35.861 INFO:tasks.workunit.client.0.smithi112.stdout: create 227/32 file after the snapshot 2023-01-24T03:33:35.862 INFO:tasks.workunit.client.0.smithi112.stdout: create 227/33 file after the snapshot 2023-01-24T03:33:35.863 INFO:tasks.workunit.client.0.smithi112.stdout: create 227/34 file after the snapshot 2023-01-24T03:33:35.864 INFO:tasks.workunit.client.0.smithi112.stdout: create 227/35 file after the snapshot 2023-01-24T03:33:35.865 INFO:tasks.workunit.client.0.smithi112.stdout: create 227/36 file after the snapshot 2023-01-24T03:33:35.866 INFO:tasks.workunit.client.0.smithi112.stdout: create 227/37 file after the snapshot 2023-01-24T03:33:35.867 INFO:tasks.workunit.client.0.smithi112.stdout: create 227/38 file after the snapshot 2023-01-24T03:33:35.868 INFO:tasks.workunit.client.0.smithi112.stdout: create 227/39 file after the snapshot 2023-01-24T03:33:35.868 INFO:tasks.workunit.client.0.smithi112.stdout: create 228/.snap/snap-subdir-test 2023-01-24T03:33:35.896 INFO:tasks.workunit.client.0.smithi112.stdout: create 228/30 file after the snapshot 2023-01-24T03:33:35.897 INFO:tasks.workunit.client.0.smithi112.stdout: create 228/31 file after the snapshot 2023-01-24T03:33:35.898 INFO:tasks.workunit.client.0.smithi112.stdout: create 228/32 file after the snapshot 2023-01-24T03:33:35.899 INFO:tasks.workunit.client.0.smithi112.stdout: create 228/33 file after the snapshot 2023-01-24T03:33:35.900 INFO:tasks.workunit.client.0.smithi112.stdout: create 228/34 file after the snapshot 2023-01-24T03:33:35.901 INFO:tasks.workunit.client.0.smithi112.stdout: create 228/35 file after the snapshot 2023-01-24T03:33:35.902 INFO:tasks.workunit.client.0.smithi112.stdout: create 228/36 file after the snapshot 2023-01-24T03:33:35.902 INFO:tasks.workunit.client.0.smithi112.stdout: create 228/37 file after the snapshot 2023-01-24T03:33:35.903 INFO:tasks.workunit.client.0.smithi112.stdout: create 228/38 file after the snapshot 2023-01-24T03:33:35.904 INFO:tasks.workunit.client.0.smithi112.stdout: create 228/39 file after the snapshot 2023-01-24T03:33:35.905 INFO:tasks.workunit.client.0.smithi112.stdout: create 229/.snap/snap-subdir-test 2023-01-24T03:33:36.718 INFO:tasks.workunit.client.0.smithi112.stdout: create 229/30 file after the snapshot 2023-01-24T03:33:36.720 INFO:tasks.workunit.client.0.smithi112.stdout: create 229/31 file after the snapshot 2023-01-24T03:33:36.721 INFO:tasks.workunit.client.0.smithi112.stdout: create 229/32 file after the snapshot 2023-01-24T03:33:36.722 INFO:tasks.workunit.client.0.smithi112.stdout: create 229/33 file after the snapshot 2023-01-24T03:33:36.723 INFO:tasks.workunit.client.0.smithi112.stdout: create 229/34 file after the snapshot 2023-01-24T03:33:36.724 INFO:tasks.workunit.client.0.smithi112.stdout: create 229/35 file after the snapshot 2023-01-24T03:33:36.725 INFO:tasks.workunit.client.0.smithi112.stdout: create 229/36 file after the snapshot 2023-01-24T03:33:36.726 INFO:tasks.workunit.client.0.smithi112.stdout: create 229/37 file after the snapshot 2023-01-24T03:33:36.727 INFO:tasks.workunit.client.0.smithi112.stdout: create 229/38 file after the snapshot 2023-01-24T03:33:36.728 INFO:tasks.workunit.client.0.smithi112.stdout: create 229/39 file after the snapshot 2023-01-24T03:33:36.729 INFO:tasks.workunit.client.0.smithi112.stdout: create 230/.snap/snap-subdir-test 2023-01-24T03:33:36.758 INFO:tasks.workunit.client.0.smithi112.stdout: create 230/30 file after the snapshot 2023-01-24T03:33:36.759 INFO:tasks.workunit.client.0.smithi112.stdout: create 230/31 file after the snapshot 2023-01-24T03:33:36.760 INFO:tasks.workunit.client.0.smithi112.stdout: create 230/32 file after the snapshot 2023-01-24T03:33:36.761 INFO:tasks.workunit.client.0.smithi112.stdout: create 230/33 file after the snapshot 2023-01-24T03:33:36.762 INFO:tasks.workunit.client.0.smithi112.stdout: create 230/34 file after the snapshot 2023-01-24T03:33:36.763 INFO:tasks.workunit.client.0.smithi112.stdout: create 230/35 file after the snapshot 2023-01-24T03:33:36.764 INFO:tasks.workunit.client.0.smithi112.stdout: create 230/36 file after the snapshot 2023-01-24T03:33:36.765 INFO:tasks.workunit.client.0.smithi112.stdout: create 230/37 file after the snapshot 2023-01-24T03:33:36.766 INFO:tasks.workunit.client.0.smithi112.stdout: create 230/38 file after the snapshot 2023-01-24T03:33:36.767 INFO:tasks.workunit.client.0.smithi112.stdout: create 230/39 file after the snapshot 2023-01-24T03:33:36.768 INFO:tasks.workunit.client.0.smithi112.stdout: create 231/.snap/snap-subdir-test 2023-01-24T03:33:36.789 INFO:tasks.workunit.client.0.smithi112.stdout: create 231/30 file after the snapshot 2023-01-24T03:33:36.791 INFO:tasks.workunit.client.0.smithi112.stdout: create 231/31 file after the snapshot 2023-01-24T03:33:36.792 INFO:tasks.workunit.client.0.smithi112.stdout: create 231/32 file after the snapshot 2023-01-24T03:33:36.793 INFO:tasks.workunit.client.0.smithi112.stdout: create 231/33 file after the snapshot 2023-01-24T03:33:36.794 INFO:tasks.workunit.client.0.smithi112.stdout: create 231/34 file after the snapshot 2023-01-24T03:33:36.795 INFO:tasks.workunit.client.0.smithi112.stdout: create 231/35 file after the snapshot 2023-01-24T03:33:36.796 INFO:tasks.workunit.client.0.smithi112.stdout: create 231/36 file after the snapshot 2023-01-24T03:33:36.797 INFO:tasks.workunit.client.0.smithi112.stdout: create 231/37 file after the snapshot 2023-01-24T03:33:36.798 INFO:tasks.workunit.client.0.smithi112.stdout: create 231/38 file after the snapshot 2023-01-24T03:33:36.799 INFO:tasks.workunit.client.0.smithi112.stdout: create 231/39 file after the snapshot 2023-01-24T03:33:36.800 INFO:tasks.workunit.client.0.smithi112.stdout: create 232/.snap/snap-subdir-test 2023-01-24T03:33:36.821 INFO:tasks.workunit.client.0.smithi112.stdout: create 232/30 file after the snapshot 2023-01-24T03:33:36.822 INFO:tasks.workunit.client.0.smithi112.stdout: create 232/31 file after the snapshot 2023-01-24T03:33:36.823 INFO:tasks.workunit.client.0.smithi112.stdout: create 232/32 file after the snapshot 2023-01-24T03:33:36.824 INFO:tasks.workunit.client.0.smithi112.stdout: create 232/33 file after the snapshot 2023-01-24T03:33:36.824 INFO:tasks.workunit.client.0.smithi112.stdout: create 232/34 file after the snapshot 2023-01-24T03:33:36.825 INFO:tasks.workunit.client.0.smithi112.stdout: create 232/35 file after the snapshot 2023-01-24T03:33:36.826 INFO:tasks.workunit.client.0.smithi112.stdout: create 232/36 file after the snapshot 2023-01-24T03:33:36.827 INFO:tasks.workunit.client.0.smithi112.stdout: create 232/37 file after the snapshot 2023-01-24T03:33:36.828 INFO:tasks.workunit.client.0.smithi112.stdout: create 232/38 file after the snapshot 2023-01-24T03:33:36.829 INFO:tasks.workunit.client.0.smithi112.stdout: create 232/39 file after the snapshot 2023-01-24T03:33:36.830 INFO:tasks.workunit.client.0.smithi112.stdout: create 233/.snap/snap-subdir-test 2023-01-24T03:33:36.850 INFO:tasks.workunit.client.0.smithi112.stdout: create 233/30 file after the snapshot 2023-01-24T03:33:36.852 INFO:tasks.workunit.client.0.smithi112.stdout: create 233/31 file after the snapshot 2023-01-24T03:33:36.853 INFO:tasks.workunit.client.0.smithi112.stdout: create 233/32 file after the snapshot 2023-01-24T03:33:36.854 INFO:tasks.workunit.client.0.smithi112.stdout: create 233/33 file after the snapshot 2023-01-24T03:33:36.855 INFO:tasks.workunit.client.0.smithi112.stdout: create 233/34 file after the snapshot 2023-01-24T03:33:36.856 INFO:tasks.workunit.client.0.smithi112.stdout: create 233/35 file after the snapshot 2023-01-24T03:33:36.857 INFO:tasks.workunit.client.0.smithi112.stdout: create 233/36 file after the snapshot 2023-01-24T03:33:36.858 INFO:tasks.workunit.client.0.smithi112.stdout: create 233/37 file after the snapshot 2023-01-24T03:33:36.859 INFO:tasks.workunit.client.0.smithi112.stdout: create 233/38 file after the snapshot 2023-01-24T03:33:36.860 INFO:tasks.workunit.client.0.smithi112.stdout: create 233/39 file after the snapshot 2023-01-24T03:33:36.861 INFO:tasks.workunit.client.0.smithi112.stdout: create 234/.snap/snap-subdir-test 2023-01-24T03:33:36.881 INFO:tasks.workunit.client.0.smithi112.stdout: create 234/30 file after the snapshot 2023-01-24T03:33:36.882 INFO:tasks.workunit.client.0.smithi112.stdout: create 234/31 file after the snapshot 2023-01-24T03:33:36.883 INFO:tasks.workunit.client.0.smithi112.stdout: create 234/32 file after the snapshot 2023-01-24T03:33:36.884 INFO:tasks.workunit.client.0.smithi112.stdout: create 234/33 file after the snapshot 2023-01-24T03:33:36.885 INFO:tasks.workunit.client.0.smithi112.stdout: create 234/34 file after the snapshot 2023-01-24T03:33:36.886 INFO:tasks.workunit.client.0.smithi112.stdout: create 234/35 file after the snapshot 2023-01-24T03:33:37.229 INFO:tasks.workunit.client.0.smithi112.stdout: create 234/36 file after the snapshot 2023-01-24T03:33:37.229 INFO:tasks.workunit.client.0.smithi112.stdout: create 234/37 file after the snapshot 2023-01-24T03:33:37.230 INFO:tasks.workunit.client.0.smithi112.stdout: create 234/38 file after the snapshot 2023-01-24T03:33:37.231 INFO:tasks.workunit.client.0.smithi112.stdout: create 234/39 file after the snapshot 2023-01-24T03:33:37.232 INFO:tasks.workunit.client.0.smithi112.stdout: create 235/.snap/snap-subdir-test 2023-01-24T03:33:37.255 INFO:tasks.workunit.client.0.smithi112.stdout: create 235/30 file after the snapshot 2023-01-24T03:33:37.256 INFO:tasks.workunit.client.0.smithi112.stdout: create 235/31 file after the snapshot 2023-01-24T03:33:37.257 INFO:tasks.workunit.client.0.smithi112.stdout: create 235/32 file after the snapshot 2023-01-24T03:33:37.258 INFO:tasks.workunit.client.0.smithi112.stdout: create 235/33 file after the snapshot 2023-01-24T03:33:37.259 INFO:tasks.workunit.client.0.smithi112.stdout: create 235/34 file after the snapshot 2023-01-24T03:33:37.260 INFO:tasks.workunit.client.0.smithi112.stdout: create 235/35 file after the snapshot 2023-01-24T03:33:37.261 INFO:tasks.workunit.client.0.smithi112.stdout: create 235/36 file after the snapshot 2023-01-24T03:33:37.262 INFO:tasks.workunit.client.0.smithi112.stdout: create 235/37 file after the snapshot 2023-01-24T03:33:37.263 INFO:tasks.workunit.client.0.smithi112.stdout: create 235/38 file after the snapshot 2023-01-24T03:33:37.264 INFO:tasks.workunit.client.0.smithi112.stdout: create 235/39 file after the snapshot 2023-01-24T03:33:37.265 INFO:tasks.workunit.client.0.smithi112.stdout: create 236/.snap/snap-subdir-test 2023-01-24T03:33:37.286 INFO:tasks.workunit.client.0.smithi112.stdout: create 236/30 file after the snapshot 2023-01-24T03:33:37.289 INFO:tasks.workunit.client.0.smithi112.stdout: create 236/31 file after the snapshot 2023-01-24T03:33:37.289 INFO:tasks.workunit.client.0.smithi112.stdout: create 236/32 file after the snapshot 2023-01-24T03:33:37.289 INFO:tasks.workunit.client.0.smithi112.stdout: create 236/33 file after the snapshot 2023-01-24T03:33:37.290 INFO:tasks.workunit.client.0.smithi112.stdout: create 236/34 file after the snapshot 2023-01-24T03:33:37.291 INFO:tasks.workunit.client.0.smithi112.stdout: create 236/35 file after the snapshot 2023-01-24T03:33:37.292 INFO:tasks.workunit.client.0.smithi112.stdout: create 236/36 file after the snapshot 2023-01-24T03:33:37.293 INFO:tasks.workunit.client.0.smithi112.stdout: create 236/37 file after the snapshot 2023-01-24T03:33:37.294 INFO:tasks.workunit.client.0.smithi112.stdout: create 236/38 file after the snapshot 2023-01-24T03:33:37.295 INFO:tasks.workunit.client.0.smithi112.stdout: create 236/39 file after the snapshot 2023-01-24T03:33:37.296 INFO:tasks.workunit.client.0.smithi112.stdout: create 237/.snap/snap-subdir-test 2023-01-24T03:33:37.318 INFO:tasks.workunit.client.0.smithi112.stdout: create 237/30 file after the snapshot 2023-01-24T03:33:37.319 INFO:tasks.workunit.client.0.smithi112.stdout: create 237/31 file after the snapshot 2023-01-24T03:33:37.320 INFO:tasks.workunit.client.0.smithi112.stdout: create 237/32 file after the snapshot 2023-01-24T03:33:37.321 INFO:tasks.workunit.client.0.smithi112.stdout: create 237/33 file after the snapshot 2023-01-24T03:33:37.322 INFO:tasks.workunit.client.0.smithi112.stdout: create 237/34 file after the snapshot 2023-01-24T03:33:37.323 INFO:tasks.workunit.client.0.smithi112.stdout: create 237/35 file after the snapshot 2023-01-24T03:33:37.324 INFO:tasks.workunit.client.0.smithi112.stdout: create 237/36 file after the snapshot 2023-01-24T03:33:37.325 INFO:tasks.workunit.client.0.smithi112.stdout: create 237/37 file after the snapshot 2023-01-24T03:33:37.326 INFO:tasks.workunit.client.0.smithi112.stdout: create 237/38 file after the snapshot 2023-01-24T03:33:37.327 INFO:tasks.workunit.client.0.smithi112.stdout: create 237/39 file after the snapshot 2023-01-24T03:33:37.328 INFO:tasks.workunit.client.0.smithi112.stdout: create 238/.snap/snap-subdir-test 2023-01-24T03:33:37.351 INFO:tasks.workunit.client.0.smithi112.stdout: create 238/30 file after the snapshot 2023-01-24T03:33:37.352 INFO:tasks.workunit.client.0.smithi112.stdout: create 238/31 file after the snapshot 2023-01-24T03:33:37.353 INFO:tasks.workunit.client.0.smithi112.stdout: create 238/32 file after the snapshot 2023-01-24T03:33:37.354 INFO:tasks.workunit.client.0.smithi112.stdout: create 238/33 file after the snapshot 2023-01-24T03:33:37.355 INFO:tasks.workunit.client.0.smithi112.stdout: create 238/34 file after the snapshot 2023-01-24T03:33:37.356 INFO:tasks.workunit.client.0.smithi112.stdout: create 238/35 file after the snapshot 2023-01-24T03:33:37.357 INFO:tasks.workunit.client.0.smithi112.stdout: create 238/36 file after the snapshot 2023-01-24T03:33:37.358 INFO:tasks.workunit.client.0.smithi112.stdout: create 238/37 file after the snapshot 2023-01-24T03:33:37.359 INFO:tasks.workunit.client.0.smithi112.stdout: create 238/38 file after the snapshot 2023-01-24T03:33:37.360 INFO:tasks.workunit.client.0.smithi112.stdout: create 238/39 file after the snapshot 2023-01-24T03:33:37.360 INFO:tasks.workunit.client.0.smithi112.stdout: create 239/.snap/snap-subdir-test 2023-01-24T03:33:37.383 INFO:tasks.workunit.client.0.smithi112.stdout: create 239/30 file after the snapshot 2023-01-24T03:33:38.032 INFO:tasks.workunit.client.0.smithi112.stdout: create 239/31 file after the snapshot 2023-01-24T03:33:38.033 INFO:tasks.workunit.client.0.smithi112.stdout: create 239/32 file after the snapshot 2023-01-24T03:33:38.034 INFO:tasks.workunit.client.0.smithi112.stdout: create 239/33 file after the snapshot 2023-01-24T03:33:38.035 INFO:tasks.workunit.client.0.smithi112.stdout: create 239/34 file after the snapshot 2023-01-24T03:33:38.036 INFO:tasks.workunit.client.0.smithi112.stdout: create 239/35 file after the snapshot 2023-01-24T03:33:38.037 INFO:tasks.workunit.client.0.smithi112.stdout: create 239/36 file after the snapshot 2023-01-24T03:33:38.038 INFO:tasks.workunit.client.0.smithi112.stdout: create 239/37 file after the snapshot 2023-01-24T03:33:38.039 INFO:tasks.workunit.client.0.smithi112.stdout: create 239/38 file after the snapshot 2023-01-24T03:33:38.040 INFO:tasks.workunit.client.0.smithi112.stdout: create 239/39 file after the snapshot 2023-01-24T03:33:38.041 INFO:tasks.workunit.client.0.smithi112.stdout: create 240/.snap/snap-subdir-test 2023-01-24T03:33:38.066 INFO:tasks.workunit.client.0.smithi112.stdout: create 240/30 file after the snapshot 2023-01-24T03:33:38.068 INFO:tasks.workunit.client.0.smithi112.stdout: create 240/31 file after the snapshot 2023-01-24T03:33:38.068 INFO:tasks.workunit.client.0.smithi112.stdout: create 240/32 file after the snapshot 2023-01-24T03:33:38.069 INFO:tasks.workunit.client.0.smithi112.stdout: create 240/33 file after the snapshot 2023-01-24T03:33:38.070 INFO:tasks.workunit.client.0.smithi112.stdout: create 240/34 file after the snapshot 2023-01-24T03:33:38.071 INFO:tasks.workunit.client.0.smithi112.stdout: create 240/35 file after the snapshot 2023-01-24T03:33:38.072 INFO:tasks.workunit.client.0.smithi112.stdout: create 240/36 file after the snapshot 2023-01-24T03:33:38.073 INFO:tasks.workunit.client.0.smithi112.stdout: create 240/37 file after the snapshot 2023-01-24T03:33:38.074 INFO:tasks.workunit.client.0.smithi112.stdout: create 240/38 file after the snapshot 2023-01-24T03:33:38.075 INFO:tasks.workunit.client.0.smithi112.stdout: create 240/39 file after the snapshot 2023-01-24T03:33:38.076 INFO:tasks.workunit.client.0.smithi112.stdout: create 241/.snap/snap-subdir-test 2023-01-24T03:33:38.101 INFO:tasks.workunit.client.0.smithi112.stdout: create 241/30 file after the snapshot 2023-01-24T03:33:38.102 INFO:tasks.workunit.client.0.smithi112.stdout: create 241/31 file after the snapshot 2023-01-24T03:33:38.103 INFO:tasks.workunit.client.0.smithi112.stdout: create 241/32 file after the snapshot 2023-01-24T03:33:38.104 INFO:tasks.workunit.client.0.smithi112.stdout: create 241/33 file after the snapshot 2023-01-24T03:33:38.105 INFO:tasks.workunit.client.0.smithi112.stdout: create 241/34 file after the snapshot 2023-01-24T03:33:38.106 INFO:tasks.workunit.client.0.smithi112.stdout: create 241/35 file after the snapshot 2023-01-24T03:33:38.106 INFO:tasks.workunit.client.0.smithi112.stdout: create 241/36 file after the snapshot 2023-01-24T03:33:38.107 INFO:tasks.workunit.client.0.smithi112.stdout: create 241/37 file after the snapshot 2023-01-24T03:33:38.108 INFO:tasks.workunit.client.0.smithi112.stdout: create 241/38 file after the snapshot 2023-01-24T03:33:38.109 INFO:tasks.workunit.client.0.smithi112.stdout: create 241/39 file after the snapshot 2023-01-24T03:33:38.110 INFO:tasks.workunit.client.0.smithi112.stdout: create 242/.snap/snap-subdir-test 2023-01-24T03:33:38.134 INFO:tasks.workunit.client.0.smithi112.stdout: create 242/30 file after the snapshot 2023-01-24T03:33:38.135 INFO:tasks.workunit.client.0.smithi112.stdout: create 242/31 file after the snapshot 2023-01-24T03:33:38.136 INFO:tasks.workunit.client.0.smithi112.stdout: create 242/32 file after the snapshot 2023-01-24T03:33:38.137 INFO:tasks.workunit.client.0.smithi112.stdout: create 242/33 file after the snapshot 2023-01-24T03:33:38.138 INFO:tasks.workunit.client.0.smithi112.stdout: create 242/34 file after the snapshot 2023-01-24T03:33:38.139 INFO:tasks.workunit.client.0.smithi112.stdout: create 242/35 file after the snapshot 2023-01-24T03:33:38.140 INFO:tasks.workunit.client.0.smithi112.stdout: create 242/36 file after the snapshot 2023-01-24T03:33:38.141 INFO:tasks.workunit.client.0.smithi112.stdout: create 242/37 file after the snapshot 2023-01-24T03:33:38.142 INFO:tasks.workunit.client.0.smithi112.stdout: create 242/38 file after the snapshot 2023-01-24T03:33:38.143 INFO:tasks.workunit.client.0.smithi112.stdout: create 242/39 file after the snapshot 2023-01-24T03:33:38.144 INFO:tasks.workunit.client.0.smithi112.stdout: create 243/.snap/snap-subdir-test 2023-01-24T03:33:38.167 INFO:tasks.workunit.client.0.smithi112.stdout: create 243/30 file after the snapshot 2023-01-24T03:33:38.168 INFO:tasks.workunit.client.0.smithi112.stdout: create 243/31 file after the snapshot 2023-01-24T03:33:38.169 INFO:tasks.workunit.client.0.smithi112.stdout: create 243/32 file after the snapshot 2023-01-24T03:33:38.170 INFO:tasks.workunit.client.0.smithi112.stdout: create 243/33 file after the snapshot 2023-01-24T03:33:38.171 INFO:tasks.workunit.client.0.smithi112.stdout: create 243/34 file after the snapshot 2023-01-24T03:33:38.172 INFO:tasks.workunit.client.0.smithi112.stdout: create 243/35 file after the snapshot 2023-01-24T03:33:38.173 INFO:tasks.workunit.client.0.smithi112.stdout: create 243/36 file after the snapshot 2023-01-24T03:33:38.174 INFO:tasks.workunit.client.0.smithi112.stdout: create 243/37 file after the snapshot 2023-01-24T03:33:38.175 INFO:tasks.workunit.client.0.smithi112.stdout: create 243/38 file after the snapshot 2023-01-24T03:33:38.176 INFO:tasks.workunit.client.0.smithi112.stdout: create 243/39 file after the snapshot 2023-01-24T03:33:38.177 INFO:tasks.workunit.client.0.smithi112.stdout: create 244/.snap/snap-subdir-test 2023-01-24T03:33:38.199 INFO:tasks.workunit.client.0.smithi112.stdout: create 244/30 file after the snapshot 2023-01-24T03:33:38.200 INFO:tasks.workunit.client.0.smithi112.stdout: create 244/31 file after the snapshot 2023-01-24T03:33:38.201 INFO:tasks.workunit.client.0.smithi112.stdout: create 244/32 file after the snapshot 2023-01-24T03:33:38.202 INFO:tasks.workunit.client.0.smithi112.stdout: create 244/33 file after the snapshot 2023-01-24T03:33:38.203 INFO:tasks.workunit.client.0.smithi112.stdout: create 244/34 file after the snapshot 2023-01-24T03:33:38.204 INFO:tasks.workunit.client.0.smithi112.stdout: create 244/35 file after the snapshot 2023-01-24T03:33:38.205 INFO:tasks.workunit.client.0.smithi112.stdout: create 244/36 file after the snapshot 2023-01-24T03:33:38.206 INFO:tasks.workunit.client.0.smithi112.stdout: create 244/37 file after the snapshot 2023-01-24T03:33:38.207 INFO:tasks.workunit.client.0.smithi112.stdout: create 244/38 file after the snapshot 2023-01-24T03:33:38.208 INFO:tasks.workunit.client.0.smithi112.stdout: create 244/39 file after the snapshot 2023-01-24T03:33:38.209 INFO:tasks.workunit.client.0.smithi112.stdout: create 245/.snap/snap-subdir-test 2023-01-24T03:33:38.230 INFO:tasks.workunit.client.0.smithi112.stdout: create 245/30 file after the snapshot 2023-01-24T03:33:38.232 INFO:tasks.workunit.client.0.smithi112.stdout: create 245/31 file after the snapshot 2023-01-24T03:33:38.233 INFO:tasks.workunit.client.0.smithi112.stdout: create 245/32 file after the snapshot 2023-01-24T03:33:38.234 INFO:tasks.workunit.client.0.smithi112.stdout: create 245/33 file after the snapshot 2023-01-24T03:33:38.235 INFO:tasks.workunit.client.0.smithi112.stdout: create 245/34 file after the snapshot 2023-01-24T03:33:38.236 INFO:tasks.workunit.client.0.smithi112.stdout: create 245/35 file after the snapshot 2023-01-24T03:33:38.237 INFO:tasks.workunit.client.0.smithi112.stdout: create 245/36 file after the snapshot 2023-01-24T03:33:38.238 INFO:tasks.workunit.client.0.smithi112.stdout: create 245/37 file after the snapshot 2023-01-24T03:33:38.239 INFO:tasks.workunit.client.0.smithi112.stdout: create 245/38 file after the snapshot 2023-01-24T03:33:38.240 INFO:tasks.workunit.client.0.smithi112.stdout: create 245/39 file after the snapshot 2023-01-24T03:33:38.241 INFO:tasks.workunit.client.0.smithi112.stdout: create 246/.snap/snap-subdir-test 2023-01-24T03:33:38.263 INFO:tasks.workunit.client.0.smithi112.stdout: create 246/30 file after the snapshot 2023-01-24T03:33:38.265 INFO:tasks.workunit.client.0.smithi112.stdout: create 246/31 file after the snapshot 2023-01-24T03:33:38.267 INFO:tasks.workunit.client.0.smithi112.stdout: create 246/32 file after the snapshot 2023-01-24T03:33:38.268 INFO:tasks.workunit.client.0.smithi112.stdout: create 246/33 file after the snapshot 2023-01-24T03:33:38.269 INFO:tasks.workunit.client.0.smithi112.stdout: create 246/34 file after the snapshot 2023-01-24T03:33:38.270 INFO:tasks.workunit.client.0.smithi112.stdout: create 246/35 file after the snapshot 2023-01-24T03:33:38.271 INFO:tasks.workunit.client.0.smithi112.stdout: create 246/36 file after the snapshot 2023-01-24T03:33:38.273 INFO:tasks.workunit.client.0.smithi112.stdout: create 246/37 file after the snapshot 2023-01-24T03:33:38.356 INFO:tasks.workunit.client.0.smithi112.stdout: create 246/38 file after the snapshot 2023-01-24T03:33:38.688 INFO:tasks.workunit.client.0.smithi112.stdout: create 246/39 file after the snapshot 2023-01-24T03:33:38.688 INFO:tasks.workunit.client.0.smithi112.stdout: create 247/.snap/snap-subdir-test 2023-01-24T03:33:38.709 INFO:tasks.workunit.client.0.smithi112.stdout: create 247/30 file after the snapshot 2023-01-24T03:33:38.710 INFO:tasks.workunit.client.0.smithi112.stdout: create 247/31 file after the snapshot 2023-01-24T03:33:38.711 INFO:tasks.workunit.client.0.smithi112.stdout: create 247/32 file after the snapshot 2023-01-24T03:33:38.712 INFO:tasks.workunit.client.0.smithi112.stdout: create 247/33 file after the snapshot 2023-01-24T03:33:38.713 INFO:tasks.workunit.client.0.smithi112.stdout: create 247/34 file after the snapshot 2023-01-24T03:33:38.714 INFO:tasks.workunit.client.0.smithi112.stdout: create 247/35 file after the snapshot 2023-01-24T03:33:38.715 INFO:tasks.workunit.client.0.smithi112.stdout: create 247/36 file after the snapshot 2023-01-24T03:33:38.716 INFO:tasks.workunit.client.0.smithi112.stdout: create 247/37 file after the snapshot 2023-01-24T03:33:38.717 INFO:tasks.workunit.client.0.smithi112.stdout: create 247/38 file after the snapshot 2023-01-24T03:33:38.718 INFO:tasks.workunit.client.0.smithi112.stdout: create 247/39 file after the snapshot 2023-01-24T03:33:38.719 INFO:tasks.workunit.client.0.smithi112.stdout: create 248/.snap/snap-subdir-test 2023-01-24T03:33:38.743 INFO:tasks.workunit.client.0.smithi112.stdout: create 248/30 file after the snapshot 2023-01-24T03:33:38.744 INFO:tasks.workunit.client.0.smithi112.stdout: create 248/31 file after the snapshot 2023-01-24T03:33:38.745 INFO:tasks.workunit.client.0.smithi112.stdout: create 248/32 file after the snapshot 2023-01-24T03:33:38.745 INFO:tasks.workunit.client.0.smithi112.stdout: create 248/33 file after the snapshot 2023-01-24T03:33:38.746 INFO:tasks.workunit.client.0.smithi112.stdout: create 248/34 file after the snapshot 2023-01-24T03:33:38.747 INFO:tasks.workunit.client.0.smithi112.stdout: create 248/35 file after the snapshot 2023-01-24T03:33:38.748 INFO:tasks.workunit.client.0.smithi112.stdout: create 248/36 file after the snapshot 2023-01-24T03:33:38.749 INFO:tasks.workunit.client.0.smithi112.stdout: create 248/37 file after the snapshot 2023-01-24T03:33:38.749 INFO:tasks.workunit.client.0.smithi112.stdout: create 248/38 file after the snapshot 2023-01-24T03:33:38.750 INFO:tasks.workunit.client.0.smithi112.stdout: create 248/39 file after the snapshot 2023-01-24T03:33:38.751 INFO:tasks.workunit.client.0.smithi112.stdout: create 249/.snap/snap-subdir-test 2023-01-24T03:33:38.773 INFO:tasks.workunit.client.0.smithi112.stdout: create 249/30 file after the snapshot 2023-01-24T03:33:38.774 INFO:tasks.workunit.client.0.smithi112.stdout: create 249/31 file after the snapshot 2023-01-24T03:33:38.775 INFO:tasks.workunit.client.0.smithi112.stdout: create 249/32 file after the snapshot 2023-01-24T03:33:38.776 INFO:tasks.workunit.client.0.smithi112.stdout: create 249/33 file after the snapshot 2023-01-24T03:33:38.777 INFO:tasks.workunit.client.0.smithi112.stdout: create 249/34 file after the snapshot 2023-01-24T03:33:38.778 INFO:tasks.workunit.client.0.smithi112.stdout: create 249/35 file after the snapshot 2023-01-24T03:33:38.779 INFO:tasks.workunit.client.0.smithi112.stdout: create 249/36 file after the snapshot 2023-01-24T03:33:38.780 INFO:tasks.workunit.client.0.smithi112.stdout: create 249/37 file after the snapshot 2023-01-24T03:33:38.781 INFO:tasks.workunit.client.0.smithi112.stdout: create 249/38 file after the snapshot 2023-01-24T03:33:38.782 INFO:tasks.workunit.client.0.smithi112.stdout: create 249/39 file after the snapshot 2023-01-24T03:33:38.783 INFO:tasks.workunit.client.0.smithi112.stdout: create 250/.snap/snap-subdir-test 2023-01-24T03:33:38.803 INFO:tasks.workunit.client.0.smithi112.stdout: create 250/30 file after the snapshot 2023-01-24T03:33:38.804 INFO:tasks.workunit.client.0.smithi112.stdout: create 250/31 file after the snapshot 2023-01-24T03:33:38.805 INFO:tasks.workunit.client.0.smithi112.stdout: create 250/32 file after the snapshot 2023-01-24T03:33:38.806 INFO:tasks.workunit.client.0.smithi112.stdout: create 250/33 file after the snapshot 2023-01-24T03:33:38.807 INFO:tasks.workunit.client.0.smithi112.stdout: create 250/34 file after the snapshot 2023-01-24T03:33:38.809 INFO:tasks.workunit.client.0.smithi112.stdout: create 250/35 file after the snapshot 2023-01-24T03:33:38.810 INFO:tasks.workunit.client.0.smithi112.stdout: create 250/36 file after the snapshot 2023-01-24T03:33:38.810 INFO:tasks.workunit.client.0.smithi112.stdout: create 250/37 file after the snapshot 2023-01-24T03:33:38.811 INFO:tasks.workunit.client.0.smithi112.stdout: create 250/38 file after the snapshot 2023-01-24T03:33:38.811 INFO:tasks.workunit.client.0.smithi112.stdout: create 250/39 file after the snapshot 2023-01-24T03:33:38.812 INFO:tasks.workunit.client.0.smithi112.stdout: create 251/.snap/snap-subdir-test 2023-01-24T03:33:38.832 INFO:tasks.workunit.client.0.smithi112.stdout: create 251/30 file after the snapshot 2023-01-24T03:33:39.091 INFO:tasks.workunit.client.0.smithi112.stdout: create 251/31 file after the snapshot 2023-01-24T03:33:39.092 INFO:tasks.workunit.client.0.smithi112.stdout: create 251/32 file after the snapshot 2023-01-24T03:33:39.093 INFO:tasks.workunit.client.0.smithi112.stdout: create 251/33 file after the snapshot 2023-01-24T03:33:39.094 INFO:tasks.workunit.client.0.smithi112.stdout: create 251/34 file after the snapshot 2023-01-24T03:33:39.095 INFO:tasks.workunit.client.0.smithi112.stdout: create 251/35 file after the snapshot 2023-01-24T03:33:39.096 INFO:tasks.workunit.client.0.smithi112.stdout: create 251/36 file after the snapshot 2023-01-24T03:33:39.097 INFO:tasks.workunit.client.0.smithi112.stdout: create 251/37 file after the snapshot 2023-01-24T03:33:39.098 INFO:tasks.workunit.client.0.smithi112.stdout: create 251/38 file after the snapshot 2023-01-24T03:33:39.099 INFO:tasks.workunit.client.0.smithi112.stdout: create 251/39 file after the snapshot 2023-01-24T03:33:39.100 INFO:tasks.workunit.client.0.smithi112.stdout: create 252/.snap/snap-subdir-test 2023-01-24T03:33:39.124 INFO:tasks.workunit.client.0.smithi112.stdout: create 252/30 file after the snapshot 2023-01-24T03:33:39.387 INFO:tasks.workunit.client.0.smithi112.stdout: create 252/31 file after the snapshot 2023-01-24T03:33:39.388 INFO:tasks.workunit.client.0.smithi112.stdout: create 252/32 file after the snapshot 2023-01-24T03:33:39.389 INFO:tasks.workunit.client.0.smithi112.stdout: create 252/33 file after the snapshot 2023-01-24T03:33:39.390 INFO:tasks.workunit.client.0.smithi112.stdout: create 252/34 file after the snapshot 2023-01-24T03:33:39.391 INFO:tasks.workunit.client.0.smithi112.stdout: create 252/35 file after the snapshot 2023-01-24T03:33:39.392 INFO:tasks.workunit.client.0.smithi112.stdout: create 252/36 file after the snapshot 2023-01-24T03:33:39.393 INFO:tasks.workunit.client.0.smithi112.stdout: create 252/37 file after the snapshot 2023-01-24T03:33:39.394 INFO:tasks.workunit.client.0.smithi112.stdout: create 252/38 file after the snapshot 2023-01-24T03:33:39.395 INFO:tasks.workunit.client.0.smithi112.stdout: create 252/39 file after the snapshot 2023-01-24T03:33:39.396 INFO:tasks.workunit.client.0.smithi112.stdout: create 253/.snap/snap-subdir-test 2023-01-24T03:33:39.423 INFO:tasks.workunit.client.0.smithi112.stdout: create 253/30 file after the snapshot 2023-01-24T03:33:39.424 INFO:tasks.workunit.client.0.smithi112.stdout: create 253/31 file after the snapshot 2023-01-24T03:33:39.425 INFO:tasks.workunit.client.0.smithi112.stdout: create 253/32 file after the snapshot 2023-01-24T03:33:39.425 INFO:tasks.workunit.client.0.smithi112.stdout: create 253/33 file after the snapshot 2023-01-24T03:33:39.426 INFO:tasks.workunit.client.0.smithi112.stdout: create 253/34 file after the snapshot 2023-01-24T03:33:39.427 INFO:tasks.workunit.client.0.smithi112.stdout: create 253/35 file after the snapshot 2023-01-24T03:33:39.428 INFO:tasks.workunit.client.0.smithi112.stdout: create 253/36 file after the snapshot 2023-01-24T03:33:39.429 INFO:tasks.workunit.client.0.smithi112.stdout: create 253/37 file after the snapshot 2023-01-24T03:33:39.430 INFO:tasks.workunit.client.0.smithi112.stdout: create 253/38 file after the snapshot 2023-01-24T03:33:39.431 INFO:tasks.workunit.client.0.smithi112.stdout: create 253/39 file after the snapshot 2023-01-24T03:33:39.432 INFO:tasks.workunit.client.0.smithi112.stdout: create 254/.snap/snap-subdir-test 2023-01-24T03:33:39.455 INFO:tasks.workunit.client.0.smithi112.stdout: create 254/30 file after the snapshot 2023-01-24T03:33:39.457 INFO:tasks.workunit.client.0.smithi112.stdout: create 254/31 file after the snapshot 2023-01-24T03:33:39.458 INFO:tasks.workunit.client.0.smithi112.stdout: create 254/32 file after the snapshot 2023-01-24T03:33:39.459 INFO:tasks.workunit.client.0.smithi112.stdout: create 254/33 file after the snapshot 2023-01-24T03:33:39.460 INFO:tasks.workunit.client.0.smithi112.stdout: create 254/34 file after the snapshot 2023-01-24T03:33:39.461 INFO:tasks.workunit.client.0.smithi112.stdout: create 254/35 file after the snapshot 2023-01-24T03:33:39.462 INFO:tasks.workunit.client.0.smithi112.stdout: create 254/36 file after the snapshot 2023-01-24T03:33:39.463 INFO:tasks.workunit.client.0.smithi112.stdout: create 254/37 file after the snapshot 2023-01-24T03:33:39.464 INFO:tasks.workunit.client.0.smithi112.stdout: create 254/38 file after the snapshot 2023-01-24T03:33:39.465 INFO:tasks.workunit.client.0.smithi112.stdout: create 254/39 file after the snapshot 2023-01-24T03:33:39.467 INFO:tasks.workunit.client.0.smithi112.stdout: create 255/.snap/snap-subdir-test 2023-01-24T03:33:39.489 INFO:tasks.workunit.client.0.smithi112.stdout: create 255/30 file after the snapshot 2023-01-24T03:33:39.490 INFO:tasks.workunit.client.0.smithi112.stdout: create 255/31 file after the snapshot 2023-01-24T03:33:39.491 INFO:tasks.workunit.client.0.smithi112.stdout: create 255/32 file after the snapshot 2023-01-24T03:33:39.492 INFO:tasks.workunit.client.0.smithi112.stdout: create 255/33 file after the snapshot 2023-01-24T03:33:39.493 INFO:tasks.workunit.client.0.smithi112.stdout: create 255/34 file after the snapshot 2023-01-24T03:33:39.494 INFO:tasks.workunit.client.0.smithi112.stdout: create 255/35 file after the snapshot 2023-01-24T03:33:39.495 INFO:tasks.workunit.client.0.smithi112.stdout: create 255/36 file after the snapshot 2023-01-24T03:33:39.496 INFO:tasks.workunit.client.0.smithi112.stdout: create 255/37 file after the snapshot 2023-01-24T03:33:39.497 INFO:tasks.workunit.client.0.smithi112.stdout: create 255/38 file after the snapshot 2023-01-24T03:33:39.498 INFO:tasks.workunit.client.0.smithi112.stdout: create 255/39 file after the snapshot 2023-01-24T03:33:39.498 INFO:tasks.workunit.client.0.smithi112.stdout: create 256/.snap/snap-subdir-test 2023-01-24T03:33:39.520 INFO:tasks.workunit.client.0.smithi112.stdout: create 256/30 file after the snapshot 2023-01-24T03:33:39.522 INFO:tasks.workunit.client.0.smithi112.stdout: create 256/31 file after the snapshot 2023-01-24T03:33:39.522 INFO:tasks.workunit.client.0.smithi112.stdout: create 256/32 file after the snapshot 2023-01-24T03:33:39.523 INFO:tasks.workunit.client.0.smithi112.stdout: create 256/33 file after the snapshot 2023-01-24T03:33:39.524 INFO:tasks.workunit.client.0.smithi112.stdout: create 256/34 file after the snapshot 2023-01-24T03:33:39.525 INFO:tasks.workunit.client.0.smithi112.stdout: create 256/35 file after the snapshot 2023-01-24T03:33:39.526 INFO:tasks.workunit.client.0.smithi112.stdout: create 256/36 file after the snapshot 2023-01-24T03:33:39.527 INFO:tasks.workunit.client.0.smithi112.stdout: create 256/37 file after the snapshot 2023-01-24T03:33:39.528 INFO:tasks.workunit.client.0.smithi112.stdout: create 256/38 file after the snapshot 2023-01-24T03:33:39.529 INFO:tasks.workunit.client.0.smithi112.stdout: create 256/39 file after the snapshot 2023-01-24T03:33:39.530 INFO:tasks.workunit.client.0.smithi112.stdout: create 257/.snap/snap-subdir-test 2023-01-24T03:33:39.551 INFO:tasks.workunit.client.0.smithi112.stdout: create 257/30 file after the snapshot 2023-01-24T03:33:39.552 INFO:tasks.workunit.client.0.smithi112.stdout: create 257/31 file after the snapshot 2023-01-24T03:33:39.553 INFO:tasks.workunit.client.0.smithi112.stdout: create 257/32 file after the snapshot 2023-01-24T03:33:39.554 INFO:tasks.workunit.client.0.smithi112.stdout: create 257/33 file after the snapshot 2023-01-24T03:33:39.555 INFO:tasks.workunit.client.0.smithi112.stdout: create 257/34 file after the snapshot 2023-01-24T03:33:39.556 INFO:tasks.workunit.client.0.smithi112.stdout: create 257/35 file after the snapshot 2023-01-24T03:33:39.557 INFO:tasks.workunit.client.0.smithi112.stdout: create 257/36 file after the snapshot 2023-01-24T03:33:39.558 INFO:tasks.workunit.client.0.smithi112.stdout: create 257/37 file after the snapshot 2023-01-24T03:33:39.559 INFO:tasks.workunit.client.0.smithi112.stdout: create 257/38 file after the snapshot 2023-01-24T03:33:39.560 INFO:tasks.workunit.client.0.smithi112.stdout: create 257/39 file after the snapshot 2023-01-24T03:33:39.561 INFO:tasks.workunit.client.0.smithi112.stdout: create 258/.snap/snap-subdir-test 2023-01-24T03:33:39.582 INFO:tasks.workunit.client.0.smithi112.stdout: create 258/30 file after the snapshot 2023-01-24T03:33:39.584 INFO:tasks.workunit.client.0.smithi112.stdout: create 258/31 file after the snapshot 2023-01-24T03:33:39.585 INFO:tasks.workunit.client.0.smithi112.stdout: create 258/32 file after the snapshot 2023-01-24T03:33:39.586 INFO:tasks.workunit.client.0.smithi112.stdout: create 258/33 file after the snapshot 2023-01-24T03:33:39.587 INFO:tasks.workunit.client.0.smithi112.stdout: create 258/34 file after the snapshot 2023-01-24T03:33:39.588 INFO:tasks.workunit.client.0.smithi112.stdout: create 258/35 file after the snapshot 2023-01-24T03:33:39.590 INFO:tasks.workunit.client.0.smithi112.stdout: create 258/36 file after the snapshot 2023-01-24T03:33:39.590 INFO:tasks.workunit.client.0.smithi112.stdout: create 258/37 file after the snapshot 2023-01-24T03:33:39.591 INFO:tasks.workunit.client.0.smithi112.stdout: create 258/38 file after the snapshot 2023-01-24T03:33:39.592 INFO:tasks.workunit.client.0.smithi112.stdout: create 258/39 file after the snapshot 2023-01-24T03:33:40.339 INFO:tasks.workunit.client.0.smithi112.stdout: create 259/.snap/snap-subdir-test 2023-01-24T03:33:40.365 INFO:tasks.workunit.client.0.smithi112.stdout: create 259/30 file after the snapshot 2023-01-24T03:33:40.366 INFO:tasks.workunit.client.0.smithi112.stdout: create 259/31 file after the snapshot 2023-01-24T03:33:40.368 INFO:tasks.workunit.client.0.smithi112.stdout: create 259/32 file after the snapshot 2023-01-24T03:33:40.368 INFO:tasks.workunit.client.0.smithi112.stdout: create 259/33 file after the snapshot 2023-01-24T03:33:40.369 INFO:tasks.workunit.client.0.smithi112.stdout: create 259/34 file after the snapshot 2023-01-24T03:33:40.370 INFO:tasks.workunit.client.0.smithi112.stdout: create 259/35 file after the snapshot 2023-01-24T03:33:40.371 INFO:tasks.workunit.client.0.smithi112.stdout: create 259/36 file after the snapshot 2023-01-24T03:33:40.372 INFO:tasks.workunit.client.0.smithi112.stdout: create 259/37 file after the snapshot 2023-01-24T03:33:40.373 INFO:tasks.workunit.client.0.smithi112.stdout: create 259/38 file after the snapshot 2023-01-24T03:33:40.374 INFO:tasks.workunit.client.0.smithi112.stdout: create 259/39 file after the snapshot 2023-01-24T03:33:40.375 INFO:tasks.workunit.client.0.smithi112.stdout: create 260/.snap/snap-subdir-test 2023-01-24T03:33:40.409 INFO:tasks.workunit.client.0.smithi112.stdout: create 260/30 file after the snapshot 2023-01-24T03:33:40.675 INFO:tasks.workunit.client.0.smithi112.stdout: create 260/31 file after the snapshot 2023-01-24T03:33:40.676 INFO:tasks.workunit.client.0.smithi112.stdout: create 260/32 file after the snapshot 2023-01-24T03:33:40.677 INFO:tasks.workunit.client.0.smithi112.stdout: create 260/33 file after the snapshot 2023-01-24T03:33:40.678 INFO:tasks.workunit.client.0.smithi112.stdout: create 260/34 file after the snapshot 2023-01-24T03:33:40.679 INFO:tasks.workunit.client.0.smithi112.stdout: create 260/35 file after the snapshot 2023-01-24T03:33:40.681 INFO:tasks.workunit.client.0.smithi112.stdout: create 260/36 file after the snapshot 2023-01-24T03:33:40.681 INFO:tasks.workunit.client.0.smithi112.stdout: create 260/37 file after the snapshot 2023-01-24T03:33:40.682 INFO:tasks.workunit.client.0.smithi112.stdout: create 260/38 file after the snapshot 2023-01-24T03:33:40.683 INFO:tasks.workunit.client.0.smithi112.stdout: create 260/39 file after the snapshot 2023-01-24T03:33:40.684 INFO:tasks.workunit.client.0.smithi112.stdout: create 261/.snap/snap-subdir-test 2023-01-24T03:33:40.717 INFO:tasks.workunit.client.0.smithi112.stdout: create 261/30 file after the snapshot 2023-01-24T03:33:40.719 INFO:tasks.workunit.client.0.smithi112.stdout: create 261/31 file after the snapshot 2023-01-24T03:33:40.719 INFO:tasks.workunit.client.0.smithi112.stdout: create 261/32 file after the snapshot 2023-01-24T03:33:40.720 INFO:tasks.workunit.client.0.smithi112.stdout: create 261/33 file after the snapshot 2023-01-24T03:33:40.721 INFO:tasks.workunit.client.0.smithi112.stdout: create 261/34 file after the snapshot 2023-01-24T03:33:40.721 INFO:tasks.workunit.client.0.smithi112.stdout: create 261/35 file after the snapshot 2023-01-24T03:33:40.722 INFO:tasks.workunit.client.0.smithi112.stdout: create 261/36 file after the snapshot 2023-01-24T03:33:40.723 INFO:tasks.workunit.client.0.smithi112.stdout: create 261/37 file after the snapshot 2023-01-24T03:33:40.724 INFO:tasks.workunit.client.0.smithi112.stdout: create 261/38 file after the snapshot 2023-01-24T03:33:40.725 INFO:tasks.workunit.client.0.smithi112.stdout: create 261/39 file after the snapshot 2023-01-24T03:33:40.726 INFO:tasks.workunit.client.0.smithi112.stdout: create 262/.snap/snap-subdir-test 2023-01-24T03:33:40.755 INFO:tasks.workunit.client.0.smithi112.stdout: create 262/30 file after the snapshot 2023-01-24T03:33:40.756 INFO:tasks.workunit.client.0.smithi112.stdout: create 262/31 file after the snapshot 2023-01-24T03:33:40.757 INFO:tasks.workunit.client.0.smithi112.stdout: create 262/32 file after the snapshot 2023-01-24T03:33:40.757 INFO:tasks.workunit.client.0.smithi112.stdout: create 262/33 file after the snapshot 2023-01-24T03:33:40.758 INFO:tasks.workunit.client.0.smithi112.stdout: create 262/34 file after the snapshot 2023-01-24T03:33:40.759 INFO:tasks.workunit.client.0.smithi112.stdout: create 262/35 file after the snapshot 2023-01-24T03:33:40.760 INFO:tasks.workunit.client.0.smithi112.stdout: create 262/36 file after the snapshot 2023-01-24T03:33:40.761 INFO:tasks.workunit.client.0.smithi112.stdout: create 262/37 file after the snapshot 2023-01-24T03:33:40.762 INFO:tasks.workunit.client.0.smithi112.stdout: create 262/38 file after the snapshot 2023-01-24T03:33:40.763 INFO:tasks.workunit.client.0.smithi112.stdout: create 262/39 file after the snapshot 2023-01-24T03:33:40.764 INFO:tasks.workunit.client.0.smithi112.stdout: create 263/.snap/snap-subdir-test 2023-01-24T03:33:40.792 INFO:tasks.workunit.client.0.smithi112.stdout: create 263/30 file after the snapshot 2023-01-24T03:33:40.793 INFO:tasks.workunit.client.0.smithi112.stdout: create 263/31 file after the snapshot 2023-01-24T03:33:40.794 INFO:tasks.workunit.client.0.smithi112.stdout: create 263/32 file after the snapshot 2023-01-24T03:33:40.795 INFO:tasks.workunit.client.0.smithi112.stdout: create 263/33 file after the snapshot 2023-01-24T03:33:40.796 INFO:tasks.workunit.client.0.smithi112.stdout: create 263/34 file after the snapshot 2023-01-24T03:33:40.797 INFO:tasks.workunit.client.0.smithi112.stdout: create 263/35 file after the snapshot 2023-01-24T03:33:40.798 INFO:tasks.workunit.client.0.smithi112.stdout: create 263/36 file after the snapshot 2023-01-24T03:33:40.799 INFO:tasks.workunit.client.0.smithi112.stdout: create 263/37 file after the snapshot 2023-01-24T03:33:40.800 INFO:tasks.workunit.client.0.smithi112.stdout: create 263/38 file after the snapshot 2023-01-24T03:33:40.801 INFO:tasks.workunit.client.0.smithi112.stdout: create 263/39 file after the snapshot 2023-01-24T03:33:40.802 INFO:tasks.workunit.client.0.smithi112.stdout: create 264/.snap/snap-subdir-test 2023-01-24T03:33:40.829 INFO:tasks.workunit.client.0.smithi112.stdout: create 264/30 file after the snapshot 2023-01-24T03:33:40.831 INFO:tasks.workunit.client.0.smithi112.stdout: create 264/31 file after the snapshot 2023-01-24T03:33:40.832 INFO:tasks.workunit.client.0.smithi112.stdout: create 264/32 file after the snapshot 2023-01-24T03:33:40.833 INFO:tasks.workunit.client.0.smithi112.stdout: create 264/33 file after the snapshot 2023-01-24T03:33:40.833 INFO:tasks.workunit.client.0.smithi112.stdout: create 264/34 file after the snapshot 2023-01-24T03:33:40.834 INFO:tasks.workunit.client.0.smithi112.stdout: create 264/35 file after the snapshot 2023-01-24T03:33:40.835 INFO:tasks.workunit.client.0.smithi112.stdout: create 264/36 file after the snapshot 2023-01-24T03:33:40.836 INFO:tasks.workunit.client.0.smithi112.stdout: create 264/37 file after the snapshot 2023-01-24T03:33:40.837 INFO:tasks.workunit.client.0.smithi112.stdout: create 264/38 file after the snapshot 2023-01-24T03:33:40.838 INFO:tasks.workunit.client.0.smithi112.stdout: create 264/39 file after the snapshot 2023-01-24T03:33:40.839 INFO:tasks.workunit.client.0.smithi112.stdout: create 265/.snap/snap-subdir-test 2023-01-24T03:33:40.868 INFO:tasks.workunit.client.0.smithi112.stdout: create 265/30 file after the snapshot 2023-01-24T03:33:40.869 INFO:tasks.workunit.client.0.smithi112.stdout: create 265/31 file after the snapshot 2023-01-24T03:33:40.870 INFO:tasks.workunit.client.0.smithi112.stdout: create 265/32 file after the snapshot 2023-01-24T03:33:40.871 INFO:tasks.workunit.client.0.smithi112.stdout: create 265/33 file after the snapshot 2023-01-24T03:33:40.872 INFO:tasks.workunit.client.0.smithi112.stdout: create 265/34 file after the snapshot 2023-01-24T03:33:40.873 INFO:tasks.workunit.client.0.smithi112.stdout: create 265/35 file after the snapshot 2023-01-24T03:33:40.874 INFO:tasks.workunit.client.0.smithi112.stdout: create 265/36 file after the snapshot 2023-01-24T03:33:40.875 INFO:tasks.workunit.client.0.smithi112.stdout: create 265/37 file after the snapshot 2023-01-24T03:33:40.876 INFO:tasks.workunit.client.0.smithi112.stdout: create 265/38 file after the snapshot 2023-01-24T03:33:40.877 INFO:tasks.workunit.client.0.smithi112.stdout: create 265/39 file after the snapshot 2023-01-24T03:33:40.878 INFO:tasks.workunit.client.0.smithi112.stdout: create 266/.snap/snap-subdir-test 2023-01-24T03:33:40.909 INFO:tasks.workunit.client.0.smithi112.stdout: create 266/30 file after the snapshot 2023-01-24T03:33:40.910 INFO:tasks.workunit.client.0.smithi112.stdout: create 266/31 file after the snapshot 2023-01-24T03:33:40.911 INFO:tasks.workunit.client.0.smithi112.stdout: create 266/32 file after the snapshot 2023-01-24T03:33:40.912 INFO:tasks.workunit.client.0.smithi112.stdout: create 266/33 file after the snapshot 2023-01-24T03:33:40.913 INFO:tasks.workunit.client.0.smithi112.stdout: create 266/34 file after the snapshot 2023-01-24T03:33:40.913 INFO:tasks.workunit.client.0.smithi112.stdout: create 266/35 file after the snapshot 2023-01-24T03:33:40.914 INFO:tasks.workunit.client.0.smithi112.stdout: create 266/36 file after the snapshot 2023-01-24T03:33:40.916 INFO:tasks.workunit.client.0.smithi112.stdout: create 266/37 file after the snapshot 2023-01-24T03:33:40.916 INFO:tasks.workunit.client.0.smithi112.stdout: create 266/38 file after the snapshot 2023-01-24T03:33:40.916 INFO:tasks.workunit.client.0.smithi112.stdout: create 266/39 file after the snapshot 2023-01-24T03:33:40.917 INFO:tasks.workunit.client.0.smithi112.stdout: create 267/.snap/snap-subdir-test 2023-01-24T03:33:40.945 INFO:tasks.workunit.client.0.smithi112.stdout: create 267/30 file after the snapshot 2023-01-24T03:33:40.945 INFO:tasks.workunit.client.0.smithi112.stdout: create 267/31 file after the snapshot 2023-01-24T03:33:40.946 INFO:tasks.workunit.client.0.smithi112.stdout: create 267/32 file after the snapshot 2023-01-24T03:33:40.948 INFO:tasks.workunit.client.0.smithi112.stdout: create 267/33 file after the snapshot 2023-01-24T03:33:40.949 INFO:tasks.workunit.client.0.smithi112.stdout: create 267/34 file after the snapshot 2023-01-24T03:33:40.949 INFO:tasks.workunit.client.0.smithi112.stdout: create 267/35 file after the snapshot 2023-01-24T03:33:40.950 INFO:tasks.workunit.client.0.smithi112.stdout: create 267/36 file after the snapshot 2023-01-24T03:33:40.951 INFO:tasks.workunit.client.0.smithi112.stdout: create 267/37 file after the snapshot 2023-01-24T03:33:40.952 INFO:tasks.workunit.client.0.smithi112.stdout: create 267/38 file after the snapshot 2023-01-24T03:33:40.953 INFO:tasks.workunit.client.0.smithi112.stdout: create 267/39 file after the snapshot 2023-01-24T03:33:40.954 INFO:tasks.workunit.client.0.smithi112.stdout: create 268/.snap/snap-subdir-test 2023-01-24T03:33:40.983 INFO:tasks.workunit.client.0.smithi112.stdout: create 268/30 file after the snapshot 2023-01-24T03:33:40.984 INFO:tasks.workunit.client.0.smithi112.stdout: create 268/31 file after the snapshot 2023-01-24T03:33:40.985 INFO:tasks.workunit.client.0.smithi112.stdout: create 268/32 file after the snapshot 2023-01-24T03:33:40.986 INFO:tasks.workunit.client.0.smithi112.stdout: create 268/33 file after the snapshot 2023-01-24T03:33:40.987 INFO:tasks.workunit.client.0.smithi112.stdout: create 268/34 file after the snapshot 2023-01-24T03:33:40.987 INFO:tasks.workunit.client.0.smithi112.stdout: create 268/35 file after the snapshot 2023-01-24T03:33:40.988 INFO:tasks.workunit.client.0.smithi112.stdout: create 268/36 file after the snapshot 2023-01-24T03:33:40.989 INFO:tasks.workunit.client.0.smithi112.stdout: create 268/37 file after the snapshot 2023-01-24T03:33:40.990 INFO:tasks.workunit.client.0.smithi112.stdout: create 268/38 file after the snapshot 2023-01-24T03:33:40.991 INFO:tasks.workunit.client.0.smithi112.stdout: create 268/39 file after the snapshot 2023-01-24T03:33:40.992 INFO:tasks.workunit.client.0.smithi112.stdout: create 269/.snap/snap-subdir-test 2023-01-24T03:33:41.021 INFO:tasks.workunit.client.0.smithi112.stdout: create 269/30 file after the snapshot 2023-01-24T03:33:41.022 INFO:tasks.workunit.client.0.smithi112.stdout: create 269/31 file after the snapshot 2023-01-24T03:33:41.022 INFO:tasks.workunit.client.0.smithi112.stdout: create 269/32 file after the snapshot 2023-01-24T03:33:41.023 INFO:tasks.workunit.client.0.smithi112.stdout: create 269/33 file after the snapshot 2023-01-24T03:33:41.024 INFO:tasks.workunit.client.0.smithi112.stdout: create 269/34 file after the snapshot 2023-01-24T03:33:41.025 INFO:tasks.workunit.client.0.smithi112.stdout: create 269/35 file after the snapshot 2023-01-24T03:33:41.026 INFO:tasks.workunit.client.0.smithi112.stdout: create 269/36 file after the snapshot 2023-01-24T03:33:41.027 INFO:tasks.workunit.client.0.smithi112.stdout: create 269/37 file after the snapshot 2023-01-24T03:33:41.027 INFO:tasks.workunit.client.0.smithi112.stdout: create 269/38 file after the snapshot 2023-01-24T03:33:41.028 INFO:tasks.workunit.client.0.smithi112.stdout: create 269/39 file after the snapshot 2023-01-24T03:33:41.029 INFO:tasks.workunit.client.0.smithi112.stdout: create 270/.snap/snap-subdir-test 2023-01-24T03:33:41.050 INFO:tasks.workunit.client.0.smithi112.stdout: create 270/30 file after the snapshot 2023-01-24T03:33:41.052 INFO:tasks.workunit.client.0.smithi112.stdout: create 270/31 file after the snapshot 2023-01-24T03:33:41.052 INFO:tasks.workunit.client.0.smithi112.stdout: create 270/32 file after the snapshot 2023-01-24T03:33:41.053 INFO:tasks.workunit.client.0.smithi112.stdout: create 270/33 file after the snapshot 2023-01-24T03:33:41.054 INFO:tasks.workunit.client.0.smithi112.stdout: create 270/34 file after the snapshot 2023-01-24T03:33:41.054 INFO:tasks.workunit.client.0.smithi112.stdout: create 270/35 file after the snapshot 2023-01-24T03:33:41.055 INFO:tasks.workunit.client.0.smithi112.stdout: create 270/36 file after the snapshot 2023-01-24T03:33:41.056 INFO:tasks.workunit.client.0.smithi112.stdout: create 270/37 file after the snapshot 2023-01-24T03:33:41.057 INFO:tasks.workunit.client.0.smithi112.stdout: create 270/38 file after the snapshot 2023-01-24T03:33:41.058 INFO:tasks.workunit.client.0.smithi112.stdout: create 270/39 file after the snapshot 2023-01-24T03:33:41.059 INFO:tasks.workunit.client.0.smithi112.stdout: create 271/.snap/snap-subdir-test 2023-01-24T03:33:41.080 INFO:tasks.workunit.client.0.smithi112.stdout: create 271/30 file after the snapshot 2023-01-24T03:33:41.081 INFO:tasks.workunit.client.0.smithi112.stdout: create 271/31 file after the snapshot 2023-01-24T03:33:41.081 INFO:tasks.workunit.client.0.smithi112.stdout: create 271/32 file after the snapshot 2023-01-24T03:33:41.082 INFO:tasks.workunit.client.0.smithi112.stdout: create 271/33 file after the snapshot 2023-01-24T03:33:41.083 INFO:tasks.workunit.client.0.smithi112.stdout: create 271/34 file after the snapshot 2023-01-24T03:33:41.084 INFO:tasks.workunit.client.0.smithi112.stdout: create 271/35 file after the snapshot 2023-01-24T03:33:41.084 INFO:tasks.workunit.client.0.smithi112.stdout: create 271/36 file after the snapshot 2023-01-24T03:33:41.085 INFO:tasks.workunit.client.0.smithi112.stdout: create 271/37 file after the snapshot 2023-01-24T03:33:41.086 INFO:tasks.workunit.client.0.smithi112.stdout: create 271/38 file after the snapshot 2023-01-24T03:33:41.087 INFO:tasks.workunit.client.0.smithi112.stdout: create 271/39 file after the snapshot 2023-01-24T03:33:41.087 INFO:tasks.workunit.client.0.smithi112.stdout: create 272/.snap/snap-subdir-test 2023-01-24T03:33:41.108 INFO:tasks.workunit.client.0.smithi112.stdout: create 272/30 file after the snapshot 2023-01-24T03:33:41.109 INFO:tasks.workunit.client.0.smithi112.stdout: create 272/31 file after the snapshot 2023-01-24T03:33:41.110 INFO:tasks.workunit.client.0.smithi112.stdout: create 272/32 file after the snapshot 2023-01-24T03:33:41.111 INFO:tasks.workunit.client.0.smithi112.stdout: create 272/33 file after the snapshot 2023-01-24T03:33:41.111 INFO:tasks.workunit.client.0.smithi112.stdout: create 272/34 file after the snapshot 2023-01-24T03:33:41.112 INFO:tasks.workunit.client.0.smithi112.stdout: create 272/35 file after the snapshot 2023-01-24T03:33:41.113 INFO:tasks.workunit.client.0.smithi112.stdout: create 272/36 file after the snapshot 2023-01-24T03:33:41.114 INFO:tasks.workunit.client.0.smithi112.stdout: create 272/37 file after the snapshot 2023-01-24T03:33:41.114 INFO:tasks.workunit.client.0.smithi112.stdout: create 272/38 file after the snapshot 2023-01-24T03:33:41.115 INFO:tasks.workunit.client.0.smithi112.stdout: create 272/39 file after the snapshot 2023-01-24T03:33:41.116 INFO:tasks.workunit.client.0.smithi112.stdout: create 273/.snap/snap-subdir-test 2023-01-24T03:33:41.137 INFO:tasks.workunit.client.0.smithi112.stdout: create 273/30 file after the snapshot 2023-01-24T03:33:41.138 INFO:tasks.workunit.client.0.smithi112.stdout: create 273/31 file after the snapshot 2023-01-24T03:33:41.139 INFO:tasks.workunit.client.0.smithi112.stdout: create 273/32 file after the snapshot 2023-01-24T03:33:41.140 INFO:tasks.workunit.client.0.smithi112.stdout: create 273/33 file after the snapshot 2023-01-24T03:33:41.140 INFO:tasks.workunit.client.0.smithi112.stdout: create 273/34 file after the snapshot 2023-01-24T03:33:41.142 INFO:tasks.workunit.client.0.smithi112.stdout: create 273/35 file after the snapshot 2023-01-24T03:33:41.142 INFO:tasks.workunit.client.0.smithi112.stdout: create 273/36 file after the snapshot 2023-01-24T03:33:41.143 INFO:tasks.workunit.client.0.smithi112.stdout: create 273/37 file after the snapshot 2023-01-24T03:33:41.144 INFO:tasks.workunit.client.0.smithi112.stdout: create 273/38 file after the snapshot 2023-01-24T03:33:41.145 INFO:tasks.workunit.client.0.smithi112.stdout: create 273/39 file after the snapshot 2023-01-24T03:33:41.145 INFO:tasks.workunit.client.0.smithi112.stdout: create 274/.snap/snap-subdir-test 2023-01-24T03:33:41.166 INFO:tasks.workunit.client.0.smithi112.stdout: create 274/30 file after the snapshot 2023-01-24T03:33:41.167 INFO:tasks.workunit.client.0.smithi112.stdout: create 274/31 file after the snapshot 2023-01-24T03:33:41.168 INFO:tasks.workunit.client.0.smithi112.stdout: create 274/32 file after the snapshot 2023-01-24T03:33:41.169 INFO:tasks.workunit.client.0.smithi112.stdout: create 274/33 file after the snapshot 2023-01-24T03:33:41.170 INFO:tasks.workunit.client.0.smithi112.stdout: create 274/34 file after the snapshot 2023-01-24T03:33:41.171 INFO:tasks.workunit.client.0.smithi112.stdout: create 274/35 file after the snapshot 2023-01-24T03:33:41.171 INFO:tasks.workunit.client.0.smithi112.stdout: create 274/36 file after the snapshot 2023-01-24T03:33:41.172 INFO:tasks.workunit.client.0.smithi112.stdout: create 274/37 file after the snapshot 2023-01-24T03:33:41.173 INFO:tasks.workunit.client.0.smithi112.stdout: create 274/38 file after the snapshot 2023-01-24T03:33:41.173 INFO:tasks.workunit.client.0.smithi112.stdout: create 274/39 file after the snapshot 2023-01-24T03:33:41.174 INFO:tasks.workunit.client.0.smithi112.stdout: create 275/.snap/snap-subdir-test 2023-01-24T03:33:41.195 INFO:tasks.workunit.client.0.smithi112.stdout: create 275/30 file after the snapshot 2023-01-24T03:33:41.196 INFO:tasks.workunit.client.0.smithi112.stdout: create 275/31 file after the snapshot 2023-01-24T03:33:41.196 INFO:tasks.workunit.client.0.smithi112.stdout: create 275/32 file after the snapshot 2023-01-24T03:33:41.197 INFO:tasks.workunit.client.0.smithi112.stdout: create 275/33 file after the snapshot 2023-01-24T03:33:41.198 INFO:tasks.workunit.client.0.smithi112.stdout: create 275/34 file after the snapshot 2023-01-24T03:33:41.198 INFO:tasks.workunit.client.0.smithi112.stdout: create 275/35 file after the snapshot 2023-01-24T03:33:41.201 INFO:tasks.workunit.client.0.smithi112.stdout: create 275/36 file after the snapshot 2023-01-24T03:33:41.201 INFO:tasks.workunit.client.0.smithi112.stdout: create 275/37 file after the snapshot 2023-01-24T03:33:41.201 INFO:tasks.workunit.client.0.smithi112.stdout: create 275/38 file after the snapshot 2023-01-24T03:33:41.202 INFO:tasks.workunit.client.0.smithi112.stdout: create 275/39 file after the snapshot 2023-01-24T03:33:41.202 INFO:tasks.workunit.client.0.smithi112.stdout: create 276/.snap/snap-subdir-test 2023-01-24T03:33:41.225 INFO:tasks.workunit.client.0.smithi112.stdout: create 276/30 file after the snapshot 2023-01-24T03:33:41.226 INFO:tasks.workunit.client.0.smithi112.stdout: create 276/31 file after the snapshot 2023-01-24T03:33:41.227 INFO:tasks.workunit.client.0.smithi112.stdout: create 276/32 file after the snapshot 2023-01-24T03:33:41.228 INFO:tasks.workunit.client.0.smithi112.stdout: create 276/33 file after the snapshot 2023-01-24T03:33:41.228 INFO:tasks.workunit.client.0.smithi112.stdout: create 276/34 file after the snapshot 2023-01-24T03:33:41.229 INFO:tasks.workunit.client.0.smithi112.stdout: create 276/35 file after the snapshot 2023-01-24T03:33:41.230 INFO:tasks.workunit.client.0.smithi112.stdout: create 276/36 file after the snapshot 2023-01-24T03:33:41.231 INFO:tasks.workunit.client.0.smithi112.stdout: create 276/37 file after the snapshot 2023-01-24T03:33:41.232 INFO:tasks.workunit.client.0.smithi112.stdout: create 276/38 file after the snapshot 2023-01-24T03:33:41.232 INFO:tasks.workunit.client.0.smithi112.stdout: create 276/39 file after the snapshot 2023-01-24T03:33:41.233 INFO:tasks.workunit.client.0.smithi112.stdout: create 277/.snap/snap-subdir-test 2023-01-24T03:33:41.831 INFO:tasks.workunit.client.0.smithi112.stdout: create 277/30 file after the snapshot 2023-01-24T03:33:41.832 INFO:tasks.workunit.client.0.smithi112.stdout: create 277/31 file after the snapshot 2023-01-24T03:33:41.833 INFO:tasks.workunit.client.0.smithi112.stdout: create 277/32 file after the snapshot 2023-01-24T03:33:41.834 INFO:tasks.workunit.client.0.smithi112.stdout: create 277/33 file after the snapshot 2023-01-24T03:33:41.835 INFO:tasks.workunit.client.0.smithi112.stdout: create 277/34 file after the snapshot 2023-01-24T03:33:41.836 INFO:tasks.workunit.client.0.smithi112.stdout: create 277/35 file after the snapshot 2023-01-24T03:33:41.837 INFO:tasks.workunit.client.0.smithi112.stdout: create 277/36 file after the snapshot 2023-01-24T03:33:41.838 INFO:tasks.workunit.client.0.smithi112.stdout: create 277/37 file after the snapshot 2023-01-24T03:33:41.839 INFO:tasks.workunit.client.0.smithi112.stdout: create 277/38 file after the snapshot 2023-01-24T03:33:41.840 INFO:tasks.workunit.client.0.smithi112.stdout: create 277/39 file after the snapshot 2023-01-24T03:33:41.840 INFO:tasks.workunit.client.0.smithi112.stdout: create 278/.snap/snap-subdir-test 2023-01-24T03:33:41.862 INFO:tasks.workunit.client.0.smithi112.stdout: create 278/30 file after the snapshot 2023-01-24T03:33:41.863 INFO:tasks.workunit.client.0.smithi112.stdout: create 278/31 file after the snapshot 2023-01-24T03:33:41.864 INFO:tasks.workunit.client.0.smithi112.stdout: create 278/32 file after the snapshot 2023-01-24T03:33:41.865 INFO:tasks.workunit.client.0.smithi112.stdout: create 278/33 file after the snapshot 2023-01-24T03:33:41.866 INFO:tasks.workunit.client.0.smithi112.stdout: create 278/34 file after the snapshot 2023-01-24T03:33:41.867 INFO:tasks.workunit.client.0.smithi112.stdout: create 278/35 file after the snapshot 2023-01-24T03:33:41.868 INFO:tasks.workunit.client.0.smithi112.stdout: create 278/36 file after the snapshot 2023-01-24T03:33:41.869 INFO:tasks.workunit.client.0.smithi112.stdout: create 278/37 file after the snapshot 2023-01-24T03:33:41.870 INFO:tasks.workunit.client.0.smithi112.stdout: create 278/38 file after the snapshot 2023-01-24T03:33:41.870 INFO:tasks.workunit.client.0.smithi112.stdout: create 278/39 file after the snapshot 2023-01-24T03:33:41.871 INFO:tasks.workunit.client.0.smithi112.stdout: create 279/.snap/snap-subdir-test 2023-01-24T03:33:41.892 INFO:tasks.workunit.client.0.smithi112.stdout: create 279/30 file after the snapshot 2023-01-24T03:33:41.893 INFO:tasks.workunit.client.0.smithi112.stdout: create 279/31 file after the snapshot 2023-01-24T03:33:41.894 INFO:tasks.workunit.client.0.smithi112.stdout: create 279/32 file after the snapshot 2023-01-24T03:33:41.895 INFO:tasks.workunit.client.0.smithi112.stdout: create 279/33 file after the snapshot 2023-01-24T03:33:41.896 INFO:tasks.workunit.client.0.smithi112.stdout: create 279/34 file after the snapshot 2023-01-24T03:33:41.897 INFO:tasks.workunit.client.0.smithi112.stdout: create 279/35 file after the snapshot 2023-01-24T03:33:41.898 INFO:tasks.workunit.client.0.smithi112.stdout: create 279/36 file after the snapshot 2023-01-24T03:33:41.899 INFO:tasks.workunit.client.0.smithi112.stdout: create 279/37 file after the snapshot 2023-01-24T03:33:41.900 INFO:tasks.workunit.client.0.smithi112.stdout: create 279/38 file after the snapshot 2023-01-24T03:33:41.901 INFO:tasks.workunit.client.0.smithi112.stdout: create 279/39 file after the snapshot 2023-01-24T03:33:41.902 INFO:tasks.workunit.client.0.smithi112.stdout: create 280/.snap/snap-subdir-test 2023-01-24T03:33:41.923 INFO:tasks.workunit.client.0.smithi112.stdout: create 280/30 file after the snapshot 2023-01-24T03:33:41.924 INFO:tasks.workunit.client.0.smithi112.stdout: create 280/31 file after the snapshot 2023-01-24T03:33:41.925 INFO:tasks.workunit.client.0.smithi112.stdout: create 280/32 file after the snapshot 2023-01-24T03:33:41.926 INFO:tasks.workunit.client.0.smithi112.stdout: create 280/33 file after the snapshot 2023-01-24T03:33:41.927 INFO:tasks.workunit.client.0.smithi112.stdout: create 280/34 file after the snapshot 2023-01-24T03:33:41.928 INFO:tasks.workunit.client.0.smithi112.stdout: create 280/35 file after the snapshot 2023-01-24T03:33:41.929 INFO:tasks.workunit.client.0.smithi112.stdout: create 280/36 file after the snapshot 2023-01-24T03:33:41.930 INFO:tasks.workunit.client.0.smithi112.stdout: create 280/37 file after the snapshot 2023-01-24T03:33:41.931 INFO:tasks.workunit.client.0.smithi112.stdout: create 280/38 file after the snapshot 2023-01-24T03:33:41.932 INFO:tasks.workunit.client.0.smithi112.stdout: create 280/39 file after the snapshot 2023-01-24T03:33:41.932 INFO:tasks.workunit.client.0.smithi112.stdout: create 281/.snap/snap-subdir-test 2023-01-24T03:33:41.955 INFO:tasks.workunit.client.0.smithi112.stdout: create 281/30 file after the snapshot 2023-01-24T03:33:41.955 INFO:tasks.workunit.client.0.smithi112.stdout: create 281/31 file after the snapshot 2023-01-24T03:33:41.956 INFO:tasks.workunit.client.0.smithi112.stdout: create 281/32 file after the snapshot 2023-01-24T03:33:41.957 INFO:tasks.workunit.client.0.smithi112.stdout: create 281/33 file after the snapshot 2023-01-24T03:33:41.958 INFO:tasks.workunit.client.0.smithi112.stdout: create 281/34 file after the snapshot 2023-01-24T03:33:41.959 INFO:tasks.workunit.client.0.smithi112.stdout: create 281/35 file after the snapshot 2023-01-24T03:33:41.959 INFO:tasks.workunit.client.0.smithi112.stdout: create 281/36 file after the snapshot 2023-01-24T03:33:41.960 INFO:tasks.workunit.client.0.smithi112.stdout: create 281/37 file after the snapshot 2023-01-24T03:33:41.961 INFO:tasks.workunit.client.0.smithi112.stdout: create 281/38 file after the snapshot 2023-01-24T03:33:41.961 INFO:tasks.workunit.client.0.smithi112.stdout: create 281/39 file after the snapshot 2023-01-24T03:33:41.962 INFO:tasks.workunit.client.0.smithi112.stdout: create 282/.snap/snap-subdir-test 2023-01-24T03:33:41.984 INFO:tasks.workunit.client.0.smithi112.stdout: create 282/30 file after the snapshot 2023-01-24T03:33:41.985 INFO:tasks.workunit.client.0.smithi112.stdout: create 282/31 file after the snapshot 2023-01-24T03:33:41.986 INFO:tasks.workunit.client.0.smithi112.stdout: create 282/32 file after the snapshot 2023-01-24T03:33:41.987 INFO:tasks.workunit.client.0.smithi112.stdout: create 282/33 file after the snapshot 2023-01-24T03:33:41.988 INFO:tasks.workunit.client.0.smithi112.stdout: create 282/34 file after the snapshot 2023-01-24T03:33:41.988 INFO:tasks.workunit.client.0.smithi112.stdout: create 282/35 file after the snapshot 2023-01-24T03:33:41.989 INFO:tasks.workunit.client.0.smithi112.stdout: create 282/36 file after the snapshot 2023-01-24T03:33:41.990 INFO:tasks.workunit.client.0.smithi112.stdout: create 282/37 file after the snapshot 2023-01-24T03:33:41.991 INFO:tasks.workunit.client.0.smithi112.stdout: create 282/38 file after the snapshot 2023-01-24T03:33:41.991 INFO:tasks.workunit.client.0.smithi112.stdout: create 282/39 file after the snapshot 2023-01-24T03:33:41.992 INFO:tasks.workunit.client.0.smithi112.stdout: create 283/.snap/snap-subdir-test 2023-01-24T03:33:42.015 INFO:tasks.workunit.client.0.smithi112.stdout: create 283/30 file after the snapshot 2023-01-24T03:33:42.016 INFO:tasks.workunit.client.0.smithi112.stdout: create 283/31 file after the snapshot 2023-01-24T03:33:42.017 INFO:tasks.workunit.client.0.smithi112.stdout: create 283/32 file after the snapshot 2023-01-24T03:33:42.018 INFO:tasks.workunit.client.0.smithi112.stdout: create 283/33 file after the snapshot 2023-01-24T03:33:42.018 INFO:tasks.workunit.client.0.smithi112.stdout: create 283/34 file after the snapshot 2023-01-24T03:33:42.019 INFO:tasks.workunit.client.0.smithi112.stdout: create 283/35 file after the snapshot 2023-01-24T03:33:42.020 INFO:tasks.workunit.client.0.smithi112.stdout: create 283/36 file after the snapshot 2023-01-24T03:33:42.021 INFO:tasks.workunit.client.0.smithi112.stdout: create 283/37 file after the snapshot 2023-01-24T03:33:42.022 INFO:tasks.workunit.client.0.smithi112.stdout: create 283/38 file after the snapshot 2023-01-24T03:33:42.022 INFO:tasks.workunit.client.0.smithi112.stdout: create 283/39 file after the snapshot 2023-01-24T03:33:42.023 INFO:tasks.workunit.client.0.smithi112.stdout: create 284/.snap/snap-subdir-test 2023-01-24T03:33:42.045 INFO:tasks.workunit.client.0.smithi112.stdout: create 284/30 file after the snapshot 2023-01-24T03:33:42.046 INFO:tasks.workunit.client.0.smithi112.stdout: create 284/31 file after the snapshot 2023-01-24T03:33:42.047 INFO:tasks.workunit.client.0.smithi112.stdout: create 284/32 file after the snapshot 2023-01-24T03:33:42.047 INFO:tasks.workunit.client.0.smithi112.stdout: create 284/33 file after the snapshot 2023-01-24T03:33:42.048 INFO:tasks.workunit.client.0.smithi112.stdout: create 284/34 file after the snapshot 2023-01-24T03:33:42.049 INFO:tasks.workunit.client.0.smithi112.stdout: create 284/35 file after the snapshot 2023-01-24T03:33:42.050 INFO:tasks.workunit.client.0.smithi112.stdout: create 284/36 file after the snapshot 2023-01-24T03:33:42.051 INFO:tasks.workunit.client.0.smithi112.stdout: create 284/37 file after the snapshot 2023-01-24T03:33:42.052 INFO:tasks.workunit.client.0.smithi112.stdout: create 284/38 file after the snapshot 2023-01-24T03:33:42.053 INFO:tasks.workunit.client.0.smithi112.stdout: create 284/39 file after the snapshot 2023-01-24T03:33:42.053 INFO:tasks.workunit.client.0.smithi112.stdout: create 285/.snap/snap-subdir-test 2023-01-24T03:33:42.074 INFO:tasks.workunit.client.0.smithi112.stdout: create 285/30 file after the snapshot 2023-01-24T03:33:42.076 INFO:tasks.workunit.client.0.smithi112.stdout: create 285/31 file after the snapshot 2023-01-24T03:33:42.077 INFO:tasks.workunit.client.0.smithi112.stdout: create 285/32 file after the snapshot 2023-01-24T03:33:42.078 INFO:tasks.workunit.client.0.smithi112.stdout: create 285/33 file after the snapshot 2023-01-24T03:33:42.079 INFO:tasks.workunit.client.0.smithi112.stdout: create 285/34 file after the snapshot 2023-01-24T03:33:42.080 INFO:tasks.workunit.client.0.smithi112.stdout: create 285/35 file after the snapshot 2023-01-24T03:33:42.081 INFO:tasks.workunit.client.0.smithi112.stdout: create 285/36 file after the snapshot 2023-01-24T03:33:42.082 INFO:tasks.workunit.client.0.smithi112.stdout: create 285/37 file after the snapshot 2023-01-24T03:33:42.083 INFO:tasks.workunit.client.0.smithi112.stdout: create 285/38 file after the snapshot 2023-01-24T03:33:42.083 INFO:tasks.workunit.client.0.smithi112.stdout: create 285/39 file after the snapshot 2023-01-24T03:33:42.084 INFO:tasks.workunit.client.0.smithi112.stdout: create 286/.snap/snap-subdir-test 2023-01-24T03:33:42.105 INFO:tasks.workunit.client.0.smithi112.stdout: create 286/30 file after the snapshot 2023-01-24T03:33:42.106 INFO:tasks.workunit.client.0.smithi112.stdout: create 286/31 file after the snapshot 2023-01-24T03:33:42.107 INFO:tasks.workunit.client.0.smithi112.stdout: create 286/32 file after the snapshot 2023-01-24T03:33:42.108 INFO:tasks.workunit.client.0.smithi112.stdout: create 286/33 file after the snapshot 2023-01-24T03:33:42.108 INFO:tasks.workunit.client.0.smithi112.stdout: create 286/34 file after the snapshot 2023-01-24T03:33:42.109 INFO:tasks.workunit.client.0.smithi112.stdout: create 286/35 file after the snapshot 2023-01-24T03:33:42.110 INFO:tasks.workunit.client.0.smithi112.stdout: create 286/36 file after the snapshot 2023-01-24T03:33:42.110 INFO:tasks.workunit.client.0.smithi112.stdout: create 286/37 file after the snapshot 2023-01-24T03:33:42.111 INFO:tasks.workunit.client.0.smithi112.stdout: create 286/38 file after the snapshot 2023-01-24T03:33:42.112 INFO:tasks.workunit.client.0.smithi112.stdout: create 286/39 file after the snapshot 2023-01-24T03:33:42.113 INFO:tasks.workunit.client.0.smithi112.stdout: create 287/.snap/snap-subdir-test 2023-01-24T03:33:42.135 INFO:tasks.workunit.client.0.smithi112.stdout: create 287/30 file after the snapshot 2023-01-24T03:33:42.136 INFO:tasks.workunit.client.0.smithi112.stdout: create 287/31 file after the snapshot 2023-01-24T03:33:42.395 INFO:tasks.workunit.client.0.smithi112.stdout: create 287/32 file after the snapshot 2023-01-24T03:33:42.396 INFO:tasks.workunit.client.0.smithi112.stdout: create 287/33 file after the snapshot 2023-01-24T03:33:42.397 INFO:tasks.workunit.client.0.smithi112.stdout: create 287/34 file after the snapshot 2023-01-24T03:33:42.398 INFO:tasks.workunit.client.0.smithi112.stdout: create 287/35 file after the snapshot 2023-01-24T03:33:42.399 INFO:tasks.workunit.client.0.smithi112.stdout: create 287/36 file after the snapshot 2023-01-24T03:33:42.399 INFO:tasks.workunit.client.0.smithi112.stdout: create 287/37 file after the snapshot 2023-01-24T03:33:42.400 INFO:tasks.workunit.client.0.smithi112.stdout: create 287/38 file after the snapshot 2023-01-24T03:33:42.401 INFO:tasks.workunit.client.0.smithi112.stdout: create 287/39 file after the snapshot 2023-01-24T03:33:42.402 INFO:tasks.workunit.client.0.smithi112.stdout: create 288/.snap/snap-subdir-test 2023-01-24T03:33:42.424 INFO:tasks.workunit.client.0.smithi112.stdout: create 288/30 file after the snapshot 2023-01-24T03:33:42.426 INFO:tasks.workunit.client.0.smithi112.stdout: create 288/31 file after the snapshot 2023-01-24T03:33:42.427 INFO:tasks.workunit.client.0.smithi112.stdout: create 288/32 file after the snapshot 2023-01-24T03:33:42.428 INFO:tasks.workunit.client.0.smithi112.stdout: create 288/33 file after the snapshot 2023-01-24T03:33:42.428 INFO:tasks.workunit.client.0.smithi112.stdout: create 288/34 file after the snapshot 2023-01-24T03:33:42.429 INFO:tasks.workunit.client.0.smithi112.stdout: create 288/35 file after the snapshot 2023-01-24T03:33:42.430 INFO:tasks.workunit.client.0.smithi112.stdout: create 288/36 file after the snapshot 2023-01-24T03:33:42.431 INFO:tasks.workunit.client.0.smithi112.stdout: create 288/37 file after the snapshot 2023-01-24T03:33:42.432 INFO:tasks.workunit.client.0.smithi112.stdout: create 288/38 file after the snapshot 2023-01-24T03:33:42.432 INFO:tasks.workunit.client.0.smithi112.stdout: create 288/39 file after the snapshot 2023-01-24T03:33:42.433 INFO:tasks.workunit.client.0.smithi112.stdout: create 289/.snap/snap-subdir-test 2023-01-24T03:33:42.454 INFO:tasks.workunit.client.0.smithi112.stdout: create 289/30 file after the snapshot 2023-01-24T03:33:42.456 INFO:tasks.workunit.client.0.smithi112.stdout: create 289/31 file after the snapshot 2023-01-24T03:33:42.456 INFO:tasks.workunit.client.0.smithi112.stdout: create 289/32 file after the snapshot 2023-01-24T03:33:42.457 INFO:tasks.workunit.client.0.smithi112.stdout: create 289/33 file after the snapshot 2023-01-24T03:33:42.458 INFO:tasks.workunit.client.0.smithi112.stdout: create 289/34 file after the snapshot 2023-01-24T03:33:42.459 INFO:tasks.workunit.client.0.smithi112.stdout: create 289/35 file after the snapshot 2023-01-24T03:33:42.460 INFO:tasks.workunit.client.0.smithi112.stdout: create 289/36 file after the snapshot 2023-01-24T03:33:42.460 INFO:tasks.workunit.client.0.smithi112.stdout: create 289/37 file after the snapshot 2023-01-24T03:33:42.461 INFO:tasks.workunit.client.0.smithi112.stdout: create 289/38 file after the snapshot 2023-01-24T03:33:42.462 INFO:tasks.workunit.client.0.smithi112.stdout: create 289/39 file after the snapshot 2023-01-24T03:33:42.463 INFO:tasks.workunit.client.0.smithi112.stdout: create 290/.snap/snap-subdir-test 2023-01-24T03:33:42.484 INFO:tasks.workunit.client.0.smithi112.stdout: create 290/30 file after the snapshot 2023-01-24T03:33:42.485 INFO:tasks.workunit.client.0.smithi112.stdout: create 290/31 file after the snapshot 2023-01-24T03:33:42.486 INFO:tasks.workunit.client.0.smithi112.stdout: create 290/32 file after the snapshot 2023-01-24T03:33:42.487 INFO:tasks.workunit.client.0.smithi112.stdout: create 290/33 file after the snapshot 2023-01-24T03:33:42.488 INFO:tasks.workunit.client.0.smithi112.stdout: create 290/34 file after the snapshot 2023-01-24T03:33:42.489 INFO:tasks.workunit.client.0.smithi112.stdout: create 290/35 file after the snapshot 2023-01-24T03:33:42.490 INFO:tasks.workunit.client.0.smithi112.stdout: create 290/36 file after the snapshot 2023-01-24T03:33:42.491 INFO:tasks.workunit.client.0.smithi112.stdout: create 290/37 file after the snapshot 2023-01-24T03:33:42.492 INFO:tasks.workunit.client.0.smithi112.stdout: create 290/38 file after the snapshot 2023-01-24T03:33:42.493 INFO:tasks.workunit.client.0.smithi112.stdout: create 290/39 file after the snapshot 2023-01-24T03:33:42.494 INFO:tasks.workunit.client.0.smithi112.stdout: create 291/.snap/snap-subdir-test 2023-01-24T03:33:42.515 INFO:tasks.workunit.client.0.smithi112.stdout: create 291/30 file after the snapshot 2023-01-24T03:33:42.516 INFO:tasks.workunit.client.0.smithi112.stdout: create 291/31 file after the snapshot 2023-01-24T03:33:42.517 INFO:tasks.workunit.client.0.smithi112.stdout: create 291/32 file after the snapshot 2023-01-24T03:33:42.518 INFO:tasks.workunit.client.0.smithi112.stdout: create 291/33 file after the snapshot 2023-01-24T03:33:42.519 INFO:tasks.workunit.client.0.smithi112.stdout: create 291/34 file after the snapshot 2023-01-24T03:33:42.519 INFO:tasks.workunit.client.0.smithi112.stdout: create 291/35 file after the snapshot 2023-01-24T03:33:42.520 INFO:tasks.workunit.client.0.smithi112.stdout: create 291/36 file after the snapshot 2023-01-24T03:33:42.521 INFO:tasks.workunit.client.0.smithi112.stdout: create 291/37 file after the snapshot 2023-01-24T03:33:42.522 INFO:tasks.workunit.client.0.smithi112.stdout: create 291/38 file after the snapshot 2023-01-24T03:33:42.523 INFO:tasks.workunit.client.0.smithi112.stdout: create 291/39 file after the snapshot 2023-01-24T03:33:42.524 INFO:tasks.workunit.client.0.smithi112.stdout: create 292/.snap/snap-subdir-test 2023-01-24T03:33:42.545 INFO:tasks.workunit.client.0.smithi112.stdout: create 292/30 file after the snapshot 2023-01-24T03:33:42.546 INFO:tasks.workunit.client.0.smithi112.stdout: create 292/31 file after the snapshot 2023-01-24T03:33:42.547 INFO:tasks.workunit.client.0.smithi112.stdout: create 292/32 file after the snapshot 2023-01-24T03:33:42.548 INFO:tasks.workunit.client.0.smithi112.stdout: create 292/33 file after the snapshot 2023-01-24T03:33:42.549 INFO:tasks.workunit.client.0.smithi112.stdout: create 292/34 file after the snapshot 2023-01-24T03:33:42.550 INFO:tasks.workunit.client.0.smithi112.stdout: create 292/35 file after the snapshot 2023-01-24T03:33:42.551 INFO:tasks.workunit.client.0.smithi112.stdout: create 292/36 file after the snapshot 2023-01-24T03:33:42.551 INFO:tasks.workunit.client.0.smithi112.stdout: create 292/37 file after the snapshot 2023-01-24T03:33:42.552 INFO:tasks.workunit.client.0.smithi112.stdout: create 292/38 file after the snapshot 2023-01-24T03:33:42.553 INFO:tasks.workunit.client.0.smithi112.stdout: create 292/39 file after the snapshot 2023-01-24T03:33:42.554 INFO:tasks.workunit.client.0.smithi112.stdout: create 293/.snap/snap-subdir-test 2023-01-24T03:33:42.575 INFO:tasks.workunit.client.0.smithi112.stdout: create 293/30 file after the snapshot 2023-01-24T03:33:42.576 INFO:tasks.workunit.client.0.smithi112.stdout: create 293/31 file after the snapshot 2023-01-24T03:33:42.577 INFO:tasks.workunit.client.0.smithi112.stdout: create 293/32 file after the snapshot 2023-01-24T03:33:42.577 INFO:tasks.workunit.client.0.smithi112.stdout: create 293/33 file after the snapshot 2023-01-24T03:33:42.578 INFO:tasks.workunit.client.0.smithi112.stdout: create 293/34 file after the snapshot 2023-01-24T03:33:42.579 INFO:tasks.workunit.client.0.smithi112.stdout: create 293/35 file after the snapshot 2023-01-24T03:33:42.580 INFO:tasks.workunit.client.0.smithi112.stdout: create 293/36 file after the snapshot 2023-01-24T03:33:42.581 INFO:tasks.workunit.client.0.smithi112.stdout: create 293/37 file after the snapshot 2023-01-24T03:33:42.582 INFO:tasks.workunit.client.0.smithi112.stdout: create 293/38 file after the snapshot 2023-01-24T03:33:42.582 INFO:tasks.workunit.client.0.smithi112.stdout: create 293/39 file after the snapshot 2023-01-24T03:33:42.583 INFO:tasks.workunit.client.0.smithi112.stdout: create 294/.snap/snap-subdir-test 2023-01-24T03:33:42.604 INFO:tasks.workunit.client.0.smithi112.stdout: create 294/30 file after the snapshot 2023-01-24T03:33:42.605 INFO:tasks.workunit.client.0.smithi112.stdout: create 294/31 file after the snapshot 2023-01-24T03:33:42.606 INFO:tasks.workunit.client.0.smithi112.stdout: create 294/32 file after the snapshot 2023-01-24T03:33:42.607 INFO:tasks.workunit.client.0.smithi112.stdout: create 294/33 file after the snapshot 2023-01-24T03:33:42.608 INFO:tasks.workunit.client.0.smithi112.stdout: create 294/34 file after the snapshot 2023-01-24T03:33:42.609 INFO:tasks.workunit.client.0.smithi112.stdout: create 294/35 file after the snapshot 2023-01-24T03:33:42.609 INFO:tasks.workunit.client.0.smithi112.stdout: create 294/36 file after the snapshot 2023-01-24T03:33:42.610 INFO:tasks.workunit.client.0.smithi112.stdout: create 294/37 file after the snapshot 2023-01-24T03:33:42.611 INFO:tasks.workunit.client.0.smithi112.stdout: create 294/38 file after the snapshot 2023-01-24T03:33:42.612 INFO:tasks.workunit.client.0.smithi112.stdout: create 294/39 file after the snapshot 2023-01-24T03:33:42.612 INFO:tasks.workunit.client.0.smithi112.stdout: create 295/.snap/snap-subdir-test 2023-01-24T03:33:42.633 INFO:tasks.workunit.client.0.smithi112.stdout: create 295/30 file after the snapshot 2023-01-24T03:33:42.634 INFO:tasks.workunit.client.0.smithi112.stdout: create 295/31 file after the snapshot 2023-01-24T03:33:42.635 INFO:tasks.workunit.client.0.smithi112.stdout: create 295/32 file after the snapshot 2023-01-24T03:33:42.636 INFO:tasks.workunit.client.0.smithi112.stdout: create 295/33 file after the snapshot 2023-01-24T03:33:42.636 INFO:tasks.workunit.client.0.smithi112.stdout: create 295/34 file after the snapshot 2023-01-24T03:33:42.637 INFO:tasks.workunit.client.0.smithi112.stdout: create 295/35 file after the snapshot 2023-01-24T03:33:42.638 INFO:tasks.workunit.client.0.smithi112.stdout: create 295/36 file after the snapshot 2023-01-24T03:33:42.639 INFO:tasks.workunit.client.0.smithi112.stdout: create 295/37 file after the snapshot 2023-01-24T03:33:42.640 INFO:tasks.workunit.client.0.smithi112.stdout: create 295/38 file after the snapshot 2023-01-24T03:33:42.641 INFO:tasks.workunit.client.0.smithi112.stdout: create 295/39 file after the snapshot 2023-01-24T03:33:42.642 INFO:tasks.workunit.client.0.smithi112.stdout: create 296/.snap/snap-subdir-test 2023-01-24T03:33:42.662 INFO:tasks.workunit.client.0.smithi112.stdout: create 296/30 file after the snapshot 2023-01-24T03:33:42.664 INFO:tasks.workunit.client.0.smithi112.stdout: create 296/31 file after the snapshot 2023-01-24T03:33:42.664 INFO:tasks.workunit.client.0.smithi112.stdout: create 296/32 file after the snapshot 2023-01-24T03:33:42.665 INFO:tasks.workunit.client.0.smithi112.stdout: create 296/33 file after the snapshot 2023-01-24T03:33:42.666 INFO:tasks.workunit.client.0.smithi112.stdout: create 296/34 file after the snapshot 2023-01-24T03:33:42.667 INFO:tasks.workunit.client.0.smithi112.stdout: create 296/35 file after the snapshot 2023-01-24T03:33:42.667 INFO:tasks.workunit.client.0.smithi112.stdout: create 296/36 file after the snapshot 2023-01-24T03:33:42.668 INFO:tasks.workunit.client.0.smithi112.stdout: create 296/37 file after the snapshot 2023-01-24T03:33:42.669 INFO:tasks.workunit.client.0.smithi112.stdout: create 296/38 file after the snapshot 2023-01-24T03:33:42.669 INFO:tasks.workunit.client.0.smithi112.stdout: create 296/39 file after the snapshot 2023-01-24T03:33:42.670 INFO:tasks.workunit.client.0.smithi112.stdout: create 297/.snap/snap-subdir-test 2023-01-24T03:33:42.691 INFO:tasks.workunit.client.0.smithi112.stdout: create 297/30 file after the snapshot 2023-01-24T03:33:42.691 INFO:tasks.workunit.client.0.smithi112.stdout: create 297/31 file after the snapshot 2023-01-24T03:33:42.692 INFO:tasks.workunit.client.0.smithi112.stdout: create 297/32 file after the snapshot 2023-01-24T03:33:42.693 INFO:tasks.workunit.client.0.smithi112.stdout: create 297/33 file after the snapshot 2023-01-24T03:33:42.693 INFO:tasks.workunit.client.0.smithi112.stdout: create 297/34 file after the snapshot 2023-01-24T03:33:42.694 INFO:tasks.workunit.client.0.smithi112.stdout: create 297/35 file after the snapshot 2023-01-24T03:33:42.695 INFO:tasks.workunit.client.0.smithi112.stdout: create 297/36 file after the snapshot 2023-01-24T03:33:42.696 INFO:tasks.workunit.client.0.smithi112.stdout: create 297/37 file after the snapshot 2023-01-24T03:33:42.696 INFO:tasks.workunit.client.0.smithi112.stdout: create 297/38 file after the snapshot 2023-01-24T03:33:42.697 INFO:tasks.workunit.client.0.smithi112.stdout: create 297/39 file after the snapshot 2023-01-24T03:33:42.698 INFO:tasks.workunit.client.0.smithi112.stdout: create 298/.snap/snap-subdir-test 2023-01-24T03:33:42.719 INFO:tasks.workunit.client.0.smithi112.stdout: create 298/30 file after the snapshot 2023-01-24T03:33:42.720 INFO:tasks.workunit.client.0.smithi112.stdout: create 298/31 file after the snapshot 2023-01-24T03:33:42.721 INFO:tasks.workunit.client.0.smithi112.stdout: create 298/32 file after the snapshot 2023-01-24T03:33:42.721 INFO:tasks.workunit.client.0.smithi112.stdout: create 298/33 file after the snapshot 2023-01-24T03:33:42.722 INFO:tasks.workunit.client.0.smithi112.stdout: create 298/34 file after the snapshot 2023-01-24T03:33:42.723 INFO:tasks.workunit.client.0.smithi112.stdout: create 298/35 file after the snapshot 2023-01-24T03:33:42.724 INFO:tasks.workunit.client.0.smithi112.stdout: create 298/36 file after the snapshot 2023-01-24T03:33:42.724 INFO:tasks.workunit.client.0.smithi112.stdout: create 298/37 file after the snapshot 2023-01-24T03:33:42.725 INFO:tasks.workunit.client.0.smithi112.stdout: create 298/38 file after the snapshot 2023-01-24T03:33:42.726 INFO:tasks.workunit.client.0.smithi112.stdout: create 298/39 file after the snapshot 2023-01-24T03:33:42.727 INFO:tasks.workunit.client.0.smithi112.stdout: create 299/.snap/snap-subdir-test 2023-01-24T03:33:42.749 INFO:tasks.workunit.client.0.smithi112.stdout: create 299/30 file after the snapshot 2023-01-24T03:33:42.750 INFO:tasks.workunit.client.0.smithi112.stdout: create 299/31 file after the snapshot 2023-01-24T03:33:42.751 INFO:tasks.workunit.client.0.smithi112.stdout: create 299/32 file after the snapshot 2023-01-24T03:33:42.752 INFO:tasks.workunit.client.0.smithi112.stdout: create 299/33 file after the snapshot 2023-01-24T03:33:42.753 INFO:tasks.workunit.client.0.smithi112.stdout: create 299/34 file after the snapshot 2023-01-24T03:33:42.753 INFO:tasks.workunit.client.0.smithi112.stdout: create 299/35 file after the snapshot 2023-01-24T03:33:42.754 INFO:tasks.workunit.client.0.smithi112.stdout: create 299/36 file after the snapshot 2023-01-24T03:33:42.755 INFO:tasks.workunit.client.0.smithi112.stdout: create 299/37 file after the snapshot 2023-01-24T03:33:42.756 INFO:tasks.workunit.client.0.smithi112.stdout: create 299/38 file after the snapshot 2023-01-24T03:33:42.756 INFO:tasks.workunit.client.0.smithi112.stdout: create 299/39 file after the snapshot 2023-01-24T03:33:42.757 INFO:tasks.workunit.client.0.smithi112.stdout:Take second snapshot .snap/test2 2023-01-24T03:33:42.801 INFO:tasks.workunit.client.0.smithi112.stdout:Copy content of .snap/test1 to copyofsnap1 ... 2023-01-24T03:33:44.983 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1' -> 'copyofsnap1/test1' 2023-01-24T03:33:44.983 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/100' -> 'copyofsnap1/test1/100' 2023-01-24T03:33:44.983 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/100/10' -> 'copyofsnap1/test1/100/10' 2023-01-24T03:33:44.984 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/100/11' -> 'copyofsnap1/test1/100/11' 2023-01-24T03:33:44.984 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/100/12' -> 'copyofsnap1/test1/100/12' 2023-01-24T03:33:44.984 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/100/13' -> 'copyofsnap1/test1/100/13' 2023-01-24T03:33:44.984 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/100/14' -> 'copyofsnap1/test1/100/14' 2023-01-24T03:33:44.985 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/100/15' -> 'copyofsnap1/test1/100/15' 2023-01-24T03:33:44.985 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/100/16' -> 'copyofsnap1/test1/100/16' 2023-01-24T03:33:44.985 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/100/17' -> 'copyofsnap1/test1/100/17' 2023-01-24T03:33:44.985 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/100/18' -> 'copyofsnap1/test1/100/18' 2023-01-24T03:33:44.985 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/100/19' -> 'copyofsnap1/test1/100/19' 2023-01-24T03:33:44.986 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/100/20' -> 'copyofsnap1/test1/100/20' 2023-01-24T03:33:44.986 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/101' -> 'copyofsnap1/test1/101' 2023-01-24T03:33:44.986 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/101/10' -> 'copyofsnap1/test1/101/10' 2023-01-24T03:33:44.986 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/101/11' -> 'copyofsnap1/test1/101/11' 2023-01-24T03:33:44.986 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/101/12' -> 'copyofsnap1/test1/101/12' 2023-01-24T03:33:44.987 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/101/13' -> 'copyofsnap1/test1/101/13' 2023-01-24T03:33:44.987 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/101/14' -> 'copyofsnap1/test1/101/14' 2023-01-24T03:33:44.987 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/101/15' -> 'copyofsnap1/test1/101/15' 2023-01-24T03:33:44.987 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/101/16' -> 'copyofsnap1/test1/101/16' 2023-01-24T03:33:44.987 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/101/17' -> 'copyofsnap1/test1/101/17' 2023-01-24T03:33:44.988 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/101/18' -> 'copyofsnap1/test1/101/18' 2023-01-24T03:33:44.988 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/101/19' -> 'copyofsnap1/test1/101/19' 2023-01-24T03:33:44.988 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/101/20' -> 'copyofsnap1/test1/101/20' 2023-01-24T03:33:44.988 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/102' -> 'copyofsnap1/test1/102' 2023-01-24T03:33:44.988 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/102/10' -> 'copyofsnap1/test1/102/10' 2023-01-24T03:33:44.989 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/102/11' -> 'copyofsnap1/test1/102/11' 2023-01-24T03:33:44.989 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/102/12' -> 'copyofsnap1/test1/102/12' 2023-01-24T03:33:44.989 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/102/13' -> 'copyofsnap1/test1/102/13' 2023-01-24T03:33:44.989 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/102/14' -> 'copyofsnap1/test1/102/14' 2023-01-24T03:33:44.989 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/102/15' -> 'copyofsnap1/test1/102/15' 2023-01-24T03:33:44.989 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/102/16' -> 'copyofsnap1/test1/102/16' 2023-01-24T03:33:44.990 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/102/17' -> 'copyofsnap1/test1/102/17' 2023-01-24T03:33:44.990 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/102/18' -> 'copyofsnap1/test1/102/18' 2023-01-24T03:33:44.990 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/102/19' -> 'copyofsnap1/test1/102/19' 2023-01-24T03:33:44.990 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/102/20' -> 'copyofsnap1/test1/102/20' 2023-01-24T03:33:44.990 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/103' -> 'copyofsnap1/test1/103' 2023-01-24T03:33:44.991 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/103/10' -> 'copyofsnap1/test1/103/10' 2023-01-24T03:33:44.991 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/103/11' -> 'copyofsnap1/test1/103/11' 2023-01-24T03:33:44.991 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/103/12' -> 'copyofsnap1/test1/103/12' 2023-01-24T03:33:44.991 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/103/13' -> 'copyofsnap1/test1/103/13' 2023-01-24T03:33:44.991 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/103/14' -> 'copyofsnap1/test1/103/14' 2023-01-24T03:33:44.992 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/103/15' -> 'copyofsnap1/test1/103/15' 2023-01-24T03:33:44.992 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/103/16' -> 'copyofsnap1/test1/103/16' 2023-01-24T03:33:44.992 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/103/17' -> 'copyofsnap1/test1/103/17' 2023-01-24T03:33:44.992 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/103/18' -> 'copyofsnap1/test1/103/18' 2023-01-24T03:33:44.992 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/103/19' -> 'copyofsnap1/test1/103/19' 2023-01-24T03:33:44.993 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/103/20' -> 'copyofsnap1/test1/103/20' 2023-01-24T03:33:44.993 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/104' -> 'copyofsnap1/test1/104' 2023-01-24T03:33:44.993 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/104/10' -> 'copyofsnap1/test1/104/10' 2023-01-24T03:33:44.993 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/104/11' -> 'copyofsnap1/test1/104/11' 2023-01-24T03:33:44.993 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/104/12' -> 'copyofsnap1/test1/104/12' 2023-01-24T03:33:44.994 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/104/13' -> 'copyofsnap1/test1/104/13' 2023-01-24T03:33:44.994 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/104/14' -> 'copyofsnap1/test1/104/14' 2023-01-24T03:33:44.994 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/104/15' -> 'copyofsnap1/test1/104/15' 2023-01-24T03:33:44.994 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/104/16' -> 'copyofsnap1/test1/104/16' 2023-01-24T03:33:44.994 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/104/17' -> 'copyofsnap1/test1/104/17' 2023-01-24T03:33:44.994 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/104/18' -> 'copyofsnap1/test1/104/18' 2023-01-24T03:33:44.995 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/104/19' -> 'copyofsnap1/test1/104/19' 2023-01-24T03:33:44.995 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/104/20' -> 'copyofsnap1/test1/104/20' 2023-01-24T03:33:44.995 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/105' -> 'copyofsnap1/test1/105' 2023-01-24T03:33:44.995 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/105/10' -> 'copyofsnap1/test1/105/10' 2023-01-24T03:33:44.996 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/105/11' -> 'copyofsnap1/test1/105/11' 2023-01-24T03:33:44.996 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/105/12' -> 'copyofsnap1/test1/105/12' 2023-01-24T03:33:44.996 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/105/13' -> 'copyofsnap1/test1/105/13' 2023-01-24T03:33:44.996 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/105/14' -> 'copyofsnap1/test1/105/14' 2023-01-24T03:33:44.996 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/105/15' -> 'copyofsnap1/test1/105/15' 2023-01-24T03:33:44.997 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/105/16' -> 'copyofsnap1/test1/105/16' 2023-01-24T03:33:44.997 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/105/17' -> 'copyofsnap1/test1/105/17' 2023-01-24T03:33:44.997 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/105/18' -> 'copyofsnap1/test1/105/18' 2023-01-24T03:33:44.997 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/105/19' -> 'copyofsnap1/test1/105/19' 2023-01-24T03:33:44.997 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/105/20' -> 'copyofsnap1/test1/105/20' 2023-01-24T03:33:44.998 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/106' -> 'copyofsnap1/test1/106' 2023-01-24T03:33:44.998 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/106/10' -> 'copyofsnap1/test1/106/10' 2023-01-24T03:33:44.998 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/106/11' -> 'copyofsnap1/test1/106/11' 2023-01-24T03:33:44.998 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/106/12' -> 'copyofsnap1/test1/106/12' 2023-01-24T03:33:44.998 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/106/13' -> 'copyofsnap1/test1/106/13' 2023-01-24T03:33:44.999 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/106/14' -> 'copyofsnap1/test1/106/14' 2023-01-24T03:33:44.999 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/106/15' -> 'copyofsnap1/test1/106/15' 2023-01-24T03:33:44.999 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/106/16' -> 'copyofsnap1/test1/106/16' 2023-01-24T03:33:46.502 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/106/17' -> 'copyofsnap1/test1/106/17' 2023-01-24T03:33:46.503 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/106/18' -> 'copyofsnap1/test1/106/18' 2023-01-24T03:33:46.503 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/106/19' -> 'copyofsnap1/test1/106/19' 2023-01-24T03:33:46.503 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/106/20' -> 'copyofsnap1/test1/106/20' 2023-01-24T03:33:46.503 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/107' -> 'copyofsnap1/test1/107' 2023-01-24T03:33:46.503 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/107/10' -> 'copyofsnap1/test1/107/10' 2023-01-24T03:33:46.504 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/107/11' -> 'copyofsnap1/test1/107/11' 2023-01-24T03:33:46.504 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/107/12' -> 'copyofsnap1/test1/107/12' 2023-01-24T03:33:46.504 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/107/13' -> 'copyofsnap1/test1/107/13' 2023-01-24T03:33:46.504 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/107/14' -> 'copyofsnap1/test1/107/14' 2023-01-24T03:33:46.505 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/107/15' -> 'copyofsnap1/test1/107/15' 2023-01-24T03:33:46.505 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/107/16' -> 'copyofsnap1/test1/107/16' 2023-01-24T03:33:46.505 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/107/17' -> 'copyofsnap1/test1/107/17' 2023-01-24T03:33:46.505 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/107/18' -> 'copyofsnap1/test1/107/18' 2023-01-24T03:33:46.505 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/107/19' -> 'copyofsnap1/test1/107/19' 2023-01-24T03:33:46.506 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/107/20' -> 'copyofsnap1/test1/107/20' 2023-01-24T03:33:46.506 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/108' -> 'copyofsnap1/test1/108' 2023-01-24T03:33:46.506 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/108/10' -> 'copyofsnap1/test1/108/10' 2023-01-24T03:33:46.506 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/108/11' -> 'copyofsnap1/test1/108/11' 2023-01-24T03:33:46.506 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/108/12' -> 'copyofsnap1/test1/108/12' 2023-01-24T03:33:46.507 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/108/13' -> 'copyofsnap1/test1/108/13' 2023-01-24T03:33:46.507 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/108/14' -> 'copyofsnap1/test1/108/14' 2023-01-24T03:33:46.507 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/108/15' -> 'copyofsnap1/test1/108/15' 2023-01-24T03:33:46.507 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/108/16' -> 'copyofsnap1/test1/108/16' 2023-01-24T03:33:46.507 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/108/17' -> 'copyofsnap1/test1/108/17' 2023-01-24T03:33:46.508 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/108/18' -> 'copyofsnap1/test1/108/18' 2023-01-24T03:33:46.508 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/108/19' -> 'copyofsnap1/test1/108/19' 2023-01-24T03:33:46.508 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/108/20' -> 'copyofsnap1/test1/108/20' 2023-01-24T03:33:46.508 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/109' -> 'copyofsnap1/test1/109' 2023-01-24T03:33:46.508 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/109/10' -> 'copyofsnap1/test1/109/10' 2023-01-24T03:33:46.508 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/109/11' -> 'copyofsnap1/test1/109/11' 2023-01-24T03:33:46.509 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/109/12' -> 'copyofsnap1/test1/109/12' 2023-01-24T03:33:46.509 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/109/13' -> 'copyofsnap1/test1/109/13' 2023-01-24T03:33:46.509 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/109/14' -> 'copyofsnap1/test1/109/14' 2023-01-24T03:33:46.509 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/109/15' -> 'copyofsnap1/test1/109/15' 2023-01-24T03:33:46.509 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/109/16' -> 'copyofsnap1/test1/109/16' 2023-01-24T03:33:46.510 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/109/17' -> 'copyofsnap1/test1/109/17' 2023-01-24T03:33:46.510 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/109/18' -> 'copyofsnap1/test1/109/18' 2023-01-24T03:33:46.510 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/109/19' -> 'copyofsnap1/test1/109/19' 2023-01-24T03:33:46.510 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/109/20' -> 'copyofsnap1/test1/109/20' 2023-01-24T03:33:46.510 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/110' -> 'copyofsnap1/test1/110' 2023-01-24T03:33:46.511 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/110/10' -> 'copyofsnap1/test1/110/10' 2023-01-24T03:33:46.511 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/110/11' -> 'copyofsnap1/test1/110/11' 2023-01-24T03:33:46.511 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/110/12' -> 'copyofsnap1/test1/110/12' 2023-01-24T03:33:46.511 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/110/13' -> 'copyofsnap1/test1/110/13' 2023-01-24T03:33:46.511 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/110/14' -> 'copyofsnap1/test1/110/14' 2023-01-24T03:33:46.511 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/110/15' -> 'copyofsnap1/test1/110/15' 2023-01-24T03:33:46.512 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/110/16' -> 'copyofsnap1/test1/110/16' 2023-01-24T03:33:46.512 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/110/17' -> 'copyofsnap1/test1/110/17' 2023-01-24T03:33:46.512 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/110/18' -> 'copyofsnap1/test1/110/18' 2023-01-24T03:33:46.512 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/110/19' -> 'copyofsnap1/test1/110/19' 2023-01-24T03:33:46.512 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/110/20' -> 'copyofsnap1/test1/110/20' 2023-01-24T03:33:46.513 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/111' -> 'copyofsnap1/test1/111' 2023-01-24T03:33:46.513 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/111/10' -> 'copyofsnap1/test1/111/10' 2023-01-24T03:33:46.513 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/111/11' -> 'copyofsnap1/test1/111/11' 2023-01-24T03:33:46.513 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/111/12' -> 'copyofsnap1/test1/111/12' 2023-01-24T03:33:46.513 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/111/13' -> 'copyofsnap1/test1/111/13' 2023-01-24T03:33:46.513 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/111/14' -> 'copyofsnap1/test1/111/14' 2023-01-24T03:33:46.514 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/111/15' -> 'copyofsnap1/test1/111/15' 2023-01-24T03:33:46.514 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/111/16' -> 'copyofsnap1/test1/111/16' 2023-01-24T03:33:46.514 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/111/17' -> 'copyofsnap1/test1/111/17' 2023-01-24T03:33:46.514 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/111/18' -> 'copyofsnap1/test1/111/18' 2023-01-24T03:33:46.515 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/111/19' -> 'copyofsnap1/test1/111/19' 2023-01-24T03:33:46.515 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/111/20' -> 'copyofsnap1/test1/111/20' 2023-01-24T03:33:46.515 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/112' -> 'copyofsnap1/test1/112' 2023-01-24T03:33:46.515 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/112/10' -> 'copyofsnap1/test1/112/10' 2023-01-24T03:33:46.515 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/112/11' -> 'copyofsnap1/test1/112/11' 2023-01-24T03:33:46.516 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/112/12' -> 'copyofsnap1/test1/112/12' 2023-01-24T03:33:46.516 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/112/13' -> 'copyofsnap1/test1/112/13' 2023-01-24T03:33:46.516 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/112/14' -> 'copyofsnap1/test1/112/14' 2023-01-24T03:33:46.516 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/112/15' -> 'copyofsnap1/test1/112/15' 2023-01-24T03:33:46.516 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/112/16' -> 'copyofsnap1/test1/112/16' 2023-01-24T03:33:46.516 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/112/17' -> 'copyofsnap1/test1/112/17' 2023-01-24T03:33:46.517 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/112/18' -> 'copyofsnap1/test1/112/18' 2023-01-24T03:33:46.517 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/112/19' -> 'copyofsnap1/test1/112/19' 2023-01-24T03:33:46.517 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/112/20' -> 'copyofsnap1/test1/112/20' 2023-01-24T03:33:46.517 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/113' -> 'copyofsnap1/test1/113' 2023-01-24T03:33:46.517 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/113/10' -> 'copyofsnap1/test1/113/10' 2023-01-24T03:33:46.518 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/113/11' -> 'copyofsnap1/test1/113/11' 2023-01-24T03:33:46.518 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/113/12' -> 'copyofsnap1/test1/113/12' 2023-01-24T03:33:46.518 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/113/13' -> 'copyofsnap1/test1/113/13' 2023-01-24T03:33:47.684 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/113/14' -> 'copyofsnap1/test1/113/14' 2023-01-24T03:33:47.684 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/113/15' -> 'copyofsnap1/test1/113/15' 2023-01-24T03:33:47.684 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/113/16' -> 'copyofsnap1/test1/113/16' 2023-01-24T03:33:47.685 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/113/17' -> 'copyofsnap1/test1/113/17' 2023-01-24T03:33:47.685 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/113/18' -> 'copyofsnap1/test1/113/18' 2023-01-24T03:33:47.685 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/113/19' -> 'copyofsnap1/test1/113/19' 2023-01-24T03:33:47.685 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/113/20' -> 'copyofsnap1/test1/113/20' 2023-01-24T03:33:47.686 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/114' -> 'copyofsnap1/test1/114' 2023-01-24T03:33:47.686 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/114/10' -> 'copyofsnap1/test1/114/10' 2023-01-24T03:33:47.686 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/114/11' -> 'copyofsnap1/test1/114/11' 2023-01-24T03:33:47.686 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/114/12' -> 'copyofsnap1/test1/114/12' 2023-01-24T03:33:47.686 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/114/13' -> 'copyofsnap1/test1/114/13' 2023-01-24T03:33:47.686 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/114/14' -> 'copyofsnap1/test1/114/14' 2023-01-24T03:33:47.687 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/114/15' -> 'copyofsnap1/test1/114/15' 2023-01-24T03:33:47.687 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/114/16' -> 'copyofsnap1/test1/114/16' 2023-01-24T03:33:47.687 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/114/17' -> 'copyofsnap1/test1/114/17' 2023-01-24T03:33:47.687 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/114/18' -> 'copyofsnap1/test1/114/18' 2023-01-24T03:33:47.687 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/114/19' -> 'copyofsnap1/test1/114/19' 2023-01-24T03:33:47.688 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/114/20' -> 'copyofsnap1/test1/114/20' 2023-01-24T03:33:47.688 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/115' -> 'copyofsnap1/test1/115' 2023-01-24T03:33:47.688 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/115/10' -> 'copyofsnap1/test1/115/10' 2023-01-24T03:33:47.688 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/115/11' -> 'copyofsnap1/test1/115/11' 2023-01-24T03:33:47.688 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/115/12' -> 'copyofsnap1/test1/115/12' 2023-01-24T03:33:47.689 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/115/13' -> 'copyofsnap1/test1/115/13' 2023-01-24T03:33:47.689 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/115/14' -> 'copyofsnap1/test1/115/14' 2023-01-24T03:33:47.689 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/115/15' -> 'copyofsnap1/test1/115/15' 2023-01-24T03:33:47.689 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/115/16' -> 'copyofsnap1/test1/115/16' 2023-01-24T03:33:47.689 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/115/17' -> 'copyofsnap1/test1/115/17' 2023-01-24T03:33:47.690 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/115/18' -> 'copyofsnap1/test1/115/18' 2023-01-24T03:33:47.690 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/115/19' -> 'copyofsnap1/test1/115/19' 2023-01-24T03:33:47.690 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/115/20' -> 'copyofsnap1/test1/115/20' 2023-01-24T03:33:47.690 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/116' -> 'copyofsnap1/test1/116' 2023-01-24T03:33:47.690 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/116/10' -> 'copyofsnap1/test1/116/10' 2023-01-24T03:33:47.691 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/116/11' -> 'copyofsnap1/test1/116/11' 2023-01-24T03:33:47.691 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/116/12' -> 'copyofsnap1/test1/116/12' 2023-01-24T03:33:47.691 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/116/13' -> 'copyofsnap1/test1/116/13' 2023-01-24T03:33:47.691 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/116/14' -> 'copyofsnap1/test1/116/14' 2023-01-24T03:33:47.691 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/116/15' -> 'copyofsnap1/test1/116/15' 2023-01-24T03:33:47.692 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/116/16' -> 'copyofsnap1/test1/116/16' 2023-01-24T03:33:47.692 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/116/17' -> 'copyofsnap1/test1/116/17' 2023-01-24T03:33:47.692 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/116/18' -> 'copyofsnap1/test1/116/18' 2023-01-24T03:33:47.692 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/116/19' -> 'copyofsnap1/test1/116/19' 2023-01-24T03:33:47.692 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/116/20' -> 'copyofsnap1/test1/116/20' 2023-01-24T03:33:47.693 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/117' -> 'copyofsnap1/test1/117' 2023-01-24T03:33:47.693 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/117/10' -> 'copyofsnap1/test1/117/10' 2023-01-24T03:33:47.693 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/117/11' -> 'copyofsnap1/test1/117/11' 2023-01-24T03:33:47.693 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/117/12' -> 'copyofsnap1/test1/117/12' 2023-01-24T03:33:47.693 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/117/13' -> 'copyofsnap1/test1/117/13' 2023-01-24T03:33:47.694 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/117/14' -> 'copyofsnap1/test1/117/14' 2023-01-24T03:33:47.694 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/117/15' -> 'copyofsnap1/test1/117/15' 2023-01-24T03:33:47.694 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/117/16' -> 'copyofsnap1/test1/117/16' 2023-01-24T03:33:47.694 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/117/17' -> 'copyofsnap1/test1/117/17' 2023-01-24T03:33:47.694 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/117/18' -> 'copyofsnap1/test1/117/18' 2023-01-24T03:33:47.695 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/117/19' -> 'copyofsnap1/test1/117/19' 2023-01-24T03:33:47.695 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/117/20' -> 'copyofsnap1/test1/117/20' 2023-01-24T03:33:47.695 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/118' -> 'copyofsnap1/test1/118' 2023-01-24T03:33:47.695 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/118/10' -> 'copyofsnap1/test1/118/10' 2023-01-24T03:33:47.695 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/118/11' -> 'copyofsnap1/test1/118/11' 2023-01-24T03:33:47.696 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/118/12' -> 'copyofsnap1/test1/118/12' 2023-01-24T03:33:47.696 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/118/13' -> 'copyofsnap1/test1/118/13' 2023-01-24T03:33:47.696 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/118/14' -> 'copyofsnap1/test1/118/14' 2023-01-24T03:33:47.696 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/118/15' -> 'copyofsnap1/test1/118/15' 2023-01-24T03:33:47.696 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/118/16' -> 'copyofsnap1/test1/118/16' 2023-01-24T03:33:47.697 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/118/17' -> 'copyofsnap1/test1/118/17' 2023-01-24T03:33:47.697 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/118/18' -> 'copyofsnap1/test1/118/18' 2023-01-24T03:33:47.697 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/118/19' -> 'copyofsnap1/test1/118/19' 2023-01-24T03:33:47.697 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/118/20' -> 'copyofsnap1/test1/118/20' 2023-01-24T03:33:47.697 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/119' -> 'copyofsnap1/test1/119' 2023-01-24T03:33:47.697 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/119/10' -> 'copyofsnap1/test1/119/10' 2023-01-24T03:33:47.698 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/119/11' -> 'copyofsnap1/test1/119/11' 2023-01-24T03:33:47.698 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/119/12' -> 'copyofsnap1/test1/119/12' 2023-01-24T03:33:47.698 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/119/13' -> 'copyofsnap1/test1/119/13' 2023-01-24T03:33:47.698 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/119/14' -> 'copyofsnap1/test1/119/14' 2023-01-24T03:33:47.698 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/119/15' -> 'copyofsnap1/test1/119/15' 2023-01-24T03:33:47.699 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/119/16' -> 'copyofsnap1/test1/119/16' 2023-01-24T03:33:47.699 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/119/17' -> 'copyofsnap1/test1/119/17' 2023-01-24T03:33:47.699 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/119/18' -> 'copyofsnap1/test1/119/18' 2023-01-24T03:33:47.699 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/119/19' -> 'copyofsnap1/test1/119/19' 2023-01-24T03:33:47.699 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/119/20' -> 'copyofsnap1/test1/119/20' 2023-01-24T03:33:47.700 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/120' -> 'copyofsnap1/test1/120' 2023-01-24T03:33:47.700 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/120/10' -> 'copyofsnap1/test1/120/10' 2023-01-24T03:33:48.229 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/120/11' -> 'copyofsnap1/test1/120/11' 2023-01-24T03:33:48.230 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/120/12' -> 'copyofsnap1/test1/120/12' 2023-01-24T03:33:48.230 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/120/13' -> 'copyofsnap1/test1/120/13' 2023-01-24T03:33:48.230 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/120/14' -> 'copyofsnap1/test1/120/14' 2023-01-24T03:33:48.230 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/120/15' -> 'copyofsnap1/test1/120/15' 2023-01-24T03:33:48.230 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/120/16' -> 'copyofsnap1/test1/120/16' 2023-01-24T03:33:48.231 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/120/17' -> 'copyofsnap1/test1/120/17' 2023-01-24T03:33:48.231 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/120/18' -> 'copyofsnap1/test1/120/18' 2023-01-24T03:33:48.231 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/120/19' -> 'copyofsnap1/test1/120/19' 2023-01-24T03:33:48.231 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/120/20' -> 'copyofsnap1/test1/120/20' 2023-01-24T03:33:48.231 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/121' -> 'copyofsnap1/test1/121' 2023-01-24T03:33:48.232 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/121/10' -> 'copyofsnap1/test1/121/10' 2023-01-24T03:33:48.232 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/121/11' -> 'copyofsnap1/test1/121/11' 2023-01-24T03:33:48.232 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/121/12' -> 'copyofsnap1/test1/121/12' 2023-01-24T03:33:48.232 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/121/13' -> 'copyofsnap1/test1/121/13' 2023-01-24T03:33:48.232 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/121/14' -> 'copyofsnap1/test1/121/14' 2023-01-24T03:33:48.233 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/121/15' -> 'copyofsnap1/test1/121/15' 2023-01-24T03:33:48.233 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/121/16' -> 'copyofsnap1/test1/121/16' 2023-01-24T03:33:48.233 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/121/17' -> 'copyofsnap1/test1/121/17' 2023-01-24T03:33:48.233 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/121/18' -> 'copyofsnap1/test1/121/18' 2023-01-24T03:33:48.233 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/121/19' -> 'copyofsnap1/test1/121/19' 2023-01-24T03:33:48.234 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/121/20' -> 'copyofsnap1/test1/121/20' 2023-01-24T03:33:48.234 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/122' -> 'copyofsnap1/test1/122' 2023-01-24T03:33:48.234 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/122/10' -> 'copyofsnap1/test1/122/10' 2023-01-24T03:33:48.234 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/122/11' -> 'copyofsnap1/test1/122/11' 2023-01-24T03:33:48.234 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/122/12' -> 'copyofsnap1/test1/122/12' 2023-01-24T03:33:48.234 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/122/13' -> 'copyofsnap1/test1/122/13' 2023-01-24T03:33:48.235 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/122/14' -> 'copyofsnap1/test1/122/14' 2023-01-24T03:33:48.235 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/122/15' -> 'copyofsnap1/test1/122/15' 2023-01-24T03:33:48.235 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/122/16' -> 'copyofsnap1/test1/122/16' 2023-01-24T03:33:48.235 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/122/17' -> 'copyofsnap1/test1/122/17' 2023-01-24T03:33:48.235 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/122/18' -> 'copyofsnap1/test1/122/18' 2023-01-24T03:33:48.236 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/122/19' -> 'copyofsnap1/test1/122/19' 2023-01-24T03:33:48.236 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/122/20' -> 'copyofsnap1/test1/122/20' 2023-01-24T03:33:48.236 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/123' -> 'copyofsnap1/test1/123' 2023-01-24T03:33:48.236 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/123/10' -> 'copyofsnap1/test1/123/10' 2023-01-24T03:33:48.236 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/123/11' -> 'copyofsnap1/test1/123/11' 2023-01-24T03:33:48.237 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/123/12' -> 'copyofsnap1/test1/123/12' 2023-01-24T03:33:48.237 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/123/13' -> 'copyofsnap1/test1/123/13' 2023-01-24T03:33:48.237 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/123/14' -> 'copyofsnap1/test1/123/14' 2023-01-24T03:33:48.237 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/123/15' -> 'copyofsnap1/test1/123/15' 2023-01-24T03:33:48.237 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/123/16' -> 'copyofsnap1/test1/123/16' 2023-01-24T03:33:48.238 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/123/17' -> 'copyofsnap1/test1/123/17' 2023-01-24T03:33:48.238 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/123/18' -> 'copyofsnap1/test1/123/18' 2023-01-24T03:33:48.238 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/123/19' -> 'copyofsnap1/test1/123/19' 2023-01-24T03:33:48.238 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/123/20' -> 'copyofsnap1/test1/123/20' 2023-01-24T03:33:48.238 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/124' -> 'copyofsnap1/test1/124' 2023-01-24T03:33:48.238 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/124/10' -> 'copyofsnap1/test1/124/10' 2023-01-24T03:33:48.239 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/124/11' -> 'copyofsnap1/test1/124/11' 2023-01-24T03:33:48.239 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/124/12' -> 'copyofsnap1/test1/124/12' 2023-01-24T03:33:48.239 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/124/13' -> 'copyofsnap1/test1/124/13' 2023-01-24T03:33:48.239 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/124/14' -> 'copyofsnap1/test1/124/14' 2023-01-24T03:33:48.239 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/124/15' -> 'copyofsnap1/test1/124/15' 2023-01-24T03:33:48.240 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/124/16' -> 'copyofsnap1/test1/124/16' 2023-01-24T03:33:48.240 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/124/17' -> 'copyofsnap1/test1/124/17' 2023-01-24T03:33:48.240 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/124/18' -> 'copyofsnap1/test1/124/18' 2023-01-24T03:33:48.240 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/124/19' -> 'copyofsnap1/test1/124/19' 2023-01-24T03:33:48.241 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/124/20' -> 'copyofsnap1/test1/124/20' 2023-01-24T03:33:48.241 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/125' -> 'copyofsnap1/test1/125' 2023-01-24T03:33:48.241 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/125/10' -> 'copyofsnap1/test1/125/10' 2023-01-24T03:33:48.241 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/125/11' -> 'copyofsnap1/test1/125/11' 2023-01-24T03:33:48.241 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/125/12' -> 'copyofsnap1/test1/125/12' 2023-01-24T03:33:48.241 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/125/13' -> 'copyofsnap1/test1/125/13' 2023-01-24T03:33:48.242 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/125/14' -> 'copyofsnap1/test1/125/14' 2023-01-24T03:33:48.242 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/125/15' -> 'copyofsnap1/test1/125/15' 2023-01-24T03:33:48.242 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/125/16' -> 'copyofsnap1/test1/125/16' 2023-01-24T03:33:48.242 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/125/17' -> 'copyofsnap1/test1/125/17' 2023-01-24T03:33:48.242 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/125/18' -> 'copyofsnap1/test1/125/18' 2023-01-24T03:33:48.243 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/125/19' -> 'copyofsnap1/test1/125/19' 2023-01-24T03:33:48.243 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/125/20' -> 'copyofsnap1/test1/125/20' 2023-01-24T03:33:48.243 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/126' -> 'copyofsnap1/test1/126' 2023-01-24T03:33:48.243 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/126/10' -> 'copyofsnap1/test1/126/10' 2023-01-24T03:33:48.243 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/126/11' -> 'copyofsnap1/test1/126/11' 2023-01-24T03:33:48.244 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/126/12' -> 'copyofsnap1/test1/126/12' 2023-01-24T03:33:48.244 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/126/13' -> 'copyofsnap1/test1/126/13' 2023-01-24T03:33:48.244 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/126/14' -> 'copyofsnap1/test1/126/14' 2023-01-24T03:33:48.246 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/126/15' -> 'copyofsnap1/test1/126/15' 2023-01-24T03:33:48.246 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/126/16' -> 'copyofsnap1/test1/126/16' 2023-01-24T03:33:48.246 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/126/17' -> 'copyofsnap1/test1/126/17' 2023-01-24T03:33:48.246 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/126/18' -> 'copyofsnap1/test1/126/18' 2023-01-24T03:33:48.246 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/126/19' -> 'copyofsnap1/test1/126/19' 2023-01-24T03:33:51.428 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/126/20' -> 'copyofsnap1/test1/126/20' 2023-01-24T03:33:51.429 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/127' -> 'copyofsnap1/test1/127' 2023-01-24T03:33:51.429 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/127/10' -> 'copyofsnap1/test1/127/10' 2023-01-24T03:33:51.430 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/127/11' -> 'copyofsnap1/test1/127/11' 2023-01-24T03:33:51.431 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/127/12' -> 'copyofsnap1/test1/127/12' 2023-01-24T03:33:51.431 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/127/13' -> 'copyofsnap1/test1/127/13' 2023-01-24T03:33:51.431 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/127/14' -> 'copyofsnap1/test1/127/14' 2023-01-24T03:33:51.431 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/127/15' -> 'copyofsnap1/test1/127/15' 2023-01-24T03:33:51.432 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/127/16' -> 'copyofsnap1/test1/127/16' 2023-01-24T03:33:51.432 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/127/17' -> 'copyofsnap1/test1/127/17' 2023-01-24T03:33:51.432 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/127/18' -> 'copyofsnap1/test1/127/18' 2023-01-24T03:33:51.432 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/127/19' -> 'copyofsnap1/test1/127/19' 2023-01-24T03:33:51.432 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/127/20' -> 'copyofsnap1/test1/127/20' 2023-01-24T03:33:51.433 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/128' -> 'copyofsnap1/test1/128' 2023-01-24T03:33:51.433 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/128/10' -> 'copyofsnap1/test1/128/10' 2023-01-24T03:33:51.433 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/128/11' -> 'copyofsnap1/test1/128/11' 2023-01-24T03:33:51.433 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/128/12' -> 'copyofsnap1/test1/128/12' 2023-01-24T03:33:51.433 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/128/13' -> 'copyofsnap1/test1/128/13' 2023-01-24T03:33:51.434 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/128/14' -> 'copyofsnap1/test1/128/14' 2023-01-24T03:33:51.434 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/128/15' -> 'copyofsnap1/test1/128/15' 2023-01-24T03:33:51.434 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/128/16' -> 'copyofsnap1/test1/128/16' 2023-01-24T03:33:51.434 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/128/17' -> 'copyofsnap1/test1/128/17' 2023-01-24T03:33:51.434 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/128/18' -> 'copyofsnap1/test1/128/18' 2023-01-24T03:33:51.435 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/128/19' -> 'copyofsnap1/test1/128/19' 2023-01-24T03:33:51.435 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/128/20' -> 'copyofsnap1/test1/128/20' 2023-01-24T03:33:51.435 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/129' -> 'copyofsnap1/test1/129' 2023-01-24T03:33:51.435 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/129/10' -> 'copyofsnap1/test1/129/10' 2023-01-24T03:33:51.436 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/129/11' -> 'copyofsnap1/test1/129/11' 2023-01-24T03:33:51.436 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/129/12' -> 'copyofsnap1/test1/129/12' 2023-01-24T03:33:51.436 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/129/13' -> 'copyofsnap1/test1/129/13' 2023-01-24T03:33:51.436 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/129/14' -> 'copyofsnap1/test1/129/14' 2023-01-24T03:33:51.437 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/129/15' -> 'copyofsnap1/test1/129/15' 2023-01-24T03:33:51.437 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/129/16' -> 'copyofsnap1/test1/129/16' 2023-01-24T03:33:51.437 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/129/17' -> 'copyofsnap1/test1/129/17' 2023-01-24T03:33:51.437 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/129/18' -> 'copyofsnap1/test1/129/18' 2023-01-24T03:33:51.437 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/129/19' -> 'copyofsnap1/test1/129/19' 2023-01-24T03:33:51.438 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/129/20' -> 'copyofsnap1/test1/129/20' 2023-01-24T03:33:51.438 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/130' -> 'copyofsnap1/test1/130' 2023-01-24T03:33:51.438 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/130/10' -> 'copyofsnap1/test1/130/10' 2023-01-24T03:33:51.438 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/130/11' -> 'copyofsnap1/test1/130/11' 2023-01-24T03:33:51.438 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/130/12' -> 'copyofsnap1/test1/130/12' 2023-01-24T03:33:51.438 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/130/13' -> 'copyofsnap1/test1/130/13' 2023-01-24T03:33:51.439 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/130/14' -> 'copyofsnap1/test1/130/14' 2023-01-24T03:33:51.439 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/130/15' -> 'copyofsnap1/test1/130/15' 2023-01-24T03:33:51.439 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/130/16' -> 'copyofsnap1/test1/130/16' 2023-01-24T03:33:51.439 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/130/17' -> 'copyofsnap1/test1/130/17' 2023-01-24T03:33:51.439 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/130/18' -> 'copyofsnap1/test1/130/18' 2023-01-24T03:33:51.440 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/130/19' -> 'copyofsnap1/test1/130/19' 2023-01-24T03:33:51.440 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/130/20' -> 'copyofsnap1/test1/130/20' 2023-01-24T03:33:51.440 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/131' -> 'copyofsnap1/test1/131' 2023-01-24T03:33:51.440 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/131/10' -> 'copyofsnap1/test1/131/10' 2023-01-24T03:33:51.440 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/131/11' -> 'copyofsnap1/test1/131/11' 2023-01-24T03:33:51.441 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/131/12' -> 'copyofsnap1/test1/131/12' 2023-01-24T03:33:51.441 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/131/13' -> 'copyofsnap1/test1/131/13' 2023-01-24T03:33:51.441 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/131/14' -> 'copyofsnap1/test1/131/14' 2023-01-24T03:33:51.441 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/131/15' -> 'copyofsnap1/test1/131/15' 2023-01-24T03:33:51.441 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/131/16' -> 'copyofsnap1/test1/131/16' 2023-01-24T03:33:51.442 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/131/17' -> 'copyofsnap1/test1/131/17' 2023-01-24T03:33:51.442 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/131/18' -> 'copyofsnap1/test1/131/18' 2023-01-24T03:33:51.442 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/131/19' -> 'copyofsnap1/test1/131/19' 2023-01-24T03:33:51.442 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/131/20' -> 'copyofsnap1/test1/131/20' 2023-01-24T03:33:51.442 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/132' -> 'copyofsnap1/test1/132' 2023-01-24T03:33:51.443 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/132/10' -> 'copyofsnap1/test1/132/10' 2023-01-24T03:33:51.443 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/132/11' -> 'copyofsnap1/test1/132/11' 2023-01-24T03:33:51.443 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/132/12' -> 'copyofsnap1/test1/132/12' 2023-01-24T03:33:51.443 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/132/13' -> 'copyofsnap1/test1/132/13' 2023-01-24T03:33:51.444 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/132/14' -> 'copyofsnap1/test1/132/14' 2023-01-24T03:33:51.444 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/132/15' -> 'copyofsnap1/test1/132/15' 2023-01-24T03:33:51.444 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/132/16' -> 'copyofsnap1/test1/132/16' 2023-01-24T03:33:51.444 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/132/17' -> 'copyofsnap1/test1/132/17' 2023-01-24T03:33:51.444 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/132/18' -> 'copyofsnap1/test1/132/18' 2023-01-24T03:33:51.444 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/132/19' -> 'copyofsnap1/test1/132/19' 2023-01-24T03:33:51.445 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/132/20' -> 'copyofsnap1/test1/132/20' 2023-01-24T03:33:51.445 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/133' -> 'copyofsnap1/test1/133' 2023-01-24T03:33:51.445 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/133/10' -> 'copyofsnap1/test1/133/10' 2023-01-24T03:33:51.445 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/133/11' -> 'copyofsnap1/test1/133/11' 2023-01-24T03:33:51.446 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/133/12' -> 'copyofsnap1/test1/133/12' 2023-01-24T03:33:51.446 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/133/13' -> 'copyofsnap1/test1/133/13' 2023-01-24T03:33:51.446 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/133/14' -> 'copyofsnap1/test1/133/14' 2023-01-24T03:33:51.446 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/133/15' -> 'copyofsnap1/test1/133/15' 2023-01-24T03:33:51.446 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/133/16' -> 'copyofsnap1/test1/133/16' 2023-01-24T03:33:52.559 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/133/17' -> 'copyofsnap1/test1/133/17' 2023-01-24T03:33:52.560 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/133/18' -> 'copyofsnap1/test1/133/18' 2023-01-24T03:33:52.560 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/133/19' -> 'copyofsnap1/test1/133/19' 2023-01-24T03:33:52.560 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/133/20' -> 'copyofsnap1/test1/133/20' 2023-01-24T03:33:52.561 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/134' -> 'copyofsnap1/test1/134' 2023-01-24T03:33:52.561 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/134/10' -> 'copyofsnap1/test1/134/10' 2023-01-24T03:33:52.561 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/134/11' -> 'copyofsnap1/test1/134/11' 2023-01-24T03:33:52.561 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/134/12' -> 'copyofsnap1/test1/134/12' 2023-01-24T03:33:52.561 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/134/13' -> 'copyofsnap1/test1/134/13' 2023-01-24T03:33:52.562 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/134/14' -> 'copyofsnap1/test1/134/14' 2023-01-24T03:33:52.562 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/134/15' -> 'copyofsnap1/test1/134/15' 2023-01-24T03:33:52.562 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/134/16' -> 'copyofsnap1/test1/134/16' 2023-01-24T03:33:52.562 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/134/17' -> 'copyofsnap1/test1/134/17' 2023-01-24T03:33:52.562 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/134/18' -> 'copyofsnap1/test1/134/18' 2023-01-24T03:33:52.563 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/134/19' -> 'copyofsnap1/test1/134/19' 2023-01-24T03:33:52.563 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/134/20' -> 'copyofsnap1/test1/134/20' 2023-01-24T03:33:52.563 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/135' -> 'copyofsnap1/test1/135' 2023-01-24T03:33:52.563 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/135/10' -> 'copyofsnap1/test1/135/10' 2023-01-24T03:33:52.563 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/135/11' -> 'copyofsnap1/test1/135/11' 2023-01-24T03:33:52.564 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/135/12' -> 'copyofsnap1/test1/135/12' 2023-01-24T03:33:52.564 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/135/13' -> 'copyofsnap1/test1/135/13' 2023-01-24T03:33:52.564 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/135/14' -> 'copyofsnap1/test1/135/14' 2023-01-24T03:33:52.564 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/135/15' -> 'copyofsnap1/test1/135/15' 2023-01-24T03:33:52.565 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/135/16' -> 'copyofsnap1/test1/135/16' 2023-01-24T03:33:52.565 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/135/17' -> 'copyofsnap1/test1/135/17' 2023-01-24T03:33:52.565 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/135/18' -> 'copyofsnap1/test1/135/18' 2023-01-24T03:33:52.565 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/135/19' -> 'copyofsnap1/test1/135/19' 2023-01-24T03:33:52.565 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/135/20' -> 'copyofsnap1/test1/135/20' 2023-01-24T03:33:52.566 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/136' -> 'copyofsnap1/test1/136' 2023-01-24T03:33:52.566 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/136/10' -> 'copyofsnap1/test1/136/10' 2023-01-24T03:33:52.566 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/136/11' -> 'copyofsnap1/test1/136/11' 2023-01-24T03:33:52.566 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/136/12' -> 'copyofsnap1/test1/136/12' 2023-01-24T03:33:52.566 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/136/13' -> 'copyofsnap1/test1/136/13' 2023-01-24T03:33:52.567 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/136/14' -> 'copyofsnap1/test1/136/14' 2023-01-24T03:33:52.567 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/136/15' -> 'copyofsnap1/test1/136/15' 2023-01-24T03:33:52.567 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/136/16' -> 'copyofsnap1/test1/136/16' 2023-01-24T03:33:52.567 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/136/17' -> 'copyofsnap1/test1/136/17' 2023-01-24T03:33:52.568 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/136/18' -> 'copyofsnap1/test1/136/18' 2023-01-24T03:33:52.568 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/136/19' -> 'copyofsnap1/test1/136/19' 2023-01-24T03:33:52.568 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/136/20' -> 'copyofsnap1/test1/136/20' 2023-01-24T03:33:52.568 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/137' -> 'copyofsnap1/test1/137' 2023-01-24T03:33:52.569 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/137/10' -> 'copyofsnap1/test1/137/10' 2023-01-24T03:33:52.569 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/137/11' -> 'copyofsnap1/test1/137/11' 2023-01-24T03:33:52.569 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/137/12' -> 'copyofsnap1/test1/137/12' 2023-01-24T03:33:52.569 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/137/13' -> 'copyofsnap1/test1/137/13' 2023-01-24T03:33:52.570 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/137/14' -> 'copyofsnap1/test1/137/14' 2023-01-24T03:33:52.570 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/137/15' -> 'copyofsnap1/test1/137/15' 2023-01-24T03:33:52.570 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/137/16' -> 'copyofsnap1/test1/137/16' 2023-01-24T03:33:52.570 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/137/17' -> 'copyofsnap1/test1/137/17' 2023-01-24T03:33:52.570 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/137/18' -> 'copyofsnap1/test1/137/18' 2023-01-24T03:33:52.571 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/137/19' -> 'copyofsnap1/test1/137/19' 2023-01-24T03:33:52.571 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/137/20' -> 'copyofsnap1/test1/137/20' 2023-01-24T03:33:52.571 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/138' -> 'copyofsnap1/test1/138' 2023-01-24T03:33:52.571 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/138/10' -> 'copyofsnap1/test1/138/10' 2023-01-24T03:33:52.571 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/138/11' -> 'copyofsnap1/test1/138/11' 2023-01-24T03:33:52.571 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/138/12' -> 'copyofsnap1/test1/138/12' 2023-01-24T03:33:52.572 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/138/13' -> 'copyofsnap1/test1/138/13' 2023-01-24T03:33:52.572 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/138/14' -> 'copyofsnap1/test1/138/14' 2023-01-24T03:33:52.572 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/138/15' -> 'copyofsnap1/test1/138/15' 2023-01-24T03:33:52.572 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/138/16' -> 'copyofsnap1/test1/138/16' 2023-01-24T03:33:52.572 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/138/17' -> 'copyofsnap1/test1/138/17' 2023-01-24T03:33:52.573 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/138/18' -> 'copyofsnap1/test1/138/18' 2023-01-24T03:33:52.573 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/138/19' -> 'copyofsnap1/test1/138/19' 2023-01-24T03:33:52.573 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/138/20' -> 'copyofsnap1/test1/138/20' 2023-01-24T03:33:52.573 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/139' -> 'copyofsnap1/test1/139' 2023-01-24T03:33:52.573 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/139/10' -> 'copyofsnap1/test1/139/10' 2023-01-24T03:33:52.574 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/139/11' -> 'copyofsnap1/test1/139/11' 2023-01-24T03:33:52.574 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/139/12' -> 'copyofsnap1/test1/139/12' 2023-01-24T03:33:52.574 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/139/13' -> 'copyofsnap1/test1/139/13' 2023-01-24T03:33:52.574 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/139/14' -> 'copyofsnap1/test1/139/14' 2023-01-24T03:33:52.574 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/139/15' -> 'copyofsnap1/test1/139/15' 2023-01-24T03:33:52.575 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/139/16' -> 'copyofsnap1/test1/139/16' 2023-01-24T03:33:52.575 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/139/17' -> 'copyofsnap1/test1/139/17' 2023-01-24T03:33:52.575 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/139/18' -> 'copyofsnap1/test1/139/18' 2023-01-24T03:33:52.575 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/139/19' -> 'copyofsnap1/test1/139/19' 2023-01-24T03:33:52.575 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/139/20' -> 'copyofsnap1/test1/139/20' 2023-01-24T03:33:52.576 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/140' -> 'copyofsnap1/test1/140' 2023-01-24T03:33:52.576 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/140/10' -> 'copyofsnap1/test1/140/10' 2023-01-24T03:33:52.576 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/140/11' -> 'copyofsnap1/test1/140/11' 2023-01-24T03:33:52.576 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/140/12' -> 'copyofsnap1/test1/140/12' 2023-01-24T03:33:52.576 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/140/13' -> 'copyofsnap1/test1/140/13' 2023-01-24T03:33:53.003 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/140/14' -> 'copyofsnap1/test1/140/14' 2023-01-24T03:33:53.004 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/140/15' -> 'copyofsnap1/test1/140/15' 2023-01-24T03:33:53.004 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/140/16' -> 'copyofsnap1/test1/140/16' 2023-01-24T03:33:53.004 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/140/17' -> 'copyofsnap1/test1/140/17' 2023-01-24T03:33:53.004 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/140/18' -> 'copyofsnap1/test1/140/18' 2023-01-24T03:33:53.005 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/140/19' -> 'copyofsnap1/test1/140/19' 2023-01-24T03:33:53.005 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/140/20' -> 'copyofsnap1/test1/140/20' 2023-01-24T03:33:53.005 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/141' -> 'copyofsnap1/test1/141' 2023-01-24T03:33:53.005 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/141/10' -> 'copyofsnap1/test1/141/10' 2023-01-24T03:33:53.005 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/141/11' -> 'copyofsnap1/test1/141/11' 2023-01-24T03:33:53.006 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/141/12' -> 'copyofsnap1/test1/141/12' 2023-01-24T03:33:53.006 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/141/13' -> 'copyofsnap1/test1/141/13' 2023-01-24T03:33:53.006 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/141/14' -> 'copyofsnap1/test1/141/14' 2023-01-24T03:33:53.006 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/141/15' -> 'copyofsnap1/test1/141/15' 2023-01-24T03:33:53.006 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/141/16' -> 'copyofsnap1/test1/141/16' 2023-01-24T03:33:53.007 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/141/17' -> 'copyofsnap1/test1/141/17' 2023-01-24T03:33:53.007 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/141/18' -> 'copyofsnap1/test1/141/18' 2023-01-24T03:33:53.007 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/141/19' -> 'copyofsnap1/test1/141/19' 2023-01-24T03:33:53.007 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/141/20' -> 'copyofsnap1/test1/141/20' 2023-01-24T03:33:53.007 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/142' -> 'copyofsnap1/test1/142' 2023-01-24T03:33:53.008 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/142/10' -> 'copyofsnap1/test1/142/10' 2023-01-24T03:33:53.008 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/142/11' -> 'copyofsnap1/test1/142/11' 2023-01-24T03:33:53.008 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/142/12' -> 'copyofsnap1/test1/142/12' 2023-01-24T03:33:53.008 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/142/13' -> 'copyofsnap1/test1/142/13' 2023-01-24T03:33:53.008 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/142/14' -> 'copyofsnap1/test1/142/14' 2023-01-24T03:33:53.009 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/142/15' -> 'copyofsnap1/test1/142/15' 2023-01-24T03:33:53.009 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/142/16' -> 'copyofsnap1/test1/142/16' 2023-01-24T03:33:53.009 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/142/17' -> 'copyofsnap1/test1/142/17' 2023-01-24T03:33:53.009 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/142/18' -> 'copyofsnap1/test1/142/18' 2023-01-24T03:33:53.009 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/142/19' -> 'copyofsnap1/test1/142/19' 2023-01-24T03:33:53.010 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/142/20' -> 'copyofsnap1/test1/142/20' 2023-01-24T03:33:53.010 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/143' -> 'copyofsnap1/test1/143' 2023-01-24T03:33:53.010 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/143/10' -> 'copyofsnap1/test1/143/10' 2023-01-24T03:33:53.011 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/143/11' -> 'copyofsnap1/test1/143/11' 2023-01-24T03:33:53.011 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/143/12' -> 'copyofsnap1/test1/143/12' 2023-01-24T03:33:53.011 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/143/13' -> 'copyofsnap1/test1/143/13' 2023-01-24T03:33:53.011 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/143/14' -> 'copyofsnap1/test1/143/14' 2023-01-24T03:33:53.011 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/143/15' -> 'copyofsnap1/test1/143/15' 2023-01-24T03:33:53.012 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/143/16' -> 'copyofsnap1/test1/143/16' 2023-01-24T03:33:53.012 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/143/17' -> 'copyofsnap1/test1/143/17' 2023-01-24T03:33:53.012 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/143/18' -> 'copyofsnap1/test1/143/18' 2023-01-24T03:33:53.012 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/143/19' -> 'copyofsnap1/test1/143/19' 2023-01-24T03:33:53.012 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/143/20' -> 'copyofsnap1/test1/143/20' 2023-01-24T03:33:53.013 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/144' -> 'copyofsnap1/test1/144' 2023-01-24T03:33:53.013 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/144/10' -> 'copyofsnap1/test1/144/10' 2023-01-24T03:33:53.013 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/144/11' -> 'copyofsnap1/test1/144/11' 2023-01-24T03:33:53.013 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/144/12' -> 'copyofsnap1/test1/144/12' 2023-01-24T03:33:53.013 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/144/13' -> 'copyofsnap1/test1/144/13' 2023-01-24T03:33:53.013 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/144/14' -> 'copyofsnap1/test1/144/14' 2023-01-24T03:33:53.014 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/144/15' -> 'copyofsnap1/test1/144/15' 2023-01-24T03:33:53.014 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/144/16' -> 'copyofsnap1/test1/144/16' 2023-01-24T03:33:53.014 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/144/17' -> 'copyofsnap1/test1/144/17' 2023-01-24T03:33:53.014 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/144/18' -> 'copyofsnap1/test1/144/18' 2023-01-24T03:33:53.014 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/144/19' -> 'copyofsnap1/test1/144/19' 2023-01-24T03:33:53.015 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/144/20' -> 'copyofsnap1/test1/144/20' 2023-01-24T03:33:53.015 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/145' -> 'copyofsnap1/test1/145' 2023-01-24T03:33:53.015 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/145/10' -> 'copyofsnap1/test1/145/10' 2023-01-24T03:33:53.015 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/145/11' -> 'copyofsnap1/test1/145/11' 2023-01-24T03:33:53.015 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/145/12' -> 'copyofsnap1/test1/145/12' 2023-01-24T03:33:53.016 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/145/13' -> 'copyofsnap1/test1/145/13' 2023-01-24T03:33:53.016 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/145/14' -> 'copyofsnap1/test1/145/14' 2023-01-24T03:33:53.016 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/145/15' -> 'copyofsnap1/test1/145/15' 2023-01-24T03:33:53.016 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/145/16' -> 'copyofsnap1/test1/145/16' 2023-01-24T03:33:53.017 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/145/17' -> 'copyofsnap1/test1/145/17' 2023-01-24T03:33:53.017 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/145/18' -> 'copyofsnap1/test1/145/18' 2023-01-24T03:33:53.017 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/145/19' -> 'copyofsnap1/test1/145/19' 2023-01-24T03:33:53.017 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/145/20' -> 'copyofsnap1/test1/145/20' 2023-01-24T03:33:53.017 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/146' -> 'copyofsnap1/test1/146' 2023-01-24T03:33:53.017 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/146/10' -> 'copyofsnap1/test1/146/10' 2023-01-24T03:33:53.018 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/146/11' -> 'copyofsnap1/test1/146/11' 2023-01-24T03:33:53.018 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/146/12' -> 'copyofsnap1/test1/146/12' 2023-01-24T03:33:53.018 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/146/13' -> 'copyofsnap1/test1/146/13' 2023-01-24T03:33:53.018 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/146/14' -> 'copyofsnap1/test1/146/14' 2023-01-24T03:33:53.018 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/146/15' -> 'copyofsnap1/test1/146/15' 2023-01-24T03:33:53.019 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/146/16' -> 'copyofsnap1/test1/146/16' 2023-01-24T03:33:53.019 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/146/17' -> 'copyofsnap1/test1/146/17' 2023-01-24T03:33:53.019 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/146/18' -> 'copyofsnap1/test1/146/18' 2023-01-24T03:33:53.019 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/146/19' -> 'copyofsnap1/test1/146/19' 2023-01-24T03:33:53.019 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/146/20' -> 'copyofsnap1/test1/146/20' 2023-01-24T03:33:53.020 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/147' -> 'copyofsnap1/test1/147' 2023-01-24T03:33:53.020 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/147/10' -> 'copyofsnap1/test1/147/10' 2023-01-24T03:33:53.020 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/147/11' -> 'copyofsnap1/test1/147/11' 2023-01-24T03:33:53.737 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/147/12' -> 'copyofsnap1/test1/147/12' 2023-01-24T03:33:53.737 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/147/13' -> 'copyofsnap1/test1/147/13' 2023-01-24T03:33:53.738 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/147/14' -> 'copyofsnap1/test1/147/14' 2023-01-24T03:33:53.738 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/147/15' -> 'copyofsnap1/test1/147/15' 2023-01-24T03:33:53.738 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/147/16' -> 'copyofsnap1/test1/147/16' 2023-01-24T03:33:53.738 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/147/17' -> 'copyofsnap1/test1/147/17' 2023-01-24T03:33:53.738 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/147/18' -> 'copyofsnap1/test1/147/18' 2023-01-24T03:33:53.739 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/147/19' -> 'copyofsnap1/test1/147/19' 2023-01-24T03:33:53.739 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/147/20' -> 'copyofsnap1/test1/147/20' 2023-01-24T03:33:53.739 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/148' -> 'copyofsnap1/test1/148' 2023-01-24T03:33:53.739 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/148/10' -> 'copyofsnap1/test1/148/10' 2023-01-24T03:33:53.740 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/148/11' -> 'copyofsnap1/test1/148/11' 2023-01-24T03:33:53.740 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/148/12' -> 'copyofsnap1/test1/148/12' 2023-01-24T03:33:53.741 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/148/13' -> 'copyofsnap1/test1/148/13' 2023-01-24T03:33:53.741 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/148/14' -> 'copyofsnap1/test1/148/14' 2023-01-24T03:33:53.741 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/148/15' -> 'copyofsnap1/test1/148/15' 2023-01-24T03:33:53.742 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/148/16' -> 'copyofsnap1/test1/148/16' 2023-01-24T03:33:53.742 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/148/17' -> 'copyofsnap1/test1/148/17' 2023-01-24T03:33:53.742 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/148/18' -> 'copyofsnap1/test1/148/18' 2023-01-24T03:33:53.743 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/148/19' -> 'copyofsnap1/test1/148/19' 2023-01-24T03:33:53.743 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/148/20' -> 'copyofsnap1/test1/148/20' 2023-01-24T03:33:53.743 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/149' -> 'copyofsnap1/test1/149' 2023-01-24T03:33:53.744 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/149/10' -> 'copyofsnap1/test1/149/10' 2023-01-24T03:33:53.744 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/149/11' -> 'copyofsnap1/test1/149/11' 2023-01-24T03:33:53.745 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/149/12' -> 'copyofsnap1/test1/149/12' 2023-01-24T03:33:53.745 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/149/13' -> 'copyofsnap1/test1/149/13' 2023-01-24T03:33:53.745 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/149/14' -> 'copyofsnap1/test1/149/14' 2023-01-24T03:33:53.746 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/149/15' -> 'copyofsnap1/test1/149/15' 2023-01-24T03:33:53.746 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/149/16' -> 'copyofsnap1/test1/149/16' 2023-01-24T03:33:53.746 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/149/17' -> 'copyofsnap1/test1/149/17' 2023-01-24T03:33:53.747 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/149/18' -> 'copyofsnap1/test1/149/18' 2023-01-24T03:33:53.747 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/149/19' -> 'copyofsnap1/test1/149/19' 2023-01-24T03:33:53.748 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/149/20' -> 'copyofsnap1/test1/149/20' 2023-01-24T03:33:53.748 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/150' -> 'copyofsnap1/test1/150' 2023-01-24T03:33:53.748 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/150/10' -> 'copyofsnap1/test1/150/10' 2023-01-24T03:33:53.748 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/150/11' -> 'copyofsnap1/test1/150/11' 2023-01-24T03:33:53.749 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/150/12' -> 'copyofsnap1/test1/150/12' 2023-01-24T03:33:53.749 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/150/13' -> 'copyofsnap1/test1/150/13' 2023-01-24T03:33:53.750 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/150/14' -> 'copyofsnap1/test1/150/14' 2023-01-24T03:33:53.750 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/150/15' -> 'copyofsnap1/test1/150/15' 2023-01-24T03:33:53.750 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/150/16' -> 'copyofsnap1/test1/150/16' 2023-01-24T03:33:53.751 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/150/17' -> 'copyofsnap1/test1/150/17' 2023-01-24T03:33:53.751 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/150/18' -> 'copyofsnap1/test1/150/18' 2023-01-24T03:33:53.751 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/150/19' -> 'copyofsnap1/test1/150/19' 2023-01-24T03:33:53.752 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/150/20' -> 'copyofsnap1/test1/150/20' 2023-01-24T03:33:53.752 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/151' -> 'copyofsnap1/test1/151' 2023-01-24T03:33:53.754 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/151/10' -> 'copyofsnap1/test1/151/10' 2023-01-24T03:33:53.754 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/151/11' -> 'copyofsnap1/test1/151/11' 2023-01-24T03:33:53.754 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/151/12' -> 'copyofsnap1/test1/151/12' 2023-01-24T03:33:53.755 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/151/13' -> 'copyofsnap1/test1/151/13' 2023-01-24T03:33:53.755 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/151/14' -> 'copyofsnap1/test1/151/14' 2023-01-24T03:33:53.755 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/151/15' -> 'copyofsnap1/test1/151/15' 2023-01-24T03:33:53.755 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/151/16' -> 'copyofsnap1/test1/151/16' 2023-01-24T03:33:53.756 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/151/17' -> 'copyofsnap1/test1/151/17' 2023-01-24T03:33:53.756 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/151/18' -> 'copyofsnap1/test1/151/18' 2023-01-24T03:33:53.756 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/151/19' -> 'copyofsnap1/test1/151/19' 2023-01-24T03:33:53.756 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/151/20' -> 'copyofsnap1/test1/151/20' 2023-01-24T03:33:53.756 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/152' -> 'copyofsnap1/test1/152' 2023-01-24T03:33:53.757 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/152/10' -> 'copyofsnap1/test1/152/10' 2023-01-24T03:33:53.757 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/152/11' -> 'copyofsnap1/test1/152/11' 2023-01-24T03:33:53.757 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/152/12' -> 'copyofsnap1/test1/152/12' 2023-01-24T03:33:53.757 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/152/13' -> 'copyofsnap1/test1/152/13' 2023-01-24T03:33:53.757 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/152/14' -> 'copyofsnap1/test1/152/14' 2023-01-24T03:33:53.758 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/152/15' -> 'copyofsnap1/test1/152/15' 2023-01-24T03:33:53.758 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/152/16' -> 'copyofsnap1/test1/152/16' 2023-01-24T03:33:53.758 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/152/17' -> 'copyofsnap1/test1/152/17' 2023-01-24T03:33:53.758 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/152/18' -> 'copyofsnap1/test1/152/18' 2023-01-24T03:33:53.758 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/152/19' -> 'copyofsnap1/test1/152/19' 2023-01-24T03:33:53.758 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/152/20' -> 'copyofsnap1/test1/152/20' 2023-01-24T03:33:53.759 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/153' -> 'copyofsnap1/test1/153' 2023-01-24T03:33:53.759 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/153/10' -> 'copyofsnap1/test1/153/10' 2023-01-24T03:33:53.759 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/153/11' -> 'copyofsnap1/test1/153/11' 2023-01-24T03:33:53.759 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/153/12' -> 'copyofsnap1/test1/153/12' 2023-01-24T03:33:53.759 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/153/13' -> 'copyofsnap1/test1/153/13' 2023-01-24T03:33:53.760 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/153/14' -> 'copyofsnap1/test1/153/14' 2023-01-24T03:33:53.760 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/153/15' -> 'copyofsnap1/test1/153/15' 2023-01-24T03:33:53.760 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/153/16' -> 'copyofsnap1/test1/153/16' 2023-01-24T03:33:53.760 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/153/17' -> 'copyofsnap1/test1/153/17' 2023-01-24T03:33:53.760 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/153/18' -> 'copyofsnap1/test1/153/18' 2023-01-24T03:33:53.761 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/153/19' -> 'copyofsnap1/test1/153/19' 2023-01-24T03:33:53.761 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/153/20' -> 'copyofsnap1/test1/153/20' 2023-01-24T03:33:54.709 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/154' -> 'copyofsnap1/test1/154' 2023-01-24T03:33:54.709 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/154/10' -> 'copyofsnap1/test1/154/10' 2023-01-24T03:33:54.710 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/154/11' -> 'copyofsnap1/test1/154/11' 2023-01-24T03:33:54.710 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/154/12' -> 'copyofsnap1/test1/154/12' 2023-01-24T03:33:54.710 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/154/13' -> 'copyofsnap1/test1/154/13' 2023-01-24T03:33:54.710 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/154/14' -> 'copyofsnap1/test1/154/14' 2023-01-24T03:33:54.710 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/154/15' -> 'copyofsnap1/test1/154/15' 2023-01-24T03:33:54.711 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/154/16' -> 'copyofsnap1/test1/154/16' 2023-01-24T03:33:54.711 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/154/17' -> 'copyofsnap1/test1/154/17' 2023-01-24T03:33:54.711 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/154/18' -> 'copyofsnap1/test1/154/18' 2023-01-24T03:33:54.711 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/154/19' -> 'copyofsnap1/test1/154/19' 2023-01-24T03:33:54.711 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/154/20' -> 'copyofsnap1/test1/154/20' 2023-01-24T03:33:54.712 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/155' -> 'copyofsnap1/test1/155' 2023-01-24T03:33:54.712 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/155/10' -> 'copyofsnap1/test1/155/10' 2023-01-24T03:33:54.712 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/155/11' -> 'copyofsnap1/test1/155/11' 2023-01-24T03:33:54.712 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/155/12' -> 'copyofsnap1/test1/155/12' 2023-01-24T03:33:54.712 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/155/13' -> 'copyofsnap1/test1/155/13' 2023-01-24T03:33:54.713 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/155/14' -> 'copyofsnap1/test1/155/14' 2023-01-24T03:33:54.713 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/155/15' -> 'copyofsnap1/test1/155/15' 2023-01-24T03:33:54.713 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/155/16' -> 'copyofsnap1/test1/155/16' 2023-01-24T03:33:54.713 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/155/17' -> 'copyofsnap1/test1/155/17' 2023-01-24T03:33:54.713 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/155/18' -> 'copyofsnap1/test1/155/18' 2023-01-24T03:33:54.713 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/155/19' -> 'copyofsnap1/test1/155/19' 2023-01-24T03:33:54.714 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/155/20' -> 'copyofsnap1/test1/155/20' 2023-01-24T03:33:54.714 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/156' -> 'copyofsnap1/test1/156' 2023-01-24T03:33:54.714 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/156/10' -> 'copyofsnap1/test1/156/10' 2023-01-24T03:33:54.714 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/156/11' -> 'copyofsnap1/test1/156/11' 2023-01-24T03:33:54.714 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/156/12' -> 'copyofsnap1/test1/156/12' 2023-01-24T03:33:54.715 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/156/13' -> 'copyofsnap1/test1/156/13' 2023-01-24T03:33:54.715 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/156/14' -> 'copyofsnap1/test1/156/14' 2023-01-24T03:33:54.715 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/156/15' -> 'copyofsnap1/test1/156/15' 2023-01-24T03:33:54.715 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/156/16' -> 'copyofsnap1/test1/156/16' 2023-01-24T03:33:54.715 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/156/17' -> 'copyofsnap1/test1/156/17' 2023-01-24T03:33:54.716 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/156/18' -> 'copyofsnap1/test1/156/18' 2023-01-24T03:33:54.716 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/156/19' -> 'copyofsnap1/test1/156/19' 2023-01-24T03:33:54.716 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/156/20' -> 'copyofsnap1/test1/156/20' 2023-01-24T03:33:54.716 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/157' -> 'copyofsnap1/test1/157' 2023-01-24T03:33:54.717 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/157/10' -> 'copyofsnap1/test1/157/10' 2023-01-24T03:33:54.717 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/157/11' -> 'copyofsnap1/test1/157/11' 2023-01-24T03:33:54.717 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/157/12' -> 'copyofsnap1/test1/157/12' 2023-01-24T03:33:54.717 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/157/13' -> 'copyofsnap1/test1/157/13' 2023-01-24T03:33:54.717 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/157/14' -> 'copyofsnap1/test1/157/14' 2023-01-24T03:33:54.717 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/157/15' -> 'copyofsnap1/test1/157/15' 2023-01-24T03:33:54.718 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/157/16' -> 'copyofsnap1/test1/157/16' 2023-01-24T03:33:54.718 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/157/17' -> 'copyofsnap1/test1/157/17' 2023-01-24T03:33:54.718 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/157/18' -> 'copyofsnap1/test1/157/18' 2023-01-24T03:33:54.718 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/157/19' -> 'copyofsnap1/test1/157/19' 2023-01-24T03:33:54.718 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/157/20' -> 'copyofsnap1/test1/157/20' 2023-01-24T03:33:54.719 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/158' -> 'copyofsnap1/test1/158' 2023-01-24T03:33:54.719 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/158/10' -> 'copyofsnap1/test1/158/10' 2023-01-24T03:33:54.719 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/158/11' -> 'copyofsnap1/test1/158/11' 2023-01-24T03:33:54.719 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/158/12' -> 'copyofsnap1/test1/158/12' 2023-01-24T03:33:54.719 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/158/13' -> 'copyofsnap1/test1/158/13' 2023-01-24T03:33:54.719 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/158/14' -> 'copyofsnap1/test1/158/14' 2023-01-24T03:33:54.720 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/158/15' -> 'copyofsnap1/test1/158/15' 2023-01-24T03:33:54.720 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/158/16' -> 'copyofsnap1/test1/158/16' 2023-01-24T03:33:54.720 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/158/17' -> 'copyofsnap1/test1/158/17' 2023-01-24T03:33:54.720 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/158/18' -> 'copyofsnap1/test1/158/18' 2023-01-24T03:33:54.720 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/158/19' -> 'copyofsnap1/test1/158/19' 2023-01-24T03:33:54.721 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/158/20' -> 'copyofsnap1/test1/158/20' 2023-01-24T03:33:54.721 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/159' -> 'copyofsnap1/test1/159' 2023-01-24T03:33:54.721 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/159/10' -> 'copyofsnap1/test1/159/10' 2023-01-24T03:33:54.721 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/159/11' -> 'copyofsnap1/test1/159/11' 2023-01-24T03:33:54.721 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/159/12' -> 'copyofsnap1/test1/159/12' 2023-01-24T03:33:54.722 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/159/13' -> 'copyofsnap1/test1/159/13' 2023-01-24T03:33:54.722 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/159/14' -> 'copyofsnap1/test1/159/14' 2023-01-24T03:33:54.722 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/159/15' -> 'copyofsnap1/test1/159/15' 2023-01-24T03:33:54.722 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/159/16' -> 'copyofsnap1/test1/159/16' 2023-01-24T03:33:54.722 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/159/17' -> 'copyofsnap1/test1/159/17' 2023-01-24T03:33:54.723 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/159/18' -> 'copyofsnap1/test1/159/18' 2023-01-24T03:33:54.723 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/159/19' -> 'copyofsnap1/test1/159/19' 2023-01-24T03:33:54.723 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/159/20' -> 'copyofsnap1/test1/159/20' 2023-01-24T03:33:54.723 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/160' -> 'copyofsnap1/test1/160' 2023-01-24T03:33:54.723 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/160/10' -> 'copyofsnap1/test1/160/10' 2023-01-24T03:33:54.724 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/160/11' -> 'copyofsnap1/test1/160/11' 2023-01-24T03:33:54.724 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/160/12' -> 'copyofsnap1/test1/160/12' 2023-01-24T03:33:54.724 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/160/13' -> 'copyofsnap1/test1/160/13' 2023-01-24T03:33:54.724 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/160/14' -> 'copyofsnap1/test1/160/14' 2023-01-24T03:33:54.724 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/160/15' -> 'copyofsnap1/test1/160/15' 2023-01-24T03:33:54.725 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/160/16' -> 'copyofsnap1/test1/160/16' 2023-01-24T03:33:54.725 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/160/17' -> 'copyofsnap1/test1/160/17' 2023-01-24T03:33:56.858 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/160/18' -> 'copyofsnap1/test1/160/18' 2023-01-24T03:33:56.859 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/160/19' -> 'copyofsnap1/test1/160/19' 2023-01-24T03:33:56.859 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/160/20' -> 'copyofsnap1/test1/160/20' 2023-01-24T03:33:56.859 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/161' -> 'copyofsnap1/test1/161' 2023-01-24T03:33:56.859 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/161/10' -> 'copyofsnap1/test1/161/10' 2023-01-24T03:33:56.860 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/161/11' -> 'copyofsnap1/test1/161/11' 2023-01-24T03:33:56.860 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/161/12' -> 'copyofsnap1/test1/161/12' 2023-01-24T03:33:56.860 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/161/13' -> 'copyofsnap1/test1/161/13' 2023-01-24T03:33:56.860 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/161/14' -> 'copyofsnap1/test1/161/14' 2023-01-24T03:33:56.860 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/161/15' -> 'copyofsnap1/test1/161/15' 2023-01-24T03:33:56.861 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/161/16' -> 'copyofsnap1/test1/161/16' 2023-01-24T03:33:56.861 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/161/17' -> 'copyofsnap1/test1/161/17' 2023-01-24T03:33:56.861 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/161/18' -> 'copyofsnap1/test1/161/18' 2023-01-24T03:33:56.861 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/161/19' -> 'copyofsnap1/test1/161/19' 2023-01-24T03:33:56.862 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/161/20' -> 'copyofsnap1/test1/161/20' 2023-01-24T03:33:56.862 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/162' -> 'copyofsnap1/test1/162' 2023-01-24T03:33:56.862 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/162/10' -> 'copyofsnap1/test1/162/10' 2023-01-24T03:33:56.862 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/162/11' -> 'copyofsnap1/test1/162/11' 2023-01-24T03:33:56.863 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/162/12' -> 'copyofsnap1/test1/162/12' 2023-01-24T03:33:56.863 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/162/13' -> 'copyofsnap1/test1/162/13' 2023-01-24T03:33:56.863 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/162/14' -> 'copyofsnap1/test1/162/14' 2023-01-24T03:33:56.863 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/162/15' -> 'copyofsnap1/test1/162/15' 2023-01-24T03:33:56.864 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/162/16' -> 'copyofsnap1/test1/162/16' 2023-01-24T03:33:56.864 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/162/17' -> 'copyofsnap1/test1/162/17' 2023-01-24T03:33:56.864 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/162/18' -> 'copyofsnap1/test1/162/18' 2023-01-24T03:33:56.864 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/162/19' -> 'copyofsnap1/test1/162/19' 2023-01-24T03:33:56.864 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/162/20' -> 'copyofsnap1/test1/162/20' 2023-01-24T03:33:56.865 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/163' -> 'copyofsnap1/test1/163' 2023-01-24T03:33:56.865 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/163/10' -> 'copyofsnap1/test1/163/10' 2023-01-24T03:33:56.865 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/163/11' -> 'copyofsnap1/test1/163/11' 2023-01-24T03:33:56.865 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/163/12' -> 'copyofsnap1/test1/163/12' 2023-01-24T03:33:56.866 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/163/13' -> 'copyofsnap1/test1/163/13' 2023-01-24T03:33:56.867 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/163/14' -> 'copyofsnap1/test1/163/14' 2023-01-24T03:33:56.867 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/163/15' -> 'copyofsnap1/test1/163/15' 2023-01-24T03:33:56.867 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/163/16' -> 'copyofsnap1/test1/163/16' 2023-01-24T03:33:56.867 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/163/17' -> 'copyofsnap1/test1/163/17' 2023-01-24T03:33:56.868 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/163/18' -> 'copyofsnap1/test1/163/18' 2023-01-24T03:33:56.868 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/163/19' -> 'copyofsnap1/test1/163/19' 2023-01-24T03:33:56.868 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/163/20' -> 'copyofsnap1/test1/163/20' 2023-01-24T03:33:56.868 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/164' -> 'copyofsnap1/test1/164' 2023-01-24T03:33:56.869 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/164/10' -> 'copyofsnap1/test1/164/10' 2023-01-24T03:33:56.869 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/164/11' -> 'copyofsnap1/test1/164/11' 2023-01-24T03:33:56.869 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/164/12' -> 'copyofsnap1/test1/164/12' 2023-01-24T03:33:56.869 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/164/13' -> 'copyofsnap1/test1/164/13' 2023-01-24T03:33:56.869 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/164/14' -> 'copyofsnap1/test1/164/14' 2023-01-24T03:33:56.870 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/164/15' -> 'copyofsnap1/test1/164/15' 2023-01-24T03:33:56.870 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/164/16' -> 'copyofsnap1/test1/164/16' 2023-01-24T03:33:56.870 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/164/17' -> 'copyofsnap1/test1/164/17' 2023-01-24T03:33:56.870 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/164/18' -> 'copyofsnap1/test1/164/18' 2023-01-24T03:33:56.870 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/164/19' -> 'copyofsnap1/test1/164/19' 2023-01-24T03:33:56.870 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/164/20' -> 'copyofsnap1/test1/164/20' 2023-01-24T03:33:56.871 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/165' -> 'copyofsnap1/test1/165' 2023-01-24T03:33:56.871 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/165/10' -> 'copyofsnap1/test1/165/10' 2023-01-24T03:33:56.871 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/165/11' -> 'copyofsnap1/test1/165/11' 2023-01-24T03:33:56.871 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/165/12' -> 'copyofsnap1/test1/165/12' 2023-01-24T03:33:56.871 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/165/13' -> 'copyofsnap1/test1/165/13' 2023-01-24T03:33:56.872 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/165/14' -> 'copyofsnap1/test1/165/14' 2023-01-24T03:33:56.872 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/165/15' -> 'copyofsnap1/test1/165/15' 2023-01-24T03:33:56.872 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/165/16' -> 'copyofsnap1/test1/165/16' 2023-01-24T03:33:56.872 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/165/17' -> 'copyofsnap1/test1/165/17' 2023-01-24T03:33:56.872 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/165/18' -> 'copyofsnap1/test1/165/18' 2023-01-24T03:33:56.873 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/165/19' -> 'copyofsnap1/test1/165/19' 2023-01-24T03:33:56.873 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/165/20' -> 'copyofsnap1/test1/165/20' 2023-01-24T03:33:56.873 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/166' -> 'copyofsnap1/test1/166' 2023-01-24T03:33:56.873 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/166/10' -> 'copyofsnap1/test1/166/10' 2023-01-24T03:33:56.873 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/166/11' -> 'copyofsnap1/test1/166/11' 2023-01-24T03:33:56.874 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/166/12' -> 'copyofsnap1/test1/166/12' 2023-01-24T03:33:56.874 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/166/13' -> 'copyofsnap1/test1/166/13' 2023-01-24T03:33:56.874 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/166/14' -> 'copyofsnap1/test1/166/14' 2023-01-24T03:33:56.874 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/166/15' -> 'copyofsnap1/test1/166/15' 2023-01-24T03:33:56.874 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/166/16' -> 'copyofsnap1/test1/166/16' 2023-01-24T03:33:56.875 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/166/17' -> 'copyofsnap1/test1/166/17' 2023-01-24T03:33:56.875 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/166/18' -> 'copyofsnap1/test1/166/18' 2023-01-24T03:33:56.875 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/166/19' -> 'copyofsnap1/test1/166/19' 2023-01-24T03:33:56.875 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/166/20' -> 'copyofsnap1/test1/166/20' 2023-01-24T03:33:56.875 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/167' -> 'copyofsnap1/test1/167' 2023-01-24T03:33:56.876 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/167/10' -> 'copyofsnap1/test1/167/10' 2023-01-24T03:33:56.876 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/167/11' -> 'copyofsnap1/test1/167/11' 2023-01-24T03:33:56.876 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/167/12' -> 'copyofsnap1/test1/167/12' 2023-01-24T03:33:56.876 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/167/13' -> 'copyofsnap1/test1/167/13' 2023-01-24T03:33:56.876 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/167/14' -> 'copyofsnap1/test1/167/14' 2023-01-24T03:33:57.243 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/167/15' -> 'copyofsnap1/test1/167/15' 2023-01-24T03:33:57.244 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/167/16' -> 'copyofsnap1/test1/167/16' 2023-01-24T03:33:57.244 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/167/17' -> 'copyofsnap1/test1/167/17' 2023-01-24T03:33:57.244 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/167/18' -> 'copyofsnap1/test1/167/18' 2023-01-24T03:33:57.245 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/167/19' -> 'copyofsnap1/test1/167/19' 2023-01-24T03:33:57.245 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/167/20' -> 'copyofsnap1/test1/167/20' 2023-01-24T03:33:57.245 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/168' -> 'copyofsnap1/test1/168' 2023-01-24T03:33:57.245 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/168/10' -> 'copyofsnap1/test1/168/10' 2023-01-24T03:33:57.245 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/168/11' -> 'copyofsnap1/test1/168/11' 2023-01-24T03:33:57.246 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/168/12' -> 'copyofsnap1/test1/168/12' 2023-01-24T03:33:57.246 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/168/13' -> 'copyofsnap1/test1/168/13' 2023-01-24T03:33:57.246 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/168/14' -> 'copyofsnap1/test1/168/14' 2023-01-24T03:33:57.246 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/168/15' -> 'copyofsnap1/test1/168/15' 2023-01-24T03:33:57.246 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/168/16' -> 'copyofsnap1/test1/168/16' 2023-01-24T03:33:57.247 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/168/17' -> 'copyofsnap1/test1/168/17' 2023-01-24T03:33:57.247 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/168/18' -> 'copyofsnap1/test1/168/18' 2023-01-24T03:33:57.247 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/168/19' -> 'copyofsnap1/test1/168/19' 2023-01-24T03:33:57.247 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/168/20' -> 'copyofsnap1/test1/168/20' 2023-01-24T03:33:57.247 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/169' -> 'copyofsnap1/test1/169' 2023-01-24T03:33:57.248 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/169/10' -> 'copyofsnap1/test1/169/10' 2023-01-24T03:33:57.248 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/169/11' -> 'copyofsnap1/test1/169/11' 2023-01-24T03:33:57.248 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/169/12' -> 'copyofsnap1/test1/169/12' 2023-01-24T03:33:57.249 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/169/13' -> 'copyofsnap1/test1/169/13' 2023-01-24T03:33:57.249 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/169/14' -> 'copyofsnap1/test1/169/14' 2023-01-24T03:33:57.250 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/169/15' -> 'copyofsnap1/test1/169/15' 2023-01-24T03:33:57.250 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/169/16' -> 'copyofsnap1/test1/169/16' 2023-01-24T03:33:57.250 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/169/17' -> 'copyofsnap1/test1/169/17' 2023-01-24T03:33:57.251 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/169/18' -> 'copyofsnap1/test1/169/18' 2023-01-24T03:33:57.251 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/169/19' -> 'copyofsnap1/test1/169/19' 2023-01-24T03:33:57.251 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/169/20' -> 'copyofsnap1/test1/169/20' 2023-01-24T03:33:57.252 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/170' -> 'copyofsnap1/test1/170' 2023-01-24T03:33:57.252 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/170/10' -> 'copyofsnap1/test1/170/10' 2023-01-24T03:33:57.253 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/170/11' -> 'copyofsnap1/test1/170/11' 2023-01-24T03:33:57.253 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/170/12' -> 'copyofsnap1/test1/170/12' 2023-01-24T03:33:57.253 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/170/13' -> 'copyofsnap1/test1/170/13' 2023-01-24T03:33:57.254 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/170/14' -> 'copyofsnap1/test1/170/14' 2023-01-24T03:33:57.254 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/170/15' -> 'copyofsnap1/test1/170/15' 2023-01-24T03:33:57.254 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/170/16' -> 'copyofsnap1/test1/170/16' 2023-01-24T03:33:57.254 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/170/17' -> 'copyofsnap1/test1/170/17' 2023-01-24T03:33:57.255 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/170/18' -> 'copyofsnap1/test1/170/18' 2023-01-24T03:33:57.255 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/170/19' -> 'copyofsnap1/test1/170/19' 2023-01-24T03:33:57.255 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/170/20' -> 'copyofsnap1/test1/170/20' 2023-01-24T03:33:57.255 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/171' -> 'copyofsnap1/test1/171' 2023-01-24T03:33:57.256 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/171/10' -> 'copyofsnap1/test1/171/10' 2023-01-24T03:33:57.256 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/171/11' -> 'copyofsnap1/test1/171/11' 2023-01-24T03:33:57.256 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/171/12' -> 'copyofsnap1/test1/171/12' 2023-01-24T03:33:57.256 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/171/13' -> 'copyofsnap1/test1/171/13' 2023-01-24T03:33:57.256 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/171/14' -> 'copyofsnap1/test1/171/14' 2023-01-24T03:33:57.257 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/171/15' -> 'copyofsnap1/test1/171/15' 2023-01-24T03:33:57.257 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/171/16' -> 'copyofsnap1/test1/171/16' 2023-01-24T03:33:57.257 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/171/17' -> 'copyofsnap1/test1/171/17' 2023-01-24T03:33:57.257 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/171/18' -> 'copyofsnap1/test1/171/18' 2023-01-24T03:33:57.257 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/171/19' -> 'copyofsnap1/test1/171/19' 2023-01-24T03:33:57.258 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/171/20' -> 'copyofsnap1/test1/171/20' 2023-01-24T03:33:57.258 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/172' -> 'copyofsnap1/test1/172' 2023-01-24T03:33:57.258 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/172/10' -> 'copyofsnap1/test1/172/10' 2023-01-24T03:33:57.258 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/172/11' -> 'copyofsnap1/test1/172/11' 2023-01-24T03:33:57.258 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/172/12' -> 'copyofsnap1/test1/172/12' 2023-01-24T03:33:57.259 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/172/13' -> 'copyofsnap1/test1/172/13' 2023-01-24T03:33:57.259 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/172/14' -> 'copyofsnap1/test1/172/14' 2023-01-24T03:33:57.259 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/172/15' -> 'copyofsnap1/test1/172/15' 2023-01-24T03:33:57.259 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/172/16' -> 'copyofsnap1/test1/172/16' 2023-01-24T03:33:57.259 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/172/17' -> 'copyofsnap1/test1/172/17' 2023-01-24T03:33:57.260 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/172/18' -> 'copyofsnap1/test1/172/18' 2023-01-24T03:33:57.260 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/172/19' -> 'copyofsnap1/test1/172/19' 2023-01-24T03:33:57.260 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/172/20' -> 'copyofsnap1/test1/172/20' 2023-01-24T03:33:57.260 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/173' -> 'copyofsnap1/test1/173' 2023-01-24T03:33:57.261 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/173/10' -> 'copyofsnap1/test1/173/10' 2023-01-24T03:33:57.261 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/173/11' -> 'copyofsnap1/test1/173/11' 2023-01-24T03:33:57.261 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/173/12' -> 'copyofsnap1/test1/173/12' 2023-01-24T03:33:57.261 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/173/13' -> 'copyofsnap1/test1/173/13' 2023-01-24T03:33:57.261 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/173/14' -> 'copyofsnap1/test1/173/14' 2023-01-24T03:33:57.262 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/173/15' -> 'copyofsnap1/test1/173/15' 2023-01-24T03:33:57.262 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/173/16' -> 'copyofsnap1/test1/173/16' 2023-01-24T03:33:57.262 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/173/17' -> 'copyofsnap1/test1/173/17' 2023-01-24T03:33:57.262 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/173/18' -> 'copyofsnap1/test1/173/18' 2023-01-24T03:33:57.262 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/173/19' -> 'copyofsnap1/test1/173/19' 2023-01-24T03:33:57.263 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/173/20' -> 'copyofsnap1/test1/173/20' 2023-01-24T03:33:57.263 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/174' -> 'copyofsnap1/test1/174' 2023-01-24T03:33:57.263 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/174/10' -> 'copyofsnap1/test1/174/10' 2023-01-24T03:33:57.263 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/174/11' -> 'copyofsnap1/test1/174/11' 2023-01-24T03:33:58.307 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/174/12' -> 'copyofsnap1/test1/174/12' 2023-01-24T03:33:58.307 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/174/13' -> 'copyofsnap1/test1/174/13' 2023-01-24T03:33:58.308 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/174/14' -> 'copyofsnap1/test1/174/14' 2023-01-24T03:33:58.308 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/174/15' -> 'copyofsnap1/test1/174/15' 2023-01-24T03:33:58.308 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/174/16' -> 'copyofsnap1/test1/174/16' 2023-01-24T03:33:58.308 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/174/17' -> 'copyofsnap1/test1/174/17' 2023-01-24T03:33:58.308 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/174/18' -> 'copyofsnap1/test1/174/18' 2023-01-24T03:33:58.309 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/174/19' -> 'copyofsnap1/test1/174/19' 2023-01-24T03:33:58.309 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/174/20' -> 'copyofsnap1/test1/174/20' 2023-01-24T03:33:58.309 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/175' -> 'copyofsnap1/test1/175' 2023-01-24T03:33:58.309 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/175/10' -> 'copyofsnap1/test1/175/10' 2023-01-24T03:33:58.309 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/175/11' -> 'copyofsnap1/test1/175/11' 2023-01-24T03:33:58.310 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/175/12' -> 'copyofsnap1/test1/175/12' 2023-01-24T03:33:58.310 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/175/13' -> 'copyofsnap1/test1/175/13' 2023-01-24T03:33:58.310 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/175/14' -> 'copyofsnap1/test1/175/14' 2023-01-24T03:33:58.310 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/175/15' -> 'copyofsnap1/test1/175/15' 2023-01-24T03:33:58.310 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/175/16' -> 'copyofsnap1/test1/175/16' 2023-01-24T03:33:58.311 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/175/17' -> 'copyofsnap1/test1/175/17' 2023-01-24T03:33:58.311 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/175/18' -> 'copyofsnap1/test1/175/18' 2023-01-24T03:33:58.311 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/175/19' -> 'copyofsnap1/test1/175/19' 2023-01-24T03:33:58.311 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/175/20' -> 'copyofsnap1/test1/175/20' 2023-01-24T03:33:58.311 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/176' -> 'copyofsnap1/test1/176' 2023-01-24T03:33:58.312 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/176/10' -> 'copyofsnap1/test1/176/10' 2023-01-24T03:33:58.312 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/176/11' -> 'copyofsnap1/test1/176/11' 2023-01-24T03:33:58.314 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/176/12' -> 'copyofsnap1/test1/176/12' 2023-01-24T03:33:58.314 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/176/13' -> 'copyofsnap1/test1/176/13' 2023-01-24T03:33:58.314 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/176/14' -> 'copyofsnap1/test1/176/14' 2023-01-24T03:33:58.314 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/176/15' -> 'copyofsnap1/test1/176/15' 2023-01-24T03:33:58.315 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/176/16' -> 'copyofsnap1/test1/176/16' 2023-01-24T03:33:58.315 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/176/17' -> 'copyofsnap1/test1/176/17' 2023-01-24T03:33:58.315 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/176/18' -> 'copyofsnap1/test1/176/18' 2023-01-24T03:33:58.315 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/176/19' -> 'copyofsnap1/test1/176/19' 2023-01-24T03:33:58.315 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/176/20' -> 'copyofsnap1/test1/176/20' 2023-01-24T03:33:58.316 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/177' -> 'copyofsnap1/test1/177' 2023-01-24T03:33:58.316 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/177/10' -> 'copyofsnap1/test1/177/10' 2023-01-24T03:33:58.316 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/177/11' -> 'copyofsnap1/test1/177/11' 2023-01-24T03:33:58.316 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/177/12' -> 'copyofsnap1/test1/177/12' 2023-01-24T03:33:58.316 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/177/13' -> 'copyofsnap1/test1/177/13' 2023-01-24T03:33:58.317 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/177/14' -> 'copyofsnap1/test1/177/14' 2023-01-24T03:33:58.317 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/177/15' -> 'copyofsnap1/test1/177/15' 2023-01-24T03:33:58.317 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/177/16' -> 'copyofsnap1/test1/177/16' 2023-01-24T03:33:58.317 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/177/17' -> 'copyofsnap1/test1/177/17' 2023-01-24T03:33:58.317 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/177/18' -> 'copyofsnap1/test1/177/18' 2023-01-24T03:33:58.318 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/177/19' -> 'copyofsnap1/test1/177/19' 2023-01-24T03:33:58.318 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/177/20' -> 'copyofsnap1/test1/177/20' 2023-01-24T03:33:58.318 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/178' -> 'copyofsnap1/test1/178' 2023-01-24T03:33:58.318 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/178/10' -> 'copyofsnap1/test1/178/10' 2023-01-24T03:33:58.318 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/178/11' -> 'copyofsnap1/test1/178/11' 2023-01-24T03:33:58.319 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/178/12' -> 'copyofsnap1/test1/178/12' 2023-01-24T03:33:58.319 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/178/13' -> 'copyofsnap1/test1/178/13' 2023-01-24T03:33:58.319 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/178/14' -> 'copyofsnap1/test1/178/14' 2023-01-24T03:33:58.319 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/178/15' -> 'copyofsnap1/test1/178/15' 2023-01-24T03:33:58.319 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/178/16' -> 'copyofsnap1/test1/178/16' 2023-01-24T03:33:58.320 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/178/17' -> 'copyofsnap1/test1/178/17' 2023-01-24T03:33:58.320 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/178/18' -> 'copyofsnap1/test1/178/18' 2023-01-24T03:33:58.320 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/178/19' -> 'copyofsnap1/test1/178/19' 2023-01-24T03:33:58.320 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/178/20' -> 'copyofsnap1/test1/178/20' 2023-01-24T03:33:58.320 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/179' -> 'copyofsnap1/test1/179' 2023-01-24T03:33:58.320 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/179/10' -> 'copyofsnap1/test1/179/10' 2023-01-24T03:33:58.321 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/179/11' -> 'copyofsnap1/test1/179/11' 2023-01-24T03:33:58.321 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/179/12' -> 'copyofsnap1/test1/179/12' 2023-01-24T03:33:58.321 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/179/13' -> 'copyofsnap1/test1/179/13' 2023-01-24T03:33:58.321 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/179/14' -> 'copyofsnap1/test1/179/14' 2023-01-24T03:33:58.321 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/179/15' -> 'copyofsnap1/test1/179/15' 2023-01-24T03:33:58.322 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/179/16' -> 'copyofsnap1/test1/179/16' 2023-01-24T03:33:58.322 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/179/17' -> 'copyofsnap1/test1/179/17' 2023-01-24T03:33:58.322 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/179/18' -> 'copyofsnap1/test1/179/18' 2023-01-24T03:33:58.322 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/179/19' -> 'copyofsnap1/test1/179/19' 2023-01-24T03:33:58.322 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/179/20' -> 'copyofsnap1/test1/179/20' 2023-01-24T03:33:58.323 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/180' -> 'copyofsnap1/test1/180' 2023-01-24T03:33:58.323 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/180/10' -> 'copyofsnap1/test1/180/10' 2023-01-24T03:33:58.323 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/180/11' -> 'copyofsnap1/test1/180/11' 2023-01-24T03:33:58.323 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/180/12' -> 'copyofsnap1/test1/180/12' 2023-01-24T03:33:58.323 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/180/13' -> 'copyofsnap1/test1/180/13' 2023-01-24T03:33:58.323 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/180/14' -> 'copyofsnap1/test1/180/14' 2023-01-24T03:33:58.324 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/180/15' -> 'copyofsnap1/test1/180/15' 2023-01-24T03:33:58.324 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/180/16' -> 'copyofsnap1/test1/180/16' 2023-01-24T03:33:58.324 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/180/17' -> 'copyofsnap1/test1/180/17' 2023-01-24T03:33:58.324 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/180/18' -> 'copyofsnap1/test1/180/18' 2023-01-24T03:33:58.324 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/180/19' -> 'copyofsnap1/test1/180/19' 2023-01-24T03:33:58.325 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/180/20' -> 'copyofsnap1/test1/180/20' 2023-01-24T03:33:59.294 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:33:59.297 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:33:59.972 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/181' -> 'copyofsnap1/test1/181' 2023-01-24T03:33:59.973 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/181/10' -> 'copyofsnap1/test1/181/10' 2023-01-24T03:33:59.973 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/181/11' -> 'copyofsnap1/test1/181/11' 2023-01-24T03:33:59.973 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/181/12' -> 'copyofsnap1/test1/181/12' 2023-01-24T03:33:59.973 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/181/13' -> 'copyofsnap1/test1/181/13' 2023-01-24T03:33:59.974 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/181/14' -> 'copyofsnap1/test1/181/14' 2023-01-24T03:33:59.974 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/181/15' -> 'copyofsnap1/test1/181/15' 2023-01-24T03:33:59.974 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/181/16' -> 'copyofsnap1/test1/181/16' 2023-01-24T03:33:59.974 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/181/17' -> 'copyofsnap1/test1/181/17' 2023-01-24T03:33:59.974 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/181/18' -> 'copyofsnap1/test1/181/18' 2023-01-24T03:33:59.975 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/181/19' -> 'copyofsnap1/test1/181/19' 2023-01-24T03:33:59.975 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/181/20' -> 'copyofsnap1/test1/181/20' 2023-01-24T03:33:59.975 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/182' -> 'copyofsnap1/test1/182' 2023-01-24T03:33:59.975 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/182/10' -> 'copyofsnap1/test1/182/10' 2023-01-24T03:33:59.975 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/182/11' -> 'copyofsnap1/test1/182/11' 2023-01-24T03:33:59.976 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/182/12' -> 'copyofsnap1/test1/182/12' 2023-01-24T03:33:59.976 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/182/13' -> 'copyofsnap1/test1/182/13' 2023-01-24T03:33:59.976 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/182/14' -> 'copyofsnap1/test1/182/14' 2023-01-24T03:33:59.976 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/182/15' -> 'copyofsnap1/test1/182/15' 2023-01-24T03:33:59.976 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/182/16' -> 'copyofsnap1/test1/182/16' 2023-01-24T03:33:59.976 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/182/17' -> 'copyofsnap1/test1/182/17' 2023-01-24T03:33:59.977 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/182/18' -> 'copyofsnap1/test1/182/18' 2023-01-24T03:33:59.977 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/182/19' -> 'copyofsnap1/test1/182/19' 2023-01-24T03:33:59.977 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/182/20' -> 'copyofsnap1/test1/182/20' 2023-01-24T03:33:59.977 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/183' -> 'copyofsnap1/test1/183' 2023-01-24T03:33:59.977 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/183/10' -> 'copyofsnap1/test1/183/10' 2023-01-24T03:33:59.978 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/183/11' -> 'copyofsnap1/test1/183/11' 2023-01-24T03:33:59.978 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/183/12' -> 'copyofsnap1/test1/183/12' 2023-01-24T03:33:59.978 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/183/13' -> 'copyofsnap1/test1/183/13' 2023-01-24T03:33:59.978 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/183/14' -> 'copyofsnap1/test1/183/14' 2023-01-24T03:33:59.978 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/183/15' -> 'copyofsnap1/test1/183/15' 2023-01-24T03:33:59.979 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/183/16' -> 'copyofsnap1/test1/183/16' 2023-01-24T03:33:59.979 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/183/17' -> 'copyofsnap1/test1/183/17' 2023-01-24T03:33:59.979 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/183/18' -> 'copyofsnap1/test1/183/18' 2023-01-24T03:33:59.979 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/183/19' -> 'copyofsnap1/test1/183/19' 2023-01-24T03:33:59.979 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/183/20' -> 'copyofsnap1/test1/183/20' 2023-01-24T03:33:59.980 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/184' -> 'copyofsnap1/test1/184' 2023-01-24T03:33:59.980 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/184/10' -> 'copyofsnap1/test1/184/10' 2023-01-24T03:33:59.980 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/184/11' -> 'copyofsnap1/test1/184/11' 2023-01-24T03:33:59.980 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/184/12' -> 'copyofsnap1/test1/184/12' 2023-01-24T03:33:59.980 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/184/13' -> 'copyofsnap1/test1/184/13' 2023-01-24T03:33:59.981 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/184/14' -> 'copyofsnap1/test1/184/14' 2023-01-24T03:33:59.981 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/184/15' -> 'copyofsnap1/test1/184/15' 2023-01-24T03:33:59.981 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/184/16' -> 'copyofsnap1/test1/184/16' 2023-01-24T03:33:59.981 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/184/17' -> 'copyofsnap1/test1/184/17' 2023-01-24T03:33:59.981 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/184/18' -> 'copyofsnap1/test1/184/18' 2023-01-24T03:33:59.982 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/184/19' -> 'copyofsnap1/test1/184/19' 2023-01-24T03:33:59.982 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/184/20' -> 'copyofsnap1/test1/184/20' 2023-01-24T03:33:59.982 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/185' -> 'copyofsnap1/test1/185' 2023-01-24T03:33:59.982 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/185/10' -> 'copyofsnap1/test1/185/10' 2023-01-24T03:33:59.982 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/185/11' -> 'copyofsnap1/test1/185/11' 2023-01-24T03:33:59.983 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/185/12' -> 'copyofsnap1/test1/185/12' 2023-01-24T03:33:59.983 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/185/13' -> 'copyofsnap1/test1/185/13' 2023-01-24T03:33:59.983 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/185/14' -> 'copyofsnap1/test1/185/14' 2023-01-24T03:33:59.983 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/185/15' -> 'copyofsnap1/test1/185/15' 2023-01-24T03:33:59.983 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/185/16' -> 'copyofsnap1/test1/185/16' 2023-01-24T03:33:59.984 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/185/17' -> 'copyofsnap1/test1/185/17' 2023-01-24T03:33:59.984 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/185/18' -> 'copyofsnap1/test1/185/18' 2023-01-24T03:33:59.984 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/185/19' -> 'copyofsnap1/test1/185/19' 2023-01-24T03:33:59.984 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/185/20' -> 'copyofsnap1/test1/185/20' 2023-01-24T03:33:59.984 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/186' -> 'copyofsnap1/test1/186' 2023-01-24T03:33:59.985 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/186/10' -> 'copyofsnap1/test1/186/10' 2023-01-24T03:33:59.985 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/186/11' -> 'copyofsnap1/test1/186/11' 2023-01-24T03:33:59.985 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/186/12' -> 'copyofsnap1/test1/186/12' 2023-01-24T03:33:59.985 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/186/13' -> 'copyofsnap1/test1/186/13' 2023-01-24T03:33:59.986 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/186/14' -> 'copyofsnap1/test1/186/14' 2023-01-24T03:33:59.986 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/186/15' -> 'copyofsnap1/test1/186/15' 2023-01-24T03:33:59.986 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/186/16' -> 'copyofsnap1/test1/186/16' 2023-01-24T03:33:59.986 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/186/17' -> 'copyofsnap1/test1/186/17' 2023-01-24T03:33:59.987 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/186/18' -> 'copyofsnap1/test1/186/18' 2023-01-24T03:33:59.987 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/186/19' -> 'copyofsnap1/test1/186/19' 2023-01-24T03:33:59.987 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/186/20' -> 'copyofsnap1/test1/186/20' 2023-01-24T03:33:59.987 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/187' -> 'copyofsnap1/test1/187' 2023-01-24T03:33:59.987 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/187/10' -> 'copyofsnap1/test1/187/10' 2023-01-24T03:33:59.988 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/187/11' -> 'copyofsnap1/test1/187/11' 2023-01-24T03:33:59.988 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/187/12' -> 'copyofsnap1/test1/187/12' 2023-01-24T03:33:59.988 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/187/13' -> 'copyofsnap1/test1/187/13' 2023-01-24T03:33:59.988 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/187/14' -> 'copyofsnap1/test1/187/14' 2023-01-24T03:33:59.988 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/187/15' -> 'copyofsnap1/test1/187/15' 2023-01-24T03:33:59.989 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/187/16' -> 'copyofsnap1/test1/187/16' 2023-01-24T03:33:59.989 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/187/17' -> 'copyofsnap1/test1/187/17' 2023-01-24T03:34:00.168 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/187/18' -> 'copyofsnap1/test1/187/18' 2023-01-24T03:34:00.168 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/187/19' -> 'copyofsnap1/test1/187/19' 2023-01-24T03:34:00.168 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/187/20' -> 'copyofsnap1/test1/187/20' 2023-01-24T03:34:00.168 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/188' -> 'copyofsnap1/test1/188' 2023-01-24T03:34:00.169 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/188/10' -> 'copyofsnap1/test1/188/10' 2023-01-24T03:34:00.169 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/188/11' -> 'copyofsnap1/test1/188/11' 2023-01-24T03:34:00.169 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/188/12' -> 'copyofsnap1/test1/188/12' 2023-01-24T03:34:00.169 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/188/13' -> 'copyofsnap1/test1/188/13' 2023-01-24T03:34:00.169 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/188/14' -> 'copyofsnap1/test1/188/14' 2023-01-24T03:34:00.170 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/188/15' -> 'copyofsnap1/test1/188/15' 2023-01-24T03:34:00.170 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/188/16' -> 'copyofsnap1/test1/188/16' 2023-01-24T03:34:00.170 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/188/17' -> 'copyofsnap1/test1/188/17' 2023-01-24T03:34:00.170 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/188/18' -> 'copyofsnap1/test1/188/18' 2023-01-24T03:34:00.170 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/188/19' -> 'copyofsnap1/test1/188/19' 2023-01-24T03:34:00.171 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/188/20' -> 'copyofsnap1/test1/188/20' 2023-01-24T03:34:00.171 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/189' -> 'copyofsnap1/test1/189' 2023-01-24T03:34:00.171 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/189/10' -> 'copyofsnap1/test1/189/10' 2023-01-24T03:34:00.171 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/189/11' -> 'copyofsnap1/test1/189/11' 2023-01-24T03:34:00.171 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/189/12' -> 'copyofsnap1/test1/189/12' 2023-01-24T03:34:00.172 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/189/13' -> 'copyofsnap1/test1/189/13' 2023-01-24T03:34:00.172 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/189/14' -> 'copyofsnap1/test1/189/14' 2023-01-24T03:34:00.172 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/189/15' -> 'copyofsnap1/test1/189/15' 2023-01-24T03:34:00.172 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/189/16' -> 'copyofsnap1/test1/189/16' 2023-01-24T03:34:00.172 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/189/17' -> 'copyofsnap1/test1/189/17' 2023-01-24T03:34:00.172 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/189/18' -> 'copyofsnap1/test1/189/18' 2023-01-24T03:34:00.173 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/189/19' -> 'copyofsnap1/test1/189/19' 2023-01-24T03:34:00.173 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/189/20' -> 'copyofsnap1/test1/189/20' 2023-01-24T03:34:00.173 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/190' -> 'copyofsnap1/test1/190' 2023-01-24T03:34:00.173 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/190/10' -> 'copyofsnap1/test1/190/10' 2023-01-24T03:34:00.173 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/190/11' -> 'copyofsnap1/test1/190/11' 2023-01-24T03:34:00.174 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/190/12' -> 'copyofsnap1/test1/190/12' 2023-01-24T03:34:00.174 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/190/13' -> 'copyofsnap1/test1/190/13' 2023-01-24T03:34:00.174 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/190/14' -> 'copyofsnap1/test1/190/14' 2023-01-24T03:34:00.174 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/190/15' -> 'copyofsnap1/test1/190/15' 2023-01-24T03:34:00.174 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/190/16' -> 'copyofsnap1/test1/190/16' 2023-01-24T03:34:00.175 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/190/17' -> 'copyofsnap1/test1/190/17' 2023-01-24T03:34:00.175 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/190/18' -> 'copyofsnap1/test1/190/18' 2023-01-24T03:34:00.175 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/190/19' -> 'copyofsnap1/test1/190/19' 2023-01-24T03:34:00.175 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/190/20' -> 'copyofsnap1/test1/190/20' 2023-01-24T03:34:00.175 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/191' -> 'copyofsnap1/test1/191' 2023-01-24T03:34:00.176 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/191/10' -> 'copyofsnap1/test1/191/10' 2023-01-24T03:34:00.176 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/191/11' -> 'copyofsnap1/test1/191/11' 2023-01-24T03:34:00.176 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/191/12' -> 'copyofsnap1/test1/191/12' 2023-01-24T03:34:00.176 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/191/13' -> 'copyofsnap1/test1/191/13' 2023-01-24T03:34:00.176 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/191/14' -> 'copyofsnap1/test1/191/14' 2023-01-24T03:34:00.177 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/191/15' -> 'copyofsnap1/test1/191/15' 2023-01-24T03:34:00.177 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/191/16' -> 'copyofsnap1/test1/191/16' 2023-01-24T03:34:00.177 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/191/17' -> 'copyofsnap1/test1/191/17' 2023-01-24T03:34:00.177 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/191/18' -> 'copyofsnap1/test1/191/18' 2023-01-24T03:34:00.177 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/191/19' -> 'copyofsnap1/test1/191/19' 2023-01-24T03:34:00.178 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/191/20' -> 'copyofsnap1/test1/191/20' 2023-01-24T03:34:00.178 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/192' -> 'copyofsnap1/test1/192' 2023-01-24T03:34:00.178 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/192/10' -> 'copyofsnap1/test1/192/10' 2023-01-24T03:34:00.178 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/192/11' -> 'copyofsnap1/test1/192/11' 2023-01-24T03:34:00.178 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/192/12' -> 'copyofsnap1/test1/192/12' 2023-01-24T03:34:00.179 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/192/13' -> 'copyofsnap1/test1/192/13' 2023-01-24T03:34:00.179 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/192/14' -> 'copyofsnap1/test1/192/14' 2023-01-24T03:34:00.179 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/192/15' -> 'copyofsnap1/test1/192/15' 2023-01-24T03:34:00.179 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/192/16' -> 'copyofsnap1/test1/192/16' 2023-01-24T03:34:00.179 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/192/17' -> 'copyofsnap1/test1/192/17' 2023-01-24T03:34:00.180 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/192/18' -> 'copyofsnap1/test1/192/18' 2023-01-24T03:34:00.180 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/192/19' -> 'copyofsnap1/test1/192/19' 2023-01-24T03:34:00.180 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/192/20' -> 'copyofsnap1/test1/192/20' 2023-01-24T03:34:00.180 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/193' -> 'copyofsnap1/test1/193' 2023-01-24T03:34:00.180 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/193/10' -> 'copyofsnap1/test1/193/10' 2023-01-24T03:34:00.180 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/193/11' -> 'copyofsnap1/test1/193/11' 2023-01-24T03:34:00.181 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/193/12' -> 'copyofsnap1/test1/193/12' 2023-01-24T03:34:00.181 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/193/13' -> 'copyofsnap1/test1/193/13' 2023-01-24T03:34:00.181 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/193/14' -> 'copyofsnap1/test1/193/14' 2023-01-24T03:34:00.181 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/193/15' -> 'copyofsnap1/test1/193/15' 2023-01-24T03:34:00.181 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/193/16' -> 'copyofsnap1/test1/193/16' 2023-01-24T03:34:00.182 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/193/17' -> 'copyofsnap1/test1/193/17' 2023-01-24T03:34:00.182 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/193/18' -> 'copyofsnap1/test1/193/18' 2023-01-24T03:34:00.182 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/193/19' -> 'copyofsnap1/test1/193/19' 2023-01-24T03:34:00.182 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/193/20' -> 'copyofsnap1/test1/193/20' 2023-01-24T03:34:00.182 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/194' -> 'copyofsnap1/test1/194' 2023-01-24T03:34:00.183 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/194/10' -> 'copyofsnap1/test1/194/10' 2023-01-24T03:34:00.183 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/194/11' -> 'copyofsnap1/test1/194/11' 2023-01-24T03:34:00.183 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/194/12' -> 'copyofsnap1/test1/194/12' 2023-01-24T03:34:00.183 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/194/13' -> 'copyofsnap1/test1/194/13' 2023-01-24T03:34:00.183 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/194/14' -> 'copyofsnap1/test1/194/14' 2023-01-24T03:34:01.976 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/194/15' -> 'copyofsnap1/test1/194/15' 2023-01-24T03:34:01.977 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/194/16' -> 'copyofsnap1/test1/194/16' 2023-01-24T03:34:01.977 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/194/17' -> 'copyofsnap1/test1/194/17' 2023-01-24T03:34:01.977 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/194/18' -> 'copyofsnap1/test1/194/18' 2023-01-24T03:34:01.978 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/194/19' -> 'copyofsnap1/test1/194/19' 2023-01-24T03:34:01.978 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/194/20' -> 'copyofsnap1/test1/194/20' 2023-01-24T03:34:01.978 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/195' -> 'copyofsnap1/test1/195' 2023-01-24T03:34:01.978 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/195/10' -> 'copyofsnap1/test1/195/10' 2023-01-24T03:34:01.978 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/195/11' -> 'copyofsnap1/test1/195/11' 2023-01-24T03:34:01.978 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/195/12' -> 'copyofsnap1/test1/195/12' 2023-01-24T03:34:01.979 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/195/13' -> 'copyofsnap1/test1/195/13' 2023-01-24T03:34:01.979 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/195/14' -> 'copyofsnap1/test1/195/14' 2023-01-24T03:34:01.979 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/195/15' -> 'copyofsnap1/test1/195/15' 2023-01-24T03:34:01.979 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/195/16' -> 'copyofsnap1/test1/195/16' 2023-01-24T03:34:01.979 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/195/17' -> 'copyofsnap1/test1/195/17' 2023-01-24T03:34:01.980 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/195/18' -> 'copyofsnap1/test1/195/18' 2023-01-24T03:34:01.980 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/195/19' -> 'copyofsnap1/test1/195/19' 2023-01-24T03:34:01.980 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/195/20' -> 'copyofsnap1/test1/195/20' 2023-01-24T03:34:01.980 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/196' -> 'copyofsnap1/test1/196' 2023-01-24T03:34:01.980 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/196/10' -> 'copyofsnap1/test1/196/10' 2023-01-24T03:34:01.981 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/196/11' -> 'copyofsnap1/test1/196/11' 2023-01-24T03:34:01.981 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/196/12' -> 'copyofsnap1/test1/196/12' 2023-01-24T03:34:01.981 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/196/13' -> 'copyofsnap1/test1/196/13' 2023-01-24T03:34:01.981 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/196/14' -> 'copyofsnap1/test1/196/14' 2023-01-24T03:34:01.981 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/196/15' -> 'copyofsnap1/test1/196/15' 2023-01-24T03:34:01.982 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/196/16' -> 'copyofsnap1/test1/196/16' 2023-01-24T03:34:01.982 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/196/17' -> 'copyofsnap1/test1/196/17' 2023-01-24T03:34:01.982 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/196/18' -> 'copyofsnap1/test1/196/18' 2023-01-24T03:34:01.982 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/196/19' -> 'copyofsnap1/test1/196/19' 2023-01-24T03:34:01.982 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/196/20' -> 'copyofsnap1/test1/196/20' 2023-01-24T03:34:01.982 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/197' -> 'copyofsnap1/test1/197' 2023-01-24T03:34:01.983 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/197/10' -> 'copyofsnap1/test1/197/10' 2023-01-24T03:34:01.983 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/197/11' -> 'copyofsnap1/test1/197/11' 2023-01-24T03:34:01.983 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/197/12' -> 'copyofsnap1/test1/197/12' 2023-01-24T03:34:01.983 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/197/13' -> 'copyofsnap1/test1/197/13' 2023-01-24T03:34:01.983 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/197/14' -> 'copyofsnap1/test1/197/14' 2023-01-24T03:34:01.984 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/197/15' -> 'copyofsnap1/test1/197/15' 2023-01-24T03:34:01.984 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/197/16' -> 'copyofsnap1/test1/197/16' 2023-01-24T03:34:01.984 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/197/17' -> 'copyofsnap1/test1/197/17' 2023-01-24T03:34:01.984 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/197/18' -> 'copyofsnap1/test1/197/18' 2023-01-24T03:34:01.984 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/197/19' -> 'copyofsnap1/test1/197/19' 2023-01-24T03:34:01.985 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/197/20' -> 'copyofsnap1/test1/197/20' 2023-01-24T03:34:01.985 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/198' -> 'copyofsnap1/test1/198' 2023-01-24T03:34:01.985 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/198/10' -> 'copyofsnap1/test1/198/10' 2023-01-24T03:34:01.985 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/198/11' -> 'copyofsnap1/test1/198/11' 2023-01-24T03:34:01.985 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/198/12' -> 'copyofsnap1/test1/198/12' 2023-01-24T03:34:01.986 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/198/13' -> 'copyofsnap1/test1/198/13' 2023-01-24T03:34:01.986 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/198/14' -> 'copyofsnap1/test1/198/14' 2023-01-24T03:34:01.986 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/198/15' -> 'copyofsnap1/test1/198/15' 2023-01-24T03:34:01.986 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/198/16' -> 'copyofsnap1/test1/198/16' 2023-01-24T03:34:01.986 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/198/17' -> 'copyofsnap1/test1/198/17' 2023-01-24T03:34:01.987 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/198/18' -> 'copyofsnap1/test1/198/18' 2023-01-24T03:34:01.987 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/198/19' -> 'copyofsnap1/test1/198/19' 2023-01-24T03:34:01.987 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/198/20' -> 'copyofsnap1/test1/198/20' 2023-01-24T03:34:01.987 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/199' -> 'copyofsnap1/test1/199' 2023-01-24T03:34:01.988 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/199/10' -> 'copyofsnap1/test1/199/10' 2023-01-24T03:34:01.988 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/199/11' -> 'copyofsnap1/test1/199/11' 2023-01-24T03:34:01.988 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/199/12' -> 'copyofsnap1/test1/199/12' 2023-01-24T03:34:01.988 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/199/13' -> 'copyofsnap1/test1/199/13' 2023-01-24T03:34:01.988 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/199/14' -> 'copyofsnap1/test1/199/14' 2023-01-24T03:34:01.989 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/199/15' -> 'copyofsnap1/test1/199/15' 2023-01-24T03:34:01.989 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/199/16' -> 'copyofsnap1/test1/199/16' 2023-01-24T03:34:01.989 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/199/17' -> 'copyofsnap1/test1/199/17' 2023-01-24T03:34:01.989 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/199/18' -> 'copyofsnap1/test1/199/18' 2023-01-24T03:34:01.989 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/199/19' -> 'copyofsnap1/test1/199/19' 2023-01-24T03:34:01.989 INFO:tasks.workunit.client.0.smithi112.stdout:'.snap/test1/199/20' -> 'copyofsnap1/test1/199/20' 2023-01-24T03:34:01.990 INFO:tasks.workunit.client.0.smithi112.stdout:Take third snapshot .snap/test3 2023-01-24T03:34:02.011 INFO:tasks.workunit.client.0.smithi112.stdout:Delete the snapshots... 2023-01-24T03:34:03.528 INFO:tasks.workunit.client.0.smithi112.stdout:.//.snap/test1 2023-01-24T03:34:03.528 INFO:tasks.workunit.client.0.smithi112.stdout:.//.snap/test2 2023-01-24T03:34:03.529 INFO:tasks.workunit.client.0.smithi112.stdout:.//.snap/test3 2023-01-24T03:34:03.532 INFO:tasks.workunit.client.0.smithi112.stdout:./274/.snap/snap-subdir-test 2023-01-24T03:34:03.536 INFO:tasks.workunit.client.0.smithi112.stdout:./166/.snap/snap-subdir-test 2023-01-24T03:34:03.539 INFO:tasks.workunit.client.0.smithi112.stdout:./256/.snap/snap-subdir-test 2023-01-24T03:34:03.543 INFO:tasks.workunit.client.0.smithi112.stdout:./247/.snap/snap-subdir-test 2023-01-24T03:34:03.547 INFO:tasks.workunit.client.0.smithi112.stdout:./248/.snap/snap-subdir-test 2023-01-24T03:34:03.551 INFO:tasks.workunit.client.0.smithi112.stdout:./147/.snap/snap-subdir-test 2023-01-24T03:34:03.556 INFO:tasks.workunit.client.0.smithi112.stdout:./151/.snap/snap-subdir-test 2023-01-24T03:34:03.559 INFO:tasks.workunit.client.0.smithi112.stdout:./170/.snap/snap-subdir-test 2023-01-24T03:34:03.563 INFO:tasks.workunit.client.0.smithi112.stdout:./167/.snap/snap-subdir-test 2023-01-24T03:34:03.567 INFO:tasks.workunit.client.0.smithi112.stdout:./140/.snap/snap-subdir-test 2023-01-24T03:34:03.571 INFO:tasks.workunit.client.0.smithi112.stdout:./142/.snap/snap-subdir-test 2023-01-24T03:34:03.575 INFO:tasks.workunit.client.0.smithi112.stdout:./150/.snap/snap-subdir-test 2023-01-24T03:34:03.578 INFO:tasks.workunit.client.0.smithi112.stdout:./273/.snap/snap-subdir-test 2023-01-24T03:34:03.697 INFO:tasks.workunit.client.0.smithi112.stdout:./269/.snap/snap-subdir-test 2023-01-24T03:34:03.701 INFO:tasks.workunit.client.0.smithi112.stdout:./105/.snap/snap-subdir-test 2023-01-24T03:34:03.706 INFO:tasks.workunit.client.0.smithi112.stdout:./210/.snap/snap-subdir-test 2023-01-24T03:34:03.710 INFO:tasks.workunit.client.0.smithi112.stdout:./265/.snap/snap-subdir-test 2023-01-24T03:34:03.713 INFO:tasks.workunit.client.0.smithi112.stdout:./218/.snap/snap-subdir-test 2023-01-24T03:34:03.717 INFO:tasks.workunit.client.0.smithi112.stdout:./290/.snap/snap-subdir-test 2023-01-24T03:34:03.720 INFO:tasks.workunit.client.0.smithi112.stdout:./149/.snap/snap-subdir-test 2023-01-24T03:34:03.725 INFO:tasks.workunit.client.0.smithi112.stdout:./109/.snap/snap-subdir-test 2023-01-24T03:34:03.728 INFO:tasks.workunit.client.0.smithi112.stdout:./292/.snap/snap-subdir-test 2023-01-24T03:34:03.732 INFO:tasks.workunit.client.0.smithi112.stdout:./242/.snap/snap-subdir-test 2023-01-24T03:34:03.736 INFO:tasks.workunit.client.0.smithi112.stdout:./158/.snap/snap-subdir-test 2023-01-24T03:34:03.742 INFO:tasks.workunit.client.0.smithi112.stdout:./243/.snap/snap-subdir-test 2023-01-24T03:34:03.746 INFO:tasks.workunit.client.0.smithi112.stdout:./106/.snap/snap-subdir-test 2023-01-24T03:34:04.628 INFO:tasks.workunit.client.0.smithi112.stdout:./250/.snap/snap-subdir-test 2023-01-24T03:34:04.632 INFO:tasks.workunit.client.0.smithi112.stdout:./221/.snap/snap-subdir-test 2023-01-24T03:34:04.637 INFO:tasks.workunit.client.0.smithi112.stdout:./244/.snap/snap-subdir-test 2023-01-24T03:34:05.082 INFO:tasks.workunit.client.0.smithi112.stdout:./258/.snap/snap-subdir-test 2023-01-24T03:34:05.869 INFO:tasks.workunit.client.0.smithi112.stdout:./268/.snap/snap-subdir-test 2023-01-24T03:34:05.873 INFO:tasks.workunit.client.0.smithi112.stdout:./189/.snap/snap-subdir-test 2023-01-24T03:34:05.877 INFO:tasks.workunit.client.0.smithi112.stdout:./285/.snap/snap-subdir-test 2023-01-24T03:34:05.881 INFO:tasks.workunit.client.0.smithi112.stdout:./208/.snap/snap-subdir-test 2023-01-24T03:34:05.886 INFO:tasks.workunit.client.0.smithi112.stdout:./288/.snap/snap-subdir-test 2023-01-24T03:34:05.890 INFO:tasks.workunit.client.0.smithi112.stdout:./209/.snap/snap-subdir-test 2023-01-24T03:34:05.894 INFO:tasks.workunit.client.0.smithi112.stdout:./253/.snap/snap-subdir-test 2023-01-24T03:34:05.898 INFO:tasks.workunit.client.0.smithi112.stdout:./249/.snap/snap-subdir-test 2023-01-24T03:34:05.903 INFO:tasks.workunit.client.0.smithi112.stdout:./133/.snap/snap-subdir-test 2023-01-24T03:34:06.439 INFO:tasks.workunit.client.0.smithi112.stdout:./291/.snap/snap-subdir-test 2023-01-24T03:34:06.443 INFO:tasks.workunit.client.0.smithi112.stdout:./226/.snap/snap-subdir-test 2023-01-24T03:34:06.446 INFO:tasks.workunit.client.0.smithi112.stdout:./173/.snap/snap-subdir-test 2023-01-24T03:34:06.449 INFO:tasks.workunit.client.0.smithi112.stdout:./174/.snap/snap-subdir-test 2023-01-24T03:34:06.453 INFO:tasks.workunit.client.0.smithi112.stdout:./299/.snap/snap-subdir-test 2023-01-24T03:34:06.456 INFO:tasks.workunit.client.0.smithi112.stdout:./262/.snap/snap-subdir-test 2023-01-24T03:34:06.460 INFO:tasks.workunit.client.0.smithi112.stdout:./231/.snap/snap-subdir-test 2023-01-24T03:34:06.463 INFO:tasks.workunit.client.0.smithi112.stdout:./214/.snap/snap-subdir-test 2023-01-24T03:34:06.466 INFO:tasks.workunit.client.0.smithi112.stdout:./281/.snap/snap-subdir-test 2023-01-24T03:34:06.469 INFO:tasks.workunit.client.0.smithi112.stdout:./289/.snap/snap-subdir-test 2023-01-24T03:34:06.473 INFO:tasks.workunit.client.0.smithi112.stdout:./283/.snap/snap-subdir-test 2023-01-24T03:34:06.476 INFO:tasks.workunit.client.0.smithi112.stdout:./232/.snap/snap-subdir-test 2023-01-24T03:34:06.479 INFO:tasks.workunit.client.0.smithi112.stdout:./298/.snap/snap-subdir-test 2023-01-24T03:34:06.483 INFO:tasks.workunit.client.0.smithi112.stdout:./233/.snap/snap-subdir-test 2023-01-24T03:34:06.486 INFO:tasks.workunit.client.0.smithi112.stdout:./155/.snap/snap-subdir-test 2023-01-24T03:34:06.490 INFO:tasks.workunit.client.0.smithi112.stdout:./114/.snap/snap-subdir-test 2023-01-24T03:34:06.494 INFO:tasks.workunit.client.0.smithi112.stdout:./159/.snap/snap-subdir-test 2023-01-24T03:34:06.497 INFO:tasks.workunit.client.0.smithi112.stdout:./187/.snap/snap-subdir-test 2023-01-24T03:34:06.500 INFO:tasks.workunit.client.0.smithi112.stdout:./185/.snap/snap-subdir-test 2023-01-24T03:34:06.503 INFO:tasks.workunit.client.0.smithi112.stdout:./293/.snap/snap-subdir-test 2023-01-24T03:34:06.507 INFO:tasks.workunit.client.0.smithi112.stdout:./228/.snap/snap-subdir-test 2023-01-24T03:34:06.510 INFO:tasks.workunit.client.0.smithi112.stdout:./120/.snap/snap-subdir-test 2023-01-24T03:34:06.513 INFO:tasks.workunit.client.0.smithi112.stdout:./110/.snap/snap-subdir-test 2023-01-24T03:34:06.516 INFO:tasks.workunit.client.0.smithi112.stdout:./267/.snap/snap-subdir-test 2023-01-24T03:34:06.519 INFO:tasks.workunit.client.0.smithi112.stdout:./202/.snap/snap-subdir-test 2023-01-24T03:34:06.810 INFO:tasks.workunit.client.0.smithi112.stdout:./137/.snap/snap-subdir-test 2023-01-24T03:34:06.814 INFO:tasks.workunit.client.0.smithi112.stdout:./246/.snap/snap-subdir-test 2023-01-24T03:34:07.723 INFO:tasks.workunit.client.0.smithi112.stdout:./219/.snap/snap-subdir-test 2023-01-24T03:34:07.728 INFO:tasks.workunit.client.0.smithi112.stdout:./198/.snap/snap-subdir-test 2023-01-24T03:34:07.998 INFO:tasks.workunit.client.0.smithi112.stdout:./136/.snap/snap-subdir-test 2023-01-24T03:34:08.002 INFO:tasks.workunit.client.0.smithi112.stdout:./270/.snap/snap-subdir-test 2023-01-24T03:34:08.006 INFO:tasks.workunit.client.0.smithi112.stdout:./125/.snap/snap-subdir-test 2023-01-24T03:34:08.011 INFO:tasks.workunit.client.0.smithi112.stdout:./194/.snap/snap-subdir-test 2023-01-24T03:34:08.016 INFO:tasks.workunit.client.0.smithi112.stdout:./119/.snap/snap-subdir-test 2023-01-24T03:34:08.020 INFO:tasks.workunit.client.0.smithi112.stdout:./297/.snap/snap-subdir-test 2023-01-24T03:34:08.025 INFO:tasks.workunit.client.0.smithi112.stdout:./230/.snap/snap-subdir-test 2023-01-24T03:34:08.029 INFO:tasks.workunit.client.0.smithi112.stdout:./260/.snap/snap-subdir-test 2023-01-24T03:34:08.033 INFO:tasks.workunit.client.0.smithi112.stdout:./157/.snap/snap-subdir-test 2023-01-24T03:34:08.038 INFO:tasks.workunit.client.0.smithi112.stdout:./172/.snap/snap-subdir-test 2023-01-24T03:34:08.302 INFO:tasks.workunit.client.0.smithi112.stdout:./277/.snap/snap-subdir-test 2023-01-24T03:34:08.307 INFO:tasks.workunit.client.0.smithi112.stdout:./186/.snap/snap-subdir-test 2023-01-24T03:34:08.311 INFO:tasks.workunit.client.0.smithi112.stdout:./237/.snap/snap-subdir-test 2023-01-24T03:34:08.316 INFO:tasks.workunit.client.0.smithi112.stdout:./190/.snap/snap-subdir-test 2023-01-24T03:34:08.320 INFO:tasks.workunit.client.0.smithi112.stdout:./213/.snap/snap-subdir-test 2023-01-24T03:34:08.325 INFO:tasks.workunit.client.0.smithi112.stdout:./104/.snap/snap-subdir-test 2023-01-24T03:34:08.329 INFO:tasks.workunit.client.0.smithi112.stdout:./164/.snap/snap-subdir-test 2023-01-24T03:34:08.333 INFO:tasks.workunit.client.0.smithi112.stdout:./112/.snap/snap-subdir-test 2023-01-24T03:34:08.338 INFO:tasks.workunit.client.0.smithi112.stdout:./295/.snap/snap-subdir-test 2023-01-24T03:34:08.342 INFO:tasks.workunit.client.0.smithi112.stdout:./272/.snap/snap-subdir-test 2023-01-24T03:34:08.346 INFO:tasks.workunit.client.0.smithi112.stdout:./100/.snap/snap-subdir-test 2023-01-24T03:34:08.350 INFO:tasks.workunit.client.0.smithi112.stdout:./134/.snap/snap-subdir-test 2023-01-24T03:34:08.355 INFO:tasks.workunit.client.0.smithi112.stdout:./200/.snap/snap-subdir-test 2023-01-24T03:34:08.358 INFO:tasks.workunit.client.0.smithi112.stdout:./115/.snap/snap-subdir-test 2023-01-24T03:34:08.363 INFO:tasks.workunit.client.0.smithi112.stdout:./284/.snap/snap-subdir-test 2023-01-24T03:34:08.367 INFO:tasks.workunit.client.0.smithi112.stdout:./280/.snap/snap-subdir-test 2023-01-24T03:34:08.370 INFO:tasks.workunit.client.0.smithi112.stdout:./116/.snap/snap-subdir-test 2023-01-24T03:34:08.374 INFO:tasks.workunit.client.0.smithi112.stdout:./245/.snap/snap-subdir-test 2023-01-24T03:34:08.378 INFO:tasks.workunit.client.0.smithi112.stdout:./195/.snap/snap-subdir-test 2023-01-24T03:34:08.382 INFO:tasks.workunit.client.0.smithi112.stdout:./138/.snap/snap-subdir-test 2023-01-24T03:34:09.327 INFO:tasks.workunit.client.0.smithi112.stdout:./135/.snap/snap-subdir-test 2023-01-24T03:34:09.332 INFO:tasks.workunit.client.0.smithi112.stdout:./117/.snap/snap-subdir-test 2023-01-24T03:34:09.337 INFO:tasks.workunit.client.0.smithi112.stdout:./241/.snap/snap-subdir-test 2023-01-24T03:34:09.342 INFO:tasks.workunit.client.0.smithi112.stdout:./143/.snap/snap-subdir-test 2023-01-24T03:34:09.346 INFO:tasks.workunit.client.0.smithi112.stdout:./180/.snap/snap-subdir-test 2023-01-24T03:34:09.350 INFO:tasks.workunit.client.0.smithi112.stdout:./126/.snap/snap-subdir-test 2023-01-24T03:34:09.355 INFO:tasks.workunit.client.0.smithi112.stdout:./161/.snap/snap-subdir-test 2023-01-24T03:34:09.359 INFO:tasks.workunit.client.0.smithi112.stdout:./204/.snap/snap-subdir-test 2023-01-24T03:34:09.363 INFO:tasks.workunit.client.0.smithi112.stdout:./176/.snap/snap-subdir-test 2023-01-24T03:34:09.367 INFO:tasks.workunit.client.0.smithi112.stdout:./255/.snap/snap-subdir-test 2023-01-24T03:34:09.371 INFO:tasks.workunit.client.0.smithi112.stdout:./215/.snap/snap-subdir-test 2023-01-24T03:34:09.376 INFO:tasks.workunit.client.0.smithi112.stdout:./111/.snap/snap-subdir-test 2023-01-24T03:34:09.380 INFO:tasks.workunit.client.0.smithi112.stdout:./238/.snap/snap-subdir-test 2023-01-24T03:34:09.384 INFO:tasks.workunit.client.0.smithi112.stdout:./191/.snap/snap-subdir-test 2023-01-24T03:34:09.652 INFO:tasks.workunit.client.0.smithi112.stdout:./184/.snap/snap-subdir-test 2023-01-24T03:34:09.656 INFO:tasks.workunit.client.0.smithi112.stdout:./192/.snap/snap-subdir-test 2023-01-24T03:34:09.662 INFO:tasks.workunit.client.0.smithi112.stdout:./103/.snap/snap-subdir-test 2023-01-24T03:34:10.527 INFO:tasks.workunit.client.0.smithi112.stdout:./128/.snap/snap-subdir-test 2023-01-24T03:34:10.532 INFO:tasks.workunit.client.0.smithi112.stdout:./229/.snap/snap-subdir-test 2023-01-24T03:34:10.536 INFO:tasks.workunit.client.0.smithi112.stdout:./144/.snap/snap-subdir-test 2023-01-24T03:34:10.542 INFO:tasks.workunit.client.0.smithi112.stdout:./146/.snap/snap-subdir-test 2023-01-24T03:34:10.545 INFO:tasks.workunit.client.0.smithi112.stdout:./227/.snap/snap-subdir-test 2023-01-24T03:34:10.550 INFO:tasks.workunit.client.0.smithi112.stdout:./278/.snap/snap-subdir-test 2023-01-24T03:34:10.554 INFO:tasks.workunit.client.0.smithi112.stdout:./141/.snap/snap-subdir-test 2023-01-24T03:34:10.559 INFO:tasks.workunit.client.0.smithi112.stdout:./234/.snap/snap-subdir-test 2023-01-24T03:34:10.563 INFO:tasks.workunit.client.0.smithi112.stdout:./225/.snap/snap-subdir-test 2023-01-24T03:34:10.567 INFO:tasks.workunit.client.0.smithi112.stdout:./182/.snap/snap-subdir-test 2023-01-24T03:34:10.572 INFO:tasks.workunit.client.0.smithi112.stdout:./107/.snap/snap-subdir-test 2023-01-24T03:34:10.576 INFO:tasks.workunit.client.0.smithi112.stdout:./178/.snap/snap-subdir-test 2023-01-24T03:34:10.580 INFO:tasks.workunit.client.0.smithi112.stdout:./266/.snap/snap-subdir-test 2023-01-24T03:34:10.699 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:34:10.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.822+0000 7f93e3a3d700 1 Processor -- start 2023-01-24T03:34:10.825 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.822+0000 7f93e3a3d700 1 -- start start 2023-01-24T03:34:10.827 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.823+0000 7f93e3a3d700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f93dc0c7e40 0x7f93dc0b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:10.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.823+0000 7f93e3a3d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f93dc155da0 0x7f93dc0ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:10.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.823+0000 7f93e3a3d700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93dc156240 con 0x7f93dc0aef20 2023-01-24T03:34:10.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.823+0000 7f93e3a3d700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93dc0af320 con 0x7f93dc0c7e40 2023-01-24T03:34:10.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.823+0000 7f93e3a3d700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93dc0b1170 con 0x7f93dc155da0 2023-01-24T03:34:10.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.823+0000 7f93e17d9700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f93dc0c7e40 0x7f93dc0b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:10.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.823+0000 7f93e17d9700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f93dc0c7e40 0x7f93dc0b0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52538/0 (socket says 172.21.15.112:52538) 2023-01-24T03:34:10.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.823+0000 7f93e17d9700 1 -- 172.21.15.112:0/1270653179 learned_addr learned my addr 172.21.15.112:0/1270653179 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:34:10.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.823+0000 7f93e0fd8700 1 --2- 172.21.15.112:0/1270653179 >> v2:172.21.15.142:3300/0 conn(0x7f93dc155da0 0x7f93dc0ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:10.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.823+0000 7f93e17d9700 1 -- 172.21.15.112:0/1270653179 >> v2:172.21.15.142:3300/0 conn(0x7f93dc155da0 msgr2=0x7f93dc0ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:10.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.823+0000 7f93e17d9700 1 --2- 172.21.15.112:0/1270653179 >> v2:172.21.15.142:3300/0 conn(0x7f93dc155da0 0x7f93dc0ae9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:10.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.823+0000 7f93e17d9700 1 -- 172.21.15.112:0/1270653179 >> v1:172.21.15.112:6789/0 conn(0x7f93dc0aef20 legacy=0x7f93dc0ca3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:10.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.824+0000 7f93e17d9700 1 -- 172.21.15.112:0/1270653179 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93dc0b1450 con 0x7f93dc0c7e40 2023-01-24T03:34:10.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.824+0000 7f93e17d9700 1 --2- 172.21.15.112:0/1270653179 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f93dc0c7e40 0x7f93dc0b0aa0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f93dc152ea0 tx=0x7f93d8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c8170c7fa2150b4 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:10.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.824+0000 7f93cbfff700 1 -- 172.21.15.112:0/1270653179 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d800e070 con 0x7f93dc0c7e40 2023-01-24T03:34:10.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.824+0000 7f93cbfff700 1 -- 172.21.15.112:0/1270653179 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93d800ba50 con 0x7f93dc0c7e40 2023-01-24T03:34:10.831 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.824+0000 7f93cbfff700 1 -- 172.21.15.112:0/1270653179 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d80092e0 con 0x7f93dc0c7e40 2023-01-24T03:34:10.831 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.824+0000 7f93e3a3d700 1 -- 172.21.15.112:0/1270653179 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f93dc0c7e40 msgr2=0x7f93dc0b0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:10.831 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.824+0000 7f93e3a3d700 1 --2- 172.21.15.112:0/1270653179 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f93dc0c7e40 0x7f93dc0b0aa0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f93dc152ea0 tx=0x7f93d8009840 comp rx=0 tx=0).stop 2023-01-24T03:34:10.831 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.824+0000 7f93e3a3d700 1 -- 172.21.15.112:0/1270653179 shutdown_connections 2023-01-24T03:34:10.831 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.824+0000 7f93e3a3d700 1 --2- 172.21.15.112:0/1270653179 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f93dc0c7e40 0x7f93dc0b0aa0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:10.832 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.824+0000 7f93e3a3d700 1 --2- 172.21.15.112:0/1270653179 >> v2:172.21.15.142:3300/0 conn(0x7f93dc155da0 0x7f93dc0ae9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:10.832 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.824+0000 7f93e3a3d700 1 -- 172.21.15.112:0/1270653179 >> 172.21.15.112:0/1270653179 conn(0x7f93dc05e2c0 msgr2=0x7f93dc0cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:10.832 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.824+0000 7f93e3a3d700 1 -- 172.21.15.112:0/1270653179 shutdown_connections 2023-01-24T03:34:10.832 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.824+0000 7f93e3a3d700 1 -- 172.21.15.112:0/1270653179 wait complete. 2023-01-24T03:34:10.832 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.825+0000 7f93e3a3d700 1 Processor -- start 2023-01-24T03:34:10.833 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.826+0000 7f93e3a3d700 1 -- start start 2023-01-24T03:34:10.835 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.826+0000 7f93e3a3d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f93dc0aef20 0x7f93dc0cb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:10.835 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.826+0000 7f93e3a3d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f93dc0c7e40 0x7f93dc0c34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:10.836 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.826+0000 7f93e3a3d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f93dc155da0 0x7f93dc0c39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:10.836 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.826+0000 7f93e3a3d700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93dc151150 con 0x7f93dc0aef20 2023-01-24T03:34:10.836 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.826+0000 7f93e3a3d700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93dc157170 con 0x7f93dc0c7e40 2023-01-24T03:34:10.836 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.826+0000 7f93e3a3d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93dc169690 con 0x7f93dc155da0 2023-01-24T03:34:10.837 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.826+0000 7f93e17d9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f93dc0aef20 0x7f93dc0cb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:10.837 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.826+0000 7f93e17d9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f93dc0aef20 0x7f93dc0cb4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52544/0 (socket says 172.21.15.112:52544) 2023-01-24T03:34:10.837 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.826+0000 7f93e17d9700 1 -- 172.21.15.112:0/3062481837 learned_addr learned my addr 172.21.15.112:0/3062481837 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:34:10.837 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.827+0000 7f93e0fd8700 1 --2- 172.21.15.112:0/3062481837 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f93dc0c7e40 0x7f93dc0c34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:10.837 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.827+0000 7f93e1fda700 1 --2- 172.21.15.112:0/3062481837 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f93dc155da0 0x7f93dc0c39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:10.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.827+0000 7f93e17d9700 1 -- 172.21.15.112:0/3062481837 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f93dc155da0 msgr2=0x7f93dc0c39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:10.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.827+0000 7f93e17d9700 1 --2- 172.21.15.112:0/3062481837 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f93dc155da0 0x7f93dc0c39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:10.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.827+0000 7f93e17d9700 1 -- 172.21.15.112:0/3062481837 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f93dc0c7e40 msgr2=0x7f93dc0c34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:10.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.827+0000 7f93e17d9700 1 --2- 172.21.15.112:0/3062481837 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f93dc0c7e40 0x7f93dc0c34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:10.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.827+0000 7f93e17d9700 1 -- 172.21.15.112:0/3062481837 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d8009ae0 con 0x7f93dc0aef20 2023-01-24T03:34:10.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.827+0000 7f93e1fda700 1 --2- 172.21.15.112:0/3062481837 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f93dc155da0 0x7f93dc0c39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:34:10.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.827+0000 7f93e0fd8700 1 --2- 172.21.15.112:0/3062481837 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f93dc0c7e40 0x7f93dc0c34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:34:10.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.827+0000 7f93e17d9700 1 --2- 172.21.15.112:0/3062481837 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f93dc0aef20 0x7f93dc0cb4a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f93d8003440 tx=0x7f93d8018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:10.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.827+0000 7f93c97fa700 1 -- 172.21.15.112:0/3062481837 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d8008930 con 0x7f93dc0aef20 2023-01-24T03:34:10.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.827+0000 7f93c97fa700 1 -- 172.21.15.112:0/3062481837 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93d8008ab0 con 0x7f93dc0aef20 2023-01-24T03:34:10.840 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.827+0000 7f93e3a3d700 1 -- 172.21.15.112:0/3062481837 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93dc0c4040 con 0x7f93dc0aef20 2023-01-24T03:34:10.840 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.827+0000 7f93c97fa700 1 -- 172.21.15.112:0/3062481837 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d8008e70 con 0x7f93dc0aef20 2023-01-24T03:34:10.840 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.827+0000 7f93e3a3d700 1 -- 172.21.15.112:0/3062481837 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93dc168d60 con 0x7f93dc0aef20 2023-01-24T03:34:10.840 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.828+0000 7f93e3a3d700 1 -- 172.21.15.112:0/3062481837 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93dc04bb80 con 0x7f93dc0aef20 2023-01-24T03:34:10.840 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.828+0000 7f93c97fa700 1 -- 172.21.15.112:0/3062481837 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f93d801c620 con 0x7f93dc0aef20 2023-01-24T03:34:10.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.829+0000 7f93c97fa700 1 --2- 172.21.15.112:0/3062481837 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f93bc0611f0 0x7f93bc0636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:10.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.829+0000 7f93c97fa700 1 -- 172.21.15.112:0/3062481837 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7039+0+0 (secure 0 0 0) 0x7f93d807f340 con 0x7f93dc0aef20 2023-01-24T03:34:10.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.829+0000 7f93e0fd8700 1 --2- 172.21.15.112:0/3062481837 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f93bc0611f0 0x7f93bc0636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:10.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.829+0000 7f93e0fd8700 1 --2- 172.21.15.112:0/3062481837 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f93bc0611f0 0x7f93bc0636a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f93dc068cc0 tx=0x7f93cc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:10.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:10.833+0000 7f93c97fa700 1 -- 172.21.15.112:0/3062481837 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f93d804f0a0 con 0x7f93dc0aef20 2023-01-24T03:34:11.054 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.052+0000 7f93e3a3d700 1 -- 172.21.15.112:0/3062481837 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93dc06efe0 con 0x7f93dc0aef20 2023-01-24T03:34:11.055 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.053+0000 7f93c97fa700 1 -- 172.21.15.112:0/3062481837 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+6112 (secure 0 0 0) 0x7f93d80526c0 con 0x7f93dc0aef20 2023-01-24T03:34:11.055 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:34:11.056 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5041,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/612058861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":612058861},{"type":"v1","addr":"172.21.15.142:6834","nonce":612058861}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:33:18.651832+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4897},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4897":{"gid":4897,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":19,"addr":"172.21.15.142:6835/11022091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":11022091},{"type":"v1","addr":"172.21.15.142:6835","nonce":11022091}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:34:11.057 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.055+0000 7f93e3a3d700 1 -- 172.21.15.112:0/3062481837 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f93bc0611f0 msgr2=0x7f93bc0636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:11.057 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.055+0000 7f93e3a3d700 1 --2- 172.21.15.112:0/3062481837 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f93bc0611f0 0x7f93bc0636a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f93dc068cc0 tx=0x7f93cc008040 comp rx=0 tx=0).stop 2023-01-24T03:34:11.057 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.055+0000 7f93e3a3d700 1 -- 172.21.15.112:0/3062481837 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f93dc0aef20 msgr2=0x7f93dc0cb4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:11.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.055+0000 7f93e3a3d700 1 --2- 172.21.15.112:0/3062481837 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f93dc0aef20 0x7f93dc0cb4a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f93d8003440 tx=0x7f93d8018000 comp rx=0 tx=0).stop 2023-01-24T03:34:11.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.055+0000 7f93e3a3d700 1 -- 172.21.15.112:0/3062481837 shutdown_connections 2023-01-24T03:34:11.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.055+0000 7f93e3a3d700 1 --2- 172.21.15.112:0/3062481837 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f93bc0611f0 0x7f93bc0636a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:11.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.055+0000 7f93e3a3d700 1 --2- 172.21.15.112:0/3062481837 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f93dc0aef20 0x7f93dc0cb4a0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:11.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.055+0000 7f93e3a3d700 1 --2- 172.21.15.112:0/3062481837 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f93dc155da0 0x7f93dc0c39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:11.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.055+0000 7f93e3a3d700 1 --2- 172.21.15.112:0/3062481837 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f93dc0c7e40 0x7f93dc0c34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:11.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.055+0000 7f93e3a3d700 1 -- 172.21.15.112:0/3062481837 >> 172.21.15.112:0/3062481837 conn(0x7f93dc05e2c0 msgr2=0x7f93dc1551d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:11.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.055+0000 7f93e3a3d700 1 -- 172.21.15.112:0/3062481837 shutdown_connections 2023-01-24T03:34:11.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.055+0000 7f93e3a3d700 1 -- 172.21.15.112:0/3062481837 wait complete. 2023-01-24T03:34:11.060 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 21 2023-01-24T03:34:11.069 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=0) 2023-01-24T03:34:11.069 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-01-24T03:34:11.069 INFO:teuthology.orchestra.run:waiting for 300 2023-01-24T03:34:11.082 INFO:tasks.ceph.mds.d:Stopped 2023-01-24T03:34:11.083 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:34:11.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.192+0000 7fafa759e700 1 Processor -- start 2023-01-24T03:34:11.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.192+0000 7fafa759e700 1 -- start start 2023-01-24T03:34:11.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.193+0000 7fafa759e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fafa80c5680 0x7fafa8156730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:11.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.193+0000 7fafa759e700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fafa8155e20 0x7fafa8156c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:11.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.193+0000 7fafa759e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fafa80c5a50 0x7fafa80cb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:11.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.193+0000 7fafa759e700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafa80c4eb0 con 0x7fafa8155e20 2023-01-24T03:34:11.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.193+0000 7fafa759e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafa80c3d10 con 0x7fafa80c5a50 2023-01-24T03:34:11.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.193+0000 7fafa759e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafa80c2e20 con 0x7fafa80c5680 2023-01-24T03:34:11.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.193+0000 7fafa5d9b700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fafa8155e20 0x7fafa8156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:11.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.193+0000 7fafa5d9b700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fafa8155e20 0x7fafa8156c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52552/0 (socket says 172.21.15.112:52552) 2023-01-24T03:34:11.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.193+0000 7fafa5d9b700 1 -- 172.21.15.112:0/238917600 learned_addr learned my addr 172.21.15.112:0/238917600 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:34:11.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.193+0000 7fafa6d9d700 1 --2- 172.21.15.112:0/238917600 >> v2:172.21.15.142:3300/0 conn(0x7fafa80c5a50 0x7fafa80cb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:11.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.193+0000 7fafa659c700 1 --2- 172.21.15.112:0/238917600 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fafa80c5680 0x7fafa8156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:11.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.193+0000 7fafa5d9b700 1 -- 172.21.15.112:0/238917600 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fafa80c5680 msgr2=0x7fafa8156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:11.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.193+0000 7fafa5d9b700 1 --2- 172.21.15.112:0/238917600 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fafa80c5680 0x7fafa8156730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:11.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.193+0000 7fafa5d9b700 1 -- 172.21.15.112:0/238917600 >> v2:172.21.15.142:3300/0 conn(0x7fafa80c5a50 msgr2=0x7fafa80cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:11.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.193+0000 7fafa5d9b700 1 --2- 172.21.15.112:0/238917600 >> v2:172.21.15.142:3300/0 conn(0x7fafa80c5a50 0x7fafa80cb870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:11.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.193+0000 7fafa5d9b700 1 -- 172.21.15.112:0/238917600 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa80c3100 con 0x7fafa8155e20 2023-01-24T03:34:11.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.193+0000 7fafa659c700 1 --2- 172.21.15.112:0/238917600 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fafa80c5680 0x7fafa8156730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-01-24T03:34:11.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.193+0000 7fafa5d9b700 1 --2- 172.21.15.112:0/238917600 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fafa8155e20 0x7fafa8156c70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7faf90006fd0 tx=0x7faf900090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37ff3dd8cc6ea877 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:11.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.194+0000 7fafa559a700 1 -- 172.21.15.112:0/238917600 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf9000b690 con 0x7fafa8155e20 2023-01-24T03:34:11.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.194+0000 7fafa559a700 1 -- 172.21.15.112:0/238917600 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf9000b7f0 con 0x7fafa8155e20 2023-01-24T03:34:11.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.194+0000 7fafa559a700 1 -- 172.21.15.112:0/238917600 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf9000bbb0 con 0x7fafa8155e20 2023-01-24T03:34:11.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.194+0000 7fafa759e700 1 -- 172.21.15.112:0/238917600 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fafa8155e20 msgr2=0x7fafa8156c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:11.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.194+0000 7fafa759e700 1 --2- 172.21.15.112:0/238917600 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fafa8155e20 0x7fafa8156c70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7faf90006fd0 tx=0x7faf900090f0 comp rx=0 tx=0).stop 2023-01-24T03:34:11.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.194+0000 7fafa759e700 1 -- 172.21.15.112:0/238917600 shutdown_connections 2023-01-24T03:34:11.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.194+0000 7fafa759e700 1 --2- 172.21.15.112:0/238917600 >> v2:172.21.15.142:3300/0 conn(0x7fafa80c5a50 0x7fafa80cb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:11.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.194+0000 7fafa759e700 1 --2- 172.21.15.112:0/238917600 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fafa80c5680 0x7fafa8156730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:11.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.194+0000 7fafa759e700 1 --2- 172.21.15.112:0/238917600 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fafa8155e20 0x7fafa8156c70 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:11.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.194+0000 7fafa759e700 1 -- 172.21.15.112:0/238917600 >> 172.21.15.112:0/238917600 conn(0x7fafa80685b0 msgr2=0x7fafa80689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:11.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.194+0000 7fafa759e700 1 -- 172.21.15.112:0/238917600 shutdown_connections 2023-01-24T03:34:11.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.194+0000 7fafa759e700 1 -- 172.21.15.112:0/238917600 wait complete. 2023-01-24T03:34:11.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.195+0000 7fafa759e700 1 Processor -- start 2023-01-24T03:34:11.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.195+0000 7fafa759e700 1 -- start start 2023-01-24T03:34:11.206 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.196+0000 7fafa759e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fafa80c5680 0x7fafa80c1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:11.206 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.196+0000 7fafa759e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafa80c5a50 0x7fafa80c1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:11.207 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.196+0000 7fafa759e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fafa8155e20 0x7fafa80bfd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:11.207 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.196+0000 7fafa759e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa81659d0 con 0x7fafa8155e20 2023-01-24T03:34:11.207 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.196+0000 7fafa759e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa81654e0 con 0x7fafa80c5a50 2023-01-24T03:34:11.207 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.196+0000 7fafa759e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafa80c13e0 con 0x7fafa80c5680 2023-01-24T03:34:11.208 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.196+0000 7fafa6d9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fafa8155e20 0x7fafa80bfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:11.208 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.196+0000 7fafa6d9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fafa8155e20 0x7fafa80bfd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52562/0 (socket says 172.21.15.112:52562) 2023-01-24T03:34:11.208 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.196+0000 7fafa6d9d700 1 -- 172.21.15.112:0/699255404 learned_addr learned my addr 172.21.15.112:0/699255404 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:34:11.208 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.196+0000 7fafa5d9b700 1 --2- 172.21.15.112:0/699255404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafa80c5a50 0x7fafa80c1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:11.208 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.196+0000 7fafa659c700 1 --2- 172.21.15.112:0/699255404 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fafa80c5680 0x7fafa80c1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:11.209 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.196+0000 7fafa6d9d700 1 -- 172.21.15.112:0/699255404 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fafa80c5680 msgr2=0x7fafa80c1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:11.209 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.196+0000 7fafa6d9d700 1 --2- 172.21.15.112:0/699255404 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fafa80c5680 0x7fafa80c1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:11.209 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.196+0000 7fafa6d9d700 1 -- 172.21.15.112:0/699255404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafa80c5a50 msgr2=0x7fafa80c1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:11.209 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.196+0000 7fafa6d9d700 1 --2- 172.21.15.112:0/699255404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafa80c5a50 0x7fafa80c1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:11.209 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.196+0000 7fafa6d9d700 1 -- 172.21.15.112:0/699255404 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf90009370 con 0x7fafa8155e20 2023-01-24T03:34:11.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.197+0000 7fafa6d9d700 1 --2- 172.21.15.112:0/699255404 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fafa8155e20 0x7fafa80bfd50 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fafa000d8c0 tx=0x7fafa0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:11.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.197+0000 7faf9e7fc700 1 -- 172.21.15.112:0/699255404 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa00125f0 con 0x7fafa8155e20 2023-01-24T03:34:11.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.197+0000 7faf9e7fc700 1 -- 172.21.15.112:0/699255404 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa0012750 con 0x7fafa8155e20 2023-01-24T03:34:11.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.197+0000 7faf9e7fc700 1 -- 172.21.15.112:0/699255404 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa0012b10 con 0x7fafa8155e20 2023-01-24T03:34:11.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.197+0000 7fafa759e700 1 -- 172.21.15.112:0/699255404 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafa80c0af0 con 0x7fafa8155e20 2023-01-24T03:34:11.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.197+0000 7fafa759e700 1 -- 172.21.15.112:0/699255404 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafa80c0d90 con 0x7fafa8155e20 2023-01-24T03:34:11.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.198+0000 7fafa759e700 1 -- 172.21.15.112:0/699255404 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafa805ea20 con 0x7fafa8155e20 2023-01-24T03:34:11.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.198+0000 7faf9e7fc700 1 -- 172.21.15.112:0/699255404 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fafa001e460 con 0x7fafa8155e20 2023-01-24T03:34:11.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.198+0000 7faf9e7fc700 1 --2- 172.21.15.112:0/699255404 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7faf8c0611c0 0x7faf8c063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:11.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.198+0000 7faf9e7fc700 1 -- 172.21.15.112:0/699255404 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7039+0+0 (secure 0 0 0) 0x7fafa007dea0 con 0x7fafa8155e20 2023-01-24T03:34:11.212 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.198+0000 7fafa659c700 1 --2- 172.21.15.112:0/699255404 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7faf8c0611c0 0x7faf8c063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:11.212 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.199+0000 7fafa659c700 1 --2- 172.21.15.112:0/699255404 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7faf8c0611c0 0x7faf8c063670 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7faf9800d6e0 tx=0x7faf98007850 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:11.212 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.203+0000 7faf9e7fc700 1 -- 172.21.15.112:0/699255404 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fafa004dba0 con 0x7fafa8155e20 2023-01-24T03:34:11.424 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.422+0000 7fafa759e700 1 -- 172.21.15.112:0/699255404 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafa804b340 con 0x7fafa8155e20 2023-01-24T03:34:11.425 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.423+0000 7faf9e7fc700 1 -- 172.21.15.112:0/699255404 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+6112 (secure 0 0 0) 0x7fafa00511c0 con 0x7fafa8155e20 2023-01-24T03:34:11.425 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:34:11.425 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5041,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/612058861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":612058861},{"type":"v1","addr":"172.21.15.142:6834","nonce":612058861}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:33:18.651832+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4897},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4897":{"gid":4897,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":19,"addr":"172.21.15.142:6835/11022091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":11022091},{"type":"v1","addr":"172.21.15.142:6835","nonce":11022091}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:34:11.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.425+0000 7fafa759e700 1 -- 172.21.15.112:0/699255404 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7faf8c0611c0 msgr2=0x7faf8c063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:11.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.425+0000 7fafa759e700 1 --2- 172.21.15.112:0/699255404 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7faf8c0611c0 0x7faf8c063670 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7faf9800d6e0 tx=0x7faf98007850 comp rx=0 tx=0).stop 2023-01-24T03:34:11.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.425+0000 7fafa759e700 1 -- 172.21.15.112:0/699255404 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fafa8155e20 msgr2=0x7fafa80bfd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:11.428 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.425+0000 7fafa759e700 1 --2- 172.21.15.112:0/699255404 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fafa8155e20 0x7fafa80bfd50 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fafa000d8c0 tx=0x7fafa0008040 comp rx=0 tx=0).stop 2023-01-24T03:34:11.428 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.425+0000 7fafa759e700 1 -- 172.21.15.112:0/699255404 shutdown_connections 2023-01-24T03:34:11.428 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.425+0000 7fafa759e700 1 --2- 172.21.15.112:0/699255404 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7faf8c0611c0 0x7faf8c063670 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:11.428 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.425+0000 7fafa759e700 1 --2- 172.21.15.112:0/699255404 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fafa8155e20 0x7fafa80bfd50 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:11.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.425+0000 7fafa759e700 1 --2- 172.21.15.112:0/699255404 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fafa80c5680 0x7fafa80c1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:11.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.425+0000 7fafa759e700 1 --2- 172.21.15.112:0/699255404 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fafa80c5a50 0x7fafa80c1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:11.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.425+0000 7fafa759e700 1 -- 172.21.15.112:0/699255404 >> 172.21.15.112:0/699255404 conn(0x7fafa80685b0 msgr2=0x7fafa80c9930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:11.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.425+0000 7fafa759e700 1 -- 172.21.15.112:0/699255404 shutdown_connections 2023-01-24T03:34:11.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:11.425+0000 7fafa759e700 1 -- 172.21.15.112:0/699255404 wait complete. 2023-01-24T03:34:11.430 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 21 2023-01-24T03:34:11.438 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-01-24T03:34:13.439 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:34:13.564 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.562+0000 7ffaf95a4700 1 Processor -- start 2023-01-24T03:34:13.565 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.562+0000 7ffaf95a4700 1 -- start start 2023-01-24T03:34:13.570 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.563+0000 7ffaf95a4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffaf40c7e40 0x7ffaf40b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:13.570 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.563+0000 7ffaf95a4700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf4156210 con 0x7ffaf4155da0 2023-01-24T03:34:13.570 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.563+0000 7ffaf95a4700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf40b1170 con 0x7ffaf40ca3e0 2023-01-24T03:34:13.571 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.563+0000 7ffaf95a4700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf40b1450 con 0x7ffaf40c7e40 2023-01-24T03:34:13.571 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.563+0000 7ffaf27fc700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7ffaf4155da0 0x7ffaf40ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:33564/0 (socket says 172.21.15.112:33564) 2023-01-24T03:34:13.571 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.563+0000 7ffaf27fc700 1 -- 172.21.15.112:0/1261067105 learned_addr learned my addr 172.21.15.112:0/1261067105 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:34:13.571 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.563+0000 7ffaf2ffd700 1 --2- 172.21.15.112:0/1261067105 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffaf40c7e40 0x7ffaf40b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:13.571 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.563+0000 7ffaf1ffb700 1 -- 172.21.15.112:0/1261067105 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4121736975 0 0) 0x7ffaf4156210 con 0x7ffaf4155da0 2023-01-24T03:34:13.572 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.563+0000 7ffaf1ffb700 1 -- 172.21.15.112:0/1261067105 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad40033f0 con 0x7ffaf4155da0 2023-01-24T03:34:13.572 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.563+0000 7ffaf1ffb700 1 -- 172.21.15.112:0/1261067105 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2359344588 0 0) 0x7ffad40033f0 con 0x7ffaf4155da0 2023-01-24T03:34:13.572 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.563+0000 7ffaf1ffb700 1 -- 172.21.15.112:0/1261067105 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffaf40c7e40 msgr2=0x7ffaf40b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:13.572 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.563+0000 7ffaf1ffb700 1 --2- 172.21.15.112:0/1261067105 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffaf40c7e40 0x7ffaf40b0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:13.572 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.563+0000 7ffaf1ffb700 1 -- 172.21.15.112:0/1261067105 >> v1:172.21.15.142:6789/0 conn(0x7ffaf40ca3e0 legacy=0x7ffaf40ca820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:13.573 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.563+0000 7ffaf1ffb700 1 -- 172.21.15.112:0/1261067105 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf40b1730 con 0x7ffaf4155da0 2023-01-24T03:34:13.573 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.563+0000 7ffaf1ffb700 1 -- 172.21.15.112:0/1261067105 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7ffadc001470 con 0x7ffaf4155da0 2023-01-24T03:34:13.573 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.563+0000 7ffaf1ffb700 1 -- 172.21.15.112:0/1261067105 >> v1:172.21.15.112:6789/0 conn(0x7ffaf4155da0 legacy=0x7ffaf40ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:13.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.563+0000 7ffaf1ffb700 1 --2- 172.21.15.112:0/1261067105 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffad4003910 0x7ffad4003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:13.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.563+0000 7ffaf1ffb700 1 --2- 172.21.15.112:0/1261067105 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffaf40c7e40 0x7ffad4004690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:13.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.563+0000 7ffaf1ffb700 1 --2- 172.21.15.112:0/1261067105 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffad4004bd0 0x7ffad4005040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:13.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.563+0000 7ffaf1ffb700 1 -- 172.21.15.112:0/1261067105 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf40b1450 con 0x7ffad4004bd0 2023-01-24T03:34:13.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.564+0000 7ffaf1ffb700 1 -- 172.21.15.112:0/1261067105 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad400b990 con 0x7ffad4003910 2023-01-24T03:34:13.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.564+0000 7ffaf1ffb700 1 -- 172.21.15.112:0/1261067105 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffad400bb50 con 0x7ffaf40c7e40 2023-01-24T03:34:13.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.564+0000 7ffaf37fe700 1 --2- 172.21.15.112:0/1261067105 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffad4004bd0 0x7ffad4005040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:13.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.564+0000 7ffaf2ffd700 1 --2- 172.21.15.112:0/1261067105 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffad4003910 0x7ffad4003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:13.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.564+0000 7ffaf27fc700 1 --2- 172.21.15.112:0/1261067105 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffaf40c7e40 0x7ffad4004690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:13.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.564+0000 7ffaf37fe700 1 -- 172.21.15.112:0/1261067105 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffaf40c7e40 msgr2=0x7ffad4004690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:13.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.564+0000 7ffaf37fe700 1 --2- 172.21.15.112:0/1261067105 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffaf40c7e40 0x7ffad4004690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:13.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.564+0000 7ffaf37fe700 1 -- 172.21.15.112:0/1261067105 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffad4003910 msgr2=0x7ffad4003ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:13.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.564+0000 7ffaf37fe700 1 --2- 172.21.15.112:0/1261067105 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffad4003910 0x7ffad4003ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:13.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.564+0000 7ffaf37fe700 1 -- 172.21.15.112:0/1261067105 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad40033f0 con 0x7ffad4004bd0 2023-01-24T03:34:13.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.564+0000 7ffaf37fe700 1 --2- 172.21.15.112:0/1261067105 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffad4004bd0 0x7ffad4005040 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7ffaec001820 tx=0x7ffaec007b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da9ac8cd6582a4a0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:13.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.564+0000 7ffaf1ffb700 1 -- 172.21.15.112:0/1261067105 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaec007ea0 con 0x7ffad4004bd0 2023-01-24T03:34:13.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.564+0000 7ffaf1ffb700 1 -- 172.21.15.112:0/1261067105 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaec012420 con 0x7ffad4004bd0 2023-01-24T03:34:13.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.564+0000 7ffaf1ffb700 1 -- 172.21.15.112:0/1261067105 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaec0127c0 con 0x7ffad4004bd0 2023-01-24T03:34:13.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.565+0000 7ffaf95a4700 1 -- 172.21.15.112:0/1261067105 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffad4004bd0 msgr2=0x7ffad4005040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:13.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.565+0000 7ffaf95a4700 1 --2- 172.21.15.112:0/1261067105 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffad4004bd0 0x7ffad4005040 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7ffaec001820 tx=0x7ffaec007b90 comp rx=0 tx=0).stop 2023-01-24T03:34:13.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.565+0000 7ffaf37fe700 1 -- 172.21.15.112:0/1261067105 reap_dead start 2023-01-24T03:34:13.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.565+0000 7ffaf95a4700 1 -- 172.21.15.112:0/1261067105 shutdown_connections 2023-01-24T03:34:13.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.565+0000 7ffaf95a4700 1 -- 172.21.15.112:0/1261067105 >> 172.21.15.112:0/1261067105 conn(0x7ffaf405e2c0 msgr2=0x7ffaf40cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:13.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.565+0000 7ffaf95a4700 1 -- 172.21.15.112:0/1261067105 shutdown_connections 2023-01-24T03:34:13.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.565+0000 7ffaf95a4700 1 -- 172.21.15.112:0/1261067105 wait complete. 2023-01-24T03:34:13.579 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.566+0000 7ffaf95a4700 1 Processor -- start 2023-01-24T03:34:13.579 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.566+0000 7ffaf95a4700 1 -- start start 2023-01-24T03:34:13.579 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.567+0000 7ffaf95a4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffaf41652d0 0x7ffaf41656a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:13.579 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.567+0000 7ffaf95a4700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffaf40c1af0 0x7ffaf4150fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:13.579 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.567+0000 7ffaf95a4700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffaf414eda0 0x7ffaf4151520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:13.580 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.567+0000 7ffaf95a4700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf40c2260 con 0x7ffaf40c1af0 2023-01-24T03:34:13.580 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.567+0000 7ffaf95a4700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf4150600 con 0x7ffaf414eda0 2023-01-24T03:34:13.580 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.567+0000 7ffaf95a4700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf4150b50 con 0x7ffaf41652d0 2023-01-24T03:34:13.580 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.567+0000 7ffaf27fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffaf40c1af0 0x7ffaf4150fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:13.580 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.567+0000 7ffaf27fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffaf40c1af0 0x7ffaf4150fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52590/0 (socket says 172.21.15.112:52590) 2023-01-24T03:34:13.581 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.567+0000 7ffaf27fc700 1 -- 172.21.15.112:0/1871369335 learned_addr learned my addr 172.21.15.112:0/1871369335 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:34:13.581 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.567+0000 7ffaf37fe700 1 --2- 172.21.15.112:0/1871369335 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffaf414eda0 0x7ffaf4151520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:13.581 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.567+0000 7ffaf2ffd700 1 --2- 172.21.15.112:0/1871369335 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffaf41652d0 0x7ffaf41656a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:13.581 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.567+0000 7ffaf27fc700 1 -- 172.21.15.112:0/1871369335 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffaf41652d0 msgr2=0x7ffaf41656a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:13.581 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.567+0000 7ffaf27fc700 1 --2- 172.21.15.112:0/1871369335 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffaf41652d0 0x7ffaf41656a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:13.582 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.567+0000 7ffaf27fc700 1 -- 172.21.15.112:0/1871369335 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffaf414eda0 msgr2=0x7ffaf4151520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:13.582 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.567+0000 7ffaf27fc700 1 --2- 172.21.15.112:0/1871369335 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffaf414eda0 0x7ffaf4151520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:13.582 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.567+0000 7ffaf27fc700 1 -- 172.21.15.112:0/1871369335 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf40c1f80 con 0x7ffaf40c1af0 2023-01-24T03:34:13.582 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.567+0000 7ffaf27fc700 1 --2- 172.21.15.112:0/1871369335 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffaf40c1af0 0x7ffaf4150fe0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7ffadc004aa0 tx=0x7ffadc001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:13.583 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.567+0000 7ffaeaffd700 1 -- 172.21.15.112:0/1871369335 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffadc001430 con 0x7ffaf40c1af0 2023-01-24T03:34:13.583 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.567+0000 7ffaeaffd700 1 -- 172.21.15.112:0/1871369335 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffadc00f660 con 0x7ffaf40c1af0 2023-01-24T03:34:13.583 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.567+0000 7ffaf95a4700 1 -- 172.21.15.112:0/1871369335 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaf4165f10 con 0x7ffaf40c1af0 2023-01-24T03:34:13.583 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.567+0000 7ffaeaffd700 1 -- 172.21.15.112:0/1871369335 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffadc00fa00 con 0x7ffaf40c1af0 2023-01-24T03:34:13.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.567+0000 7ffaf95a4700 1 -- 172.21.15.112:0/1871369335 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaf4165be0 con 0x7ffaf40c1af0 2023-01-24T03:34:13.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.568+0000 7ffaf95a4700 1 -- 172.21.15.112:0/1871369335 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaf4166150 con 0x7ffaf40c1af0 2023-01-24T03:34:13.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.569+0000 7ffaeaffd700 1 -- 172.21.15.112:0/1871369335 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7ffadc00fdb0 con 0x7ffaf40c1af0 2023-01-24T03:34:13.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.569+0000 7ffaeaffd700 1 --2- 172.21.15.112:0/1871369335 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ffad80611f0 0x7ffad80636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:13.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.569+0000 7ffaeaffd700 1 -- 172.21.15.112:0/1871369335 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7039+0+0 (secure 0 0 0) 0x7ffadc07d940 con 0x7ffaf40c1af0 2023-01-24T03:34:13.586 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.569+0000 7ffaf2ffd700 1 --2- 172.21.15.112:0/1871369335 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ffad80611f0 0x7ffad80636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:13.586 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.569+0000 7ffaf2ffd700 1 --2- 172.21.15.112:0/1871369335 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ffad80611f0 0x7ffad80636a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ffae40094d0 tx=0x7ffae400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:13.586 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.573+0000 7ffaeaffd700 1 -- 172.21.15.112:0/1871369335 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7ffaf4166150 con 0x7ffaf40c1af0 2023-01-24T03:34:13.796 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.794+0000 7ffaf95a4700 1 -- 172.21.15.112:0/1871369335 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffaf404bb80 con 0x7ffaf40c1af0 2023-01-24T03:34:13.797 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.795+0000 7ffaeaffd700 1 -- 172.21.15.112:0/1871369335 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+6112 (secure 0 0 0) 0x7ffadc04d6a0 con 0x7ffaf40c1af0 2023-01-24T03:34:13.797 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:34:13.798 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5041,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/612058861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":612058861},{"type":"v1","addr":"172.21.15.142:6834","nonce":612058861}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:33:18.651832+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4897},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4897":{"gid":4897,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":19,"addr":"172.21.15.142:6835/11022091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":11022091},{"type":"v1","addr":"172.21.15.142:6835","nonce":11022091}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:34:13.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.797+0000 7ffaf95a4700 1 -- 172.21.15.112:0/1871369335 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ffad80611f0 msgr2=0x7ffad80636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:13.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.797+0000 7ffaf95a4700 1 --2- 172.21.15.112:0/1871369335 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ffad80611f0 0x7ffad80636a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ffae40094d0 tx=0x7ffae400b040 comp rx=0 tx=0).stop 2023-01-24T03:34:13.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.797+0000 7ffaf95a4700 1 -- 172.21.15.112:0/1871369335 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffaf40c1af0 msgr2=0x7ffaf4150fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:13.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.797+0000 7ffaf95a4700 1 --2- 172.21.15.112:0/1871369335 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffaf40c1af0 0x7ffaf4150fe0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7ffadc004aa0 tx=0x7ffadc001e70 comp rx=0 tx=0).stop 2023-01-24T03:34:13.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.797+0000 7ffaf95a4700 1 -- 172.21.15.112:0/1871369335 shutdown_connections 2023-01-24T03:34:13.801 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.797+0000 7ffaf95a4700 1 --2- 172.21.15.112:0/1871369335 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ffad80611f0 0x7ffad80636a0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:13.801 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.797+0000 7ffaf95a4700 1 --2- 172.21.15.112:0/1871369335 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffaf414eda0 0x7ffaf4151520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:13.801 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.797+0000 7ffaf95a4700 1 --2- 172.21.15.112:0/1871369335 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ffaf41652d0 0x7ffaf41656a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:13.801 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.797+0000 7ffaf95a4700 1 --2- 172.21.15.112:0/1871369335 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffaf40c1af0 0x7ffaf4150fe0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:13.801 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.797+0000 7ffaf95a4700 1 -- 172.21.15.112:0/1871369335 >> 172.21.15.112:0/1871369335 conn(0x7ffaf405e2c0 msgr2=0x7ffaf40ae9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:13.802 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.797+0000 7ffaf95a4700 1 -- 172.21.15.112:0/1871369335 shutdown_connections 2023-01-24T03:34:13.802 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:13.797+0000 7ffaf95a4700 1 -- 172.21.15.112:0/1871369335 wait complete. 2023-01-24T03:34:13.802 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 21 2023-01-24T03:34:13.811 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-01-24T03:34:15.812 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:34:15.935 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.933+0000 7faef0fbf700 1 Processor -- start 2023-01-24T03:34:15.936 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.933+0000 7faef0fbf700 1 -- start start 2023-01-24T03:34:15.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.934+0000 7faef0fbf700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faeec155db0 0x7faeec14efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:15.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.934+0000 7faef0fbf700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeec0c54d0 con 0x7faeec06d320 2023-01-24T03:34:15.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.934+0000 7faef0fbf700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeec0c6070 con 0x7faeec06cf50 2023-01-24T03:34:15.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.934+0000 7faef0fbf700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeec0c49c0 con 0x7faeec155db0 2023-01-24T03:34:15.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.934+0000 7faee9d9b700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7faeec06d320 0x7faeec067f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:33588/0 (socket says 172.21.15.112:33588) 2023-01-24T03:34:15.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.934+0000 7faee9d9b700 1 -- 172.21.15.112:0/83385280 learned_addr learned my addr 172.21.15.112:0/83385280 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:34:15.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.935+0000 7faeead9d700 1 --2- 172.21.15.112:0/83385280 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faeec155db0 0x7faeec14efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:15.944 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.935+0000 7faee959a700 1 -- 172.21.15.112:0/83385280 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 543258815 0 0) 0x7faeec0c54d0 con 0x7faeec06d320 2023-01-24T03:34:15.944 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.935+0000 7faee959a700 1 -- 172.21.15.112:0/83385280 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faecc0033f0 con 0x7faeec06d320 2023-01-24T03:34:15.944 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.935+0000 7faee959a700 1 -- 172.21.15.112:0/83385280 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3770384050 0 0) 0x7faeec0c6070 con 0x7faeec06cf50 2023-01-24T03:34:15.944 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.935+0000 7faee959a700 1 -- 172.21.15.112:0/83385280 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faeec0c54d0 con 0x7faeec06cf50 2023-01-24T03:34:15.945 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.935+0000 7faee959a700 1 -- 172.21.15.112:0/83385280 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1592276841 0 0) 0x7faecc0033f0 con 0x7faeec06d320 2023-01-24T03:34:15.945 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.935+0000 7faee959a700 1 -- 172.21.15.112:0/83385280 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faeec155db0 msgr2=0x7faeec14efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:15.945 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.935+0000 7faee959a700 1 --2- 172.21.15.112:0/83385280 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faeec155db0 0x7faeec14efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:15.945 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.935+0000 7faee959a700 1 -- 172.21.15.112:0/83385280 >> v1:172.21.15.142:6789/0 conn(0x7faeec06cf50 legacy=0x7faeec0c5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:15.945 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.935+0000 7faee959a700 1 -- 172.21.15.112:0/83385280 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeec068920 con 0x7faeec06d320 2023-01-24T03:34:15.946 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.935+0000 7faee959a700 1 -- 172.21.15.112:0/83385280 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7faed4001470 con 0x7faeec06d320 2023-01-24T03:34:15.946 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.935+0000 7faee959a700 1 -- 172.21.15.112:0/83385280 >> v1:172.21.15.112:6789/0 conn(0x7faeec06d320 legacy=0x7faeec067f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:15.946 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.935+0000 7faee959a700 1 --2- 172.21.15.112:0/83385280 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faecc003aa0 0x7faecc003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:15.946 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.935+0000 7faee959a700 1 --2- 172.21.15.112:0/83385280 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faecc004690 0x7faecc004b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:15.946 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.935+0000 7faee959a700 1 --2- 172.21.15.112:0/83385280 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faecc005040 0x7faecc00b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:15.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.935+0000 7faee959a700 1 -- 172.21.15.112:0/83385280 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeec0c49c0 con 0x7faecc004690 2023-01-24T03:34:15.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.935+0000 7faee959a700 1 -- 172.21.15.112:0/83385280 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faecc00bf30 con 0x7faecc005040 2023-01-24T03:34:15.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.935+0000 7faee959a700 1 -- 172.21.15.112:0/83385280 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faecc00c0f0 con 0x7faecc003aa0 2023-01-24T03:34:15.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.935+0000 7faeead9d700 1 --2- 172.21.15.112:0/83385280 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faeec155db0 0x7faeec14efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:34:15.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.935+0000 7faee9d9b700 1 --2- 172.21.15.112:0/83385280 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faecc004690 0x7faecc004b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:15.948 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.935+0000 7faeea59c700 1 --2- 172.21.15.112:0/83385280 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faecc003aa0 0x7faecc003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:15.948 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.935+0000 7faeead9d700 1 --2- 172.21.15.112:0/83385280 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faecc005040 0x7faecc00b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:15.948 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.935+0000 7faee9d9b700 1 -- 172.21.15.112:0/83385280 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faecc003aa0 msgr2=0x7faecc003e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:15.948 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.935+0000 7faee9d9b700 1 --2- 172.21.15.112:0/83385280 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faecc003aa0 0x7faecc003e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:15.948 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.935+0000 7faee9d9b700 1 -- 172.21.15.112:0/83385280 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faecc005040 msgr2=0x7faecc00b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:15.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.935+0000 7faee9d9b700 1 --2- 172.21.15.112:0/83385280 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faecc005040 0x7faecc00b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:15.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.935+0000 7faee9d9b700 1 -- 172.21.15.112:0/83385280 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faecc0033f0 con 0x7faecc004690 2023-01-24T03:34:15.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.935+0000 7faeea59c700 1 --2- 172.21.15.112:0/83385280 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faecc003aa0 0x7faecc003e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:34:15.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.936+0000 7faee9d9b700 1 --2- 172.21.15.112:0/83385280 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faecc004690 0x7faecc004b00 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7faed400c190 tx=0x7faed4001e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d356ad0df19f9951 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:15.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.936+0000 7faee959a700 1 -- 172.21.15.112:0/83385280 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed400ca00 con 0x7faecc004690 2023-01-24T03:34:15.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.936+0000 7faee959a700 1 -- 172.21.15.112:0/83385280 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed400cb80 con 0x7faecc004690 2023-01-24T03:34:15.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.936+0000 7faee959a700 1 -- 172.21.15.112:0/83385280 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed40070d0 con 0x7faecc004690 2023-01-24T03:34:15.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.936+0000 7faef0fbf700 1 -- 172.21.15.112:0/83385280 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faecc004690 msgr2=0x7faecc004b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:15.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.936+0000 7faef0fbf700 1 --2- 172.21.15.112:0/83385280 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faecc004690 0x7faecc004b00 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7faed400c190 tx=0x7faed4001e10 comp rx=0 tx=0).stop 2023-01-24T03:34:15.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.936+0000 7faeead9d700 1 -- 172.21.15.112:0/83385280 reap_dead start 2023-01-24T03:34:15.951 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.936+0000 7faef0fbf700 1 -- 172.21.15.112:0/83385280 shutdown_connections 2023-01-24T03:34:15.951 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.936+0000 7faef0fbf700 1 -- 172.21.15.112:0/83385280 >> 172.21.15.112:0/83385280 conn(0x7faeec0c9820 msgr2=0x7faeec0c9c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:15.951 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.936+0000 7faef0fbf700 1 -- 172.21.15.112:0/83385280 shutdown_connections 2023-01-24T03:34:15.951 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.936+0000 7faef0fbf700 1 -- 172.21.15.112:0/83385280 wait complete. 2023-01-24T03:34:15.951 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.937+0000 7faef0fbf700 1 Processor -- start 2023-01-24T03:34:15.952 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.937+0000 7faef0fbf700 1 -- start start 2023-01-24T03:34:15.952 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.938+0000 7faef0fbf700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faecc004690 0x7faeec156540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:15.952 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.938+0000 7faef0fbf700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faeec153bb0 0x7faeec158f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:15.952 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.938+0000 7faef0fbf700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faeec14ab30 0x7faeec14afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:15.952 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.938+0000 7faef0fbf700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeec153720 con 0x7faeec153bb0 2023-01-24T03:34:15.953 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.938+0000 7faef0fbf700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeec0c0b90 con 0x7faeec14ab30 2023-01-24T03:34:15.953 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.938+0000 7faef0fbf700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeec0c0e30 con 0x7faecc004690 2023-01-24T03:34:15.953 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.938+0000 7faee9d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faeec153bb0 0x7faeec158f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:15.953 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.938+0000 7faee9d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faeec153bb0 0x7faeec158f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52612/0 (socket says 172.21.15.112:52612) 2023-01-24T03:34:15.953 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.938+0000 7faee9d9b700 1 -- 172.21.15.112:0/1012613860 learned_addr learned my addr 172.21.15.112:0/1012613860 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:34:15.954 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.938+0000 7faeea59c700 1 --2- 172.21.15.112:0/1012613860 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faecc004690 0x7faeec156540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:15.954 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.938+0000 7faeead9d700 1 --2- 172.21.15.112:0/1012613860 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faeec14ab30 0x7faeec14afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:15.954 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.938+0000 7faee9d9b700 1 -- 172.21.15.112:0/1012613860 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faecc004690 msgr2=0x7faeec156540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:15.954 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.938+0000 7faee9d9b700 1 --2- 172.21.15.112:0/1012613860 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faecc004690 0x7faeec156540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:15.954 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.938+0000 7faee9d9b700 1 -- 172.21.15.112:0/1012613860 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faeec14ab30 msgr2=0x7faeec14afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:15.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.938+0000 7faee9d9b700 1 --2- 172.21.15.112:0/1012613860 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faeec14ab30 0x7faeec14afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:15.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.938+0000 7faee9d9b700 1 -- 172.21.15.112:0/1012613860 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeec1578b0 con 0x7faeec153bb0 2023-01-24T03:34:15.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.938+0000 7faee9d9b700 1 --2- 172.21.15.112:0/1012613860 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faeec153bb0 0x7faeec158f20 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7faecc00bf00 tx=0x7faed4015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:15.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.939+0000 7faee27fc700 1 -- 172.21.15.112:0/1012613860 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed4001430 con 0x7faeec153bb0 2023-01-24T03:34:15.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.939+0000 7faee27fc700 1 -- 172.21.15.112:0/1012613860 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed40015b0 con 0x7faeec153bb0 2023-01-24T03:34:15.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.939+0000 7faee27fc700 1 -- 172.21.15.112:0/1012613860 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed4004900 con 0x7faeec153bb0 2023-01-24T03:34:15.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.939+0000 7faef0fbf700 1 -- 172.21.15.112:0/1012613860 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeec157b90 con 0x7faeec153bb0 2023-01-24T03:34:15.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.939+0000 7faef0fbf700 1 -- 172.21.15.112:0/1012613860 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeec156a80 con 0x7faeec153bb0 2023-01-24T03:34:15.957 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.940+0000 7faef0fbf700 1 -- 172.21.15.112:0/1012613860 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeec153570 con 0x7faeec153bb0 2023-01-24T03:34:15.957 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.940+0000 7faee27fc700 1 -- 172.21.15.112:0/1012613860 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7faed4010070 con 0x7faeec153bb0 2023-01-24T03:34:15.958 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.940+0000 7faee27fc700 1 --2- 172.21.15.112:0/1012613860 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7faed00611f0 0x7faed00636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:15.958 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.940+0000 7faee27fc700 1 -- 172.21.15.112:0/1012613860 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7039+0+0 (secure 0 0 0) 0x7faed407ecf0 con 0x7faeec153bb0 2023-01-24T03:34:15.958 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.940+0000 7faeea59c700 1 --2- 172.21.15.112:0/1012613860 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7faed00611f0 0x7faed00636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:15.958 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.941+0000 7faeea59c700 1 --2- 172.21.15.112:0/1012613860 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7faed00611f0 0x7faed00636a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7faedc007da0 tx=0x7faedc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:15.958 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:15.945+0000 7faee27fc700 1 -- 172.21.15.112:0/1012613860 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7faed404ea50 con 0x7faeec153bb0 2023-01-24T03:34:16.169 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:16.167+0000 7faef0fbf700 1 -- 172.21.15.112:0/1012613860 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faeec153210 con 0x7faeec153bb0 2023-01-24T03:34:16.170 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:16.168+0000 7faee27fc700 1 -- 172.21.15.112:0/1012613860 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+6112 (secure 0 0 0) 0x7faeec153210 con 0x7faeec153bb0 2023-01-24T03:34:16.170 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:34:16.170 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5041,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/612058861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":612058861},{"type":"v1","addr":"172.21.15.142:6834","nonce":612058861}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:33:18.651832+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4897},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4897":{"gid":4897,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":19,"addr":"172.21.15.142:6835/11022091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":11022091},{"type":"v1","addr":"172.21.15.142:6835","nonce":11022091}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:34:16.172 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:16.170+0000 7faef0fbf700 1 -- 172.21.15.112:0/1012613860 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7faed00611f0 msgr2=0x7faed00636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:16.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:16.170+0000 7faef0fbf700 1 --2- 172.21.15.112:0/1012613860 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7faed00611f0 0x7faed00636a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7faedc007da0 tx=0x7faedc009040 comp rx=0 tx=0).stop 2023-01-24T03:34:16.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:16.170+0000 7faef0fbf700 1 -- 172.21.15.112:0/1012613860 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faeec153bb0 msgr2=0x7faeec158f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:16.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:16.170+0000 7faef0fbf700 1 --2- 172.21.15.112:0/1012613860 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faeec153bb0 0x7faeec158f20 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7faecc00bf00 tx=0x7faed4015000 comp rx=0 tx=0).stop 2023-01-24T03:34:16.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:16.170+0000 7faef0fbf700 1 -- 172.21.15.112:0/1012613860 shutdown_connections 2023-01-24T03:34:16.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:16.170+0000 7faef0fbf700 1 --2- 172.21.15.112:0/1012613860 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7faed00611f0 0x7faed00636a0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:16.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:16.170+0000 7faef0fbf700 1 --2- 172.21.15.112:0/1012613860 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7faeec14ab30 0x7faeec14afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:16.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:16.170+0000 7faef0fbf700 1 --2- 172.21.15.112:0/1012613860 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7faecc004690 0x7faeec156540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:16.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:16.170+0000 7faef0fbf700 1 --2- 172.21.15.112:0/1012613860 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faeec153bb0 0x7faeec158f20 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:16.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:16.170+0000 7faef0fbf700 1 -- 172.21.15.112:0/1012613860 >> 172.21.15.112:0/1012613860 conn(0x7faeec0c9820 msgr2=0x7faeec0c6520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:16.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:16.170+0000 7faef0fbf700 1 -- 172.21.15.112:0/1012613860 shutdown_connections 2023-01-24T03:34:16.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:16.170+0000 7faef0fbf700 1 -- 172.21.15.112:0/1012613860 wait complete. 2023-01-24T03:34:16.175 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 21 2023-01-24T03:34:16.184 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-01-24T03:34:18.185 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:34:18.309 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.307+0000 7fe1e3fff700 1 Processor -- start 2023-01-24T03:34:18.310 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.307+0000 7fe1e3fff700 1 -- start start 2023-01-24T03:34:18.313 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.308+0000 7fe1e3fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fe1e40c5680 0x7fe1e4156730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:18.313 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.308+0000 7fe1e3fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe1e4155e20 0x7fe1e4156c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:18.314 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.308+0000 7fe1e3fff700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1e40c3d10 con 0x7fe1e40c5a50 2023-01-24T03:34:18.314 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.308+0000 7fe1e3fff700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1e40c4eb0 con 0x7fe1e40c5680 2023-01-24T03:34:18.314 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.308+0000 7fe1e3fff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e40c2e20 con 0x7fe1e4155e20 2023-01-24T03:34:18.314 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.308+0000 7fe1e37fe700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fe1e40c5a50 0x7fe1e40cb870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:33616/0 (socket says 172.21.15.112:33616) 2023-01-24T03:34:18.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.308+0000 7fe1e37fe700 1 -- 172.21.15.112:0/2125537796 learned_addr learned my addr 172.21.15.112:0/2125537796 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:34:18.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.308+0000 7fe1e1ffb700 1 -- 172.21.15.112:0/2125537796 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318397746 0 0) 0x7fe1e40c3d10 con 0x7fe1e40c5a50 2023-01-24T03:34:18.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.308+0000 7fe1e27fc700 1 --2- 172.21.15.112:0/2125537796 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe1e4155e20 0x7fe1e4156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:18.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.308+0000 7fe1e2ffd700 1 --2- 172.21.15.112:0/2125537796 >> v2:172.21.15.142:3300/0 conn(0x7fe1e40c5680 0x7fe1e4156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:18.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.308+0000 7fe1e1ffb700 1 -- 172.21.15.112:0/2125537796 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1c40033f0 con 0x7fe1e40c5a50 2023-01-24T03:34:18.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.309+0000 7fe1e1ffb700 1 -- 172.21.15.112:0/2125537796 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1813999323 0 0) 0x7fe1c40033f0 con 0x7fe1e40c5a50 2023-01-24T03:34:18.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.309+0000 7fe1e1ffb700 1 -- 172.21.15.112:0/2125537796 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe1e4155e20 msgr2=0x7fe1e4156c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:18.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.309+0000 7fe1e1ffb700 1 --2- 172.21.15.112:0/2125537796 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe1e4155e20 0x7fe1e4156c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:18.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.309+0000 7fe1e1ffb700 1 -- 172.21.15.112:0/2125537796 >> v2:172.21.15.142:3300/0 conn(0x7fe1e40c5680 msgr2=0x7fe1e4156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:18.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.309+0000 7fe1e1ffb700 1 --2- 172.21.15.112:0/2125537796 >> v2:172.21.15.142:3300/0 conn(0x7fe1e40c5680 0x7fe1e4156730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:18.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.309+0000 7fe1e1ffb700 1 -- 172.21.15.112:0/2125537796 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e40c3100 con 0x7fe1e40c5a50 2023-01-24T03:34:18.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.309+0000 7fe1e1ffb700 1 -- 172.21.15.112:0/2125537796 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7fe1dc001470 con 0x7fe1e40c5a50 2023-01-24T03:34:18.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.309+0000 7fe1e1ffb700 1 -- 172.21.15.112:0/2125537796 >> v1:172.21.15.112:6789/0 conn(0x7fe1e40c5a50 legacy=0x7fe1e40cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:18.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.309+0000 7fe1e1ffb700 1 --2- 172.21.15.112:0/2125537796 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe1e4155e20 0x7fe1c40038e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:18.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.309+0000 7fe1e1ffb700 1 --2- 172.21.15.112:0/2125537796 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe1c4003e60 0x7fe1c4004690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:18.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.309+0000 7fe1e1ffb700 1 --2- 172.21.15.112:0/2125537796 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe1c4004bd0 0x7fe1c4005040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:18.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.309+0000 7fe1e1ffb700 1 -- 172.21.15.112:0/2125537796 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e40c4eb0 con 0x7fe1c4003e60 2023-01-24T03:34:18.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.309+0000 7fe1e1ffb700 1 -- 172.21.15.112:0/2125537796 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e40c2e20 con 0x7fe1c4004bd0 2023-01-24T03:34:18.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.309+0000 7fe1e1ffb700 1 -- 172.21.15.112:0/2125537796 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c400b910 con 0x7fe1e4155e20 2023-01-24T03:34:18.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.309+0000 7fe1e27fc700 1 --2- 172.21.15.112:0/2125537796 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe1c4003e60 0x7fe1c4004690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:18.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.309+0000 7fe1e2ffd700 1 --2- 172.21.15.112:0/2125537796 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe1e4155e20 0x7fe1c40038e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:18.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.309+0000 7fe1e37fe700 1 --2- 172.21.15.112:0/2125537796 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe1c4004bd0 0x7fe1c4005040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:18.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.309+0000 7fe1e27fc700 1 -- 172.21.15.112:0/2125537796 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe1e4155e20 msgr2=0x7fe1c40038e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:18.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.309+0000 7fe1e27fc700 1 --2- 172.21.15.112:0/2125537796 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe1e4155e20 0x7fe1c40038e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:18.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.309+0000 7fe1e27fc700 1 -- 172.21.15.112:0/2125537796 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe1c4004bd0 msgr2=0x7fe1c4005040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:18.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.309+0000 7fe1e27fc700 1 --2- 172.21.15.112:0/2125537796 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe1c4004bd0 0x7fe1c4005040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:18.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.309+0000 7fe1e27fc700 1 -- 172.21.15.112:0/2125537796 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c40033f0 con 0x7fe1c4003e60 2023-01-24T03:34:18.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.309+0000 7fe1e27fc700 1 --2- 172.21.15.112:0/2125537796 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe1c4003e60 0x7fe1c4004690 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc009a80 tx=0x7fe1cc009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db7d8483ad1b4bca server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:18.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.309+0000 7fe1e1ffb700 1 -- 172.21.15.112:0/2125537796 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc009430 con 0x7fe1c4003e60 2023-01-24T03:34:18.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.309+0000 7fe1e1ffb700 1 -- 172.21.15.112:0/2125537796 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1cc00e5e0 con 0x7fe1c4003e60 2023-01-24T03:34:18.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.309+0000 7fe1e1ffb700 1 -- 172.21.15.112:0/2125537796 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc00e9a0 con 0x7fe1c4003e60 2023-01-24T03:34:18.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.310+0000 7fe1e3fff700 1 -- 172.21.15.112:0/2125537796 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe1c4003e60 msgr2=0x7fe1c4004690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:18.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.310+0000 7fe1e3fff700 1 --2- 172.21.15.112:0/2125537796 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe1c4003e60 0x7fe1c4004690 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc009a80 tx=0x7fe1cc009060 comp rx=0 tx=0).stop 2023-01-24T03:34:18.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.310+0000 7fe1e37fe700 1 -- 172.21.15.112:0/2125537796 reap_dead start 2023-01-24T03:34:18.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.310+0000 7fe1e3fff700 1 -- 172.21.15.112:0/2125537796 shutdown_connections 2023-01-24T03:34:18.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.310+0000 7fe1e3fff700 1 -- 172.21.15.112:0/2125537796 >> 172.21.15.112:0/2125537796 conn(0x7fe1e40685b0 msgr2=0x7fe1e40689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:18.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.310+0000 7fe1e3fff700 1 -- 172.21.15.112:0/2125537796 shutdown_connections 2023-01-24T03:34:18.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.310+0000 7fe1e3fff700 1 -- 172.21.15.112:0/2125537796 wait complete. 2023-01-24T03:34:18.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.311+0000 7fe1e3fff700 1 Processor -- start 2023-01-24T03:34:18.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.311+0000 7fe1e3fff700 1 -- start start 2023-01-24T03:34:18.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.312+0000 7fe1e3fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe1c4003e60 0x7fe1e4165d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:18.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.312+0000 7fe1e3fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe1e40c27e0 0x7fe1e40cac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:18.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.312+0000 7fe1e3fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe1e40cb190 0x7fe1e40c0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:18.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.312+0000 7fe1e3fff700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e4168840 con 0x7fe1c4003e60 2023-01-24T03:34:18.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.312+0000 7fe1e3fff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e4168b20 con 0x7fe1e40cb190 2023-01-24T03:34:18.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.312+0000 7fe1e3fff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e4168e00 con 0x7fe1e40c27e0 2023-01-24T03:34:18.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.312+0000 7fe1e2ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe1c4003e60 0x7fe1e4165d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:18.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.312+0000 7fe1e2ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe1c4003e60 0x7fe1e4165d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52638/0 (socket says 172.21.15.112:52638) 2023-01-24T03:34:18.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.312+0000 7fe1e2ffd700 1 -- 172.21.15.112:0/1349151428 learned_addr learned my addr 172.21.15.112:0/1349151428 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:34:18.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.312+0000 7fe1e27fc700 1 --2- 172.21.15.112:0/1349151428 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe1e40c27e0 0x7fe1e40cac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:18.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.312+0000 7fe1e37fe700 1 --2- 172.21.15.112:0/1349151428 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe1e40cb190 0x7fe1e40c0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:18.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.312+0000 7fe1e2ffd700 1 -- 172.21.15.112:0/1349151428 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe1e40c27e0 msgr2=0x7fe1e40cac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:18.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.312+0000 7fe1e2ffd700 1 --2- 172.21.15.112:0/1349151428 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe1e40c27e0 0x7fe1e40cac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:18.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.312+0000 7fe1e2ffd700 1 -- 172.21.15.112:0/1349151428 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe1e40cb190 msgr2=0x7fe1e40c0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:18.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.312+0000 7fe1e2ffd700 1 --2- 172.21.15.112:0/1349151428 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe1e40cb190 0x7fe1e40c0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:18.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.312+0000 7fe1e2ffd700 1 -- 172.21.15.112:0/1349151428 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e41690e0 con 0x7fe1c4003e60 2023-01-24T03:34:18.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.312+0000 7fe1e2ffd700 1 --2- 172.21.15.112:0/1349151428 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe1c4003e60 0x7fe1e4165d50 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe1d4011f80 tx=0x7fe1d400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:18.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.313+0000 7fe1daffd700 1 -- 172.21.15.112:0/1349151428 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d400e040 con 0x7fe1c4003e60 2023-01-24T03:34:18.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.313+0000 7fe1daffd700 1 -- 172.21.15.112:0/1349151428 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1d400f040 con 0x7fe1c4003e60 2023-01-24T03:34:18.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.313+0000 7fe1e3fff700 1 -- 172.21.15.112:0/1349151428 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1e41693c0 con 0x7fe1c4003e60 2023-01-24T03:34:18.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.313+0000 7fe1daffd700 1 -- 172.21.15.112:0/1349151428 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d4015430 con 0x7fe1c4003e60 2023-01-24T03:34:18.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.313+0000 7fe1e3fff700 1 -- 172.21.15.112:0/1349151428 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1e41696a0 con 0x7fe1c4003e60 2023-01-24T03:34:18.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.313+0000 7fe1e3fff700 1 -- 172.21.15.112:0/1349151428 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1e40cb560 con 0x7fe1c4003e60 2023-01-24T03:34:18.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.314+0000 7fe1daffd700 1 -- 172.21.15.112:0/1349151428 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fe1d4015820 con 0x7fe1c4003e60 2023-01-24T03:34:18.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.314+0000 7fe1daffd700 1 --2- 172.21.15.112:0/1349151428 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fe1c80611c0 0x7fe1c8063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:18.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.314+0000 7fe1daffd700 1 -- 172.21.15.112:0/1349151428 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7039+0+0 (secure 0 0 0) 0x7fe1d407e6d0 con 0x7fe1c4003e60 2023-01-24T03:34:18.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.314+0000 7fe1e27fc700 1 --2- 172.21.15.112:0/1349151428 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fe1c80611c0 0x7fe1c8063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:18.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.315+0000 7fe1e27fc700 1 --2- 172.21.15.112:0/1349151428 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fe1c80611c0 0x7fe1c8063670 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe1e415fc80 tx=0x7fe1cc01a000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:18.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.318+0000 7fe1daffd700 1 -- 172.21.15.112:0/1349151428 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fe1d404e430 con 0x7fe1c4003e60 2023-01-24T03:34:18.541 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.539+0000 7fe1e3fff700 1 -- 172.21.15.112:0/1349151428 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1e405ea20 con 0x7fe1c4003e60 2023-01-24T03:34:18.541 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.539+0000 7fe1daffd700 1 -- 172.21.15.112:0/1349151428 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+6112 (secure 0 0 0) 0x7fe1d4051a50 con 0x7fe1c4003e60 2023-01-24T03:34:18.542 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:34:18.542 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5041,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/612058861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":612058861},{"type":"v1","addr":"172.21.15.142:6834","nonce":612058861}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:33:18.651832+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4897},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4897":{"gid":4897,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":19,"addr":"172.21.15.142:6835/11022091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":11022091},{"type":"v1","addr":"172.21.15.142:6835","nonce":11022091}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:34:18.543 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.541+0000 7fe1e3fff700 1 -- 172.21.15.112:0/1349151428 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fe1c80611c0 msgr2=0x7fe1c8063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:18.544 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.541+0000 7fe1e3fff700 1 --2- 172.21.15.112:0/1349151428 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fe1c80611c0 0x7fe1c8063670 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe1e415fc80 tx=0x7fe1cc01a000 comp rx=0 tx=0).stop 2023-01-24T03:34:18.544 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.541+0000 7fe1e3fff700 1 -- 172.21.15.112:0/1349151428 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe1c4003e60 msgr2=0x7fe1e4165d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:18.544 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.541+0000 7fe1e3fff700 1 --2- 172.21.15.112:0/1349151428 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe1c4003e60 0x7fe1e4165d50 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe1d4011f80 tx=0x7fe1d400b040 comp rx=0 tx=0).stop 2023-01-24T03:34:18.544 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.541+0000 7fe1e3fff700 1 -- 172.21.15.112:0/1349151428 shutdown_connections 2023-01-24T03:34:18.545 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.541+0000 7fe1e3fff700 1 --2- 172.21.15.112:0/1349151428 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fe1c80611c0 0x7fe1c8063670 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:18.545 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.541+0000 7fe1e3fff700 1 --2- 172.21.15.112:0/1349151428 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe1c4003e60 0x7fe1e4165d50 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:18.545 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.541+0000 7fe1e3fff700 1 --2- 172.21.15.112:0/1349151428 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe1e40cb190 0x7fe1e40c0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:18.545 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.541+0000 7fe1e3fff700 1 --2- 172.21.15.112:0/1349151428 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fe1e40c27e0 0x7fe1e40cac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:18.545 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.541+0000 7fe1e3fff700 1 -- 172.21.15.112:0/1349151428 >> 172.21.15.112:0/1349151428 conn(0x7fe1e40685b0 msgr2=0x7fe1e4153080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:18.546 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.541+0000 7fe1e3fff700 1 -- 172.21.15.112:0/1349151428 shutdown_connections 2023-01-24T03:34:18.546 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:18.541+0000 7fe1e3fff700 1 -- 172.21.15.112:0/1349151428 wait complete. 2023-01-24T03:34:18.546 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 21 2023-01-24T03:34:18.555 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-01-24T03:34:20.556 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:34:20.696 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.680+0000 7f650899d700 1 Processor -- start 2023-01-24T03:34:20.697 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.680+0000 7f650899d700 1 -- start start 2023-01-24T03:34:20.697 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.681+0000 7f650899d700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f6504156e40 0x7f65040cb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:20.697 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.681+0000 7f650899d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6504155e20 0x7f65040cbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:20.697 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.681+0000 7f650899d700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65040c3d10 con 0x7f65040c5680 2023-01-24T03:34:20.698 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.681+0000 7f650899d700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65040c4eb0 con 0x7f6504156e40 2023-01-24T03:34:20.698 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.681+0000 7f650899d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65040cc2f0 con 0x7f6504155e20 2023-01-24T03:34:20.698 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.681+0000 7f6502d9d700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f65040c5680 0x7f6504156730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:33638/0 (socket says 172.21.15.112:33638) 2023-01-24T03:34:20.698 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.681+0000 7f6502d9d700 1 -- 172.21.15.112:0/232817874 learned_addr learned my addr 172.21.15.112:0/232817874 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:34:20.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.681+0000 7f650259c700 1 --2- 172.21.15.112:0/232817874 >> v2:172.21.15.112:3300/0 conn(0x7f6504156e40 0x7f65040cb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:20.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.681+0000 7f650359e700 1 --2- 172.21.15.112:0/232817874 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6504155e20 0x7f65040cbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:20.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.681+0000 7f650259c700 1 -- 172.21.15.112:0/232817874 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6504155e20 msgr2=0x7f65040cbdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:20.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.681+0000 7f650259c700 1 --2- 172.21.15.112:0/232817874 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6504155e20 0x7f65040cbdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:20.700 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.681+0000 7f650259c700 1 -- 172.21.15.112:0/232817874 >> v1:172.21.15.112:6789/0 conn(0x7f65040c5680 legacy=0x7f6504156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:20.700 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.681+0000 7f650259c700 1 -- 172.21.15.112:0/232817874 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65040c2e20 con 0x7f6504156e40 2023-01-24T03:34:20.700 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.681+0000 7f6501d9b700 1 -- 172.21.15.112:0/232817874 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1739331622 0 0) 0x7f65040c3d10 con 0x7f65040c5680 2023-01-24T03:34:20.700 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.682+0000 7f650259c700 1 --2- 172.21.15.112:0/232817874 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6504156e40 0x7f65040cb870 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f64f8009760 tx=0x7f64f8009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8abc4431f153a64e server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:20.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.682+0000 7f6501d9b700 1 -- 172.21.15.112:0/232817874 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f8011630 con 0x7f6504156e40 2023-01-24T03:34:20.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.682+0000 7f6501d9b700 1 -- 172.21.15.112:0/232817874 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64f8011790 con 0x7f6504156e40 2023-01-24T03:34:20.702 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.682+0000 7f6501d9b700 1 -- 172.21.15.112:0/232817874 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f8011b50 con 0x7f6504156e40 2023-01-24T03:34:20.702 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.682+0000 7f650899d700 1 -- 172.21.15.112:0/232817874 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6504156e40 msgr2=0x7f65040cb870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:20.702 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.682+0000 7f650899d700 1 --2- 172.21.15.112:0/232817874 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6504156e40 0x7f65040cb870 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f64f8009760 tx=0x7f64f8009060 comp rx=0 tx=0).stop 2023-01-24T03:34:20.702 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.682+0000 7f650899d700 1 -- 172.21.15.112:0/232817874 shutdown_connections 2023-01-24T03:34:20.702 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.682+0000 7f650899d700 1 --2- 172.21.15.112:0/232817874 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6504155e20 0x7f65040cbdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:20.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.682+0000 7f650899d700 1 --2- 172.21.15.112:0/232817874 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6504156e40 0x7f65040cb870 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:20.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.682+0000 7f650899d700 1 -- 172.21.15.112:0/232817874 >> 172.21.15.112:0/232817874 conn(0x7f65040685b0 msgr2=0x7f65040689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:20.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.682+0000 7f650899d700 1 -- 172.21.15.112:0/232817874 shutdown_connections 2023-01-24T03:34:20.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.682+0000 7f650899d700 1 -- 172.21.15.112:0/232817874 wait complete. 2023-01-24T03:34:20.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.683+0000 7f650899d700 1 Processor -- start 2023-01-24T03:34:20.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.684+0000 7f650899d700 1 -- start start 2023-01-24T03:34:20.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.684+0000 7f650899d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f65040c5680 0x7f6504153f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:20.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.684+0000 7f650899d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6504155e20 0x7f65040c7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:20.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.684+0000 7f650899d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6504156e40 0x7f65040c61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:20.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.684+0000 7f650899d700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6504168480 con 0x7f6504155e20 2023-01-24T03:34:20.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.684+0000 7f650899d700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65040b1280 con 0x7f6504156e40 2023-01-24T03:34:20.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.684+0000 7f650899d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65040c7d70 con 0x7f65040c5680 2023-01-24T03:34:20.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.684+0000 7f650259c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6504155e20 0x7f65040c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:20.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.684+0000 7f650259c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6504155e20 0x7f65040c7350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52656/0 (socket says 172.21.15.112:52656) 2023-01-24T03:34:20.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.684+0000 7f650259c700 1 -- 172.21.15.112:0/2545459628 learned_addr learned my addr 172.21.15.112:0/2545459628 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:34:20.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.685+0000 7f6502d9d700 1 --2- 172.21.15.112:0/2545459628 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f65040c5680 0x7f6504153f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:20.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.685+0000 7f650359e700 1 --2- 172.21.15.112:0/2545459628 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6504156e40 0x7f65040c61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:20.707 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.685+0000 7f650259c700 1 -- 172.21.15.112:0/2545459628 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f65040c5680 msgr2=0x7f6504153f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:20.707 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.685+0000 7f650259c700 1 --2- 172.21.15.112:0/2545459628 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f65040c5680 0x7f6504153f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:20.707 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.685+0000 7f650259c700 1 -- 172.21.15.112:0/2545459628 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6504156e40 msgr2=0x7f65040c61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:20.707 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.685+0000 7f650259c700 1 --2- 172.21.15.112:0/2545459628 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6504156e40 0x7f65040c61b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:20.708 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.685+0000 7f650259c700 1 -- 172.21.15.112:0/2545459628 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f80092a0 con 0x7f6504155e20 2023-01-24T03:34:20.708 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.685+0000 7f650259c700 1 --2- 172.21.15.112:0/2545459628 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6504155e20 0x7f65040c7350 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f64f8009e00 tx=0x7f64f8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:20.708 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.685+0000 7f64eaffd700 1 -- 172.21.15.112:0/2545459628 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f8011900 con 0x7f6504155e20 2023-01-24T03:34:20.708 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.685+0000 7f64eaffd700 1 -- 172.21.15.112:0/2545459628 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64f8011a60 con 0x7f6504155e20 2023-01-24T03:34:20.709 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.685+0000 7f650899d700 1 -- 172.21.15.112:0/2545459628 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65041686d0 con 0x7f6504155e20 2023-01-24T03:34:20.709 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.685+0000 7f64eaffd700 1 -- 172.21.15.112:0/2545459628 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f801a680 con 0x7f6504155e20 2023-01-24T03:34:20.709 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.685+0000 7f650899d700 1 -- 172.21.15.112:0/2545459628 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6504168c90 con 0x7f6504155e20 2023-01-24T03:34:20.709 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.686+0000 7f650899d700 1 -- 172.21.15.112:0/2545459628 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f650405ea20 con 0x7f6504155e20 2023-01-24T03:34:20.709 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.686+0000 7f64eaffd700 1 -- 172.21.15.112:0/2545459628 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f64f800f050 con 0x7f6504155e20 2023-01-24T03:34:20.710 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.686+0000 7f64eaffd700 1 --2- 172.21.15.112:0/2545459628 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f64e40611c0 0x7f64e4063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:20.710 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.687+0000 7f64eaffd700 1 -- 172.21.15.112:0/2545459628 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7039+0+0 (secure 0 0 0) 0x7f64f807f520 con 0x7f6504155e20 2023-01-24T03:34:20.710 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.687+0000 7f6502d9d700 1 --2- 172.21.15.112:0/2545459628 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f64e40611c0 0x7f64e4063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:20.710 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.687+0000 7f6502d9d700 1 --2- 172.21.15.112:0/2545459628 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f64e40611c0 0x7f64e4063670 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f64f4001d60 tx=0x7f64f4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:20.711 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.691+0000 7f64eaffd700 1 -- 172.21.15.112:0/2545459628 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f64f804f190 con 0x7f6504155e20 2023-01-24T03:34:20.914 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.910+0000 7f650899d700 1 -- 172.21.15.112:0/2545459628 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65040c3100 con 0x7f6504155e20 2023-01-24T03:34:20.914 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.911+0000 7f64eaffd700 1 -- 172.21.15.112:0/2545459628 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+6112 (secure 0 0 0) 0x7f64f80527b0 con 0x7f6504155e20 2023-01-24T03:34:20.915 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:34:20.915 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5041,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/612058861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":612058861},{"type":"v1","addr":"172.21.15.142:6834","nonce":612058861}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:33:18.651832+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4897},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4897":{"gid":4897,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":19,"addr":"172.21.15.142:6835/11022091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":11022091},{"type":"v1","addr":"172.21.15.142:6835","nonce":11022091}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:34:20.917 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.913+0000 7f650899d700 1 -- 172.21.15.112:0/2545459628 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f64e40611c0 msgr2=0x7f64e4063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:20.917 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.913+0000 7f650899d700 1 --2- 172.21.15.112:0/2545459628 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f64e40611c0 0x7f64e4063670 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f64f4001d60 tx=0x7f64f4009040 comp rx=0 tx=0).stop 2023-01-24T03:34:20.917 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.913+0000 7f650899d700 1 -- 172.21.15.112:0/2545459628 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6504155e20 msgr2=0x7f65040c7350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:20.917 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.913+0000 7f650899d700 1 --2- 172.21.15.112:0/2545459628 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6504155e20 0x7f65040c7350 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f64f8009e00 tx=0x7f64f8017000 comp rx=0 tx=0).stop 2023-01-24T03:34:20.918 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.913+0000 7f650899d700 1 -- 172.21.15.112:0/2545459628 shutdown_connections 2023-01-24T03:34:20.918 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.913+0000 7f650899d700 1 --2- 172.21.15.112:0/2545459628 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f64e40611c0 0x7f64e4063670 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:20.918 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.913+0000 7f650899d700 1 --2- 172.21.15.112:0/2545459628 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6504156e40 0x7f65040c61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:20.918 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.913+0000 7f650899d700 1 --2- 172.21.15.112:0/2545459628 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f65040c5680 0x7f6504153f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:20.919 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.913+0000 7f650899d700 1 --2- 172.21.15.112:0/2545459628 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6504155e20 0x7f65040c7350 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:20.919 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.913+0000 7f650899d700 1 -- 172.21.15.112:0/2545459628 >> 172.21.15.112:0/2545459628 conn(0x7f65040685b0 msgr2=0x7f650414ca20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:20.919 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.913+0000 7f650899d700 1 -- 172.21.15.112:0/2545459628 shutdown_connections 2023-01-24T03:34:20.919 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:20.913+0000 7f650899d700 1 -- 172.21.15.112:0/2545459628 wait complete. 2023-01-24T03:34:20.919 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 21 2023-01-24T03:34:20.927 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-01-24T03:34:22.929 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:34:23.053 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.050+0000 7f9e5e63b700 1 Processor -- start 2023-01-24T03:34:23.053 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.050+0000 7f9e5e63b700 1 -- start start 2023-01-24T03:34:23.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.050+0000 7f9e5e63b700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f9e580c7e40 0x7f9e580b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:23.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.050+0000 7f9e5e63b700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e58156240 con 0x7f9e58155da0 2023-01-24T03:34:23.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.050+0000 7f9e5e63b700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e580b1170 con 0x7f9e580ca3e0 2023-01-24T03:34:23.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.050+0000 7f9e5e63b700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e580b1450 con 0x7f9e580c7e40 2023-01-24T03:34:23.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.050+0000 7f9e4ffff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f9e58155da0 0x7f9e580ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:33654/0 (socket says 172.21.15.112:33654) 2023-01-24T03:34:23.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.050+0000 7f9e4ffff700 1 -- 172.21.15.112:0/4196479667 learned_addr learned my addr 172.21.15.112:0/4196479667 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:34:23.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.051+0000 7f9e57fff700 1 --2- 172.21.15.112:0/4196479667 >> v2:172.21.15.142:3300/0 conn(0x7f9e580c7e40 0x7f9e580b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:23.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.051+0000 7f9e577fe700 1 -- 172.21.15.112:0/4196479667 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 499558897 0 0) 0x7f9e58156240 con 0x7f9e58155da0 2023-01-24T03:34:23.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.051+0000 7f9e577fe700 1 -- 172.21.15.112:0/4196479667 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e380033f0 con 0x7f9e58155da0 2023-01-24T03:34:23.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.051+0000 7f9e577fe700 1 -- 172.21.15.112:0/4196479667 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1676372032 0 0) 0x7f9e380033f0 con 0x7f9e58155da0 2023-01-24T03:34:23.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.051+0000 7f9e577fe700 1 -- 172.21.15.112:0/4196479667 >> v2:172.21.15.142:3300/0 conn(0x7f9e580c7e40 msgr2=0x7f9e580b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:23.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.051+0000 7f9e577fe700 1 --2- 172.21.15.112:0/4196479667 >> v2:172.21.15.142:3300/0 conn(0x7f9e580c7e40 0x7f9e580b0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:23.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.051+0000 7f9e577fe700 1 -- 172.21.15.112:0/4196479667 >> v1:172.21.15.142:6789/0 conn(0x7f9e580ca3e0 legacy=0x7f9e580ca7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:23.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.051+0000 7f9e577fe700 1 -- 172.21.15.112:0/4196479667 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e580b1730 con 0x7f9e58155da0 2023-01-24T03:34:23.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.051+0000 7f9e577fe700 1 -- 172.21.15.112:0/4196479667 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7f9e40001470 con 0x7f9e58155da0 2023-01-24T03:34:23.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.051+0000 7f9e577fe700 1 -- 172.21.15.112:0/4196479667 >> v1:172.21.15.112:6789/0 conn(0x7f9e58155da0 legacy=0x7f9e580ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:23.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.051+0000 7f9e577fe700 1 --2- 172.21.15.112:0/4196479667 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9e380038e0 0x7f9e38003cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:23.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.051+0000 7f9e577fe700 1 --2- 172.21.15.112:0/4196479667 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e38004690 0x7f9e38004b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:23.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.051+0000 7f9e577fe700 1 --2- 172.21.15.112:0/4196479667 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e38005040 0x7f9e3800b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:23.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.051+0000 7f9e577fe700 1 -- 172.21.15.112:0/4196479667 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e580b1450 con 0x7f9e380038e0 2023-01-24T03:34:23.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.051+0000 7f9e577fe700 1 -- 172.21.15.112:0/4196479667 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3800bfe0 con 0x7f9e38005040 2023-01-24T03:34:23.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.051+0000 7f9e577fe700 1 -- 172.21.15.112:0/4196479667 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3800c1a0 con 0x7f9e38004690 2023-01-24T03:34:23.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.051+0000 7f9e57fff700 1 --2- 172.21.15.112:0/4196479667 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9e380038e0 0x7f9e38003cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:23.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.051+0000 7f9e5cbd8700 1 --2- 172.21.15.112:0/4196479667 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e38005040 0x7f9e3800b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:23.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.051+0000 7f9e4ffff700 1 --2- 172.21.15.112:0/4196479667 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e38004690 0x7f9e38004b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:23.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.052+0000 7f9e57fff700 1 -- 172.21.15.112:0/4196479667 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e38004690 msgr2=0x7f9e38004b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:23.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.052+0000 7f9e57fff700 1 --2- 172.21.15.112:0/4196479667 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e38004690 0x7f9e38004b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:23.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.052+0000 7f9e57fff700 1 -- 172.21.15.112:0/4196479667 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e38005040 msgr2=0x7f9e3800b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:23.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.052+0000 7f9e57fff700 1 --2- 172.21.15.112:0/4196479667 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e38005040 0x7f9e3800b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:23.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.052+0000 7f9e5cbd8700 1 -- 172.21.15.112:0/4196479667 reap_dead start 2023-01-24T03:34:23.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.052+0000 7f9e57fff700 1 -- 172.21.15.112:0/4196479667 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e380033f0 con 0x7f9e380038e0 2023-01-24T03:34:23.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.052+0000 7f9e4ffff700 1 --2- 172.21.15.112:0/4196479667 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e38004690 0x7f9e38004b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:34:23.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.052+0000 7f9e57fff700 1 --2- 172.21.15.112:0/4196479667 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9e380038e0 0x7f9e38003cf0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f9e48009a80 tx=0x7f9e48009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bf6530621b2bee7 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:23.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.052+0000 7f9e577fe700 1 -- 172.21.15.112:0/4196479667 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e48009430 con 0x7f9e380038e0 2023-01-24T03:34:23.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.052+0000 7f9e577fe700 1 -- 172.21.15.112:0/4196479667 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e4800e640 con 0x7f9e380038e0 2023-01-24T03:34:23.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.052+0000 7f9e577fe700 1 -- 172.21.15.112:0/4196479667 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4800ea00 con 0x7f9e380038e0 2023-01-24T03:34:23.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.052+0000 7f9e5e63b700 1 -- 172.21.15.112:0/4196479667 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9e380038e0 msgr2=0x7f9e38003cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:23.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.052+0000 7f9e5e63b700 1 --2- 172.21.15.112:0/4196479667 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9e380038e0 0x7f9e38003cf0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f9e48009a80 tx=0x7f9e48009060 comp rx=0 tx=0).stop 2023-01-24T03:34:23.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.052+0000 7f9e5e63b700 1 -- 172.21.15.112:0/4196479667 shutdown_connections 2023-01-24T03:34:23.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.052+0000 7f9e5e63b700 1 --2- 172.21.15.112:0/4196479667 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9e380038e0 0x7f9e38003cf0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:23.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.052+0000 7f9e5e63b700 1 -- 172.21.15.112:0/4196479667 >> 172.21.15.112:0/4196479667 conn(0x7f9e5805e2c0 msgr2=0x7f9e580cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:23.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.052+0000 7f9e5e63b700 1 -- 172.21.15.112:0/4196479667 shutdown_connections 2023-01-24T03:34:23.069 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.052+0000 7f9e5e63b700 1 -- 172.21.15.112:0/4196479667 wait complete. 2023-01-24T03:34:23.069 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.054+0000 7f9e5e63b700 1 Processor -- start 2023-01-24T03:34:23.069 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.054+0000 7f9e5e63b700 1 -- start start 2023-01-24T03:34:23.069 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.054+0000 7f9e5e63b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9e380038e0 0x7f9e58154350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:23.069 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.054+0000 7f9e5e63b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e580c0b50 0x7f9e580bfc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:23.070 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.054+0000 7f9e5e63b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e581550a0 0x7f9e580c0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:23.070 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.054+0000 7f9e5e63b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e581518b0 con 0x7f9e380038e0 2023-01-24T03:34:23.070 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.054+0000 7f9e5e63b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e58165560 con 0x7f9e581550a0 2023-01-24T03:34:23.070 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.054+0000 7f9e5e63b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e58165840 con 0x7f9e580c0b50 2023-01-24T03:34:23.070 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.054+0000 7f9e57fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9e380038e0 0x7f9e58154350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:23.070 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.054+0000 7f9e57fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9e380038e0 0x7f9e58154350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52676/0 (socket says 172.21.15.112:52676) 2023-01-24T03:34:23.071 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.054+0000 7f9e57fff700 1 -- 172.21.15.112:0/2572107905 learned_addr learned my addr 172.21.15.112:0/2572107905 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:34:23.071 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.055+0000 7f9e57fff700 1 -- 172.21.15.112:0/2572107905 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e580c0b50 msgr2=0x7f9e580bfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:23.071 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.055+0000 7f9e57fff700 1 --2- 172.21.15.112:0/2572107905 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e580c0b50 0x7f9e580bfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:23.071 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.055+0000 7f9e57fff700 1 -- 172.21.15.112:0/2572107905 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e581550a0 msgr2=0x7f9e580c0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:23.071 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.055+0000 7f9e57fff700 1 --2- 172.21.15.112:0/2572107905 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e581550a0 0x7f9e580c0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:23.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.055+0000 7f9e57fff700 1 -- 172.21.15.112:0/2572107905 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e58165b20 con 0x7f9e380038e0 2023-01-24T03:34:23.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.055+0000 7f9e57fff700 1 --2- 172.21.15.112:0/2572107905 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9e380038e0 0x7f9e58154350 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f9e3800bfb0 tx=0x7f9e4801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:23.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.055+0000 7f9e54ff9700 1 -- 172.21.15.112:0/2572107905 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e48021070 con 0x7f9e380038e0 2023-01-24T03:34:23.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.055+0000 7f9e54ff9700 1 -- 172.21.15.112:0/2572107905 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e48009100 con 0x7f9e380038e0 2023-01-24T03:34:23.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.055+0000 7f9e5e63b700 1 -- 172.21.15.112:0/2572107905 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e58165e00 con 0x7f9e380038e0 2023-01-24T03:34:23.073 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.055+0000 7f9e54ff9700 1 -- 172.21.15.112:0/2572107905 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e48013a20 con 0x7f9e380038e0 2023-01-24T03:34:23.073 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.055+0000 7f9e5e63b700 1 -- 172.21.15.112:0/2572107905 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e58152cf0 con 0x7f9e380038e0 2023-01-24T03:34:23.073 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.056+0000 7f9e5e63b700 1 -- 172.21.15.112:0/2572107905 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e580bfa70 con 0x7f9e380038e0 2023-01-24T03:34:23.073 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.056+0000 7f9e54ff9700 1 -- 172.21.15.112:0/2572107905 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f9e4802e020 con 0x7f9e380038e0 2023-01-24T03:34:23.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.056+0000 7f9e54ff9700 1 --2- 172.21.15.112:0/2572107905 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9e3c0611c0 0x7f9e3c063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:23.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.056+0000 7f9e54ff9700 1 -- 172.21.15.112:0/2572107905 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7039+0+0 (secure 0 0 0) 0x7f9e4807f2b0 con 0x7f9e380038e0 2023-01-24T03:34:23.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.056+0000 7f9e4ffff700 1 --2- 172.21.15.112:0/2572107905 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9e3c0611c0 0x7f9e3c063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:23.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.057+0000 7f9e4ffff700 1 --2- 172.21.15.112:0/2572107905 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9e3c0611c0 0x7f9e3c063670 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9e40000e10 tx=0x7f9e40001c50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:23.075 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.060+0000 7f9e54ff9700 1 -- 172.21.15.112:0/2572107905 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f9e4804f010 con 0x7f9e380038e0 2023-01-24T03:34:23.285 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.282+0000 7f9e5e63b700 1 -- 172.21.15.112:0/2572107905 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e5804bb80 con 0x7f9e380038e0 2023-01-24T03:34:23.286 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.283+0000 7f9e54ff9700 1 -- 172.21.15.112:0/2572107905 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+6112 (secure 0 0 0) 0x7f9e48052630 con 0x7f9e380038e0 2023-01-24T03:34:23.287 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:34:23.287 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5041,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/612058861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":612058861},{"type":"v1","addr":"172.21.15.142:6834","nonce":612058861}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:33:18.651832+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4897},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4897":{"gid":4897,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":19,"addr":"172.21.15.142:6835/11022091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":11022091},{"type":"v1","addr":"172.21.15.142:6835","nonce":11022091}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:34:23.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.285+0000 7f9e5e63b700 1 -- 172.21.15.112:0/2572107905 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9e3c0611c0 msgr2=0x7f9e3c063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:23.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.285+0000 7f9e5e63b700 1 --2- 172.21.15.112:0/2572107905 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9e3c0611c0 0x7f9e3c063670 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9e40000e10 tx=0x7f9e40001c50 comp rx=0 tx=0).stop 2023-01-24T03:34:23.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.285+0000 7f9e5e63b700 1 -- 172.21.15.112:0/2572107905 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9e380038e0 msgr2=0x7f9e58154350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:23.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.285+0000 7f9e5e63b700 1 --2- 172.21.15.112:0/2572107905 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9e380038e0 0x7f9e58154350 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f9e3800bfb0 tx=0x7f9e4801a000 comp rx=0 tx=0).stop 2023-01-24T03:34:23.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.285+0000 7f9e5e63b700 1 -- 172.21.15.112:0/2572107905 shutdown_connections 2023-01-24T03:34:23.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.285+0000 7f9e5e63b700 1 --2- 172.21.15.112:0/2572107905 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9e3c0611c0 0x7f9e3c063670 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:23.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.285+0000 7f9e5e63b700 1 --2- 172.21.15.112:0/2572107905 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9e380038e0 0x7f9e58154350 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:23.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.285+0000 7f9e5e63b700 1 --2- 172.21.15.112:0/2572107905 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9e581550a0 0x7f9e580c0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:23.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.285+0000 7f9e5e63b700 1 --2- 172.21.15.112:0/2572107905 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9e580c0b50 0x7f9e580bfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:23.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.285+0000 7f9e5e63b700 1 -- 172.21.15.112:0/2572107905 >> 172.21.15.112:0/2572107905 conn(0x7f9e5805e2c0 msgr2=0x7f9e580cb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:23.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.285+0000 7f9e5e63b700 1 -- 172.21.15.112:0/2572107905 shutdown_connections 2023-01-24T03:34:23.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:23.285+0000 7f9e5e63b700 1 -- 172.21.15.112:0/2572107905 wait complete. 2023-01-24T03:34:23.292 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 21 2023-01-24T03:34:23.299 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-01-24T03:34:25.300 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:34:25.425 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.422+0000 7fb7fe968700 1 Processor -- start 2023-01-24T03:34:25.425 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.422+0000 7fb7fe968700 1 -- start start 2023-01-24T03:34:25.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.423+0000 7fb7fe968700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb7f80c5680 0x7fb7f8156730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:25.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.423+0000 7fb7fe968700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb7f80c5a50 0x7fb7f8156c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:25.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.423+0000 7fb7fe968700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7f80c3d10 con 0x7fb7f8155e20 2023-01-24T03:34:25.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.423+0000 7fb7fe968700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7f80c4eb0 con 0x7fb7f80c5a50 2023-01-24T03:34:25.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.423+0000 7fb7fe968700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f80c2e20 con 0x7fb7f80c5680 2023-01-24T03:34:25.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.423+0000 7fb7fd165700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fb7f8155e20 0x7fb7f80cb870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:33674/0 (socket says 172.21.15.112:33674) 2023-01-24T03:34:25.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.423+0000 7fb7fd165700 1 -- 172.21.15.112:0/584379319 learned_addr learned my addr 172.21.15.112:0/584379319 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:34:25.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.423+0000 7fb7fd966700 1 --2- 172.21.15.112:0/584379319 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb7f80c5680 0x7fb7f8156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:25.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.423+0000 7fb7fe167700 1 --2- 172.21.15.112:0/584379319 >> v2:172.21.15.142:3300/0 conn(0x7fb7f80c5a50 0x7fb7f8156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:25.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.423+0000 7fb7fc964700 1 -- 172.21.15.112:0/584379319 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2725614414 0 0) 0x7fb7f80c3d10 con 0x7fb7f8155e20 2023-01-24T03:34:25.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.423+0000 7fb7fc964700 1 -- 172.21.15.112:0/584379319 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7e40033f0 con 0x7fb7f8155e20 2023-01-24T03:34:25.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.424+0000 7fb7fc964700 1 -- 172.21.15.112:0/584379319 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 965126145 0 0) 0x7fb7e40033f0 con 0x7fb7f8155e20 2023-01-24T03:34:25.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.424+0000 7fb7fc964700 1 -- 172.21.15.112:0/584379319 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb7f80c5680 msgr2=0x7fb7f8156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:25.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.424+0000 7fb7fc964700 1 --2- 172.21.15.112:0/584379319 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb7f80c5680 0x7fb7f8156730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:25.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.424+0000 7fb7fc964700 1 -- 172.21.15.112:0/584379319 >> v2:172.21.15.142:3300/0 conn(0x7fb7f80c5a50 msgr2=0x7fb7f8156c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:25.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.424+0000 7fb7fc964700 1 --2- 172.21.15.112:0/584379319 >> v2:172.21.15.142:3300/0 conn(0x7fb7f80c5a50 0x7fb7f8156c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:25.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.424+0000 7fb7fc964700 1 -- 172.21.15.112:0/584379319 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f80c3100 con 0x7fb7f8155e20 2023-01-24T03:34:25.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.424+0000 7fb7fc964700 1 -- 172.21.15.112:0/584379319 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7fb7ec001470 con 0x7fb7f8155e20 2023-01-24T03:34:25.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.424+0000 7fb7fc964700 1 -- 172.21.15.112:0/584379319 >> v1:172.21.15.112:6789/0 conn(0x7fb7f8155e20 legacy=0x7fb7f80cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:25.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.424+0000 7fb7fc964700 1 --2- 172.21.15.112:0/584379319 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb7f80c5680 0x7fb7e40038e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:25.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.424+0000 7fb7fc964700 1 --2- 172.21.15.112:0/584379319 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7e4003e60 0x7fb7e4004690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:25.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.424+0000 7fb7fc964700 1 --2- 172.21.15.112:0/584379319 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb7e4004bd0 0x7fb7e4005040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:25.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.424+0000 7fb7fc964700 1 -- 172.21.15.112:0/584379319 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f80c4eb0 con 0x7fb7e4004bd0 2023-01-24T03:34:25.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.424+0000 7fb7fc964700 1 -- 172.21.15.112:0/584379319 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f80c2e20 con 0x7fb7e4003e60 2023-01-24T03:34:25.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.424+0000 7fb7fc964700 1 -- 172.21.15.112:0/584379319 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e400b910 con 0x7fb7f80c5680 2023-01-24T03:34:25.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.424+0000 7fb7fe167700 1 --2- 172.21.15.112:0/584379319 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb7e4004bd0 0x7fb7e4005040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:25.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.424+0000 7fb7fd966700 1 --2- 172.21.15.112:0/584379319 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb7f80c5680 0x7fb7e40038e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:25.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.424+0000 7fb7fd165700 1 --2- 172.21.15.112:0/584379319 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7e4003e60 0x7fb7e4004690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:25.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.424+0000 7fb7fe167700 1 -- 172.21.15.112:0/584379319 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb7f80c5680 msgr2=0x7fb7e40038e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:25.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.424+0000 7fb7fe167700 1 --2- 172.21.15.112:0/584379319 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb7f80c5680 0x7fb7e40038e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:25.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.424+0000 7fb7fe167700 1 -- 172.21.15.112:0/584379319 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7e4003e60 msgr2=0x7fb7e4004690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:25.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.424+0000 7fb7fe167700 1 --2- 172.21.15.112:0/584379319 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7e4003e60 0x7fb7e4004690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:25.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.424+0000 7fb7fe167700 1 -- 172.21.15.112:0/584379319 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e40033f0 con 0x7fb7e4004bd0 2023-01-24T03:34:25.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.424+0000 7fb7fd165700 1 --2- 172.21.15.112:0/584379319 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7e4003e60 0x7fb7e4004690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:34:25.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.424+0000 7fb7fe167700 1 --2- 172.21.15.112:0/584379319 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb7e4004bd0 0x7fb7e4005040 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0009a80 tx=0x7fb7f0009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bd3b1b3df20d83a server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:25.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.424+0000 7fb7fc964700 1 -- 172.21.15.112:0/584379319 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f0009430 con 0x7fb7e4004bd0 2023-01-24T03:34:25.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.424+0000 7fb7fc964700 1 -- 172.21.15.112:0/584379319 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7f000e5e0 con 0x7fb7e4004bd0 2023-01-24T03:34:25.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.424+0000 7fb7fc964700 1 -- 172.21.15.112:0/584379319 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f000e9a0 con 0x7fb7e4004bd0 2023-01-24T03:34:25.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.425+0000 7fb7fe968700 1 -- 172.21.15.112:0/584379319 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb7e4004bd0 msgr2=0x7fb7e4005040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:25.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.425+0000 7fb7fe968700 1 --2- 172.21.15.112:0/584379319 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb7e4004bd0 0x7fb7e4005040 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0009a80 tx=0x7fb7f0009060 comp rx=0 tx=0).stop 2023-01-24T03:34:25.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.425+0000 7fb7fe167700 1 -- 172.21.15.112:0/584379319 reap_dead start 2023-01-24T03:34:25.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.425+0000 7fb7fe968700 1 -- 172.21.15.112:0/584379319 shutdown_connections 2023-01-24T03:34:25.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.425+0000 7fb7fe968700 1 -- 172.21.15.112:0/584379319 >> 172.21.15.112:0/584379319 conn(0x7fb7f80685b0 msgr2=0x7fb7f80689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:25.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.425+0000 7fb7fe968700 1 -- 172.21.15.112:0/584379319 shutdown_connections 2023-01-24T03:34:25.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.425+0000 7fb7fe968700 1 -- 172.21.15.112:0/584379319 wait complete. 2023-01-24T03:34:25.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.426+0000 7fb7fe968700 1 Processor -- start 2023-01-24T03:34:25.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.426+0000 7fb7fe968700 1 -- start start 2023-01-24T03:34:25.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.427+0000 7fb7fe968700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7f814cb80 0x7fb7f81653d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:25.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.427+0000 7fb7fe968700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb7f8165910 0x7fb7f814b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:25.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.427+0000 7fb7fe968700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb7f8151420 0x7fb7f814bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:25.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.427+0000 7fb7fe968700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f80c8ea0 con 0x7fb7f8165910 2023-01-24T03:34:25.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.427+0000 7fb7fe968700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f8164ce0 con 0x7fb7f814cb80 2023-01-24T03:34:25.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.427+0000 7fb7fe968700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f8164f80 con 0x7fb7f8151420 2023-01-24T03:34:25.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.427+0000 7fb7fd165700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb7f8165910 0x7fb7f814b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:25.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.427+0000 7fb7fd165700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb7f8165910 0x7fb7f814b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52700/0 (socket says 172.21.15.112:52700) 2023-01-24T03:34:25.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.427+0000 7fb7fd165700 1 -- 172.21.15.112:0/2374898857 learned_addr learned my addr 172.21.15.112:0/2374898857 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:34:25.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.427+0000 7fb7fd966700 1 --2- 172.21.15.112:0/2374898857 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7f814cb80 0x7fb7f81653d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:25.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.427+0000 7fb7fe167700 1 --2- 172.21.15.112:0/2374898857 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb7f8151420 0x7fb7f814bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:25.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.427+0000 7fb7fd165700 1 -- 172.21.15.112:0/2374898857 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb7f8151420 msgr2=0x7fb7f814bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:25.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.427+0000 7fb7fd165700 1 --2- 172.21.15.112:0/2374898857 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb7f8151420 0x7fb7f814bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:25.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.427+0000 7fb7fd165700 1 -- 172.21.15.112:0/2374898857 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7f814cb80 msgr2=0x7fb7f81653d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:25.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.427+0000 7fb7fd165700 1 --2- 172.21.15.112:0/2374898857 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7f814cb80 0x7fb7f81653d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:25.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.427+0000 7fb7fd165700 1 -- 172.21.15.112:0/2374898857 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f814fba0 con 0x7fb7f8165910 2023-01-24T03:34:25.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.427+0000 7fb7fd165700 1 --2- 172.21.15.112:0/2374898857 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb7f8165910 0x7fb7f814b990 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec000fd0 tx=0x7fb7ec0014f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:25.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.427+0000 7fb7f5ffb700 1 -- 172.21.15.112:0/2374898857 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7ec010490 con 0x7fb7f8165910 2023-01-24T03:34:25.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.427+0000 7fb7f5ffb700 1 -- 172.21.15.112:0/2374898857 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7ec010040 con 0x7fb7f8165910 2023-01-24T03:34:25.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.427+0000 7fb7fe968700 1 -- 172.21.15.112:0/2374898857 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7f80c61b0 con 0x7fb7f8165910 2023-01-24T03:34:25.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.427+0000 7fb7f5ffb700 1 -- 172.21.15.112:0/2374898857 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7ec010800 con 0x7fb7f8165910 2023-01-24T03:34:25.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.428+0000 7fb7fe968700 1 -- 172.21.15.112:0/2374898857 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7f80c6400 con 0x7fb7f8165910 2023-01-24T03:34:25.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.428+0000 7fb7fe968700 1 -- 172.21.15.112:0/2374898857 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7f81650f0 con 0x7fb7f8165910 2023-01-24T03:34:25.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.430+0000 7fb7f5ffb700 1 -- 172.21.15.112:0/2374898857 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fb7ec010b70 con 0x7fb7f8165910 2023-01-24T03:34:25.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.430+0000 7fb7f5ffb700 1 --2- 172.21.15.112:0/2374898857 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb7d8061270 0x7fb7d8063720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:25.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.430+0000 7fb7f5ffb700 1 -- 172.21.15.112:0/2374898857 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7039+0+0 (secure 0 0 0) 0x7fb7ec07d110 con 0x7fb7f8165910 2023-01-24T03:34:25.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.430+0000 7fb7fd966700 1 --2- 172.21.15.112:0/2374898857 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb7d8061270 0x7fb7d8063720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:25.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.430+0000 7fb7fd966700 1 --2- 172.21.15.112:0/2374898857 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb7d8061270 0x7fb7d8063720 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb7e80094d0 tx=0x7fb7e800b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:25.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.433+0000 7fb7f5ffb700 1 -- 172.21.15.112:0/2374898857 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fb7ec04ce50 con 0x7fb7f8165910 2023-01-24T03:34:25.656 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.653+0000 7fb7fe968700 1 -- 172.21.15.112:0/2374898857 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7f81650f0 con 0x7fb7f8165910 2023-01-24T03:34:25.657 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.654+0000 7fb7f5ffb700 1 -- 172.21.15.112:0/2374898857 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+6112 (secure 0 0 0) 0x7fb7f81650f0 con 0x7fb7f8165910 2023-01-24T03:34:25.657 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:34:25.657 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5041,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/612058861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":612058861},{"type":"v1","addr":"172.21.15.142:6834","nonce":612058861}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:33:18.651832+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4897},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4897":{"gid":4897,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":19,"addr":"172.21.15.142:6835/11022091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":11022091},{"type":"v1","addr":"172.21.15.142:6835","nonce":11022091}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:34:25.659 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.656+0000 7fb7fe968700 1 -- 172.21.15.112:0/2374898857 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb7d8061270 msgr2=0x7fb7d8063720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:25.659 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.656+0000 7fb7fe968700 1 --2- 172.21.15.112:0/2374898857 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb7d8061270 0x7fb7d8063720 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb7e80094d0 tx=0x7fb7e800b040 comp rx=0 tx=0).stop 2023-01-24T03:34:25.659 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.656+0000 7fb7fe968700 1 -- 172.21.15.112:0/2374898857 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb7f8165910 msgr2=0x7fb7f814b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:25.660 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.656+0000 7fb7fe968700 1 --2- 172.21.15.112:0/2374898857 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb7f8165910 0x7fb7f814b990 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec000fd0 tx=0x7fb7ec0014f0 comp rx=0 tx=0).stop 2023-01-24T03:34:25.660 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.656+0000 7fb7fe968700 1 -- 172.21.15.112:0/2374898857 shutdown_connections 2023-01-24T03:34:25.660 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.656+0000 7fb7fe968700 1 --2- 172.21.15.112:0/2374898857 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb7d8061270 0x7fb7d8063720 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:25.660 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.656+0000 7fb7fe968700 1 --2- 172.21.15.112:0/2374898857 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb7f8151420 0x7fb7f814bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:25.660 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.656+0000 7fb7fe968700 1 --2- 172.21.15.112:0/2374898857 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7f814cb80 0x7fb7f81653d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:25.661 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.656+0000 7fb7fe968700 1 --2- 172.21.15.112:0/2374898857 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb7f8165910 0x7fb7f814b990 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:25.661 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.656+0000 7fb7fe968700 1 -- 172.21.15.112:0/2374898857 >> 172.21.15.112:0/2374898857 conn(0x7fb7f80685b0 msgr2=0x7fb7f80cbb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:25.661 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.656+0000 7fb7fe968700 1 -- 172.21.15.112:0/2374898857 shutdown_connections 2023-01-24T03:34:25.661 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:25.656+0000 7fb7fe968700 1 -- 172.21.15.112:0/2374898857 wait complete. 2023-01-24T03:34:25.662 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 21 2023-01-24T03:34:25.670 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-01-24T03:34:27.671 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:34:27.796 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.793+0000 7fde71bc5700 1 Processor -- start 2023-01-24T03:34:27.796 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.793+0000 7fde71bc5700 1 -- start start 2023-01-24T03:34:27.798 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.794+0000 7fde71bc5700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fde6c155bf0 0x7fde6c068160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:27.798 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.794+0000 7fde71bc5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fde6c0c3f30 0x7fde6c0c2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:27.798 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.794+0000 7fde71bc5700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde6c0686a0 con 0x7fde6c067d90 2023-01-24T03:34:27.798 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.794+0000 7fde71bc5700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde6c156070 con 0x7fde6c155bf0 2023-01-24T03:34:27.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.794+0000 7fde71bc5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c0c5300 con 0x7fde6c0c3f30 2023-01-24T03:34:27.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.794+0000 7fde6bfff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fde6c155bf0 0x7fde6c068160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:27.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.794+0000 7fde713c4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fde6c0c3f30 0x7fde6c0c2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:27.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.794+0000 7fde6bfff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fde6c155bf0 0x7fde6c068160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.112:45204/0 (socket says 172.21.15.112:45204) 2023-01-24T03:34:27.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.794+0000 7fde713c4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fde6c0c3f30 0x7fde6c0c2c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.112:40014/0 (socket says 172.21.15.112:40014) 2023-01-24T03:34:27.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.794+0000 7fde6bfff700 1 -- 172.21.15.112:0/3006020898 learned_addr learned my addr 172.21.15.112:0/3006020898 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:34:27.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.794+0000 7fde70bc3700 1 --1- 172.21.15.112:0/3006020898 >> v1:172.21.15.142:6789/0 conn(0x7fde6c067d90 0x7fde6c0c5790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.112:47486/0 (socket says 172.21.15.112:47486) 2023-01-24T03:34:27.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.794+0000 7fde713c4700 1 -- 172.21.15.112:0/3006020898 >> v2:172.21.15.142:3300/0 conn(0x7fde6c155bf0 msgr2=0x7fde6c068160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:27.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.794+0000 7fde713c4700 1 --2- 172.21.15.112:0/3006020898 >> v2:172.21.15.142:3300/0 conn(0x7fde6c155bf0 0x7fde6c068160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:27.801 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.794+0000 7fde713c4700 1 -- 172.21.15.112:0/3006020898 >> v1:172.21.15.142:6789/0 conn(0x7fde6c067d90 legacy=0x7fde6c0c5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:27.801 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.794+0000 7fde713c4700 1 -- 172.21.15.112:0/3006020898 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde6c0c5ea0 con 0x7fde6c0c3f30 2023-01-24T03:34:27.801 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.794+0000 7fde6bfff700 1 --2- 172.21.15.112:0/3006020898 >> v2:172.21.15.142:3300/0 conn(0x7fde6c155bf0 0x7fde6c068160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:34:27.801 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.795+0000 7fde713c4700 1 --2- 172.21.15.112:0/3006020898 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fde6c0c3f30 0x7fde6c0c2c40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fde64009e60 tx=0x7fde64009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9988d5c966f651c7 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:27.805 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.796+0000 7fde6b7fe700 1 -- 172.21.15.112:0/3006020898 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6400e070 con 0x7fde6c0c3f30 2023-01-24T03:34:27.805 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.796+0000 7fde6b7fe700 1 -- 172.21.15.112:0/3006020898 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde6400bbf0 con 0x7fde6c0c3f30 2023-01-24T03:34:27.805 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.796+0000 7fde6b7fe700 1 -- 172.21.15.112:0/3006020898 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde64012430 con 0x7fde6c0c3f30 2023-01-24T03:34:27.805 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.796+0000 7fde71bc5700 1 -- 172.21.15.112:0/3006020898 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fde6c0c3f30 msgr2=0x7fde6c0c2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:27.806 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.796+0000 7fde71bc5700 1 --2- 172.21.15.112:0/3006020898 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fde6c0c3f30 0x7fde6c0c2c40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fde64009e60 tx=0x7fde64009780 comp rx=0 tx=0).stop 2023-01-24T03:34:27.806 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.796+0000 7fde71bc5700 1 -- 172.21.15.112:0/3006020898 shutdown_connections 2023-01-24T03:34:27.806 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.796+0000 7fde71bc5700 1 --2- 172.21.15.112:0/3006020898 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fde6c0c3f30 0x7fde6c0c2c40 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:27.806 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.796+0000 7fde71bc5700 1 --2- 172.21.15.112:0/3006020898 >> v2:172.21.15.142:3300/0 conn(0x7fde6c155bf0 0x7fde6c068160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:27.806 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.796+0000 7fde71bc5700 1 -- 172.21.15.112:0/3006020898 >> 172.21.15.112:0/3006020898 conn(0x7fde6c0ca150 msgr2=0x7fde6c0c6120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:27.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.796+0000 7fde71bc5700 1 -- 172.21.15.112:0/3006020898 shutdown_connections 2023-01-24T03:34:27.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.796+0000 7fde71bc5700 1 -- 172.21.15.112:0/3006020898 wait complete. 2023-01-24T03:34:27.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.797+0000 7fde71bc5700 1 Processor -- start 2023-01-24T03:34:27.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.798+0000 7fde71bc5700 1 -- start start 2023-01-24T03:34:27.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.798+0000 7fde71bc5700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fde6c067d90 0x7fde6c152210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:27.808 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.798+0000 7fde71bc5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fde6c0c3f30 0x7fde6c14ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:27.808 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.798+0000 7fde71bc5700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fde6c1659b0 0x7fde6c154750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:27.808 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.798+0000 7fde71bc5700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c159540 con 0x7fde6c067d90 2023-01-24T03:34:27.808 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.798+0000 7fde71bc5700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c165ee0 con 0x7fde6c1659b0 2023-01-24T03:34:27.808 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.798+0000 7fde71bc5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c14d410 con 0x7fde6c0c3f30 2023-01-24T03:34:27.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.798+0000 7fde70bc3700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fde6c067d90 0x7fde6c152210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:27.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.799+0000 7fde70bc3700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fde6c067d90 0x7fde6c152210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52716/0 (socket says 172.21.15.112:52716) 2023-01-24T03:34:27.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.799+0000 7fde70bc3700 1 -- 172.21.15.112:0/1341449675 learned_addr learned my addr 172.21.15.112:0/1341449675 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:34:27.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.799+0000 7fde6bfff700 1 --2- 172.21.15.112:0/1341449675 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fde6c0c3f30 0x7fde6c14ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:27.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.799+0000 7fde713c4700 1 --2- 172.21.15.112:0/1341449675 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fde6c1659b0 0x7fde6c154750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:27.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.799+0000 7fde70bc3700 1 -- 172.21.15.112:0/1341449675 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fde6c0c3f30 msgr2=0x7fde6c14ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:27.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.799+0000 7fde70bc3700 1 --2- 172.21.15.112:0/1341449675 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fde6c0c3f30 0x7fde6c14ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:27.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.799+0000 7fde70bc3700 1 -- 172.21.15.112:0/1341449675 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fde6c1659b0 msgr2=0x7fde6c154750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:27.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.799+0000 7fde70bc3700 1 --2- 172.21.15.112:0/1341449675 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fde6c1659b0 0x7fde6c154750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:27.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.799+0000 7fde70bc3700 1 -- 172.21.15.112:0/1341449675 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde58009370 con 0x7fde6c067d90 2023-01-24T03:34:27.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.799+0000 7fde70bc3700 1 --2- 172.21.15.112:0/1341449675 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fde6c067d90 0x7fde6c152210 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fde6000c4c0 tx=0x7fde60007db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:27.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.799+0000 7fde68ff9700 1 -- 172.21.15.112:0/1341449675 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde600072e0 con 0x7fde6c067d90 2023-01-24T03:34:27.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.799+0000 7fde68ff9700 1 -- 172.21.15.112:0/1341449675 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde60007440 con 0x7fde6c067d90 2023-01-24T03:34:27.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.799+0000 7fde71bc5700 1 -- 172.21.15.112:0/1341449675 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde64009a20 con 0x7fde6c067d90 2023-01-24T03:34:27.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.799+0000 7fde68ff9700 1 -- 172.21.15.112:0/1341449675 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde60004900 con 0x7fde6c067d90 2023-01-24T03:34:27.812 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.799+0000 7fde71bc5700 1 -- 172.21.15.112:0/1341449675 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde6c151880 con 0x7fde6c067d90 2023-01-24T03:34:27.812 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.800+0000 7fde71bc5700 1 -- 172.21.15.112:0/1341449675 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde6c05ea20 con 0x7fde6c067d90 2023-01-24T03:34:27.812 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.800+0000 7fde68ff9700 1 -- 172.21.15.112:0/1341449675 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fde60010070 con 0x7fde6c067d90 2023-01-24T03:34:27.812 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.801+0000 7fde68ff9700 1 --2- 172.21.15.112:0/1341449675 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fde540611c0 0x7fde54063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:27.812 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.801+0000 7fde68ff9700 1 -- 172.21.15.112:0/1341449675 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7039+0+0 (secure 0 0 0) 0x7fde6007cf30 con 0x7fde6c067d90 2023-01-24T03:34:27.813 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.801+0000 7fde6bfff700 1 --2- 172.21.15.112:0/1341449675 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fde540611c0 0x7fde54063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:27.813 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.801+0000 7fde6bfff700 1 --2- 172.21.15.112:0/1341449675 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fde540611c0 0x7fde54063670 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fde6c0c50e0 tx=0x7fde580090f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:27.814 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:27.805+0000 7fde68ff9700 1 -- 172.21.15.112:0/1341449675 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fde600ad1d0 con 0x7fde6c067d90 2023-01-24T03:34:28.027 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:28.024+0000 7fde71bc5700 1 -- 172.21.15.112:0/1341449675 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde6c06a8a0 con 0x7fde6c067d90 2023-01-24T03:34:28.028 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:28.025+0000 7fde68ff9700 1 -- 172.21.15.112:0/1341449675 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+6112 (secure 0 0 0) 0x7fde6004cc90 con 0x7fde6c067d90 2023-01-24T03:34:28.028 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:34:28.029 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5041,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/612058861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":612058861},{"type":"v1","addr":"172.21.15.142:6834","nonce":612058861}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:33:18.651832+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4897},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4897":{"gid":4897,"name":"d","rank":0,"incarnation":16,"state":"up:active","state_seq":19,"addr":"172.21.15.142:6835/11022091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":11022091},{"type":"v1","addr":"172.21.15.142:6835","nonce":11022091}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:34:28.030 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:28.027+0000 7fde71bc5700 1 -- 172.21.15.112:0/1341449675 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fde540611c0 msgr2=0x7fde54063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:28.030 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:28.027+0000 7fde71bc5700 1 --2- 172.21.15.112:0/1341449675 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fde540611c0 0x7fde54063670 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fde6c0c50e0 tx=0x7fde580090f0 comp rx=0 tx=0).stop 2023-01-24T03:34:28.031 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:28.027+0000 7fde71bc5700 1 -- 172.21.15.112:0/1341449675 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fde6c067d90 msgr2=0x7fde6c152210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:28.031 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:28.027+0000 7fde71bc5700 1 --2- 172.21.15.112:0/1341449675 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fde6c067d90 0x7fde6c152210 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fde6000c4c0 tx=0x7fde60007db0 comp rx=0 tx=0).stop 2023-01-24T03:34:28.031 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:28.027+0000 7fde71bc5700 1 -- 172.21.15.112:0/1341449675 shutdown_connections 2023-01-24T03:34:28.031 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:28.027+0000 7fde71bc5700 1 --2- 172.21.15.112:0/1341449675 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fde540611c0 0x7fde54063670 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:28.031 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:28.027+0000 7fde71bc5700 1 --2- 172.21.15.112:0/1341449675 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fde6c067d90 0x7fde6c152210 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:28.031 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:28.027+0000 7fde71bc5700 1 --2- 172.21.15.112:0/1341449675 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fde6c1659b0 0x7fde6c154750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:28.032 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:28.027+0000 7fde71bc5700 1 --2- 172.21.15.112:0/1341449675 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fde6c0c3f30 0x7fde6c14ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:28.032 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:28.027+0000 7fde71bc5700 1 -- 172.21.15.112:0/1341449675 >> 172.21.15.112:0/1341449675 conn(0x7fde6c0ca150 msgr2=0x7fde6c14e9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:28.032 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:28.027+0000 7fde71bc5700 1 -- 172.21.15.112:0/1341449675 shutdown_connections 2023-01-24T03:34:28.032 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:28.027+0000 7fde71bc5700 1 -- 172.21.15.112:0/1341449675 wait complete. 2023-01-24T03:34:28.033 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 21 2023-01-24T03:34:28.041 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-01-24T03:34:29.358 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:34:29.361 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:34:30.042 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:34:30.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.163+0000 7f1578c9c700 1 Processor -- start 2023-01-24T03:34:30.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.163+0000 7f1578c9c700 1 -- start start 2023-01-24T03:34:30.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.164+0000 7f1578c9c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f15740b0aa0 0x7f1574155da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:30.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.164+0000 7f1578c9c700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15740b0f40 con 0x7f15740ca3e0 2023-01-24T03:34:30.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.164+0000 7f1578c9c700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15740b1170 con 0x7f15740c7e40 2023-01-24T03:34:30.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.164+0000 7f1578c9c700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15740b1450 con 0x7f15740b0aa0 2023-01-24T03:34:30.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.164+0000 7f1572d9d700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f15740ca3e0 0x7f15740ca7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:33716/0 (socket says 172.21.15.112:33716) 2023-01-24T03:34:30.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.164+0000 7f1572d9d700 1 -- 172.21.15.112:0/2210260653 learned_addr learned my addr 172.21.15.112:0/2210260653 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:34:30.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.164+0000 7f1571d9b700 1 --2- 172.21.15.112:0/2210260653 >> v2:172.21.15.142:3300/0 conn(0x7f15740b0aa0 0x7f1574155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:30.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.164+0000 7f157159a700 1 -- 172.21.15.112:0/2210260653 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 829476275 0 0) 0x7f15740b0f40 con 0x7f15740ca3e0 2023-01-24T03:34:30.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.164+0000 7f157159a700 1 -- 172.21.15.112:0/2210260653 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15540033f0 con 0x7f15740ca3e0 2023-01-24T03:34:30.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.164+0000 7f157159a700 1 -- 172.21.15.112:0/2210260653 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2688949935 0 0) 0x7f15540033f0 con 0x7f15740ca3e0 2023-01-24T03:34:30.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.164+0000 7f157159a700 1 -- 172.21.15.112:0/2210260653 >> v2:172.21.15.142:3300/0 conn(0x7f15740b0aa0 msgr2=0x7f1574155da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:30.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.164+0000 7f157159a700 1 --2- 172.21.15.112:0/2210260653 >> v2:172.21.15.142:3300/0 conn(0x7f15740b0aa0 0x7f1574155da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:30.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.164+0000 7f157159a700 1 -- 172.21.15.112:0/2210260653 >> v1:172.21.15.142:6789/0 conn(0x7f15740c7e40 legacy=0x7f15740ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:30.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.164+0000 7f157159a700 1 -- 172.21.15.112:0/2210260653 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15740b1730 con 0x7f15740ca3e0 2023-01-24T03:34:30.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.164+0000 7f157159a700 1 -- 172.21.15.112:0/2210260653 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7f1564001470 con 0x7f15740ca3e0 2023-01-24T03:34:30.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.164+0000 7f157159a700 1 -- 172.21.15.112:0/2210260653 >> v1:172.21.15.112:6789/0 conn(0x7f15740ca3e0 legacy=0x7f15740ca7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:30.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.164+0000 7f157159a700 1 --2- 172.21.15.112:0/2210260653 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f15540038e0 0x7f1554003cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:30.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.164+0000 7f157159a700 1 --2- 172.21.15.112:0/2210260653 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1554004690 0x7f1554004b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:30.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.164+0000 7f157159a700 1 --2- 172.21.15.112:0/2210260653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1554005040 0x7f155400b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:30.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.164+0000 7f157159a700 1 -- 172.21.15.112:0/2210260653 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15740b1450 con 0x7f15540038e0 2023-01-24T03:34:30.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.165+0000 7f157159a700 1 -- 172.21.15.112:0/2210260653 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f155400bfe0 con 0x7f1554004690 2023-01-24T03:34:30.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.165+0000 7f157159a700 1 -- 172.21.15.112:0/2210260653 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f155400c1a0 con 0x7f1554005040 2023-01-24T03:34:30.178 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.165+0000 7f157259c700 1 --2- 172.21.15.112:0/2210260653 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f15540038e0 0x7f1554003cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:30.178 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.165+0000 7f1571d9b700 1 --2- 172.21.15.112:0/2210260653 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1554004690 0x7f1554004b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:30.178 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.165+0000 7f1572d9d700 1 --2- 172.21.15.112:0/2210260653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1554005040 0x7f155400b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:30.178 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.165+0000 7f157259c700 1 -- 172.21.15.112:0/2210260653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1554005040 msgr2=0x7f155400b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:30.178 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.165+0000 7f157259c700 1 --2- 172.21.15.112:0/2210260653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1554005040 0x7f155400b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:30.179 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.165+0000 7f157259c700 1 -- 172.21.15.112:0/2210260653 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1554004690 msgr2=0x7f1554004b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:30.179 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.165+0000 7f157259c700 1 --2- 172.21.15.112:0/2210260653 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1554004690 0x7f1554004b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:30.179 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.165+0000 7f157259c700 1 -- 172.21.15.112:0/2210260653 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15540033f0 con 0x7f15540038e0 2023-01-24T03:34:30.179 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.165+0000 7f1572d9d700 1 -- 172.21.15.112:0/2210260653 reap_dead start 2023-01-24T03:34:30.179 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.165+0000 7f157259c700 1 --2- 172.21.15.112:0/2210260653 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f15540038e0 0x7f1554003cf0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f156c001e30 tx=0x7f156c007f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f6a58e5c03cac3 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:30.180 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.165+0000 7f157159a700 1 -- 172.21.15.112:0/2210260653 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156c00c5d0 con 0x7f15540038e0 2023-01-24T03:34:30.180 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.165+0000 7f157159a700 1 -- 172.21.15.112:0/2210260653 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f156c00c750 con 0x7f15540038e0 2023-01-24T03:34:30.180 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.165+0000 7f157159a700 1 -- 172.21.15.112:0/2210260653 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156c00cb10 con 0x7f15540038e0 2023-01-24T03:34:30.180 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.166+0000 7f1578c9c700 1 -- 172.21.15.112:0/2210260653 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f15540038e0 msgr2=0x7f1554003cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:30.180 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.166+0000 7f1578c9c700 1 --2- 172.21.15.112:0/2210260653 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f15540038e0 0x7f1554003cf0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f156c001e30 tx=0x7f156c007f30 comp rx=0 tx=0).stop 2023-01-24T03:34:30.181 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.166+0000 7f1578c9c700 1 -- 172.21.15.112:0/2210260653 shutdown_connections 2023-01-24T03:34:30.181 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.166+0000 7f1578c9c700 1 --2- 172.21.15.112:0/2210260653 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f15540038e0 0x7f1554003cf0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:30.181 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.166+0000 7f1578c9c700 1 -- 172.21.15.112:0/2210260653 >> 172.21.15.112:0/2210260653 conn(0x7f157405e2c0 msgr2=0x7f15740cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:30.181 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.166+0000 7f1578c9c700 1 -- 172.21.15.112:0/2210260653 shutdown_connections 2023-01-24T03:34:30.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.166+0000 7f1578c9c700 1 -- 172.21.15.112:0/2210260653 wait complete. 2023-01-24T03:34:30.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.167+0000 7f1578c9c700 1 Processor -- start 2023-01-24T03:34:30.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.167+0000 7f1578c9c700 1 -- start start 2023-01-24T03:34:30.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.167+0000 7f1578c9c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f15540038e0 0x7f15740c23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:30.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.167+0000 7f1578c9c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15740c5680 0x7f157414fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:30.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.167+0000 7f1578c9c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1574165570 0x7f15741659e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:30.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.167+0000 7f1578c9c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1574152290 con 0x7f15540038e0 2023-01-24T03:34:30.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.167+0000 7f1578c9c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15741510a0 con 0x7f15740c5680 2023-01-24T03:34:30.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.167+0000 7f1578c9c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15740adfd0 con 0x7f1574165570 2023-01-24T03:34:30.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.168+0000 7f157259c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f15540038e0 0x7f15740c23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:30.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.168+0000 7f157259c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f15540038e0 0x7f15740c23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52736/0 (socket says 172.21.15.112:52736) 2023-01-24T03:34:30.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.168+0000 7f157259c700 1 -- 172.21.15.112:0/2293767259 learned_addr learned my addr 172.21.15.112:0/2293767259 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:34:30.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.168+0000 7f1572d9d700 1 --2- 172.21.15.112:0/2293767259 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1574165570 0x7f15741659e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:30.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.168+0000 7f1571d9b700 1 --2- 172.21.15.112:0/2293767259 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15740c5680 0x7f157414fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:30.185 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.168+0000 7f157259c700 1 -- 172.21.15.112:0/2293767259 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1574165570 msgr2=0x7f15741659e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:30.185 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.168+0000 7f157259c700 1 --2- 172.21.15.112:0/2293767259 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1574165570 0x7f15741659e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:30.185 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.168+0000 7f157259c700 1 -- 172.21.15.112:0/2293767259 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15740c5680 msgr2=0x7f157414fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:30.185 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.168+0000 7f157259c700 1 --2- 172.21.15.112:0/2293767259 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15740c5680 0x7f157414fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:30.185 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.168+0000 7f157259c700 1 -- 172.21.15.112:0/2293767259 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1574169600 con 0x7f15540038e0 2023-01-24T03:34:30.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.168+0000 7f157259c700 1 --2- 172.21.15.112:0/2293767259 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f15540038e0 0x7f15740c23d0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f156c00c3c0 tx=0x7f156c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:30.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.168+0000 7f156a7fc700 1 -- 172.21.15.112:0/2293767259 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156c00cec0 con 0x7f15540038e0 2023-01-24T03:34:30.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.168+0000 7f156a7fc700 1 -- 172.21.15.112:0/2293767259 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f156c012b10 con 0x7f15540038e0 2023-01-24T03:34:30.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.168+0000 7f156a7fc700 1 -- 172.21.15.112:0/2293767259 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156c006a20 con 0x7f15540038e0 2023-01-24T03:34:30.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.168+0000 7f1578c9c700 1 -- 172.21.15.112:0/2293767259 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15740ae270 con 0x7f15540038e0 2023-01-24T03:34:30.187 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.168+0000 7f1578c9c700 1 -- 172.21.15.112:0/2293767259 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15741698e0 con 0x7f15540038e0 2023-01-24T03:34:30.187 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.169+0000 7f1578c9c700 1 -- 172.21.15.112:0/2293767259 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1574151660 con 0x7f15540038e0 2023-01-24T03:34:30.187 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.169+0000 7f156a7fc700 1 -- 172.21.15.112:0/2293767259 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f156c010050 con 0x7f15540038e0 2023-01-24T03:34:30.187 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.170+0000 7f156a7fc700 1 --2- 172.21.15.112:0/2293767259 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f15580611c0 0x7f1558063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:30.187 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.170+0000 7f156a7fc700 1 -- 172.21.15.112:0/2293767259 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 7125+0+0 (secure 0 0 0) 0x7f156c07f890 con 0x7f15540038e0 2023-01-24T03:34:30.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.170+0000 7f1571d9b700 1 --2- 172.21.15.112:0/2293767259 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f15580611c0 0x7f1558063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:30.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.170+0000 7f1571d9b700 1 --2- 172.21.15.112:0/2293767259 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f15580611c0 0x7f1558063670 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1574157510 tx=0x7f155c009060 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:30.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.174+0000 7f156a7fc700 1 -- 172.21.15.112:0/2293767259 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f156c04f590 con 0x7f15540038e0 2023-01-24T03:34:30.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.393+0000 7f1578c9c700 1 -- 172.21.15.112:0/2293767259 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15741514b0 con 0x7f15540038e0 2023-01-24T03:34:30.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.393+0000 7f156a7fc700 1 -- 172.21.15.112:0/2293767259 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5357 (secure 0 0 0) 0x7f156c052bb0 con 0x7f15540038e0 2023-01-24T03:34:30.397 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:34:30.397 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:34:28.624819+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"b","rank":0,"incarnation":22,"state":"up:replay","state_seq":1,"addr":"172.21.15.142:6834/612058861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":612058861},{"type":"v1","addr":"172.21.15.142:6834","nonce":612058861}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:34:30.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.396+0000 7f1578c9c700 1 -- 172.21.15.112:0/2293767259 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f15580611c0 msgr2=0x7f1558063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:30.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.396+0000 7f1578c9c700 1 --2- 172.21.15.112:0/2293767259 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f15580611c0 0x7f1558063670 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1574157510 tx=0x7f155c009060 comp rx=0 tx=0).stop 2023-01-24T03:34:30.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.396+0000 7f1578c9c700 1 -- 172.21.15.112:0/2293767259 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f15540038e0 msgr2=0x7f15740c23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:30.400 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.396+0000 7f1578c9c700 1 --2- 172.21.15.112:0/2293767259 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f15540038e0 0x7f15740c23d0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f156c00c3c0 tx=0x7f156c016000 comp rx=0 tx=0).stop 2023-01-24T03:34:30.400 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.396+0000 7f1578c9c700 1 -- 172.21.15.112:0/2293767259 shutdown_connections 2023-01-24T03:34:30.400 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.396+0000 7f1578c9c700 1 --2- 172.21.15.112:0/2293767259 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f15580611c0 0x7f1558063670 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:30.400 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.396+0000 7f1578c9c700 1 --2- 172.21.15.112:0/2293767259 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f15540038e0 0x7f15740c23d0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:30.400 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.396+0000 7f1578c9c700 1 --2- 172.21.15.112:0/2293767259 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1574165570 0x7f15741659e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:30.401 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.396+0000 7f1578c9c700 1 --2- 172.21.15.112:0/2293767259 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15740c5680 0x7f157414fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:30.401 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.396+0000 7f1578c9c700 1 -- 172.21.15.112:0/2293767259 >> 172.21.15.112:0/2293767259 conn(0x7f157405e2c0 msgr2=0x7f15740ae9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:30.401 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.396+0000 7f1578c9c700 1 -- 172.21.15.112:0/2293767259 shutdown_connections 2023-01-24T03:34:30.401 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.396+0000 7f1578c9c700 1 -- 172.21.15.112:0/2293767259 wait complete. 2023-01-24T03:34:30.402 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 22 2023-01-24T03:34:30.410 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2023-01-24T03:34:30.410 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-01-24T03:34:30.411 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:34:30.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.518+0000 7fb803f80700 1 Processor -- start 2023-01-24T03:34:30.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.519+0000 7fb803f80700 1 -- start start 2023-01-24T03:34:30.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.519+0000 7fb803f80700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb7fc0c4790 0x7fb7fc0c4f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:30.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.519+0000 7fb803f80700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7fc05df40 con 0x7fb7fc14f2c0 2023-01-24T03:34:30.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.519+0000 7fb803f80700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7fc155920 con 0x7fb7fc0c40e0 2023-01-24T03:34:30.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.519+0000 7fb803f80700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7fc1552f0 con 0x7fb7fc0c4790 2023-01-24T03:34:30.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.519+0000 7fb801d1c700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fb7fc14f2c0 0x7fb7fc067f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:33732/0 (socket says 172.21.15.112:33732) 2023-01-24T03:34:30.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.519+0000 7fb801d1c700 1 -- 172.21.15.112:0/3102147096 learned_addr learned my addr 172.21.15.112:0/3102147096 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:34:30.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.520+0000 7fb80251d700 1 --2- 172.21.15.112:0/3102147096 >> v2:172.21.15.142:3300/0 conn(0x7fb7fc0c4790 0x7fb7fc0c4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:30.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.520+0000 7fb800d1a700 1 -- 172.21.15.112:0/3102147096 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4212379647 0 0) 0x7fb7fc05df40 con 0x7fb7fc14f2c0 2023-01-24T03:34:30.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.520+0000 7fb800d1a700 1 -- 172.21.15.112:0/3102147096 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7e80033f0 con 0x7fb7fc14f2c0 2023-01-24T03:34:30.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.520+0000 7fb800d1a700 1 -- 172.21.15.112:0/3102147096 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3167142954 0 0) 0x7fb7e80033f0 con 0x7fb7fc14f2c0 2023-01-24T03:34:30.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.520+0000 7fb800d1a700 1 -- 172.21.15.112:0/3102147096 >> v2:172.21.15.142:3300/0 conn(0x7fb7fc0c4790 msgr2=0x7fb7fc0c4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:30.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.520+0000 7fb800d1a700 1 --2- 172.21.15.112:0/3102147096 >> v2:172.21.15.142:3300/0 conn(0x7fb7fc0c4790 0x7fb7fc0c4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:30.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.520+0000 7fb800d1a700 1 -- 172.21.15.112:0/3102147096 >> v1:172.21.15.142:6789/0 conn(0x7fb7fc0c40e0 legacy=0x7fb7fc15dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:30.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.520+0000 7fb800d1a700 1 -- 172.21.15.112:0/3102147096 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7fc05e190 con 0x7fb7fc14f2c0 2023-01-24T03:34:30.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.520+0000 7fb800d1a700 1 -- 172.21.15.112:0/3102147096 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7fb7ec0017f0 con 0x7fb7fc14f2c0 2023-01-24T03:34:30.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.520+0000 7fb800d1a700 1 -- 172.21.15.112:0/3102147096 >> v1:172.21.15.112:6789/0 conn(0x7fb7fc14f2c0 legacy=0x7fb7fc067f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:30.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.520+0000 7fb800d1a700 1 --2- 172.21.15.112:0/3102147096 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb7e80038e0 0x7fb7e8003cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:30.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.520+0000 7fb800d1a700 1 --2- 172.21.15.112:0/3102147096 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7e8004690 0x7fb7e8004b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:30.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.520+0000 7fb800d1a700 1 --2- 172.21.15.112:0/3102147096 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb7e8005040 0x7fb7e800b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:30.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.520+0000 7fb800d1a700 1 -- 172.21.15.112:0/3102147096 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7fc1552f0 con 0x7fb7e8005040 2023-01-24T03:34:30.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.520+0000 7fb800d1a700 1 -- 172.21.15.112:0/3102147096 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e800bf50 con 0x7fb7e8004690 2023-01-24T03:34:30.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.520+0000 7fb800d1a700 1 -- 172.21.15.112:0/3102147096 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e800c110 con 0x7fb7e80038e0 2023-01-24T03:34:30.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.520+0000 7fb80251d700 1 --2- 172.21.15.112:0/3102147096 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb7e8005040 0x7fb7e800b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:30.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.520+0000 7fb801d1c700 1 --2- 172.21.15.112:0/3102147096 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb7e80038e0 0x7fb7e8003cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:30.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.520+0000 7fb80151b700 1 --2- 172.21.15.112:0/3102147096 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7e8004690 0x7fb7e8004b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:30.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.520+0000 7fb80251d700 1 -- 172.21.15.112:0/3102147096 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb7e80038e0 msgr2=0x7fb7e8003cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:30.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.520+0000 7fb80251d700 1 --2- 172.21.15.112:0/3102147096 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb7e80038e0 0x7fb7e8003cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:30.533 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.520+0000 7fb80251d700 1 -- 172.21.15.112:0/3102147096 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7e8004690 msgr2=0x7fb7e8004b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:30.533 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.520+0000 7fb80251d700 1 --2- 172.21.15.112:0/3102147096 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7e8004690 0x7fb7e8004b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:30.533 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.520+0000 7fb80251d700 1 -- 172.21.15.112:0/3102147096 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e80033f0 con 0x7fb7e8005040 2023-01-24T03:34:30.533 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.521+0000 7fb80251d700 1 -- 172.21.15.112:0/3102147096 reap_dead start 2023-01-24T03:34:30.533 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.521+0000 7fb80251d700 1 --2- 172.21.15.112:0/3102147096 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb7e8005040 0x7fb7e800b8e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb7f4009a80 tx=0x7fb7f4009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3c23d52fb8a8c48 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:30.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.521+0000 7fb800d1a700 1 -- 172.21.15.112:0/3102147096 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f4009430 con 0x7fb7e8005040 2023-01-24T03:34:30.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.521+0000 7fb800d1a700 1 -- 172.21.15.112:0/3102147096 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7f400e640 con 0x7fb7e8005040 2023-01-24T03:34:30.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.521+0000 7fb800d1a700 1 -- 172.21.15.112:0/3102147096 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f400ea00 con 0x7fb7e8005040 2023-01-24T03:34:30.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.521+0000 7fb803f80700 1 -- 172.21.15.112:0/3102147096 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb7e8005040 msgr2=0x7fb7e800b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:30.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.521+0000 7fb803f80700 1 --2- 172.21.15.112:0/3102147096 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb7e8005040 0x7fb7e800b8e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb7f4009a80 tx=0x7fb7f4009060 comp rx=0 tx=0).stop 2023-01-24T03:34:30.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.521+0000 7fb803f80700 1 -- 172.21.15.112:0/3102147096 shutdown_connections 2023-01-24T03:34:30.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.521+0000 7fb803f80700 1 --2- 172.21.15.112:0/3102147096 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb7e8005040 0x7fb7e800b8e0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:30.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.521+0000 7fb803f80700 1 -- 172.21.15.112:0/3102147096 >> 172.21.15.112:0/3102147096 conn(0x7fb7fc06daf0 msgr2=0x7fb7fc05d830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:30.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.521+0000 7fb803f80700 1 -- 172.21.15.112:0/3102147096 shutdown_connections 2023-01-24T03:34:30.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.521+0000 7fb803f80700 1 -- 172.21.15.112:0/3102147096 wait complete. 2023-01-24T03:34:30.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.522+0000 7fb803f80700 1 Processor -- start 2023-01-24T03:34:30.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.522+0000 7fb803f80700 1 -- start start 2023-01-24T03:34:30.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.523+0000 7fb803f80700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb7e8005040 0x7fb7fc159ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:30.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.523+0000 7fb803f80700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7fc166160 0x7fb7fc156680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:30.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.523+0000 7fb803f80700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb7fc06b3a0 0x7fb7fc168a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:30.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.523+0000 7fb803f80700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7fc0c9ae0 con 0x7fb7e8005040 2023-01-24T03:34:30.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.523+0000 7fb803f80700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7fc15b850 con 0x7fb7fc166160 2023-01-24T03:34:30.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.523+0000 7fb803f80700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7fc165cd0 con 0x7fb7fc06b3a0 2023-01-24T03:34:30.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.523+0000 7fb801d1c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb7e8005040 0x7fb7fc159ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:30.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.523+0000 7fb801d1c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb7e8005040 0x7fb7fc159ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52756/0 (socket says 172.21.15.112:52756) 2023-01-24T03:34:30.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.523+0000 7fb801d1c700 1 -- 172.21.15.112:0/3560115850 learned_addr learned my addr 172.21.15.112:0/3560115850 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:34:30.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.523+0000 7fb80151b700 1 --2- 172.21.15.112:0/3560115850 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7fc166160 0x7fb7fc156680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:30.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.523+0000 7fb80251d700 1 --2- 172.21.15.112:0/3560115850 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb7fc06b3a0 0x7fb7fc168a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:30.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.523+0000 7fb801d1c700 1 -- 172.21.15.112:0/3560115850 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb7fc06b3a0 msgr2=0x7fb7fc168a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:30.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.523+0000 7fb801d1c700 1 --2- 172.21.15.112:0/3560115850 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb7fc06b3a0 0x7fb7fc168a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:30.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.523+0000 7fb801d1c700 1 -- 172.21.15.112:0/3560115850 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7fc166160 msgr2=0x7fb7fc156680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:30.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.523+0000 7fb801d1c700 1 --2- 172.21.15.112:0/3560115850 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7fc166160 0x7fb7fc156680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:30.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.523+0000 7fb801d1c700 1 -- 172.21.15.112:0/3560115850 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7fc15b290 con 0x7fb7e8005040 2023-01-24T03:34:30.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.523+0000 7fb80151b700 1 --2- 172.21.15.112:0/3560115850 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7fc166160 0x7fb7fc156680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-01-24T03:34:30.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.523+0000 7fb801d1c700 1 --2- 172.21.15.112:0/3560115850 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb7e8005040 0x7fb7fc159ff0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec007690 tx=0x7fb7ec001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:30.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.524+0000 7fb7f9ffb700 1 -- 172.21.15.112:0/3560115850 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7ec00fae0 con 0x7fb7e8005040 2023-01-24T03:34:30.540 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.524+0000 7fb7f9ffb700 1 -- 172.21.15.112:0/3560115850 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7ec00fc60 con 0x7fb7e8005040 2023-01-24T03:34:30.540 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.524+0000 7fb803f80700 1 -- 172.21.15.112:0/3560115850 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7fc15b570 con 0x7fb7e8005040 2023-01-24T03:34:30.540 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.524+0000 7fb7f9ffb700 1 -- 172.21.15.112:0/3560115850 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7ec0133b0 con 0x7fb7e8005040 2023-01-24T03:34:30.540 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.524+0000 7fb803f80700 1 -- 172.21.15.112:0/3560115850 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7fc165960 con 0x7fb7e8005040 2023-01-24T03:34:30.540 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.525+0000 7fb803f80700 1 -- 172.21.15.112:0/3560115850 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7fc165b20 con 0x7fb7e8005040 2023-01-24T03:34:30.541 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.525+0000 7fb7f9ffb700 1 -- 172.21.15.112:0/3560115850 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fb7ec01c030 con 0x7fb7e8005040 2023-01-24T03:34:30.542 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.526+0000 7fb7f9ffb700 1 --2- 172.21.15.112:0/3560115850 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb7dc061240 0x7fb7dc0636f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:30.542 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.526+0000 7fb7f9ffb700 1 -- 172.21.15.112:0/3560115850 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 7125+0+0 (secure 0 0 0) 0x7fb7ec07da30 con 0x7fb7e8005040 2023-01-24T03:34:30.542 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.526+0000 7fb80151b700 1 --2- 172.21.15.112:0/3560115850 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb7dc061240 0x7fb7dc0636f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:30.542 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.526+0000 7fb80151b700 1 --2- 172.21.15.112:0/3560115850 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb7dc061240 0x7fb7dc0636f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0000f60 tx=0x7fb7f0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:30.543 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.529+0000 7fb7f9ffb700 1 -- 172.21.15.112:0/3560115850 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fb7ec04d6c0 con 0x7fb7e8005040 2023-01-24T03:34:30.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.748+0000 7fb803f80700 1 -- 172.21.15.112:0/3560115850 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7fc1654e0 con 0x7fb7e8005040 2023-01-24T03:34:30.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.748+0000 7fb7f9ffb700 1 -- 172.21.15.112:0/3560115850 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5357 (secure 0 0 0) 0x7fb7fc1654e0 con 0x7fb7e8005040 2023-01-24T03:34:30.752 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:34:30.752 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:34:28.624819+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"b","rank":0,"incarnation":22,"state":"up:replay","state_seq":1,"addr":"172.21.15.142:6834/612058861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":612058861},{"type":"v1","addr":"172.21.15.142:6834","nonce":612058861}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:34:30.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.751+0000 7fb803f80700 1 -- 172.21.15.112:0/3560115850 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb7dc061240 msgr2=0x7fb7dc0636f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:30.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.751+0000 7fb803f80700 1 --2- 172.21.15.112:0/3560115850 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb7dc061240 0x7fb7dc0636f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0000f60 tx=0x7fb7f0009040 comp rx=0 tx=0).stop 2023-01-24T03:34:30.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.751+0000 7fb803f80700 1 -- 172.21.15.112:0/3560115850 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb7e8005040 msgr2=0x7fb7fc159ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:30.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.751+0000 7fb803f80700 1 --2- 172.21.15.112:0/3560115850 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb7e8005040 0x7fb7fc159ff0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec007690 tx=0x7fb7ec001250 comp rx=0 tx=0).stop 2023-01-24T03:34:30.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.751+0000 7fb803f80700 1 -- 172.21.15.112:0/3560115850 shutdown_connections 2023-01-24T03:34:30.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.751+0000 7fb803f80700 1 --2- 172.21.15.112:0/3560115850 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb7dc061240 0x7fb7dc0636f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:30.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.751+0000 7fb803f80700 1 --2- 172.21.15.112:0/3560115850 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb7e8005040 0x7fb7fc159ff0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:30.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.751+0000 7fb803f80700 1 --2- 172.21.15.112:0/3560115850 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb7fc06b3a0 0x7fb7fc168a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:30.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.751+0000 7fb803f80700 1 --2- 172.21.15.112:0/3560115850 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb7fc166160 0x7fb7fc156680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:30.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.751+0000 7fb803f80700 1 -- 172.21.15.112:0/3560115850 >> 172.21.15.112:0/3560115850 conn(0x7fb7fc06daf0 msgr2=0x7fb7fc05d830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:30.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.751+0000 7fb803f80700 1 -- 172.21.15.112:0/3560115850 shutdown_connections 2023-01-24T03:34:30.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:30.751+0000 7fb803f80700 1 -- 172.21.15.112:0/3560115850 wait complete. 2023-01-24T03:34:30.757 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 22 2023-01-24T03:34:30.765 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 22, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-01-24T03:30:20.971710+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 22, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5041": { "addr": "172.21.15.142:6834/612058861", "addrs": { "addrvec": [ { "addr": "172.21.15.142:6832", "nonce": 612058861, "type": "v2" }, { "addr": "172.21.15.142:6834", "nonce": 612058861, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5041, "incarnation": 22, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 32, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-01-24T03:34:28.624819+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5041 } } } ], "standbys": [ { "addr": "172.21.15.142:6837/2181345869", "addrs": { "addrvec": [ { "addr": "172.21.15.142:6836", "nonce": 2181345869, "type": "v2" }, { "addr": "172.21.15.142:6837", "nonce": 2181345869, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6835/330343927", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6834", "nonce": 330343927, "type": "v2" }, { "addr": "172.21.15.112:6835", "nonce": 330343927, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6837/12176817", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6836", "nonce": 12176817, "type": "v2" }, { "addr": "172.21.15.112:6837", "nonce": 12176817, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4276, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6839/1110983841", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6838", "nonce": 1110983841, "type": "v2" }, { "addr": "172.21.15.112:6839", "nonce": 1110983841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4281, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-01-24T03:34:32.766 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:34:32.892 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.889+0000 7f390a310700 1 Processor -- start 2023-01-24T03:34:32.893 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.889+0000 7f390a310700 1 -- start start 2023-01-24T03:34:32.898 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.890+0000 7f390a310700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39040caaf0 0x7f3904155da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:32.898 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.890+0000 7f390a310700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39040b0f40 con 0x7f39040c7e40 2023-01-24T03:34:32.898 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.890+0000 7f390a310700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39040b1170 con 0x7f39040b0aa0 2023-01-24T03:34:32.898 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.890+0000 7f390a310700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39040b1450 con 0x7f39040caaf0 2023-01-24T03:34:32.899 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.890+0000 7f3903fff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f39040c7e40 0x7f39040ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:33756/0 (socket says 172.21.15.112:33756) 2023-01-24T03:34:32.899 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.890+0000 7f3903fff700 1 -- 172.21.15.112:0/4194104653 learned_addr learned my addr 172.21.15.112:0/4194104653 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:34:32.899 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.890+0000 7f39088ad700 1 --2- 172.21.15.112:0/4194104653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39040caaf0 0x7f3904155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:32.900 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.890+0000 7f3902ffd700 1 -- 172.21.15.112:0/4194104653 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1641984089 0 0) 0x7f39040b0f40 con 0x7f39040c7e40 2023-01-24T03:34:32.900 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.890+0000 7f3902ffd700 1 -- 172.21.15.112:0/4194104653 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38f00033f0 con 0x7f39040c7e40 2023-01-24T03:34:32.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.891+0000 7f3902ffd700 1 -- 172.21.15.112:0/4194104653 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2778656048 0 0) 0x7f38f00033f0 con 0x7f39040c7e40 2023-01-24T03:34:32.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.891+0000 7f3902ffd700 1 -- 172.21.15.112:0/4194104653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39040caaf0 msgr2=0x7f3904155da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:32.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.891+0000 7f3902ffd700 1 --2- 172.21.15.112:0/4194104653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39040caaf0 0x7f3904155da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:32.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.891+0000 7f3902ffd700 1 -- 172.21.15.112:0/4194104653 >> v1:172.21.15.142:6789/0 conn(0x7f39040b0aa0 legacy=0x7f39040ca3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:32.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.891+0000 7f3902ffd700 1 -- 172.21.15.112:0/4194104653 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39040b1730 con 0x7f39040c7e40 2023-01-24T03:34:32.902 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.891+0000 7f3902ffd700 1 -- 172.21.15.112:0/4194104653 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7f38f4001470 con 0x7f39040c7e40 2023-01-24T03:34:32.902 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.891+0000 7f3902ffd700 1 -- 172.21.15.112:0/4194104653 >> v1:172.21.15.112:6789/0 conn(0x7f39040c7e40 legacy=0x7f39040ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:32.902 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.891+0000 7f3902ffd700 1 --2- 172.21.15.112:0/4194104653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39040caaf0 0x7f38f0003910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:32.902 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.891+0000 7f3902ffd700 1 --2- 172.21.15.112:0/4194104653 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f38f0003e50 0x7f38f0004690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:32.902 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.891+0000 7f3902ffd700 1 --2- 172.21.15.112:0/4194104653 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f38f0004bd0 0x7f38f0005040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:32.903 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.891+0000 7f3902ffd700 1 -- 172.21.15.112:0/4194104653 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39040b1450 con 0x7f38f0003e50 2023-01-24T03:34:32.903 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.891+0000 7f3902ffd700 1 -- 172.21.15.112:0/4194104653 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f000b9c0 con 0x7f38f0004bd0 2023-01-24T03:34:32.903 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.891+0000 7f3902ffd700 1 -- 172.21.15.112:0/4194104653 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38f000bb80 con 0x7f39040caaf0 2023-01-24T03:34:32.903 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.891+0000 7f39037fe700 1 --2- 172.21.15.112:0/4194104653 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f38f0003e50 0x7f38f0004690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:32.903 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.891+0000 7f3903fff700 1 --2- 172.21.15.112:0/4194104653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39040caaf0 0x7f38f0003910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:32.904 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.891+0000 7f39088ad700 1 --2- 172.21.15.112:0/4194104653 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f38f0004bd0 0x7f38f0005040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:32.904 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.891+0000 7f39037fe700 1 -- 172.21.15.112:0/4194104653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39040caaf0 msgr2=0x7f38f0003910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:32.904 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.891+0000 7f39037fe700 1 --2- 172.21.15.112:0/4194104653 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f39040caaf0 0x7f38f0003910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:32.904 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.891+0000 7f39037fe700 1 -- 172.21.15.112:0/4194104653 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f38f0004bd0 msgr2=0x7f38f0005040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:32.904 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.891+0000 7f39037fe700 1 --2- 172.21.15.112:0/4194104653 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f38f0004bd0 0x7f38f0005040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:32.905 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.891+0000 7f39037fe700 1 -- 172.21.15.112:0/4194104653 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f00033f0 con 0x7f38f0003e50 2023-01-24T03:34:32.905 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.891+0000 7f39088ad700 1 --2- 172.21.15.112:0/4194104653 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f38f0004bd0 0x7f38f0005040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:34:32.905 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.891+0000 7f39037fe700 1 --2- 172.21.15.112:0/4194104653 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f38f0003e50 0x7f38f0004690 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f38f8001820 tx=0x7f38f8007b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e0d9f30886832c6 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:32.905 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.891+0000 7f3902ffd700 1 -- 172.21.15.112:0/4194104653 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f8007ea0 con 0x7f38f0003e50 2023-01-24T03:34:32.905 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.891+0000 7f3902ffd700 1 -- 172.21.15.112:0/4194104653 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38f8012420 con 0x7f38f0003e50 2023-01-24T03:34:32.906 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.891+0000 7f3902ffd700 1 -- 172.21.15.112:0/4194104653 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f80127c0 con 0x7f38f0003e50 2023-01-24T03:34:32.906 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.892+0000 7f390a310700 1 -- 172.21.15.112:0/4194104653 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f38f0003e50 msgr2=0x7f38f0004690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:32.906 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.892+0000 7f390a310700 1 --2- 172.21.15.112:0/4194104653 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f38f0003e50 0x7f38f0004690 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f38f8001820 tx=0x7f38f8007b90 comp rx=0 tx=0).stop 2023-01-24T03:34:32.906 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.892+0000 7f39088ad700 1 -- 172.21.15.112:0/4194104653 reap_dead start 2023-01-24T03:34:32.906 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.892+0000 7f390a310700 1 -- 172.21.15.112:0/4194104653 shutdown_connections 2023-01-24T03:34:32.907 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.892+0000 7f390a310700 1 -- 172.21.15.112:0/4194104653 >> 172.21.15.112:0/4194104653 conn(0x7f390405e2c0 msgr2=0x7f39040cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:32.907 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.892+0000 7f390a310700 1 -- 172.21.15.112:0/4194104653 shutdown_connections 2023-01-24T03:34:32.907 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.892+0000 7f390a310700 1 -- 172.21.15.112:0/4194104653 wait complete. 2023-01-24T03:34:32.907 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.893+0000 7f390a310700 1 Processor -- start 2023-01-24T03:34:32.907 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.893+0000 7f390a310700 1 -- start start 2023-01-24T03:34:32.908 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.894+0000 7f390a310700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f38f0003e50 0x7f3904150760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:32.908 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.894+0000 7f390a310700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3904152a90 0x7f3904165440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:32.908 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.894+0000 7f390a310700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39040c09d0 0x7f39040c4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:32.908 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.894+0000 7f390a310700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39040c1070 con 0x7f39040c09d0 2023-01-24T03:34:32.908 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.894+0000 7f390a310700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3904165980 con 0x7f38f0003e50 2023-01-24T03:34:32.909 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.894+0000 7f390a310700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39041502d0 con 0x7f3904152a90 2023-01-24T03:34:32.909 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.894+0000 7f39088ad700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39040c09d0 0x7f39040c4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:32.909 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.894+0000 7f3903fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f38f0003e50 0x7f3904150760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:32.909 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.894+0000 7f39037fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3904152a90 0x7f3904165440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:32.909 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.894+0000 7f3903fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f38f0003e50 0x7f3904150760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.112:45274/0 (socket says 172.21.15.112:45274) 2023-01-24T03:34:32.910 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.894+0000 7f39088ad700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39040c09d0 0x7f39040c4de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52784/0 (socket says 172.21.15.112:52784) 2023-01-24T03:34:32.910 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.894+0000 7f3903fff700 1 -- 172.21.15.112:0/2873536032 learned_addr learned my addr 172.21.15.112:0/2873536032 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:34:32.910 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.894+0000 7f39088ad700 1 -- 172.21.15.112:0/2873536032 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3904152a90 msgr2=0x7f3904165440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:32.910 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.894+0000 7f39088ad700 1 --2- 172.21.15.112:0/2873536032 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3904152a90 0x7f3904165440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:32.910 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.894+0000 7f39088ad700 1 -- 172.21.15.112:0/2873536032 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f38f0003e50 msgr2=0x7f3904150760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:32.911 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.894+0000 7f39088ad700 1 --2- 172.21.15.112:0/2873536032 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f38f0003e50 0x7f3904150760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:32.911 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.894+0000 7f39088ad700 1 -- 172.21.15.112:0/2873536032 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39040c1280 con 0x7f39040c09d0 2023-01-24T03:34:32.911 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.894+0000 7f39088ad700 1 --2- 172.21.15.112:0/2873536032 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39040c09d0 0x7f39040c4de0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f38f000bd10 tx=0x7f38fc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:32.911 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.894+0000 7f38ebfff700 1 -- 172.21.15.112:0/2873536032 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc01b070 con 0x7f39040c09d0 2023-01-24T03:34:32.911 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.895+0000 7f38ebfff700 1 -- 172.21.15.112:0/2873536032 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38fc00f040 con 0x7f39040c09d0 2023-01-24T03:34:32.912 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.895+0000 7f38ebfff700 1 -- 172.21.15.112:0/2873536032 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc00e050 con 0x7f39040c09d0 2023-01-24T03:34:32.912 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.895+0000 7f390a310700 1 -- 172.21.15.112:0/2873536032 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39040c1560 con 0x7f39040c09d0 2023-01-24T03:34:32.912 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.895+0000 7f390a310700 1 -- 172.21.15.112:0/2873536032 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f390414ff60 con 0x7f39040c09d0 2023-01-24T03:34:32.913 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.896+0000 7f390a310700 1 -- 172.21.15.112:0/2873536032 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f390404bb80 con 0x7f39040c09d0 2023-01-24T03:34:32.913 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.896+0000 7f38ebfff700 1 -- 172.21.15.112:0/2873536032 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f38fc015440 con 0x7f39040c09d0 2023-01-24T03:34:32.914 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.896+0000 7f38ebfff700 1 --2- 172.21.15.112:0/2873536032 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f38e40611f0 0x7f38e40636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:32.914 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.896+0000 7f38ebfff700 1 -- 172.21.15.112:0/2873536032 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 7125+0+0 (secure 0 0 0) 0x7f38fc07e310 con 0x7f39040c09d0 2023-01-24T03:34:32.914 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.896+0000 7f3903fff700 1 --2- 172.21.15.112:0/2873536032 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f38e40611f0 0x7f38e40636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:32.914 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.897+0000 7f3903fff700 1 --2- 172.21.15.112:0/2873536032 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f38e40611f0 0x7f38e40636a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f38f4004710 tx=0x7f38f4001e70 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:32.914 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:32.900+0000 7f38ebfff700 1 -- 172.21.15.112:0/2873536032 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f38fc04dfb0 con 0x7f39040c09d0 2023-01-24T03:34:33.127 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:33.124+0000 7f390a310700 1 -- 172.21.15.112:0/2873536032 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3904150410 con 0x7f39040c09d0 2023-01-24T03:34:33.128 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:33.125+0000 7f38ebfff700 1 -- 172.21.15.112:0/2873536032 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5357 (secure 0 0 0) 0x7f38fc0515d0 con 0x7f39040c09d0 2023-01-24T03:34:33.128 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:34:33.129 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:34:28.624819+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"b","rank":0,"incarnation":22,"state":"up:replay","state_seq":1,"addr":"172.21.15.142:6834/612058861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":612058861},{"type":"v1","addr":"172.21.15.142:6834","nonce":612058861}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:34:33.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:33.127+0000 7f390a310700 1 -- 172.21.15.112:0/2873536032 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f38e40611f0 msgr2=0x7f38e40636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:33.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:33.127+0000 7f390a310700 1 --2- 172.21.15.112:0/2873536032 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f38e40611f0 0x7f38e40636a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f38f4004710 tx=0x7f38f4001e70 comp rx=0 tx=0).stop 2023-01-24T03:34:33.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:33.127+0000 7f390a310700 1 -- 172.21.15.112:0/2873536032 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39040c09d0 msgr2=0x7f39040c4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:33.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:33.127+0000 7f390a310700 1 --2- 172.21.15.112:0/2873536032 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39040c09d0 0x7f39040c4de0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f38f000bd10 tx=0x7f38fc00b040 comp rx=0 tx=0).stop 2023-01-24T03:34:33.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:33.127+0000 7f390a310700 1 -- 172.21.15.112:0/2873536032 shutdown_connections 2023-01-24T03:34:33.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:33.127+0000 7f390a310700 1 --2- 172.21.15.112:0/2873536032 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f38e40611f0 0x7f38e40636a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:33.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:33.127+0000 7f390a310700 1 --2- 172.21.15.112:0/2873536032 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f39040c09d0 0x7f39040c4de0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:33.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:33.127+0000 7f390a310700 1 --2- 172.21.15.112:0/2873536032 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f38f0003e50 0x7f3904150760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:33.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:33.127+0000 7f390a310700 1 --2- 172.21.15.112:0/2873536032 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3904152a90 0x7f3904165440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:33.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:33.127+0000 7f390a310700 1 -- 172.21.15.112:0/2873536032 >> 172.21.15.112:0/2873536032 conn(0x7f390405e2c0 msgr2=0x7f39040c60d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:33.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:33.127+0000 7f390a310700 1 -- 172.21.15.112:0/2873536032 shutdown_connections 2023-01-24T03:34:33.133 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:33.127+0000 7f390a310700 1 -- 172.21.15.112:0/2873536032 wait complete. 2023-01-24T03:34:33.133 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 22 2023-01-24T03:34:33.141 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:34:35.143 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:34:35.267 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.264+0000 7f673aba0700 1 Processor -- start 2023-01-24T03:34:35.268 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.264+0000 7f673aba0700 1 -- start start 2023-01-24T03:34:35.277 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.265+0000 7f673aba0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f67340c7e40 0x7f67340b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:35.277 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.265+0000 7f673aba0700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f6734155da0 0x7f67340ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:35.278 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.265+0000 7f673aba0700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6734156210 con 0x7f67340aef20 2023-01-24T03:34:35.278 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.265+0000 7f673aba0700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67340b1170 con 0x7f6734155da0 2023-01-24T03:34:35.279 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.265+0000 7f673aba0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67340b1450 con 0x7f67340c7e40 2023-01-24T03:34:35.279 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.265+0000 7f6733fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f6734155da0 0x7f67340ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:35.279 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.265+0000 7f6733fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f6734155da0 0x7f67340ae9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.112:45288/0 (socket says 172.21.15.112:45288) 2023-01-24T03:34:35.280 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.265+0000 7f6733fff700 1 -- 172.21.15.112:0/1022141972 learned_addr learned my addr 172.21.15.112:0/1022141972 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:34:35.281 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.265+0000 7f673893c700 1 --2- 172.21.15.112:0/1022141972 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f67340c7e40 0x7f67340b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:35.281 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.266+0000 7f6733fff700 1 -- 172.21.15.112:0/1022141972 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f67340c7e40 msgr2=0x7f67340b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:35.281 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.266+0000 7f6733fff700 1 --2- 172.21.15.112:0/1022141972 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f67340c7e40 0x7f67340b0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:35.282 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.266+0000 7f6733fff700 1 -- 172.21.15.112:0/1022141972 >> v1:172.21.15.142:6789/0 conn(0x7f67340aef20 legacy=0x7f67340ca3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:35.282 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.266+0000 7f6733fff700 1 -- 172.21.15.112:0/1022141972 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67340b1730 con 0x7f6734155da0 2023-01-24T03:34:35.282 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.266+0000 7f67337fe700 1 -- 172.21.15.112:0/1022141972 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2034716076 0 0) 0x7f6734156210 con 0x7f67340aef20 2023-01-24T03:34:35.283 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.266+0000 7f6733fff700 1 --2- 172.21.15.112:0/1022141972 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6734155da0 0x7f67340ae9e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6728006fd0 tx=0x7f67280090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=345c7d9ef70565f8 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:35.283 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.266+0000 7f67337fe700 1 -- 172.21.15.112:0/1022141972 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6728011630 con 0x7f6734155da0 2023-01-24T03:34:35.284 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.266+0000 7f67337fe700 1 -- 172.21.15.112:0/1022141972 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67280117d0 con 0x7f6734155da0 2023-01-24T03:34:35.284 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.266+0000 7f67337fe700 1 -- 172.21.15.112:0/1022141972 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6728011b90 con 0x7f6734155da0 2023-01-24T03:34:35.284 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.266+0000 7f673aba0700 1 -- 172.21.15.112:0/1022141972 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6734155da0 msgr2=0x7f67340ae9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:35.285 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.266+0000 7f673aba0700 1 --2- 172.21.15.112:0/1022141972 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6734155da0 0x7f67340ae9e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6728006fd0 tx=0x7f67280090f0 comp rx=0 tx=0).stop 2023-01-24T03:34:35.285 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.266+0000 7f673aba0700 1 -- 172.21.15.112:0/1022141972 shutdown_connections 2023-01-24T03:34:35.285 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.266+0000 7f673aba0700 1 --2- 172.21.15.112:0/1022141972 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f67340c7e40 0x7f67340b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:35.286 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.266+0000 7f673aba0700 1 --2- 172.21.15.112:0/1022141972 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6734155da0 0x7f67340ae9e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:35.286 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.267+0000 7f673aba0700 1 -- 172.21.15.112:0/1022141972 >> 172.21.15.112:0/1022141972 conn(0x7f673405e2c0 msgr2=0x7f67340cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:35.287 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.267+0000 7f673aba0700 1 -- 172.21.15.112:0/1022141972 shutdown_connections 2023-01-24T03:34:35.287 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.267+0000 7f673aba0700 1 -- 172.21.15.112:0/1022141972 wait complete. 2023-01-24T03:34:35.287 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.267+0000 7f673aba0700 1 Processor -- start 2023-01-24T03:34:35.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.268+0000 7f673aba0700 1 -- start start 2023-01-24T03:34:35.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.268+0000 7f673aba0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f67340aef20 0x7f67341656a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:35.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.268+0000 7f673aba0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f67340c7e40 0x7f6734154860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:35.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.268+0000 7f673aba0700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6734155da0 0x7f67340c0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:35.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.268+0000 7f673aba0700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67341665d0 con 0x7f67340c7e40 2023-01-24T03:34:35.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.268+0000 7f673aba0700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6734156f70 con 0x7f6734155da0 2023-01-24T03:34:35.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.268+0000 7f673aba0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67340c2ad0 con 0x7f67340aef20 2023-01-24T03:34:35.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.268+0000 7f6733fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f67340c7e40 0x7f6734154860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:35.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.269+0000 7f6733fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f67340c7e40 0x7f6734154860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52800/0 (socket says 172.21.15.112:52800) 2023-01-24T03:34:35.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.269+0000 7f6733fff700 1 -- 172.21.15.112:0/3325367045 learned_addr learned my addr 172.21.15.112:0/3325367045 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:34:35.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.269+0000 7f673893c700 1 --2- 172.21.15.112:0/3325367045 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f67340aef20 0x7f67341656a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:35.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.269+0000 7f673913d700 1 --2- 172.21.15.112:0/3325367045 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6734155da0 0x7f67340c0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:35.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.269+0000 7f6733fff700 1 -- 172.21.15.112:0/3325367045 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f67340aef20 msgr2=0x7f67341656a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:35.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.269+0000 7f6733fff700 1 --2- 172.21.15.112:0/3325367045 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f67340aef20 0x7f67341656a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:35.292 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.269+0000 7f6733fff700 1 -- 172.21.15.112:0/3325367045 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6734155da0 msgr2=0x7f67340c0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:35.292 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.269+0000 7f6733fff700 1 --2- 172.21.15.112:0/3325367045 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6734155da0 0x7f67340c0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:35.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.269+0000 7f6733fff700 1 -- 172.21.15.112:0/3325367045 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6728009370 con 0x7f67340c7e40 2023-01-24T03:34:35.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.269+0000 7f6733fff700 1 --2- 172.21.15.112:0/3325367045 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f67340c7e40 0x7f6734154860 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f6734089340 tx=0x7f6728016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:35.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.269+0000 7f6730ff9700 1 -- 172.21.15.112:0/3325367045 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6728011940 con 0x7f67340c7e40 2023-01-24T03:34:35.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.269+0000 7f6730ff9700 1 -- 172.21.15.112:0/3325367045 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6728020070 con 0x7f67340c7e40 2023-01-24T03:34:35.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.269+0000 7f673aba0700 1 -- 172.21.15.112:0/3325367045 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6734165d30 con 0x7f67340c7e40 2023-01-24T03:34:35.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.269+0000 7f6730ff9700 1 -- 172.21.15.112:0/3325367045 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6728027480 con 0x7f67340c7e40 2023-01-24T03:34:35.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.269+0000 7f673aba0700 1 -- 172.21.15.112:0/3325367045 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67341662f0 con 0x7f67340c7e40 2023-01-24T03:34:35.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.270+0000 7f673aba0700 1 -- 172.21.15.112:0/3325367045 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f673404bb80 con 0x7f67340c7e40 2023-01-24T03:34:35.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.270+0000 7f6730ff9700 1 -- 172.21.15.112:0/3325367045 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f6728017020 con 0x7f67340c7e40 2023-01-24T03:34:35.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.271+0000 7f6730ff9700 1 --2- 172.21.15.112:0/3325367045 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f67140611c0 0x7f6714063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:35.295 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.271+0000 7f6730ff9700 1 -- 172.21.15.112:0/3325367045 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 7125+0+0 (secure 0 0 0) 0x7f6728022070 con 0x7f67340c7e40 2023-01-24T03:34:35.295 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.271+0000 7f673893c700 1 --2- 172.21.15.112:0/3325367045 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f67140611c0 0x7f6714063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:35.295 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.271+0000 7f673893c700 1 --2- 172.21.15.112:0/3325367045 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f67140611c0 0x7f6714063670 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6724009c30 tx=0x7f6724009060 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:35.296 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.276+0000 7f6730ff9700 1 -- 172.21.15.112:0/3325367045 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f672804f2f0 con 0x7f67340c7e40 2023-01-24T03:34:35.495 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.492+0000 7f673aba0700 1 -- 172.21.15.112:0/3325367045 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f673406efe0 con 0x7f67340c7e40 2023-01-24T03:34:35.496 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.493+0000 7f6730ff9700 1 -- 172.21.15.112:0/3325367045 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5357 (secure 0 0 0) 0x7f672802c020 con 0x7f67340c7e40 2023-01-24T03:34:35.496 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:34:35.497 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:34:28.624819+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"b","rank":0,"incarnation":22,"state":"up:replay","state_seq":1,"addr":"172.21.15.142:6834/612058861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":612058861},{"type":"v1","addr":"172.21.15.142:6834","nonce":612058861}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:34:35.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.495+0000 7f673aba0700 1 -- 172.21.15.112:0/3325367045 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f67140611c0 msgr2=0x7f6714063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:35.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.495+0000 7f673aba0700 1 --2- 172.21.15.112:0/3325367045 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f67140611c0 0x7f6714063670 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6724009c30 tx=0x7f6724009060 comp rx=0 tx=0).stop 2023-01-24T03:34:35.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.495+0000 7f673aba0700 1 -- 172.21.15.112:0/3325367045 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f67340c7e40 msgr2=0x7f6734154860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:35.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.495+0000 7f673aba0700 1 --2- 172.21.15.112:0/3325367045 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f67340c7e40 0x7f6734154860 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f6734089340 tx=0x7f6728016000 comp rx=0 tx=0).stop 2023-01-24T03:34:35.502 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.495+0000 7f673aba0700 1 -- 172.21.15.112:0/3325367045 shutdown_connections 2023-01-24T03:34:35.502 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.495+0000 7f673aba0700 1 --2- 172.21.15.112:0/3325367045 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f67140611c0 0x7f6714063670 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:35.502 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.495+0000 7f673aba0700 1 --2- 172.21.15.112:0/3325367045 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6734155da0 0x7f67340c0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:35.502 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.495+0000 7f673aba0700 1 --2- 172.21.15.112:0/3325367045 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f67340aef20 0x7f67341656a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:35.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.495+0000 7f673aba0700 1 --2- 172.21.15.112:0/3325367045 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f67340c7e40 0x7f6734154860 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:35.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.495+0000 7f673aba0700 1 -- 172.21.15.112:0/3325367045 >> 172.21.15.112:0/3325367045 conn(0x7f673405e2c0 msgr2=0x7f6734157270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:35.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.495+0000 7f673aba0700 1 -- 172.21.15.112:0/3325367045 shutdown_connections 2023-01-24T03:34:35.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:35.495+0000 7f673aba0700 1 -- 172.21.15.112:0/3325367045 wait complete. 2023-01-24T03:34:35.504 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 22 2023-01-24T03:34:35.510 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:34:37.511 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:34:37.636 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.634+0000 7f03d3036700 1 Processor -- start 2023-01-24T03:34:37.636 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.634+0000 7f03d3036700 1 -- start start 2023-01-24T03:34:37.642 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.634+0000 7f03d3036700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03cc14f2c0 0x7f03cc0c4f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:37.642 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.634+0000 7f03d3036700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03cc05df40 con 0x7f03cc0c4790 2023-01-24T03:34:37.642 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.634+0000 7f03d3036700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03cc155920 con 0x7f03cc0c40e0 2023-01-24T03:34:37.642 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.634+0000 7f03d3036700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc1552f0 con 0x7f03cc14f2c0 2023-01-24T03:34:37.642 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.634+0000 7f03d15d3700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f03cc0c4790 0x7f03cc15fc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:33802/0 (socket says 172.21.15.112:33802) 2023-01-24T03:34:37.643 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.634+0000 7f03d15d3700 1 -- 172.21.15.112:0/418047655 learned_addr learned my addr 172.21.15.112:0/418047655 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:34:37.643 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.635+0000 7f03d0dd2700 1 --2- 172.21.15.112:0/418047655 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03cc14f2c0 0x7f03cc0c4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:37.643 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.635+0000 7f03cb7fe700 1 -- 172.21.15.112:0/418047655 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2073744072 0 0) 0x7f03cc05df40 con 0x7f03cc0c4790 2023-01-24T03:34:37.643 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.635+0000 7f03cb7fe700 1 -- 172.21.15.112:0/418047655 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03b80033f0 con 0x7f03cc0c4790 2023-01-24T03:34:37.644 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.635+0000 7f03cb7fe700 1 -- 172.21.15.112:0/418047655 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1494591319 0 0) 0x7f03b80033f0 con 0x7f03cc0c4790 2023-01-24T03:34:37.644 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.635+0000 7f03cb7fe700 1 -- 172.21.15.112:0/418047655 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03cc14f2c0 msgr2=0x7f03cc0c4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:37.644 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.635+0000 7f03cb7fe700 1 --2- 172.21.15.112:0/418047655 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03cc14f2c0 0x7f03cc0c4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:37.644 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.635+0000 7f03cb7fe700 1 -- 172.21.15.112:0/418047655 >> v1:172.21.15.142:6789/0 conn(0x7f03cc0c40e0 legacy=0x7f03cc067f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:37.644 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.635+0000 7f03cb7fe700 1 -- 172.21.15.112:0/418047655 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03cc05e190 con 0x7f03cc0c4790 2023-01-24T03:34:37.645 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.635+0000 7f03cb7fe700 1 -- 172.21.15.112:0/418047655 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7f03c40017f0 con 0x7f03cc0c4790 2023-01-24T03:34:37.645 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.635+0000 7f03cb7fe700 1 -- 172.21.15.112:0/418047655 >> v1:172.21.15.112:6789/0 conn(0x7f03cc0c4790 legacy=0x7f03cc15fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:37.645 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.635+0000 7f03cb7fe700 1 --2- 172.21.15.112:0/418047655 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03cc14f2c0 0x7f03b8003910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:37.645 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.635+0000 7f03cb7fe700 1 --2- 172.21.15.112:0/418047655 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03b8003e50 0x7f03b8004690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:37.645 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.635+0000 7f03cb7fe700 1 --2- 172.21.15.112:0/418047655 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f03b8004bd0 0x7f03b8005040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:37.645 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.635+0000 7f03cb7fe700 1 -- 172.21.15.112:0/418047655 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc1552f0 con 0x7f03b8004bd0 2023-01-24T03:34:37.646 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.635+0000 7f03cb7fe700 1 -- 172.21.15.112:0/418047655 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b800b9c0 con 0x7f03b8003e50 2023-01-24T03:34:37.646 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.635+0000 7f03cb7fe700 1 -- 172.21.15.112:0/418047655 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03b800bb80 con 0x7f03cc14f2c0 2023-01-24T03:34:37.646 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.635+0000 7f03d15d3700 1 --2- 172.21.15.112:0/418047655 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f03b8004bd0 0x7f03b8005040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:37.646 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.635+0000 7f03d0dd2700 1 --2- 172.21.15.112:0/418047655 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03cc14f2c0 0x7f03b8003910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:37.646 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.635+0000 7f03cbfff700 1 --2- 172.21.15.112:0/418047655 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03b8003e50 0x7f03b8004690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:37.647 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.636+0000 7f03d15d3700 1 -- 172.21.15.112:0/418047655 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03cc14f2c0 msgr2=0x7f03b8003910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:37.647 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.636+0000 7f03d15d3700 1 --2- 172.21.15.112:0/418047655 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03cc14f2c0 0x7f03b8003910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:37.647 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.636+0000 7f03d15d3700 1 -- 172.21.15.112:0/418047655 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03b8003e50 msgr2=0x7f03b8004690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:37.647 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.636+0000 7f03d15d3700 1 --2- 172.21.15.112:0/418047655 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03b8003e50 0x7f03b8004690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:37.647 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.636+0000 7f03d15d3700 1 -- 172.21.15.112:0/418047655 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b80033f0 con 0x7f03b8004bd0 2023-01-24T03:34:37.648 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.636+0000 7f03d15d3700 1 --2- 172.21.15.112:0/418047655 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f03b8004bd0 0x7f03b8005040 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f03c400c5f0 tx=0x7f03c4001d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22543d2507a44235 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:37.648 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.636+0000 7f03cb7fe700 1 -- 172.21.15.112:0/418047655 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c4001fc0 con 0x7f03b8004bd0 2023-01-24T03:34:37.648 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.636+0000 7f03cb7fe700 1 -- 172.21.15.112:0/418047655 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c4005b20 con 0x7f03b8004bd0 2023-01-24T03:34:37.648 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.636+0000 7f03cb7fe700 1 -- 172.21.15.112:0/418047655 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c40070d0 con 0x7f03b8004bd0 2023-01-24T03:34:37.648 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.636+0000 7f03d3036700 1 -- 172.21.15.112:0/418047655 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f03b8004bd0 msgr2=0x7f03b8005040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:37.649 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.636+0000 7f03d3036700 1 --2- 172.21.15.112:0/418047655 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f03b8004bd0 0x7f03b8005040 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f03c400c5f0 tx=0x7f03c4001d90 comp rx=0 tx=0).stop 2023-01-24T03:34:37.649 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.636+0000 7f03d15d3700 1 -- 172.21.15.112:0/418047655 reap_dead start 2023-01-24T03:34:37.649 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.636+0000 7f03d3036700 1 -- 172.21.15.112:0/418047655 shutdown_connections 2023-01-24T03:34:37.649 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.636+0000 7f03d3036700 1 -- 172.21.15.112:0/418047655 >> 172.21.15.112:0/418047655 conn(0x7f03cc06daf0 msgr2=0x7f03cc05d830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:37.649 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.636+0000 7f03d3036700 1 -- 172.21.15.112:0/418047655 shutdown_connections 2023-01-24T03:34:37.650 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.636+0000 7f03d3036700 1 -- 172.21.15.112:0/418047655 wait complete. 2023-01-24T03:34:37.650 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.637+0000 7f03d3036700 1 Processor -- start 2023-01-24T03:34:37.650 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.638+0000 7f03d3036700 1 -- start start 2023-01-24T03:34:37.650 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.638+0000 7f03d3036700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f03cc0c5c30 0x7f03cc1652c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:37.650 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.638+0000 7f03d3036700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03cc1528d0 0x7f03cc165800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:37.651 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.638+0000 7f03d3036700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03cc0c6000 0x7f03cc15a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:37.651 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.638+0000 7f03d3036700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc06b620 con 0x7f03cc0c5c30 2023-01-24T03:34:37.651 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.638+0000 7f03d3036700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc168560 con 0x7f03cc1528d0 2023-01-24T03:34:37.651 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.638+0000 7f03d3036700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc169380 con 0x7f03cc0c6000 2023-01-24T03:34:37.651 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.638+0000 7f03d0dd2700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f03cc0c5c30 0x7f03cc1652c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:37.652 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.638+0000 7f03d0dd2700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f03cc0c5c30 0x7f03cc1652c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52826/0 (socket says 172.21.15.112:52826) 2023-01-24T03:34:37.652 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.638+0000 7f03d0dd2700 1 -- 172.21.15.112:0/3148383856 learned_addr learned my addr 172.21.15.112:0/3148383856 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:34:37.652 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.639+0000 7f03d0dd2700 1 -- 172.21.15.112:0/3148383856 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03cc0c6000 msgr2=0x7f03cc15a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:37.652 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.639+0000 7f03d0dd2700 1 --2- 172.21.15.112:0/3148383856 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03cc0c6000 0x7f03cc15a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:37.652 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.639+0000 7f03d0dd2700 1 -- 172.21.15.112:0/3148383856 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03cc1528d0 msgr2=0x7f03cc165800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:37.653 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.639+0000 7f03d0dd2700 1 --2- 172.21.15.112:0/3148383856 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03cc1528d0 0x7f03cc165800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:37.653 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.639+0000 7f03d0dd2700 1 -- 172.21.15.112:0/3148383856 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03cc153df0 con 0x7f03cc0c5c30 2023-01-24T03:34:37.653 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.639+0000 7f03d0dd2700 1 --2- 172.21.15.112:0/3148383856 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f03cc0c5c30 0x7f03cc1652c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f03bc0119f0 tx=0x7f03bc009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:37.653 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.639+0000 7f03c8ff9700 1 -- 172.21.15.112:0/3148383856 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03bc00f8e0 con 0x7f03cc0c5c30 2023-01-24T03:34:37.653 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.639+0000 7f03c8ff9700 1 -- 172.21.15.112:0/3148383856 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03bc00fa40 con 0x7f03cc0c5c30 2023-01-24T03:34:37.654 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.639+0000 7f03d3036700 1 -- 172.21.15.112:0/3148383856 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03cc168770 con 0x7f03cc0c5c30 2023-01-24T03:34:37.654 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.639+0000 7f03c8ff9700 1 -- 172.21.15.112:0/3148383856 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03bc00fde0 con 0x7f03cc0c5c30 2023-01-24T03:34:37.654 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.639+0000 7f03d3036700 1 -- 172.21.15.112:0/3148383856 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03cc168d30 con 0x7f03cc0c5c30 2023-01-24T03:34:37.654 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.640+0000 7f03d3036700 1 -- 172.21.15.112:0/3148383856 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03cc04bb80 con 0x7f03cc0c5c30 2023-01-24T03:34:37.655 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.640+0000 7f03c8ff9700 1 -- 172.21.15.112:0/3148383856 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f03bc014650 con 0x7f03cc0c5c30 2023-01-24T03:34:37.655 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.641+0000 7f03c8ff9700 1 --2- 172.21.15.112:0/3148383856 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f03ac0611f0 0x7f03ac0636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:37.656 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.641+0000 7f03c8ff9700 1 -- 172.21.15.112:0/3148383856 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 7125+0+0 (secure 0 0 0) 0x7f03bc018030 con 0x7f03cc0c5c30 2023-01-24T03:34:37.656 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.641+0000 7f03cbfff700 1 --2- 172.21.15.112:0/3148383856 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f03ac0611f0 0x7f03ac0636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:37.656 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.641+0000 7f03cbfff700 1 --2- 172.21.15.112:0/3148383856 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f03ac0611f0 0x7f03ac0636a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f03c0007fd0 tx=0x7f03c0007b90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:37.656 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.645+0000 7f03c8ff9700 1 -- 172.21.15.112:0/3148383856 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f03bc04de20 con 0x7f03cc0c5c30 2023-01-24T03:34:37.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.865+0000 7f03d3036700 1 -- 172.21.15.112:0/3148383856 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03cc14eee0 con 0x7f03cc0c5c30 2023-01-24T03:34:37.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.866+0000 7f03c8ff9700 1 -- 172.21.15.112:0/3148383856 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5361 (secure 0 0 0) 0x7f03cc14eee0 con 0x7f03cc0c5c30 2023-01-24T03:34:37.868 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:34:37.869 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:34:35.810866+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"b","rank":0,"incarnation":22,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.142:6834/612058861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":612058861},{"type":"v1","addr":"172.21.15.142:6834","nonce":612058861}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:34:37.871 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.868+0000 7f03d3036700 1 -- 172.21.15.112:0/3148383856 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f03ac0611f0 msgr2=0x7f03ac0636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:37.871 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.868+0000 7f03d3036700 1 --2- 172.21.15.112:0/3148383856 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f03ac0611f0 0x7f03ac0636a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f03c0007fd0 tx=0x7f03c0007b90 comp rx=0 tx=0).stop 2023-01-24T03:34:37.872 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.868+0000 7f03d3036700 1 -- 172.21.15.112:0/3148383856 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f03cc0c5c30 msgr2=0x7f03cc1652c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:37.872 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.868+0000 7f03d3036700 1 --2- 172.21.15.112:0/3148383856 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f03cc0c5c30 0x7f03cc1652c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f03bc0119f0 tx=0x7f03bc009060 comp rx=0 tx=0).stop 2023-01-24T03:34:37.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.868+0000 7f03d3036700 1 -- 172.21.15.112:0/3148383856 shutdown_connections 2023-01-24T03:34:37.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.868+0000 7f03d3036700 1 --2- 172.21.15.112:0/3148383856 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f03ac0611f0 0x7f03ac0636a0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:37.874 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.868+0000 7f03d3036700 1 --2- 172.21.15.112:0/3148383856 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f03cc0c5c30 0x7f03cc1652c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:37.874 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.868+0000 7f03d3036700 1 --2- 172.21.15.112:0/3148383856 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03cc0c6000 0x7f03cc15a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:37.874 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.868+0000 7f03d3036700 1 --2- 172.21.15.112:0/3148383856 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03cc1528d0 0x7f03cc165800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:37.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.868+0000 7f03d3036700 1 -- 172.21.15.112:0/3148383856 >> 172.21.15.112:0/3148383856 conn(0x7f03cc06daf0 msgr2=0x7f03cc0c9810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:37.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.868+0000 7f03d3036700 1 -- 172.21.15.112:0/3148383856 shutdown_connections 2023-01-24T03:34:37.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:37.868+0000 7f03d3036700 1 -- 172.21.15.112:0/3148383856 wait complete. 2023-01-24T03:34:37.876 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 23 2023-01-24T03:34:37.882 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:34:39.883 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:34:40.006 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.004+0000 7fd11c8fa700 1 Processor -- start 2023-01-24T03:34:40.007 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.004+0000 7fd11c8fa700 1 -- start start 2023-01-24T03:34:40.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.005+0000 7fd11c8fa700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1180c7e40 0x7fd1180b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:40.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.005+0000 7fd11c8fa700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd118156210 con 0x7fd1180ca3e0 2023-01-24T03:34:40.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.005+0000 7fd11c8fa700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1180b1170 con 0x7fd118155da0 2023-01-24T03:34:40.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.005+0000 7fd11c8fa700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1180b1450 con 0x7fd1180c7e40 2023-01-24T03:34:40.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.005+0000 7fd116d9d700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fd1180ca3e0 0x7fd1180ca820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:33822/0 (socket says 172.21.15.112:33822) 2023-01-24T03:34:40.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.005+0000 7fd116d9d700 1 -- 172.21.15.112:0/2644956516 learned_addr learned my addr 172.21.15.112:0/2644956516 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:34:40.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.006+0000 7fd11559a700 1 -- 172.21.15.112:0/2644956516 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1396036996 0 0) 0x7fd118156210 con 0x7fd1180ca3e0 2023-01-24T03:34:40.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.006+0000 7fd11559a700 1 -- 172.21.15.112:0/2644956516 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0f80033f0 con 0x7fd1180ca3e0 2023-01-24T03:34:40.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.006+0000 7fd11659c700 1 --2- 172.21.15.112:0/2644956516 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1180c7e40 0x7fd1180b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:40.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.006+0000 7fd11559a700 1 -- 172.21.15.112:0/2644956516 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3244164029 0 0) 0x7fd0f80033f0 con 0x7fd1180ca3e0 2023-01-24T03:34:40.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.006+0000 7fd11559a700 1 -- 172.21.15.112:0/2644956516 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1180c7e40 msgr2=0x7fd1180b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:40.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.006+0000 7fd11559a700 1 --2- 172.21.15.112:0/2644956516 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1180c7e40 0x7fd1180b0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:40.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.006+0000 7fd11559a700 1 -- 172.21.15.112:0/2644956516 >> v1:172.21.15.142:6789/0 conn(0x7fd118155da0 legacy=0x7fd1180ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:40.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.006+0000 7fd11559a700 1 -- 172.21.15.112:0/2644956516 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1180b1730 con 0x7fd1180ca3e0 2023-01-24T03:34:40.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.006+0000 7fd11559a700 1 -- 172.21.15.112:0/2644956516 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7fd110001470 con 0x7fd1180ca3e0 2023-01-24T03:34:40.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.006+0000 7fd11559a700 1 -- 172.21.15.112:0/2644956516 >> v1:172.21.15.112:6789/0 conn(0x7fd1180ca3e0 legacy=0x7fd1180ca820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:40.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.006+0000 7fd11559a700 1 --2- 172.21.15.112:0/2644956516 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0f8003910 0x7fd0f8003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:40.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.006+0000 7fd11559a700 1 --2- 172.21.15.112:0/2644956516 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd0f8004690 0x7fd0f8004b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:40.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.006+0000 7fd11559a700 1 --2- 172.21.15.112:0/2644956516 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1180c7e40 0x7fd0f8005040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:40.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.006+0000 7fd11559a700 1 -- 172.21.15.112:0/2644956516 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1180b1450 con 0x7fd0f8003910 2023-01-24T03:34:40.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.006+0000 7fd11559a700 1 -- 172.21.15.112:0/2644956516 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f800b990 con 0x7fd0f8004690 2023-01-24T03:34:40.015 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.006+0000 7fd11559a700 1 -- 172.21.15.112:0/2644956516 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f800bb50 con 0x7fd1180c7e40 2023-01-24T03:34:40.015 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.006+0000 7fd11659c700 1 --2- 172.21.15.112:0/2644956516 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0f8003910 0x7fd0f8003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:40.015 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.006+0000 7fd116d9d700 1 --2- 172.21.15.112:0/2644956516 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1180c7e40 0x7fd0f8005040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:40.015 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.006+0000 7fd115d9b700 1 --2- 172.21.15.112:0/2644956516 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd0f8004690 0x7fd0f8004b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:40.015 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.006+0000 7fd11659c700 1 -- 172.21.15.112:0/2644956516 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1180c7e40 msgr2=0x7fd0f8005040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:40.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.006+0000 7fd11659c700 1 --2- 172.21.15.112:0/2644956516 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1180c7e40 0x7fd0f8005040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:40.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.006+0000 7fd11659c700 1 -- 172.21.15.112:0/2644956516 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd0f8004690 msgr2=0x7fd0f8004b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:40.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.006+0000 7fd11659c700 1 --2- 172.21.15.112:0/2644956516 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd0f8004690 0x7fd0f8004b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:40.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.006+0000 7fd11659c700 1 -- 172.21.15.112:0/2644956516 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f80033f0 con 0x7fd0f8003910 2023-01-24T03:34:40.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.006+0000 7fd116d9d700 1 --2- 172.21.15.112:0/2644956516 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1180c7e40 0x7fd0f8005040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:34:40.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.006+0000 7fd115d9b700 1 --2- 172.21.15.112:0/2644956516 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd0f8004690 0x7fd0f8004b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:34:40.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.007+0000 7fd11659c700 1 --2- 172.21.15.112:0/2644956516 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0f8003910 0x7fd0f8003ce0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fd10800e210 tx=0x7fd10800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d259f3fdd5dee9ae server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:40.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.007+0000 7fd11559a700 1 -- 172.21.15.112:0/2644956516 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd10800eac0 con 0x7fd0f8003910 2023-01-24T03:34:40.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.007+0000 7fd11559a700 1 -- 172.21.15.112:0/2644956516 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd10800f040 con 0x7fd0f8003910 2023-01-24T03:34:40.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.007+0000 7fd11559a700 1 -- 172.21.15.112:0/2644956516 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd10800ee70 con 0x7fd0f8003910 2023-01-24T03:34:40.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.007+0000 7fd11c8fa700 1 -- 172.21.15.112:0/2644956516 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0f8003910 msgr2=0x7fd0f8003ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:40.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.007+0000 7fd11c8fa700 1 --2- 172.21.15.112:0/2644956516 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd0f8003910 0x7fd0f8003ce0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fd10800e210 tx=0x7fd10800b040 comp rx=0 tx=0).stop 2023-01-24T03:34:40.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.007+0000 7fd116d9d700 1 -- 172.21.15.112:0/2644956516 reap_dead start 2023-01-24T03:34:40.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.007+0000 7fd11c8fa700 1 -- 172.21.15.112:0/2644956516 shutdown_connections 2023-01-24T03:34:40.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.007+0000 7fd11c8fa700 1 -- 172.21.15.112:0/2644956516 >> 172.21.15.112:0/2644956516 conn(0x7fd11805e2c0 msgr2=0x7fd1180cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:40.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.007+0000 7fd11c8fa700 1 -- 172.21.15.112:0/2644956516 shutdown_connections 2023-01-24T03:34:40.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.007+0000 7fd11c8fa700 1 -- 172.21.15.112:0/2644956516 wait complete. 2023-01-24T03:34:40.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.008+0000 7fd11c8fa700 1 Processor -- start 2023-01-24T03:34:40.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.008+0000 7fd11c8fa700 1 -- start start 2023-01-24T03:34:40.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.009+0000 7fd11c8fa700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd0f8003910 0x7fd11814fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:40.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.009+0000 7fd11c8fa700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd118155530 0x7fd1181542b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:40.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.009+0000 7fd11c8fa700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd1181547f0 0x7fd118151880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:40.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.009+0000 7fd11c8fa700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1180c0250 con 0x7fd118155530 2023-01-24T03:34:40.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.009+0000 7fd11c8fa700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd118166220 con 0x7fd1181547f0 2023-01-24T03:34:40.023 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.009+0000 7fd11c8fa700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1180bf410 con 0x7fd0f8003910 2023-01-24T03:34:40.023 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.009+0000 7fd115d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd118155530 0x7fd1181542b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:40.023 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.009+0000 7fd115d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd118155530 0x7fd1181542b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52846/0 (socket says 172.21.15.112:52846) 2023-01-24T03:34:40.023 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.009+0000 7fd115d9b700 1 -- 172.21.15.112:0/2474704364 learned_addr learned my addr 172.21.15.112:0/2474704364 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:34:40.023 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.009+0000 7fd11659c700 1 --2- 172.21.15.112:0/2474704364 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd0f8003910 0x7fd11814fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:40.024 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.009+0000 7fd116d9d700 1 --2- 172.21.15.112:0/2474704364 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd1181547f0 0x7fd118151880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:40.024 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.009+0000 7fd115d9b700 1 -- 172.21.15.112:0/2474704364 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd0f8003910 msgr2=0x7fd11814fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:40.024 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.009+0000 7fd115d9b700 1 --2- 172.21.15.112:0/2474704364 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd0f8003910 0x7fd11814fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:40.024 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.009+0000 7fd115d9b700 1 -- 172.21.15.112:0/2474704364 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd1181547f0 msgr2=0x7fd118151880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:40.024 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.009+0000 7fd115d9b700 1 --2- 172.21.15.112:0/2474704364 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd1181547f0 0x7fd118151880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:40.025 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.009+0000 7fd115d9b700 1 -- 172.21.15.112:0/2474704364 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1181550e0 con 0x7fd118155530 2023-01-24T03:34:40.025 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.010+0000 7fd115d9b700 1 --2- 172.21.15.112:0/2474704364 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd118155530 0x7fd1181542b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fd10000e6b0 tx=0x7fd100009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:40.025 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.010+0000 7fd10e7fc700 1 -- 172.21.15.112:0/2474704364 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1000115c0 con 0x7fd118155530 2023-01-24T03:34:40.025 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.010+0000 7fd11c8fa700 1 -- 172.21.15.112:0/2474704364 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd118165350 con 0x7fd118155530 2023-01-24T03:34:40.026 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.010+0000 7fd10e7fc700 1 -- 172.21.15.112:0/2474704364 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd100011720 con 0x7fd118155530 2023-01-24T03:34:40.026 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.010+0000 7fd10e7fc700 1 -- 172.21.15.112:0/2474704364 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd100011aa0 con 0x7fd118155530 2023-01-24T03:34:40.026 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.010+0000 7fd11c8fa700 1 -- 172.21.15.112:0/2474704364 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1181658f0 con 0x7fd118155530 2023-01-24T03:34:40.026 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.011+0000 7fd11c8fa700 1 -- 172.21.15.112:0/2474704364 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd118154bc0 con 0x7fd118155530 2023-01-24T03:34:40.026 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.011+0000 7fd10e7fc700 1 -- 172.21.15.112:0/2474704364 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fd100014480 con 0x7fd118155530 2023-01-24T03:34:40.026 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.012+0000 7fd10e7fc700 1 --2- 172.21.15.112:0/2474704364 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd0fc0612c0 0x7fd0fc063770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:40.027 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.012+0000 7fd11659c700 1 --2- 172.21.15.112:0/2474704364 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd0fc0612c0 0x7fd0fc063770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:40.027 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.012+0000 7fd10e7fc700 1 -- 172.21.15.112:0/2474704364 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 7125+0+0 (secure 0 0 0) 0x7fd10007cf90 con 0x7fd118155530 2023-01-24T03:34:40.027 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.012+0000 7fd11659c700 1 --2- 172.21.15.112:0/2474704364 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd0fc0612c0 0x7fd0fc063770 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd10800afd0 tx=0x7fd10801a000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:40.027 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.016+0000 7fd10e7fc700 1 -- 172.21.15.112:0/2474704364 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fd10004b1d0 con 0x7fd118155530 2023-01-24T03:34:40.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.238+0000 7fd11c8fa700 1 -- 172.21.15.112:0/2474704364 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd118089130 con 0x7fd118155530 2023-01-24T03:34:40.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.238+0000 7fd10e7fc700 1 -- 172.21.15.112:0/2474704364 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5361 (secure 0 0 0) 0x7fd100019020 con 0x7fd118155530 2023-01-24T03:34:40.241 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:34:40.241 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:34:35.810866+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"b","rank":0,"incarnation":22,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.142:6834/612058861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":612058861},{"type":"v1","addr":"172.21.15.142:6834","nonce":612058861}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:34:40.243 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.240+0000 7fd11c8fa700 1 -- 172.21.15.112:0/2474704364 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd0fc0612c0 msgr2=0x7fd0fc063770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:40.243 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.240+0000 7fd11c8fa700 1 --2- 172.21.15.112:0/2474704364 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd0fc0612c0 0x7fd0fc063770 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd10800afd0 tx=0x7fd10801a000 comp rx=0 tx=0).stop 2023-01-24T03:34:40.243 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.240+0000 7fd11c8fa700 1 -- 172.21.15.112:0/2474704364 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd118155530 msgr2=0x7fd1181542b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:40.243 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.240+0000 7fd11c8fa700 1 --2- 172.21.15.112:0/2474704364 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd118155530 0x7fd1181542b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fd10000e6b0 tx=0x7fd100009040 comp rx=0 tx=0).stop 2023-01-24T03:34:40.244 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.241+0000 7fd11c8fa700 1 -- 172.21.15.112:0/2474704364 shutdown_connections 2023-01-24T03:34:40.244 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.241+0000 7fd11c8fa700 1 --2- 172.21.15.112:0/2474704364 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd0fc0612c0 0x7fd0fc063770 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:40.244 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.241+0000 7fd11c8fa700 1 --2- 172.21.15.112:0/2474704364 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd1181547f0 0x7fd118151880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:40.244 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.241+0000 7fd11c8fa700 1 --2- 172.21.15.112:0/2474704364 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd0f8003910 0x7fd11814fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:40.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.241+0000 7fd11c8fa700 1 --2- 172.21.15.112:0/2474704364 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd118155530 0x7fd1181542b0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:40.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.241+0000 7fd11c8fa700 1 -- 172.21.15.112:0/2474704364 >> 172.21.15.112:0/2474704364 conn(0x7fd11805e2c0 msgr2=0x7fd118068150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:40.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.241+0000 7fd11c8fa700 1 -- 172.21.15.112:0/2474704364 shutdown_connections 2023-01-24T03:34:40.246 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:40.241+0000 7fd11c8fa700 1 -- 172.21.15.112:0/2474704364 wait complete. 2023-01-24T03:34:40.246 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 23 2023-01-24T03:34:40.254 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:34:42.255 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:34:42.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.379+0000 7fd4a7fff700 1 Processor -- start 2023-01-24T03:34:42.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.379+0000 7fd4a7fff700 1 -- start start 2023-01-24T03:34:42.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.380+0000 7fd4a7fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4a80c5680 0x7fd4a8156730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:42.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.380+0000 7fd4a7fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd4a80c5a50 0x7fd4a8156c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:42.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.380+0000 7fd4a7fff700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4a80c3d10 con 0x7fd4a8155e20 2023-01-24T03:34:42.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.380+0000 7fd4a7fff700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4a80c4eb0 con 0x7fd4a80c5a50 2023-01-24T03:34:42.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.380+0000 7fd4a7fff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a80c2e20 con 0x7fd4a80c5680 2023-01-24T03:34:42.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.380+0000 7fd4a77fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd4a80c5a50 0x7fd4a8156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:42.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.380+0000 7fd4a6ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4a80c5680 0x7fd4a8156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:42.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.380+0000 7fd4a77fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd4a80c5a50 0x7fd4a8156c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.112:45352/0 (socket says 172.21.15.112:45352) 2023-01-24T03:34:42.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.380+0000 7fd4a6ffd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4a80c5680 0x7fd4a8156730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.112:40156/0 (socket says 172.21.15.112:40156) 2023-01-24T03:34:42.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.380+0000 7fd4a77fe700 1 -- 172.21.15.112:0/2578347239 learned_addr learned my addr 172.21.15.112:0/2578347239 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:34:42.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.380+0000 7fd4a67fc700 1 --1- 172.21.15.112:0/2578347239 >> v1:172.21.15.142:6789/0 conn(0x7fd4a8155e20 0x7fd4a80cb870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.112:47634/0 (socket says 172.21.15.112:47634) 2023-01-24T03:34:42.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.380+0000 7fd4a6ffd700 1 -- 172.21.15.112:0/2578347239 >> v2:172.21.15.142:3300/0 conn(0x7fd4a80c5a50 msgr2=0x7fd4a8156c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:42.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.380+0000 7fd4a6ffd700 1 --2- 172.21.15.112:0/2578347239 >> v2:172.21.15.142:3300/0 conn(0x7fd4a80c5a50 0x7fd4a8156c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:42.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.380+0000 7fd4a6ffd700 1 -- 172.21.15.112:0/2578347239 >> v1:172.21.15.142:6789/0 conn(0x7fd4a8155e20 legacy=0x7fd4a80cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:42.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.380+0000 7fd4a6ffd700 1 -- 172.21.15.112:0/2578347239 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a80c3100 con 0x7fd4a80c5680 2023-01-24T03:34:42.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.380+0000 7fd4a77fe700 1 --2- 172.21.15.112:0/2578347239 >> v2:172.21.15.142:3300/0 conn(0x7fd4a80c5a50 0x7fd4a8156c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:34:42.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.381+0000 7fd4a6ffd700 1 --2- 172.21.15.112:0/2578347239 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4a80c5680 0x7fd4a8156730 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd498009e90 tx=0x7fd498009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b245294bec229805 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:42.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.381+0000 7fd4a5ffb700 1 -- 172.21.15.112:0/2578347239 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49800e070 con 0x7fd4a80c5680 2023-01-24T03:34:42.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.381+0000 7fd4a5ffb700 1 -- 172.21.15.112:0/2578347239 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd49800bc80 con 0x7fd4a80c5680 2023-01-24T03:34:42.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.381+0000 7fd4a5ffb700 1 -- 172.21.15.112:0/2578347239 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd498012430 con 0x7fd4a80c5680 2023-01-24T03:34:42.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.381+0000 7fd4a7fff700 1 -- 172.21.15.112:0/2578347239 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4a80c5680 msgr2=0x7fd4a8156730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:42.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.381+0000 7fd4a7fff700 1 --2- 172.21.15.112:0/2578347239 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4a80c5680 0x7fd4a8156730 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd498009e90 tx=0x7fd498009780 comp rx=0 tx=0).stop 2023-01-24T03:34:42.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.381+0000 7fd4a7fff700 1 -- 172.21.15.112:0/2578347239 shutdown_connections 2023-01-24T03:34:42.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.381+0000 7fd4a7fff700 1 --2- 172.21.15.112:0/2578347239 >> v2:172.21.15.142:3300/0 conn(0x7fd4a80c5a50 0x7fd4a8156c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:42.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.381+0000 7fd4a7fff700 1 --2- 172.21.15.112:0/2578347239 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4a80c5680 0x7fd4a8156730 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:42.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.381+0000 7fd4a7fff700 1 -- 172.21.15.112:0/2578347239 >> 172.21.15.112:0/2578347239 conn(0x7fd4a80685b0 msgr2=0x7fd4a80689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:42.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.381+0000 7fd4a7fff700 1 -- 172.21.15.112:0/2578347239 shutdown_connections 2023-01-24T03:34:42.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.381+0000 7fd4a7fff700 1 -- 172.21.15.112:0/2578347239 wait complete. 2023-01-24T03:34:42.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.383+0000 7fd4a7fff700 1 Processor -- start 2023-01-24T03:34:42.393 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.383+0000 7fd4a7fff700 1 -- start start 2023-01-24T03:34:42.393 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.383+0000 7fd4a7fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4a80c5680 0x7fd4a80bf9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:42.393 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.383+0000 7fd4a7fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd4a80c5a50 0x7fd4a8165850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:42.393 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.383+0000 7fd4a7fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4a8155e20 0x7fd4a80c0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:42.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.383+0000 7fd4a7fff700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a80c2320 con 0x7fd4a80c5a50 2023-01-24T03:34:42.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.383+0000 7fd4a7fff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a8152af0 con 0x7fd4a8155e20 2023-01-24T03:34:42.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.383+0000 7fd4a7fff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a8151cc0 con 0x7fd4a80c5680 2023-01-24T03:34:42.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.383+0000 7fd4a67fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd4a80c5a50 0x7fd4a8165850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:42.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.384+0000 7fd4a67fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd4a80c5a50 0x7fd4a8165850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52862/0 (socket says 172.21.15.112:52862) 2023-01-24T03:34:42.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.384+0000 7fd4a67fc700 1 -- 172.21.15.112:0/3697188182 learned_addr learned my addr 172.21.15.112:0/3697188182 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:34:42.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.384+0000 7fd4a6ffd700 1 --2- 172.21.15.112:0/3697188182 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4a80c5680 0x7fd4a80bf9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:42.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.384+0000 7fd4a77fe700 1 --2- 172.21.15.112:0/3697188182 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4a8155e20 0x7fd4a80c0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:42.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.384+0000 7fd4a67fc700 1 -- 172.21.15.112:0/3697188182 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4a80c5680 msgr2=0x7fd4a80bf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:42.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.384+0000 7fd4a67fc700 1 --2- 172.21.15.112:0/3697188182 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4a80c5680 0x7fd4a80bf9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:42.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.384+0000 7fd4a67fc700 1 -- 172.21.15.112:0/3697188182 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4a8155e20 msgr2=0x7fd4a80c0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:42.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.384+0000 7fd4a67fc700 1 --2- 172.21.15.112:0/3697188182 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4a8155e20 0x7fd4a80c0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:42.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.384+0000 7fd4a67fc700 1 -- 172.21.15.112:0/3697188182 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a0009370 con 0x7fd4a80c5a50 2023-01-24T03:34:42.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.384+0000 7fd4a67fc700 1 --2- 172.21.15.112:0/3697188182 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd4a80c5a50 0x7fd4a8165850 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd49000c4c0 tx=0x7fd490007dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:42.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.384+0000 7fd49effd700 1 -- 172.21.15.112:0/3697188182 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4900072e0 con 0x7fd4a80c5a50 2023-01-24T03:34:42.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.384+0000 7fd49effd700 1 -- 172.21.15.112:0/3697188182 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd490007460 con 0x7fd4a80c5a50 2023-01-24T03:34:42.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.384+0000 7fd4a7fff700 1 -- 172.21.15.112:0/3697188182 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd498009a20 con 0x7fd4a80c5a50 2023-01-24T03:34:42.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.384+0000 7fd49effd700 1 -- 172.21.15.112:0/3697188182 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd490004900 con 0x7fd4a80c5a50 2023-01-24T03:34:42.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.384+0000 7fd4a7fff700 1 -- 172.21.15.112:0/3697188182 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4a8151ef0 con 0x7fd4a80c5a50 2023-01-24T03:34:42.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.385+0000 7fd4a7fff700 1 -- 172.21.15.112:0/3697188182 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4a805ea20 con 0x7fd4a80c5a50 2023-01-24T03:34:42.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.385+0000 7fd49effd700 1 -- 172.21.15.112:0/3697188182 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fd490010070 con 0x7fd4a80c5a50 2023-01-24T03:34:42.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.386+0000 7fd49effd700 1 --2- 172.21.15.112:0/3697188182 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd48c0611c0 0x7fd48c063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:42.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.386+0000 7fd49effd700 1 -- 172.21.15.112:0/3697188182 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 7125+0+0 (secure 0 0 0) 0x7fd49007cfb0 con 0x7fd4a80c5a50 2023-01-24T03:34:42.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.386+0000 7fd4a6ffd700 1 --2- 172.21.15.112:0/3697188182 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd48c0611c0 0x7fd48c063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:42.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.386+0000 7fd4a6ffd700 1 --2- 172.21.15.112:0/3697188182 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd48c0611c0 0x7fd48c063670 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd498009e90 tx=0x7fd498019000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:42.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.390+0000 7fd49effd700 1 -- 172.21.15.112:0/3697188182 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fd49004ccb0 con 0x7fd4a80c5a50 2023-01-24T03:34:42.613 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.611+0000 7fd4a7fff700 1 -- 172.21.15.112:0/3697188182 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4a8155180 con 0x7fd4a80c5a50 2023-01-24T03:34:42.614 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.612+0000 7fd49effd700 1 -- 172.21.15.112:0/3697188182 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5358 (secure 0 0 0) 0x7fd4900502d0 con 0x7fd4a80c5a50 2023-01-24T03:34:42.615 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:34:42.615 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:34:40.823914+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"b","rank":0,"incarnation":22,"state":"up:rejoin","state_seq":20,"addr":"172.21.15.142:6834/612058861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":612058861},{"type":"v1","addr":"172.21.15.142:6834","nonce":612058861}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:34:42.616 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.614+0000 7fd4a7fff700 1 -- 172.21.15.112:0/3697188182 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd48c0611c0 msgr2=0x7fd48c063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:42.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.614+0000 7fd4a7fff700 1 --2- 172.21.15.112:0/3697188182 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd48c0611c0 0x7fd48c063670 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd498009e90 tx=0x7fd498019000 comp rx=0 tx=0).stop 2023-01-24T03:34:42.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.614+0000 7fd4a7fff700 1 -- 172.21.15.112:0/3697188182 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd4a80c5a50 msgr2=0x7fd4a8165850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:42.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.614+0000 7fd4a7fff700 1 --2- 172.21.15.112:0/3697188182 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd4a80c5a50 0x7fd4a8165850 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd49000c4c0 tx=0x7fd490007dc0 comp rx=0 tx=0).stop 2023-01-24T03:34:42.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.614+0000 7fd4a7fff700 1 -- 172.21.15.112:0/3697188182 shutdown_connections 2023-01-24T03:34:42.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.614+0000 7fd4a7fff700 1 --2- 172.21.15.112:0/3697188182 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd48c0611c0 0x7fd48c063670 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:42.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.614+0000 7fd4a7fff700 1 --2- 172.21.15.112:0/3697188182 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd4a8155e20 0x7fd4a80c0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:42.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.614+0000 7fd4a7fff700 1 --2- 172.21.15.112:0/3697188182 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd4a80c5680 0x7fd4a80bf9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:42.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.614+0000 7fd4a7fff700 1 --2- 172.21.15.112:0/3697188182 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd4a80c5a50 0x7fd4a8165850 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:42.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.614+0000 7fd4a7fff700 1 -- 172.21.15.112:0/3697188182 >> 172.21.15.112:0/3697188182 conn(0x7fd4a80685b0 msgr2=0x7fd4a8153b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:42.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.614+0000 7fd4a7fff700 1 -- 172.21.15.112:0/3697188182 shutdown_connections 2023-01-24T03:34:42.619 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:42.614+0000 7fd4a7fff700 1 -- 172.21.15.112:0/3697188182 wait complete. 2023-01-24T03:34:42.619 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 24 2023-01-24T03:34:42.628 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 24, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-01-24T03:30:20.971710+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 24, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5041": { "addr": "172.21.15.142:6834/612058861", "addrs": { "addrvec": [ { "addr": "172.21.15.142:6832", "nonce": 612058861, "type": "v2" }, { "addr": "172.21.15.142:6834", "nonce": 612058861, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5041, "incarnation": 22, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:rejoin", "state_seq": 20 } }, "last_failure": 0, "last_failure_osd_epoch": 32, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-01-24T03:34:40.823914+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5041 } } } ], "standbys": [ { "addr": "172.21.15.142:6837/2181345869", "addrs": { "addrvec": [ { "addr": "172.21.15.142:6836", "nonce": 2181345869, "type": "v2" }, { "addr": "172.21.15.142:6837", "nonce": 2181345869, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6835/330343927", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6834", "nonce": 330343927, "type": "v2" }, { "addr": "172.21.15.112:6835", "nonce": 330343927, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6837/12176817", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6836", "nonce": 12176817, "type": "v2" }, { "addr": "172.21.15.112:6837", "nonce": 12176817, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4276, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6839/1110983841", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6838", "nonce": 1110983841, "type": "v2" }, { "addr": "172.21.15.112:6839", "nonce": 1110983841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4281, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-01-24T03:34:44.629 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:34:44.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.750+0000 7ff4ec944700 1 Processor -- start 2023-01-24T03:34:44.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.750+0000 7ff4ec944700 1 -- start start 2023-01-24T03:34:44.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.751+0000 7ff4ec944700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7ff4e80c7e40 0x7ff4e80b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:44.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.751+0000 7ff4ec944700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7ff4e80ca3e0 0x7ff4e80ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:44.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.751+0000 7ff4ec944700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4e8156240 con 0x7ff4e8155da0 2023-01-24T03:34:44.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.751+0000 7ff4ec944700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4e80b1170 con 0x7ff4e80c7e40 2023-01-24T03:34:44.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.751+0000 7ff4ec944700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4e80b1450 con 0x7ff4e80ca3e0 2023-01-24T03:34:44.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.751+0000 7ff4e6d9d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7ff4e80ca3e0 0x7ff4e80ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:44.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.751+0000 7ff4e6d9d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7ff4e80ca3e0 0x7ff4e80ca7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.112:45362/0 (socket says 172.21.15.112:45362) 2023-01-24T03:34:44.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.751+0000 7ff4e6d9d700 1 -- 172.21.15.112:0/751728323 learned_addr learned my addr 172.21.15.112:0/751728323 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:34:44.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.751+0000 7ff4e659c700 1 --2- 172.21.15.112:0/751728323 >> v2:172.21.15.112:3300/0 conn(0x7ff4e80c7e40 0x7ff4e80b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:44.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.751+0000 7ff4e5d9b700 1 --1- 172.21.15.112:0/751728323 >> v1:172.21.15.142:6789/0 conn(0x7ff4e8155da0 0x7ff4e80ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.112:47646/0 (socket says 172.21.15.112:47646) 2023-01-24T03:34:44.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.752+0000 7ff4e659c700 1 -- 172.21.15.112:0/751728323 >> v2:172.21.15.142:3300/0 conn(0x7ff4e80ca3e0 msgr2=0x7ff4e80ca7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:44.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.752+0000 7ff4e659c700 1 --2- 172.21.15.112:0/751728323 >> v2:172.21.15.142:3300/0 conn(0x7ff4e80ca3e0 0x7ff4e80ca7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:44.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.752+0000 7ff4e659c700 1 -- 172.21.15.112:0/751728323 >> v1:172.21.15.142:6789/0 conn(0x7ff4e8155da0 legacy=0x7ff4e80ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:44.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.752+0000 7ff4e659c700 1 -- 172.21.15.112:0/751728323 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e80b1730 con 0x7ff4e80c7e40 2023-01-24T03:34:44.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.752+0000 7ff4e659c700 1 --2- 172.21.15.112:0/751728323 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff4e80c7e40 0x7ff4e80b0aa0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7ff4e805e810 tx=0x7ff4d8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1dbaab531ddb121 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:44.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.752+0000 7ff4e559a700 1 -- 172.21.15.112:0/751728323 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d800e070 con 0x7ff4e80c7e40 2023-01-24T03:34:44.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.752+0000 7ff4e559a700 1 -- 172.21.15.112:0/751728323 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4d800bb60 con 0x7ff4e80c7e40 2023-01-24T03:34:44.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.752+0000 7ff4e559a700 1 -- 172.21.15.112:0/751728323 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d8012430 con 0x7ff4e80c7e40 2023-01-24T03:34:44.762 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.753+0000 7ff4ec944700 1 -- 172.21.15.112:0/751728323 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff4e80c7e40 msgr2=0x7ff4e80b0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:44.762 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.753+0000 7ff4ec944700 1 --2- 172.21.15.112:0/751728323 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff4e80c7e40 0x7ff4e80b0aa0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7ff4e805e810 tx=0x7ff4d8009780 comp rx=0 tx=0).stop 2023-01-24T03:34:44.762 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.753+0000 7ff4ec944700 1 -- 172.21.15.112:0/751728323 shutdown_connections 2023-01-24T03:34:44.762 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.753+0000 7ff4ec944700 1 --2- 172.21.15.112:0/751728323 >> v2:172.21.15.142:3300/0 conn(0x7ff4e80ca3e0 0x7ff4e80ca7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:44.762 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.753+0000 7ff4ec944700 1 --2- 172.21.15.112:0/751728323 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff4e80c7e40 0x7ff4e80b0aa0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:44.763 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.753+0000 7ff4ec944700 1 -- 172.21.15.112:0/751728323 >> 172.21.15.112:0/751728323 conn(0x7ff4e805e2c0 msgr2=0x7ff4e80cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:44.763 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.753+0000 7ff4ec944700 1 -- 172.21.15.112:0/751728323 shutdown_connections 2023-01-24T03:34:44.763 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.753+0000 7ff4ec944700 1 -- 172.21.15.112:0/751728323 wait complete. 2023-01-24T03:34:44.763 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.754+0000 7ff4ec944700 1 Processor -- start 2023-01-24T03:34:44.763 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.754+0000 7ff4ec944700 1 -- start start 2023-01-24T03:34:44.764 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.754+0000 7ff4ec944700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff4e80c7e40 0x7ff4e8151840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:44.764 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.754+0000 7ff4ec944700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff4e80ca3e0 0x7ff4e814fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:44.764 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.754+0000 7ff4ec944700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff4e8155da0 0x7ff4e80c1b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:44.764 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.754+0000 7ff4ec944700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e8166790 con 0x7ff4e80ca3e0 2023-01-24T03:34:44.764 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.754+0000 7ff4ec944700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e8166c20 con 0x7ff4e80c7e40 2023-01-24T03:34:44.765 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.754+0000 7ff4ec944700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e81520f0 con 0x7ff4e8155da0 2023-01-24T03:34:44.765 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.754+0000 7ff4e5d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff4e80ca3e0 0x7ff4e814fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:44.765 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.755+0000 7ff4e5d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff4e80ca3e0 0x7ff4e814fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52874/0 (socket says 172.21.15.112:52874) 2023-01-24T03:34:44.765 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.755+0000 7ff4e5d9b700 1 -- 172.21.15.112:0/366847260 learned_addr learned my addr 172.21.15.112:0/366847260 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:34:44.765 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.755+0000 7ff4e6d9d700 1 --2- 172.21.15.112:0/366847260 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff4e8155da0 0x7ff4e80c1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:44.765 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.755+0000 7ff4e659c700 1 --2- 172.21.15.112:0/366847260 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff4e80c7e40 0x7ff4e8151840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:44.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.755+0000 7ff4e5d9b700 1 -- 172.21.15.112:0/366847260 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff4e8155da0 msgr2=0x7ff4e80c1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:44.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.755+0000 7ff4e5d9b700 1 --2- 172.21.15.112:0/366847260 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff4e8155da0 0x7ff4e80c1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:44.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.755+0000 7ff4e5d9b700 1 -- 172.21.15.112:0/366847260 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff4e80c7e40 msgr2=0x7ff4e8151840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:44.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.755+0000 7ff4e5d9b700 1 --2- 172.21.15.112:0/366847260 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff4e80c7e40 0x7ff4e8151840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:44.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.755+0000 7ff4e5d9b700 1 -- 172.21.15.112:0/366847260 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d8009a20 con 0x7ff4e80ca3e0 2023-01-24T03:34:44.767 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.755+0000 7ff4e659c700 1 --2- 172.21.15.112:0/366847260 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff4e80c7e40 0x7ff4e8151840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:34:44.767 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.755+0000 7ff4e5d9b700 1 --2- 172.21.15.112:0/366847260 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff4e80ca3e0 0x7ff4e814fc60 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0007fd0 tx=0x7ff4d0007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:44.767 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.755+0000 7ff4de7fc700 1 -- 172.21.15.112:0/366847260 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d00072e0 con 0x7ff4e80ca3e0 2023-01-24T03:34:44.767 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.755+0000 7ff4de7fc700 1 -- 172.21.15.112:0/366847260 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4d0007440 con 0x7ff4e80ca3e0 2023-01-24T03:34:44.767 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.755+0000 7ff4ec944700 1 -- 172.21.15.112:0/366847260 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4e8165e60 con 0x7ff4e80ca3e0 2023-01-24T03:34:44.768 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.755+0000 7ff4de7fc700 1 -- 172.21.15.112:0/366847260 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d0004900 con 0x7ff4e80ca3e0 2023-01-24T03:34:44.768 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.755+0000 7ff4ec944700 1 -- 172.21.15.112:0/366847260 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4e8166420 con 0x7ff4e80ca3e0 2023-01-24T03:34:44.768 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.756+0000 7ff4ec944700 1 -- 172.21.15.112:0/366847260 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4e8166940 con 0x7ff4e80ca3e0 2023-01-24T03:34:44.768 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.757+0000 7ff4de7fc700 1 -- 172.21.15.112:0/366847260 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7ff4d0010070 con 0x7ff4e80ca3e0 2023-01-24T03:34:44.768 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.757+0000 7ff4de7fc700 1 --2- 172.21.15.112:0/366847260 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff4cc0611c0 0x7ff4cc063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:44.769 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.757+0000 7ff4de7fc700 1 -- 172.21.15.112:0/366847260 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 7125+0+0 (secure 0 0 0) 0x7ff4d007d4a0 con 0x7ff4e80ca3e0 2023-01-24T03:34:44.769 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.757+0000 7ff4e659c700 1 --2- 172.21.15.112:0/366847260 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff4cc0611c0 0x7ff4cc063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:44.769 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.757+0000 7ff4e659c700 1 --2- 172.21.15.112:0/366847260 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff4cc0611c0 0x7ff4cc063670 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff4d80097c0 tx=0x7ff4d8019000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:44.770 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.761+0000 7ff4de7fc700 1 -- 172.21.15.112:0/366847260 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7ff4d004d1c0 con 0x7ff4e80ca3e0 2023-01-24T03:34:44.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.981+0000 7ff4ec944700 1 -- 172.21.15.112:0/366847260 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4e81668d0 con 0x7ff4e80ca3e0 2023-01-24T03:34:44.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.982+0000 7ff4de7fc700 1 -- 172.21.15.112:0/366847260 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5358 (secure 0 0 0) 0x7ff4d0021090 con 0x7ff4e80ca3e0 2023-01-24T03:34:44.984 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:34:44.985 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:34:40.823914+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"b","rank":0,"incarnation":22,"state":"up:rejoin","state_seq":20,"addr":"172.21.15.142:6834/612058861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":612058861},{"type":"v1","addr":"172.21.15.142:6834","nonce":612058861}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:34:44.986 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.984+0000 7ff4ec944700 1 -- 172.21.15.112:0/366847260 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff4cc0611c0 msgr2=0x7ff4cc063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:44.986 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.984+0000 7ff4ec944700 1 --2- 172.21.15.112:0/366847260 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff4cc0611c0 0x7ff4cc063670 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff4d80097c0 tx=0x7ff4d8019000 comp rx=0 tx=0).stop 2023-01-24T03:34:44.987 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.984+0000 7ff4ec944700 1 -- 172.21.15.112:0/366847260 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff4e80ca3e0 msgr2=0x7ff4e814fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:44.987 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.984+0000 7ff4ec944700 1 --2- 172.21.15.112:0/366847260 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff4e80ca3e0 0x7ff4e814fc60 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0007fd0 tx=0x7ff4d0007df0 comp rx=0 tx=0).stop 2023-01-24T03:34:44.987 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.984+0000 7ff4ec944700 1 -- 172.21.15.112:0/366847260 shutdown_connections 2023-01-24T03:34:44.987 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.984+0000 7ff4ec944700 1 --2- 172.21.15.112:0/366847260 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff4cc0611c0 0x7ff4cc063670 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:44.987 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.984+0000 7ff4ec944700 1 --2- 172.21.15.112:0/366847260 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff4e8155da0 0x7ff4e80c1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:44.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.984+0000 7ff4ec944700 1 --2- 172.21.15.112:0/366847260 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff4e80c7e40 0x7ff4e8151840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:44.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.984+0000 7ff4ec944700 1 --2- 172.21.15.112:0/366847260 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff4e80ca3e0 0x7ff4e814fc60 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:44.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.984+0000 7ff4ec944700 1 -- 172.21.15.112:0/366847260 >> 172.21.15.112:0/366847260 conn(0x7ff4e805e2c0 msgr2=0x7ff4e80ae9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:44.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.984+0000 7ff4ec944700 1 -- 172.21.15.112:0/366847260 shutdown_connections 2023-01-24T03:34:44.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:44.984+0000 7ff4ec944700 1 -- 172.21.15.112:0/366847260 wait complete. 2023-01-24T03:34:44.989 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 24 2023-01-24T03:34:44.998 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:34:46.999 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:34:47.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.121+0000 7f9d89d29700 1 Processor -- start 2023-01-24T03:34:47.124 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.121+0000 7f9d89d29700 1 -- start start 2023-01-24T03:34:47.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.122+0000 7f9d89d29700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f9d840b0aa0 0x7f9d84155da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:47.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.122+0000 7f9d89d29700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f9d840ca3e0 0x7f9d840ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:47.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.122+0000 7f9d89d29700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d840b0f40 con 0x7f9d840c7e40 2023-01-24T03:34:47.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.122+0000 7f9d89d29700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d840b1170 con 0x7f9d840ca3e0 2023-01-24T03:34:47.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.122+0000 7f9d89d29700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d840b1450 con 0x7f9d840b0aa0 2023-01-24T03:34:47.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.122+0000 7f9d83fff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f9d840ca3e0 0x7f9d840ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:47.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.122+0000 7f9d83fff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f9d840ca3e0 0x7f9d840ca7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52882/0 (socket says 172.21.15.112:52882) 2023-01-24T03:34:47.133 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.122+0000 7f9d83fff700 1 -- 172.21.15.112:0/2161612511 learned_addr learned my addr 172.21.15.112:0/2161612511 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:34:47.133 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.123+0000 7f9d82ffd700 1 --2- 172.21.15.112:0/2161612511 >> v2:172.21.15.142:3300/0 conn(0x7f9d840b0aa0 0x7f9d84155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:47.133 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.123+0000 7f9d83fff700 1 -- 172.21.15.112:0/2161612511 >> v2:172.21.15.142:3300/0 conn(0x7f9d840b0aa0 msgr2=0x7f9d84155da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:47.133 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.123+0000 7f9d83fff700 1 --2- 172.21.15.112:0/2161612511 >> v2:172.21.15.142:3300/0 conn(0x7f9d840b0aa0 0x7f9d84155da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:47.133 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.123+0000 7f9d83fff700 1 -- 172.21.15.112:0/2161612511 >> v1:172.21.15.142:6789/0 conn(0x7f9d840c7e40 legacy=0x7f9d840ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:47.134 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.123+0000 7f9d83fff700 1 -- 172.21.15.112:0/2161612511 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d840b1730 con 0x7f9d840ca3e0 2023-01-24T03:34:47.134 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.123+0000 7f9d82ffd700 1 --2- 172.21.15.112:0/2161612511 >> v2:172.21.15.142:3300/0 conn(0x7f9d840b0aa0 0x7f9d84155da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:34:47.134 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.123+0000 7f9d83fff700 1 --2- 172.21.15.112:0/2161612511 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d840ca3e0 0x7f9d840ca7e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9d84152ea0 tx=0x7f9d7c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b74c9e37b11f2d66 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:47.134 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.123+0000 7f9d827fc700 1 -- 172.21.15.112:0/2161612511 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c011630 con 0x7f9d840ca3e0 2023-01-24T03:34:47.134 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.123+0000 7f9d827fc700 1 -- 172.21.15.112:0/2161612511 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d7c0117d0 con 0x7f9d840ca3e0 2023-01-24T03:34:47.135 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.123+0000 7f9d827fc700 1 -- 172.21.15.112:0/2161612511 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c011b90 con 0x7f9d840ca3e0 2023-01-24T03:34:47.135 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.123+0000 7f9d89d29700 1 -- 172.21.15.112:0/2161612511 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d840ca3e0 msgr2=0x7f9d840ca7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:47.135 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.123+0000 7f9d89d29700 1 --2- 172.21.15.112:0/2161612511 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d840ca3e0 0x7f9d840ca7e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9d84152ea0 tx=0x7f9d7c0090f0 comp rx=0 tx=0).stop 2023-01-24T03:34:47.135 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.124+0000 7f9d89d29700 1 -- 172.21.15.112:0/2161612511 shutdown_connections 2023-01-24T03:34:47.135 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.124+0000 7f9d89d29700 1 --2- 172.21.15.112:0/2161612511 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d840ca3e0 0x7f9d840ca7e0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:47.136 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.124+0000 7f9d89d29700 1 --2- 172.21.15.112:0/2161612511 >> v2:172.21.15.142:3300/0 conn(0x7f9d840b0aa0 0x7f9d84155da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:47.136 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.124+0000 7f9d89d29700 1 -- 172.21.15.112:0/2161612511 >> 172.21.15.112:0/2161612511 conn(0x7f9d8405e2c0 msgr2=0x7f9d840cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:47.136 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.124+0000 7f9d89d29700 1 -- 172.21.15.112:0/2161612511 shutdown_connections 2023-01-24T03:34:47.136 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.124+0000 7f9d89d29700 1 -- 172.21.15.112:0/2161612511 wait complete. 2023-01-24T03:34:47.136 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.125+0000 7f9d89d29700 1 Processor -- start 2023-01-24T03:34:47.137 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.125+0000 7f9d89d29700 1 -- start start 2023-01-24T03:34:47.137 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.125+0000 7f9d89d29700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d840b0aa0 0x7f9d841698b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:47.137 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.125+0000 7f9d89d29700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d840c7e40 0x7f9d840cb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:47.137 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.126+0000 7f9d89d29700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d840ca3e0 0x7f9d840c4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:47.137 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.126+0000 7f9d89d29700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d841668b0 con 0x7f9d840b0aa0 2023-01-24T03:34:47.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.126+0000 7f9d89d29700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d84169df0 con 0x7f9d840c7e40 2023-01-24T03:34:47.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.126+0000 7f9d89d29700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d84165a50 con 0x7f9d840ca3e0 2023-01-24T03:34:47.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.126+0000 7f9d837fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d840b0aa0 0x7f9d841698b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:47.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.126+0000 7f9d837fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d840b0aa0 0x7f9d841698b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52886/0 (socket says 172.21.15.112:52886) 2023-01-24T03:34:47.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.126+0000 7f9d837fe700 1 -- 172.21.15.112:0/550819458 learned_addr learned my addr 172.21.15.112:0/550819458 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:34:47.139 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.126+0000 7f9d82ffd700 1 --2- 172.21.15.112:0/550819458 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d840c7e40 0x7f9d840cb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:47.139 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.126+0000 7f9d83fff700 1 --2- 172.21.15.112:0/550819458 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d840ca3e0 0x7f9d840c4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:47.139 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.126+0000 7f9d837fe700 1 -- 172.21.15.112:0/550819458 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d840ca3e0 msgr2=0x7f9d840c4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:47.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.126+0000 7f9d837fe700 1 --2- 172.21.15.112:0/550819458 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d840ca3e0 0x7f9d840c4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:47.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.126+0000 7f9d837fe700 1 -- 172.21.15.112:0/550819458 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d840c7e40 msgr2=0x7f9d840cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:47.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.126+0000 7f9d837fe700 1 --2- 172.21.15.112:0/550819458 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d840c7e40 0x7f9d840cb4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:47.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.126+0000 7f9d837fe700 1 -- 172.21.15.112:0/550819458 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d7c009370 con 0x7f9d840b0aa0 2023-01-24T03:34:47.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.126+0000 7f9d837fe700 1 --2- 172.21.15.112:0/550819458 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d840b0aa0 0x7f9d841698b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f9d7400c7c0 tx=0x7f9d740015c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:47.141 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.126+0000 7f9d6b7fe700 1 -- 172.21.15.112:0/550819458 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d74012630 con 0x7f9d840b0aa0 2023-01-24T03:34:47.141 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.126+0000 7f9d6b7fe700 1 -- 172.21.15.112:0/550819458 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d74012790 con 0x7f9d840b0aa0 2023-01-24T03:34:47.141 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.126+0000 7f9d89d29700 1 -- 172.21.15.112:0/550819458 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d84165d30 con 0x7f9d840b0aa0 2023-01-24T03:34:47.141 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.126+0000 7f9d6b7fe700 1 -- 172.21.15.112:0/550819458 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d74012b50 con 0x7f9d840b0aa0 2023-01-24T03:34:47.141 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.126+0000 7f9d89d29700 1 -- 172.21.15.112:0/550819458 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d841662f0 con 0x7f9d840b0aa0 2023-01-24T03:34:47.142 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.127+0000 7f9d89d29700 1 -- 172.21.15.112:0/550819458 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d8404bb80 con 0x7f9d840b0aa0 2023-01-24T03:34:47.142 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.127+0000 7f9d6b7fe700 1 -- 172.21.15.112:0/550819458 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f9d74018020 con 0x7f9d840b0aa0 2023-01-24T03:34:47.142 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.128+0000 7f9d6b7fe700 1 --2- 172.21.15.112:0/550819458 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9d640611c0 0x7f9d64063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:47.142 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.128+0000 7f9d6b7fe700 1 -- 172.21.15.112:0/550819458 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 7125+0+0 (secure 0 0 0) 0x7f9d7407d910 con 0x7f9d840b0aa0 2023-01-24T03:34:47.142 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.128+0000 7f9d82ffd700 1 --2- 172.21.15.112:0/550819458 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9d640611c0 0x7f9d64063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:47.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.128+0000 7f9d82ffd700 1 --2- 172.21.15.112:0/550819458 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9d640611c0 0x7f9d64063670 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9d841563a0 tx=0x7f9d78008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:47.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.132+0000 7f9d6b7fe700 1 -- 172.21.15.112:0/550819458 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f9d7404d610 con 0x7f9d840b0aa0 2023-01-24T03:34:47.353 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.351+0000 7f9d89d29700 1 -- 172.21.15.112:0/550819458 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d8406efe0 con 0x7f9d840b0aa0 2023-01-24T03:34:47.354 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.352+0000 7f9d6b7fe700 1 -- 172.21.15.112:0/550819458 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+5358 (secure 0 0 0) 0x7f9d74050c30 con 0x7f9d840b0aa0 2023-01-24T03:34:47.354 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:34:47.355 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:34:46.842740+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":22,"addr":"172.21.15.142:6834/612058861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":612058861},{"type":"v1","addr":"172.21.15.142:6834","nonce":612058861}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:34:47.356 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.354+0000 7f9d89d29700 1 -- 172.21.15.112:0/550819458 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9d640611c0 msgr2=0x7f9d64063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:47.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.354+0000 7f9d89d29700 1 --2- 172.21.15.112:0/550819458 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9d640611c0 0x7f9d64063670 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9d841563a0 tx=0x7f9d78008040 comp rx=0 tx=0).stop 2023-01-24T03:34:47.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.354+0000 7f9d89d29700 1 -- 172.21.15.112:0/550819458 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d840b0aa0 msgr2=0x7f9d841698b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:47.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.354+0000 7f9d89d29700 1 --2- 172.21.15.112:0/550819458 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d840b0aa0 0x7f9d841698b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f9d7400c7c0 tx=0x7f9d740015c0 comp rx=0 tx=0).stop 2023-01-24T03:34:47.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.354+0000 7f9d89d29700 1 -- 172.21.15.112:0/550819458 shutdown_connections 2023-01-24T03:34:47.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.354+0000 7f9d89d29700 1 --2- 172.21.15.112:0/550819458 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9d640611c0 0x7f9d64063670 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:47.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.354+0000 7f9d89d29700 1 --2- 172.21.15.112:0/550819458 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9d840b0aa0 0x7f9d841698b0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:47.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.354+0000 7f9d89d29700 1 --2- 172.21.15.112:0/550819458 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9d840ca3e0 0x7f9d840c4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:47.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.354+0000 7f9d89d29700 1 --2- 172.21.15.112:0/550819458 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d840c7e40 0x7f9d840cb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:47.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.354+0000 7f9d89d29700 1 -- 172.21.15.112:0/550819458 >> 172.21.15.112:0/550819458 conn(0x7f9d8405e2c0 msgr2=0x7f9d840c8210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:47.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.354+0000 7f9d89d29700 1 -- 172.21.15.112:0/550819458 shutdown_connections 2023-01-24T03:34:47.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:47.354+0000 7f9d89d29700 1 -- 172.21.15.112:0/550819458 wait complete. 2023-01-24T03:34:47.359 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 25 2023-01-24T03:34:47.368 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=4897 2023-01-24T03:34:47.369 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before reviving mds.d 2023-01-24T03:34:52.014 INFO:tasks.workunit.client.0.smithi112.stdout:./240/.snap/snap-subdir-test 2023-01-24T03:34:52.020 INFO:tasks.workunit.client.0.smithi112.stdout:./156/.snap/snap-subdir-test 2023-01-24T03:34:52.025 INFO:tasks.workunit.client.0.smithi112.stdout:./131/.snap/snap-subdir-test 2023-01-24T03:34:52.029 INFO:tasks.workunit.client.0.smithi112.stdout:./123/.snap/snap-subdir-test 2023-01-24T03:34:52.034 INFO:tasks.workunit.client.0.smithi112.stdout:./160/.snap/snap-subdir-test 2023-01-24T03:34:52.039 INFO:tasks.workunit.client.0.smithi112.stdout:./165/.snap/snap-subdir-test 2023-01-24T03:34:52.043 INFO:tasks.workunit.client.0.smithi112.stdout:./101/.snap/snap-subdir-test 2023-01-24T03:34:52.048 INFO:tasks.workunit.client.0.smithi112.stdout:./148/.snap/snap-subdir-test 2023-01-24T03:34:52.052 INFO:tasks.workunit.client.0.smithi112.stdout:./168/.snap/snap-subdir-test 2023-01-24T03:34:52.056 INFO:tasks.workunit.client.0.smithi112.stdout:./179/.snap/snap-subdir-test 2023-01-24T03:34:52.061 INFO:tasks.workunit.client.0.smithi112.stdout:./257/.snap/snap-subdir-test 2023-01-24T03:34:52.065 INFO:tasks.workunit.client.0.smithi112.stdout:./252/.snap/snap-subdir-test 2023-01-24T03:34:52.070 INFO:tasks.workunit.client.0.smithi112.stdout:./132/.snap/snap-subdir-test 2023-01-24T03:34:52.074 INFO:tasks.workunit.client.0.smithi112.stdout:./236/.snap/snap-subdir-test 2023-01-24T03:34:52.077 INFO:tasks.workunit.client.0.smithi112.stdout:./211/.snap/snap-subdir-test 2023-01-24T03:34:52.082 INFO:tasks.workunit.client.0.smithi112.stdout:./199/.snap/snap-subdir-test 2023-01-24T03:34:52.086 INFO:tasks.workunit.client.0.smithi112.stdout:./205/.snap/snap-subdir-test 2023-01-24T03:34:52.091 INFO:tasks.workunit.client.0.smithi112.stdout:./163/.snap/snap-subdir-test 2023-01-24T03:34:52.095 INFO:tasks.workunit.client.0.smithi112.stdout:./261/.snap/snap-subdir-test 2023-01-24T03:34:52.100 INFO:tasks.workunit.client.0.smithi112.stdout:./201/.snap/snap-subdir-test 2023-01-24T03:34:52.104 INFO:tasks.workunit.client.0.smithi112.stdout:./282/.snap/snap-subdir-test 2023-01-24T03:34:52.108 INFO:tasks.workunit.client.0.smithi112.stdout:./296/.snap/snap-subdir-test 2023-01-24T03:34:52.112 INFO:tasks.workunit.client.0.smithi112.stdout:./217/.snap/snap-subdir-test 2023-01-24T03:34:52.117 INFO:tasks.workunit.client.0.smithi112.stdout:./153/.snap/snap-subdir-test 2023-01-24T03:34:52.121 INFO:tasks.workunit.client.0.smithi112.stdout:./206/.snap/snap-subdir-test 2023-01-24T03:34:52.125 INFO:tasks.workunit.client.0.smithi112.stdout:./122/.snap/snap-subdir-test 2023-01-24T03:34:52.129 INFO:tasks.workunit.client.0.smithi112.stdout:./162/.snap/snap-subdir-test 2023-01-24T03:34:52.133 INFO:tasks.workunit.client.0.smithi112.stdout:./197/.snap/snap-subdir-test 2023-01-24T03:34:52.137 INFO:tasks.workunit.client.0.smithi112.stdout:./216/.snap/snap-subdir-test 2023-01-24T03:34:53.688 INFO:tasks.ceph.mds.b.smithi142.stderr:2023-01-24T03:34:53.686+0000 7fa3410de700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-01-24T03:34:54.173 INFO:tasks.workunit.client.0.smithi112.stdout:./254/.snap/snap-subdir-test 2023-01-24T03:34:54.176 INFO:tasks.workunit.client.0.smithi112.stdout:./113/.snap/snap-subdir-test 2023-01-24T03:34:54.181 INFO:tasks.workunit.client.0.smithi112.stdout:./152/.snap/snap-subdir-test 2023-01-24T03:34:54.185 INFO:tasks.workunit.client.0.smithi112.stdout:./121/.snap/snap-subdir-test 2023-01-24T03:34:54.188 INFO:tasks.workunit.client.0.smithi112.stdout:./118/.snap/snap-subdir-test 2023-01-24T03:34:54.192 INFO:tasks.workunit.client.0.smithi112.stdout:./196/.snap/snap-subdir-test 2023-01-24T03:34:54.195 INFO:tasks.workunit.client.0.smithi112.stdout:./124/.snap/snap-subdir-test 2023-01-24T03:34:54.199 INFO:tasks.workunit.client.0.smithi112.stdout:./239/.snap/snap-subdir-test 2023-01-24T03:34:54.204 INFO:tasks.workunit.client.0.smithi112.stdout:./188/.snap/snap-subdir-test 2023-01-24T03:34:54.207 INFO:tasks.workunit.client.0.smithi112.stdout:./287/.snap/snap-subdir-test 2023-01-24T03:34:54.211 INFO:tasks.workunit.client.0.smithi112.stdout:./275/.snap/snap-subdir-test 2023-01-24T03:34:54.215 INFO:tasks.workunit.client.0.smithi112.stdout:./259/.snap/snap-subdir-test 2023-01-24T03:34:54.219 INFO:tasks.workunit.client.0.smithi112.stdout:./224/.snap/snap-subdir-test 2023-01-24T03:34:54.222 INFO:tasks.workunit.client.0.smithi112.stdout:./264/.snap/snap-subdir-test 2023-01-24T03:34:54.225 INFO:tasks.workunit.client.0.smithi112.stdout:./145/.snap/snap-subdir-test 2023-01-24T03:34:54.229 INFO:tasks.workunit.client.0.smithi112.stdout:./279/.snap/snap-subdir-test 2023-01-24T03:34:54.232 INFO:tasks.workunit.client.0.smithi112.stdout:./263/.snap/snap-subdir-test 2023-01-24T03:34:54.236 INFO:tasks.workunit.client.0.smithi112.stdout:./130/.snap/snap-subdir-test 2023-01-24T03:34:54.239 INFO:tasks.workunit.client.0.smithi112.stdout:./193/.snap/snap-subdir-test 2023-01-24T03:34:54.242 INFO:tasks.workunit.client.0.smithi112.stdout:./223/.snap/snap-subdir-test 2023-01-24T03:34:54.245 INFO:tasks.workunit.client.0.smithi112.stdout:./207/.snap/snap-subdir-test 2023-01-24T03:34:54.248 INFO:tasks.workunit.client.0.smithi112.stdout:./276/.snap/snap-subdir-test 2023-01-24T03:34:54.252 INFO:tasks.workunit.client.0.smithi112.stdout:./271/.snap/snap-subdir-test 2023-01-24T03:34:54.255 INFO:tasks.workunit.client.0.smithi112.stdout:./251/.snap/snap-subdir-test 2023-01-24T03:34:54.258 INFO:tasks.workunit.client.0.smithi112.stdout:./175/.snap/snap-subdir-test 2023-01-24T03:34:54.261 INFO:tasks.workunit.client.0.smithi112.stdout:./222/.snap/snap-subdir-test 2023-01-24T03:34:54.264 INFO:tasks.workunit.client.0.smithi112.stdout:./220/.snap/snap-subdir-test 2023-01-24T03:34:54.488 INFO:tasks.workunit.client.0.smithi112.stdout:./203/.snap/snap-subdir-test 2023-01-24T03:34:54.492 INFO:tasks.workunit.client.0.smithi112.stdout:./181/.snap/snap-subdir-test 2023-01-24T03:34:54.496 INFO:tasks.workunit.client.0.smithi112.stdout:./286/.snap/snap-subdir-test 2023-01-24T03:34:54.501 INFO:tasks.workunit.client.0.smithi112.stdout:./127/.snap/snap-subdir-test 2023-01-24T03:34:54.505 INFO:tasks.workunit.client.0.smithi112.stdout:./108/.snap/snap-subdir-test 2023-01-24T03:34:54.510 INFO:tasks.workunit.client.0.smithi112.stdout:./183/.snap/snap-subdir-test 2023-01-24T03:34:54.514 INFO:tasks.workunit.client.0.smithi112.stdout:./294/.snap/snap-subdir-test 2023-01-24T03:34:54.518 INFO:tasks.workunit.client.0.smithi112.stdout:./235/.snap/snap-subdir-test 2023-01-24T03:34:54.521 INFO:tasks.workunit.client.0.smithi112.stdout:./177/.snap/snap-subdir-test 2023-01-24T03:34:54.526 INFO:tasks.workunit.client.0.smithi112.stdout:./169/.snap/snap-subdir-test 2023-01-24T03:34:54.529 INFO:tasks.workunit.client.0.smithi112.stdout:./212/.snap/snap-subdir-test 2023-01-24T03:34:54.533 INFO:tasks.workunit.client.0.smithi112.stdout:./154/.snap/snap-subdir-test 2023-01-24T03:34:54.537 INFO:tasks.workunit.client.0.smithi112.stdout:./171/.snap/snap-subdir-test 2023-01-24T03:34:54.541 INFO:tasks.workunit.client.0.smithi112.stdout:./139/.snap/snap-subdir-test 2023-01-24T03:34:54.545 INFO:tasks.workunit.client.0.smithi112.stdout:./102/.snap/snap-subdir-test 2023-01-24T03:34:54.549 INFO:tasks.workunit.client.0.smithi112.stdout:./129/.snap/snap-subdir-test 2023-01-24T03:34:55.370 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2023-01-24T03:34:55.371 INFO:tasks.ceph.mds.d:Restarting daemon 2023-01-24T03:34:55.371 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-01-24T03:34:55.374 INFO:tasks.ceph.mds.d:Started 2023-01-24T03:34:55.374 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:34:55.450 INFO:tasks.ceph.mds.d.smithi142.stdout:starting mds.d at 2023-01-24T03:34:55.502 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.500+0000 7f1af0d6d700 1 Processor -- start 2023-01-24T03:34:55.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.500+0000 7f1af0d6d700 1 -- start start 2023-01-24T03:34:55.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.501+0000 7f1af0d6d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1aec0c7e40 0x7f1aec0b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:55.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.501+0000 7f1af0d6d700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f1aec0ca3e0 0x7f1aec0ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:55.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.501+0000 7f1af0d6d700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aec156240 con 0x7f1aec155da0 2023-01-24T03:34:55.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.501+0000 7f1af0d6d700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aec0b1170 con 0x7f1aec0ca3e0 2023-01-24T03:34:55.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.501+0000 7f1af0d6d700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aec0b1450 con 0x7f1aec0c7e40 2023-01-24T03:34:55.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.502+0000 7f1aead9d700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f1aec0ca3e0 0x7f1aec0ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:55.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.502+0000 7f1aead9d700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f1aec0ca3e0 0x7f1aec0ca7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52904/0 (socket says 172.21.15.112:52904) 2023-01-24T03:34:55.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.502+0000 7f1aead9d700 1 -- 172.21.15.112:0/674289706 learned_addr learned my addr 172.21.15.112:0/674289706 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:34:55.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.502+0000 7f1aea59c700 1 --2- 172.21.15.112:0/674289706 >> v2:172.21.15.142:3300/0 conn(0x7f1aec0c7e40 0x7f1aec0b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:55.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.502+0000 7f1aead9d700 1 -- 172.21.15.112:0/674289706 >> v2:172.21.15.142:3300/0 conn(0x7f1aec0c7e40 msgr2=0x7f1aec0b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:55.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.502+0000 7f1aead9d700 1 --2- 172.21.15.112:0/674289706 >> v2:172.21.15.142:3300/0 conn(0x7f1aec0c7e40 0x7f1aec0b0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:55.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.502+0000 7f1aead9d700 1 -- 172.21.15.112:0/674289706 >> v1:172.21.15.142:6789/0 conn(0x7f1aec155da0 legacy=0x7f1aec0ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:55.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.502+0000 7f1aead9d700 1 -- 172.21.15.112:0/674289706 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aec0b1730 con 0x7f1aec0ca3e0 2023-01-24T03:34:55.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.502+0000 7f1aead9d700 1 --2- 172.21.15.112:0/674289706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1aec0ca3e0 0x7f1aec0ca7e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f1aec089cf0 tx=0x7f1ae40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7832561c80dd4d server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:55.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.502+0000 7f1ae959a700 1 -- 172.21.15.112:0/674289706 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae4011630 con 0x7f1aec0ca3e0 2023-01-24T03:34:55.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.502+0000 7f1ae959a700 1 -- 172.21.15.112:0/674289706 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ae40117b0 con 0x7f1aec0ca3e0 2023-01-24T03:34:55.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.502+0000 7f1ae959a700 1 -- 172.21.15.112:0/674289706 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae4011b70 con 0x7f1aec0ca3e0 2023-01-24T03:34:55.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.503+0000 7f1af0d6d700 1 -- 172.21.15.112:0/674289706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1aec0ca3e0 msgr2=0x7f1aec0ca7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:55.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.503+0000 7f1af0d6d700 1 --2- 172.21.15.112:0/674289706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1aec0ca3e0 0x7f1aec0ca7e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f1aec089cf0 tx=0x7f1ae40090f0 comp rx=0 tx=0).stop 2023-01-24T03:34:55.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.503+0000 7f1af0d6d700 1 -- 172.21.15.112:0/674289706 shutdown_connections 2023-01-24T03:34:55.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.503+0000 7f1af0d6d700 1 --2- 172.21.15.112:0/674289706 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1aec0ca3e0 0x7f1aec0ca7e0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:55.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.503+0000 7f1af0d6d700 1 --2- 172.21.15.112:0/674289706 >> v2:172.21.15.142:3300/0 conn(0x7f1aec0c7e40 0x7f1aec0b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:55.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.503+0000 7f1af0d6d700 1 -- 172.21.15.112:0/674289706 >> 172.21.15.112:0/674289706 conn(0x7f1aec05e2c0 msgr2=0x7f1aec0cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:55.510 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.503+0000 7f1af0d6d700 1 -- 172.21.15.112:0/674289706 shutdown_connections 2023-01-24T03:34:55.510 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.503+0000 7f1af0d6d700 1 -- 172.21.15.112:0/674289706 wait complete. 2023-01-24T03:34:55.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.504+0000 7f1af0d6d700 1 Processor -- start 2023-01-24T03:34:55.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.504+0000 7f1af0d6d700 1 -- start start 2023-01-24T03:34:55.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.505+0000 7f1af0d6d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1aec0c7e40 0x7f1aec151140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:55.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.505+0000 7f1af0d6d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1aec0ca3e0 0x7f1aec155390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:55.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.505+0000 7f1af0d6d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1aec155da0 0x7f1aec14ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:55.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.505+0000 7f1af0d6d700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aec1505b0 con 0x7f1aec0ca3e0 2023-01-24T03:34:55.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.505+0000 7f1af0d6d700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aec165e40 con 0x7f1aec0c7e40 2023-01-24T03:34:55.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.505+0000 7f1af0d6d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aec0c34a0 con 0x7f1aec155da0 2023-01-24T03:34:55.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.505+0000 7f1ae9d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1aec0ca3e0 0x7f1aec155390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:55.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.505+0000 7f1ae9d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1aec0ca3e0 0x7f1aec155390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52908/0 (socket says 172.21.15.112:52908) 2023-01-24T03:34:55.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.505+0000 7f1ae9d9b700 1 -- 172.21.15.112:0/2895487744 learned_addr learned my addr 172.21.15.112:0/2895487744 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:34:55.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.505+0000 7f1aea59c700 1 --2- 172.21.15.112:0/2895487744 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1aec0c7e40 0x7f1aec151140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:55.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.505+0000 7f1aead9d700 1 --2- 172.21.15.112:0/2895487744 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1aec155da0 0x7f1aec14ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:55.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.505+0000 7f1ae9d9b700 1 -- 172.21.15.112:0/2895487744 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1aec155da0 msgr2=0x7f1aec14ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:55.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.505+0000 7f1ae9d9b700 1 --2- 172.21.15.112:0/2895487744 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1aec155da0 0x7f1aec14ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:55.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.505+0000 7f1ae9d9b700 1 -- 172.21.15.112:0/2895487744 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1aec0c7e40 msgr2=0x7f1aec151140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:55.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.505+0000 7f1ae9d9b700 1 --2- 172.21.15.112:0/2895487744 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1aec0c7e40 0x7f1aec151140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:55.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.505+0000 7f1ae9d9b700 1 -- 172.21.15.112:0/2895487744 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae4009370 con 0x7f1aec0ca3e0 2023-01-24T03:34:55.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.505+0000 7f1aea59c700 1 --2- 172.21.15.112:0/2895487744 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1aec0c7e40 0x7f1aec151140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-01-24T03:34:55.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.505+0000 7f1aead9d700 1 --2- 172.21.15.112:0/2895487744 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1aec155da0 0x7f1aec14ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-01-24T03:34:55.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.505+0000 7f1ae9d9b700 1 --2- 172.21.15.112:0/2895487744 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1aec0ca3e0 0x7f1aec155390 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f1ad400c440 tx=0x7f1ad4007de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:55.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.505+0000 7f1ae27fc700 1 -- 172.21.15.112:0/2895487744 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad4011590 con 0x7f1aec0ca3e0 2023-01-24T03:34:55.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.505+0000 7f1ae27fc700 1 -- 172.21.15.112:0/2895487744 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad4011710 con 0x7f1aec0ca3e0 2023-01-24T03:34:55.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.505+0000 7f1af0d6d700 1 -- 172.21.15.112:0/2895487744 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aec165b80 con 0x7f1aec0ca3e0 2023-01-24T03:34:55.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.506+0000 7f1ae27fc700 1 -- 172.21.15.112:0/2895487744 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad4011ad0 con 0x7f1aec0ca3e0 2023-01-24T03:34:55.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.506+0000 7f1af0d6d700 1 -- 172.21.15.112:0/2895487744 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aec0c2e50 con 0x7f1aec0ca3e0 2023-01-24T03:34:55.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.506+0000 7f1af0d6d700 1 -- 172.21.15.112:0/2895487744 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aec0c3610 con 0x7f1aec0ca3e0 2023-01-24T03:34:55.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.507+0000 7f1ae27fc700 1 -- 172.21.15.112:0/2895487744 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f1ad4010050 con 0x7f1aec0ca3e0 2023-01-24T03:34:55.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.507+0000 7f1ae27fc700 1 --2- 172.21.15.112:0/2895487744 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f1ad00611c0 0x7f1ad0063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:55.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.507+0000 7f1ae27fc700 1 -- 172.21.15.112:0/2895487744 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 7125+0+0 (secure 0 0 0) 0x7f1ad407d450 con 0x7f1aec0ca3e0 2023-01-24T03:34:55.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.507+0000 7f1aea59c700 1 --2- 172.21.15.112:0/2895487744 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f1ad00611c0 0x7f1ad0063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:55.519 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.507+0000 7f1aea59c700 1 --2- 172.21.15.112:0/2895487744 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f1ad00611c0 0x7f1ad0063670 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1adc007460 tx=0x7f1adc00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:55.519 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.511+0000 7f1ae27fc700 1 -- 172.21.15.112:0/2895487744 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f1ad404d150 con 0x7f1aec0ca3e0 2023-01-24T03:34:55.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.734+0000 7f1af0d6d700 1 -- 172.21.15.112:0/2895487744 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1aec04bb80 con 0x7f1aec0ca3e0 2023-01-24T03:34:55.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.735+0000 7f1ae27fc700 1 -- 172.21.15.112:0/2895487744 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+5358 (secure 0 0 0) 0x7f1ad4050770 con 0x7f1aec0ca3e0 2023-01-24T03:34:55.737 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:34:55.737 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:34:46.842740+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":22,"addr":"172.21.15.142:6834/612058861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":612058861},{"type":"v1","addr":"172.21.15.142:6834","nonce":612058861}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:34:55.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.736+0000 7f1af0d6d700 1 -- 172.21.15.112:0/2895487744 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f1ad00611c0 msgr2=0x7f1ad0063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:55.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.736+0000 7f1af0d6d700 1 --2- 172.21.15.112:0/2895487744 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f1ad00611c0 0x7f1ad0063670 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1adc007460 tx=0x7f1adc00b040 comp rx=0 tx=0).stop 2023-01-24T03:34:55.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.736+0000 7f1af0d6d700 1 -- 172.21.15.112:0/2895487744 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1aec0ca3e0 msgr2=0x7f1aec155390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:55.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.736+0000 7f1af0d6d700 1 --2- 172.21.15.112:0/2895487744 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1aec0ca3e0 0x7f1aec155390 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f1ad400c440 tx=0x7f1ad4007de0 comp rx=0 tx=0).stop 2023-01-24T03:34:55.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.736+0000 7f1af0d6d700 1 -- 172.21.15.112:0/2895487744 shutdown_connections 2023-01-24T03:34:55.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.736+0000 7f1af0d6d700 1 --2- 172.21.15.112:0/2895487744 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f1ad00611c0 0x7f1ad0063670 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:55.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.736+0000 7f1af0d6d700 1 --2- 172.21.15.112:0/2895487744 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1aec155da0 0x7f1aec14ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:55.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.736+0000 7f1af0d6d700 1 --2- 172.21.15.112:0/2895487744 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1aec0c7e40 0x7f1aec151140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:55.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.736+0000 7f1af0d6d700 1 --2- 172.21.15.112:0/2895487744 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1aec0ca3e0 0x7f1aec155390 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:55.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.736+0000 7f1af0d6d700 1 -- 172.21.15.112:0/2895487744 >> 172.21.15.112:0/2895487744 conn(0x7f1aec05e2c0 msgr2=0x7f1aec0cb100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:55.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.737+0000 7f1af0d6d700 1 -- 172.21.15.112:0/2895487744 shutdown_connections 2023-01-24T03:34:55.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:55.737+0000 7f1af0d6d700 1 -- 172.21.15.112:0/2895487744 wait complete. 2023-01-24T03:34:55.741 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 25 2023-01-24T03:34:55.750 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2023-01-24T03:34:57.751 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:34:57.895 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.892+0000 7f6bdd87c700 1 Processor -- start 2023-01-24T03:34:57.895 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.892+0000 7f6bdd87c700 1 -- start start 2023-01-24T03:34:57.898 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.893+0000 7f6bdd87c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f6bd814c290 0x7f6bd806ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:57.898 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.893+0000 7f6bdd87c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bd814c660 0x7f6bd80c1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:57.898 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.893+0000 7f6bdd87c700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bd80c18f0 con 0x7f6bd80c22c0 2023-01-24T03:34:57.898 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.893+0000 7f6bdd87c700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bd80c1030 con 0x7f6bd814c290 2023-01-24T03:34:57.899 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.893+0000 7f6bdd87c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd80c2740 con 0x7f6bd814c660 2023-01-24T03:34:57.899 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.893+0000 7f6bd77fe700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f6bd80c22c0 0x7f6bd8155710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:33920/0 (socket says 172.21.15.112:33920) 2023-01-24T03:34:57.899 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.893+0000 7f6bd77fe700 1 -- 172.21.15.112:0/2871397731 learned_addr learned my addr 172.21.15.112:0/2871397731 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:34:57.899 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.893+0000 7f6bd67fc700 1 --2- 172.21.15.112:0/2871397731 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bd814c660 0x7f6bd80c1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:57.900 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.893+0000 7f6bd6ffd700 1 --2- 172.21.15.112:0/2871397731 >> v2:172.21.15.142:3300/0 conn(0x7f6bd814c290 0x7f6bd806ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:57.900 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.893+0000 7f6bd5ffb700 1 -- 172.21.15.112:0/2871397731 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2015379321 0 0) 0x7f6bd80c18f0 con 0x7f6bd80c22c0 2023-01-24T03:34:57.900 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.893+0000 7f6bd5ffb700 1 -- 172.21.15.112:0/2871397731 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bc40033f0 con 0x7f6bd80c22c0 2023-01-24T03:34:57.900 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.893+0000 7f6bd6ffd700 1 -- 172.21.15.112:0/2871397731 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bd814c660 msgr2=0x7f6bd80c1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:34:57.900 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.893+0000 7f6bd6ffd700 1 --2- 172.21.15.112:0/2871397731 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bd814c660 0x7f6bd80c1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:57.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.893+0000 7f6bd6ffd700 1 -- 172.21.15.112:0/2871397731 >> v1:172.21.15.112:6789/0 conn(0x7f6bd80c22c0 legacy=0x7f6bd8155710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:57.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.893+0000 7f6bd6ffd700 1 -- 172.21.15.112:0/2871397731 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd806bad0 con 0x7f6bd814c290 2023-01-24T03:34:57.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.893+0000 7f6bd67fc700 1 --2- 172.21.15.112:0/2871397731 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bd814c660 0x7f6bd80c1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:34:57.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.893+0000 7f6bd6ffd700 1 --2- 172.21.15.112:0/2871397731 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6bd814c290 0x7f6bd806ec80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8009790 tx=0x7f6bc8009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c94438ef96de62ab server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:57.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.894+0000 7f6bd5ffb700 1 -- 172.21.15.112:0/2871397731 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc80105b0 con 0x7f6bd814c290 2023-01-24T03:34:57.902 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.894+0000 7f6bd5ffb700 1 -- 172.21.15.112:0/2871397731 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc8010710 con 0x7f6bd814c290 2023-01-24T03:34:57.902 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.894+0000 7f6bd5ffb700 1 -- 172.21.15.112:0/2871397731 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc8010a90 con 0x7f6bd814c290 2023-01-24T03:34:57.902 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.894+0000 7f6bdd87c700 1 -- 172.21.15.112:0/2871397731 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6bd814c290 msgr2=0x7f6bd806ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:57.902 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.894+0000 7f6bdd87c700 1 --2- 172.21.15.112:0/2871397731 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6bd814c290 0x7f6bd806ec80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8009790 tx=0x7f6bc8009060 comp rx=0 tx=0).stop 2023-01-24T03:34:57.902 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.894+0000 7f6bdd87c700 1 -- 172.21.15.112:0/2871397731 shutdown_connections 2023-01-24T03:34:57.903 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.894+0000 7f6bdd87c700 1 --2- 172.21.15.112:0/2871397731 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6bd814c290 0x7f6bd806ec80 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:57.903 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.894+0000 7f6bdd87c700 1 --2- 172.21.15.112:0/2871397731 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bd814c660 0x7f6bd80c1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:57.903 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.894+0000 7f6bdd87c700 1 -- 172.21.15.112:0/2871397731 >> 172.21.15.112:0/2871397731 conn(0x7f6bd8067f20 msgr2=0x7f6bd8068320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:57.903 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.895+0000 7f6bdd87c700 1 -- 172.21.15.112:0/2871397731 shutdown_connections 2023-01-24T03:34:57.903 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.895+0000 7f6bdd87c700 1 -- 172.21.15.112:0/2871397731 wait complete. 2023-01-24T03:34:57.906 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.896+0000 7f6bdd87c700 1 Processor -- start 2023-01-24T03:34:57.906 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.896+0000 7f6bdd87c700 1 -- start start 2023-01-24T03:34:57.906 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.897+0000 7f6bdd87c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6bd80c22c0 0x7f6bd8153d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:57.906 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.897+0000 7f6bdd87c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6bd814c290 0x7f6bd8066fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:57.907 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.897+0000 7f6bdd87c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bd814c660 0x7f6bd81498a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:57.907 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.897+0000 7f6bdd87c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd8166410 con 0x7f6bd814c290 2023-01-24T03:34:57.907 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.897+0000 7f6bdd87c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd8166f90 con 0x7f6bd80c22c0 2023-01-24T03:34:57.907 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.897+0000 7f6bdd87c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd814a310 con 0x7f6bd814c660 2023-01-24T03:34:57.907 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.897+0000 7f6bd67fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6bd814c290 0x7f6bd8066fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:57.908 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.897+0000 7f6bd6ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6bd80c22c0 0x7f6bd8153d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:57.908 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.897+0000 7f6bd67fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6bd814c290 0x7f6bd8066fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:52936/0 (socket says 172.21.15.112:52936) 2023-01-24T03:34:57.908 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.897+0000 7f6bd6ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6bd80c22c0 0x7f6bd8153d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.112:45428/0 (socket says 172.21.15.112:45428) 2023-01-24T03:34:57.908 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.897+0000 7f6bd67fc700 1 -- 172.21.15.112:0/2272336849 learned_addr learned my addr 172.21.15.112:0/2272336849 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:34:57.908 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.897+0000 7f6bd77fe700 1 --2- 172.21.15.112:0/2272336849 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bd814c660 0x7f6bd81498a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:57.909 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.897+0000 7f6bd67fc700 1 -- 172.21.15.112:0/2272336849 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bd814c660 msgr2=0x7f6bd81498a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:57.909 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.897+0000 7f6bd67fc700 1 --2- 172.21.15.112:0/2272336849 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bd814c660 0x7f6bd81498a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:57.909 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.897+0000 7f6bd67fc700 1 -- 172.21.15.112:0/2272336849 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6bd80c22c0 msgr2=0x7f6bd8153d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:57.909 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.897+0000 7f6bd67fc700 1 --2- 172.21.15.112:0/2272336849 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6bd80c22c0 0x7f6bd8153d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:57.909 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.897+0000 7f6bd67fc700 1 -- 172.21.15.112:0/2272336849 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bcc0092a0 con 0x7f6bd814c290 2023-01-24T03:34:57.910 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.897+0000 7f6bd6ffd700 1 --2- 172.21.15.112:0/2272336849 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6bd80c22c0 0x7f6bd8153d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:34:57.910 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.897+0000 7f6bd77fe700 1 --2- 172.21.15.112:0/2272336849 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bd814c660 0x7f6bd81498a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:34:57.910 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.898+0000 7f6bd67fc700 1 --2- 172.21.15.112:0/2272336849 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6bd814c290 0x7f6bd8066fd0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc00ea50 tx=0x7f6bcc009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:57.910 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.898+0000 7f6bbeffd700 1 -- 172.21.15.112:0/2272336849 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc00c640 con 0x7f6bd814c290 2023-01-24T03:34:57.910 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.898+0000 7f6bbeffd700 1 -- 172.21.15.112:0/2272336849 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bcc00c7a0 con 0x7f6bd814c290 2023-01-24T03:34:57.911 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.898+0000 7f6bdd87c700 1 -- 172.21.15.112:0/2272336849 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bc80092a0 con 0x7f6bd814c290 2023-01-24T03:34:57.911 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.898+0000 7f6bbeffd700 1 -- 172.21.15.112:0/2272336849 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc00cb60 con 0x7f6bd814c290 2023-01-24T03:34:57.911 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.898+0000 7f6bdd87c700 1 -- 172.21.15.112:0/2272336849 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bd8166940 con 0x7f6bd814c290 2023-01-24T03:34:57.911 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.899+0000 7f6bdd87c700 1 -- 172.21.15.112:0/2272336849 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bd804bb80 con 0x7f6bd814c290 2023-01-24T03:34:57.911 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.899+0000 7f6bbeffd700 1 -- 172.21.15.112:0/2272336849 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f6bcc019030 con 0x7f6bd814c290 2023-01-24T03:34:57.912 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.900+0000 7f6bbeffd700 1 --2- 172.21.15.112:0/2272336849 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f6bb80611f0 0x7f6bb80636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:34:57.912 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.900+0000 7f6bbeffd700 1 -- 172.21.15.112:0/2272336849 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 7125+0+0 (secure 0 0 0) 0x7f6bcc07da30 con 0x7f6bd814c290 2023-01-24T03:34:57.912 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.900+0000 7f6bd6ffd700 1 --2- 172.21.15.112:0/2272336849 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f6bb80611f0 0x7f6bb80636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:34:57.912 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.900+0000 7f6bd6ffd700 1 --2- 172.21.15.112:0/2272336849 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f6bb80611f0 0x7f6bb80636a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8009fd0 tx=0x7f6bc8016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:34:57.913 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:57.904+0000 7f6bbeffd700 1 -- 172.21.15.112:0/2272336849 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f6bcc04d6d0 con 0x7f6bd814c290 2023-01-24T03:34:58.179 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:58.176+0000 7f6bdd87c700 1 -- 172.21.15.112:0/2272336849 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6bd806c150 con 0x7f6bd814c290 2023-01-24T03:34:58.179 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:58.176+0000 7f6bbeffd700 1 -- 172.21.15.112:0/2272336849 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+6115 (secure 0 0 0) 0x7f6bcc050cf0 con 0x7f6bd814c290 2023-01-24T03:34:58.180 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:34:58.180 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4907,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/824478388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":824478388},{"type":"v1","addr":"172.21.15.142:6835","nonce":824478388}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:34:46.842740+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":22,"addr":"172.21.15.142:6834/612058861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":612058861},{"type":"v1","addr":"172.21.15.142:6834","nonce":612058861}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:34:58.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:58.179+0000 7f6bdd87c700 1 -- 172.21.15.112:0/2272336849 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f6bb80611f0 msgr2=0x7f6bb80636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:58.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:58.179+0000 7f6bdd87c700 1 --2- 172.21.15.112:0/2272336849 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f6bb80611f0 0x7f6bb80636a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8009fd0 tx=0x7f6bc8016000 comp rx=0 tx=0).stop 2023-01-24T03:34:58.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:58.179+0000 7f6bdd87c700 1 -- 172.21.15.112:0/2272336849 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6bd814c290 msgr2=0x7f6bd8066fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:34:58.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:58.179+0000 7f6bdd87c700 1 --2- 172.21.15.112:0/2272336849 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6bd814c290 0x7f6bd8066fd0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc00ea50 tx=0x7f6bcc009060 comp rx=0 tx=0).stop 2023-01-24T03:34:58.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:58.179+0000 7f6bdd87c700 1 -- 172.21.15.112:0/2272336849 shutdown_connections 2023-01-24T03:34:58.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:58.179+0000 7f6bdd87c700 1 --2- 172.21.15.112:0/2272336849 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f6bb80611f0 0x7f6bb80636a0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:58.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:58.179+0000 7f6bdd87c700 1 --2- 172.21.15.112:0/2272336849 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f6bd814c660 0x7f6bd81498a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:58.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:58.179+0000 7f6bdd87c700 1 --2- 172.21.15.112:0/2272336849 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6bd80c22c0 0x7f6bd8153d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:58.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:58.179+0000 7f6bdd87c700 1 --2- 172.21.15.112:0/2272336849 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6bd814c290 0x7f6bd8066fd0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:34:58.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:58.179+0000 7f6bdd87c700 1 -- 172.21.15.112:0/2272336849 >> 172.21.15.112:0/2272336849 conn(0x7f6bd8067f20 msgr2=0x7f6bd80c7820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:34:58.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:58.179+0000 7f6bdd87c700 1 -- 172.21.15.112:0/2272336849 shutdown_connections 2023-01-24T03:34:58.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:34:58.179+0000 7f6bdd87c700 1 -- 172.21.15.112:0/2272336849 wait complete. 2023-01-24T03:34:58.185 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 26 2023-01-24T03:34:58.195 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2023-01-24T03:34:58.196 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 68 secs before thrashing 2023-01-24T03:34:59.435 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:34:59.437 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:35:01.798 INFO:tasks.workunit.client.0.smithi112.stdout:Delete all the files and directories ... 2023-01-24T03:35:03.872 INFO:tasks.workunit.client.0.smithi112.stdout:removed './100/19' 2023-01-24T03:35:03.872 INFO:tasks.workunit.client.0.smithi112.stdout:removed './100/15' 2023-01-24T03:35:03.873 INFO:tasks.workunit.client.0.smithi112.stdout:removed './100/31' 2023-01-24T03:35:03.873 INFO:tasks.workunit.client.0.smithi112.stdout:removed './100/36' 2023-01-24T03:35:03.873 INFO:tasks.workunit.client.0.smithi112.stdout:removed './100/20' 2023-01-24T03:35:03.873 INFO:tasks.workunit.client.0.smithi112.stdout:removed './100/12' 2023-01-24T03:35:03.873 INFO:tasks.workunit.client.0.smithi112.stdout:removed './100/39' 2023-01-24T03:35:03.874 INFO:tasks.workunit.client.0.smithi112.stdout:removed './100/10' 2023-01-24T03:35:03.874 INFO:tasks.workunit.client.0.smithi112.stdout:removed './100/34' 2023-01-24T03:35:03.874 INFO:tasks.workunit.client.0.smithi112.stdout:removed './100/38' 2023-01-24T03:35:03.874 INFO:tasks.workunit.client.0.smithi112.stdout:removed './100/11' 2023-01-24T03:35:03.874 INFO:tasks.workunit.client.0.smithi112.stdout:removed './100/16' 2023-01-24T03:35:03.875 INFO:tasks.workunit.client.0.smithi112.stdout:removed './100/32' 2023-01-24T03:35:03.875 INFO:tasks.workunit.client.0.smithi112.stdout:removed './100/30' 2023-01-24T03:35:03.875 INFO:tasks.workunit.client.0.smithi112.stdout:removed './100/14' 2023-01-24T03:35:03.875 INFO:tasks.workunit.client.0.smithi112.stdout:removed './100/18' 2023-01-24T03:35:03.875 INFO:tasks.workunit.client.0.smithi112.stdout:removed './100/13' 2023-01-24T03:35:03.876 INFO:tasks.workunit.client.0.smithi112.stdout:removed './100/33' 2023-01-24T03:35:03.876 INFO:tasks.workunit.client.0.smithi112.stdout:removed './100/17' 2023-01-24T03:35:03.876 INFO:tasks.workunit.client.0.smithi112.stdout:removed './100/35' 2023-01-24T03:35:03.876 INFO:tasks.workunit.client.0.smithi112.stdout:removed './100/37' 2023-01-24T03:35:03.876 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './100' 2023-01-24T03:35:03.877 INFO:tasks.workunit.client.0.smithi112.stdout:removed './101/19' 2023-01-24T03:35:03.877 INFO:tasks.workunit.client.0.smithi112.stdout:removed './101/15' 2023-01-24T03:35:03.877 INFO:tasks.workunit.client.0.smithi112.stdout:removed './101/31' 2023-01-24T03:35:03.877 INFO:tasks.workunit.client.0.smithi112.stdout:removed './101/36' 2023-01-24T03:35:03.877 INFO:tasks.workunit.client.0.smithi112.stdout:removed './101/20' 2023-01-24T03:35:03.878 INFO:tasks.workunit.client.0.smithi112.stdout:removed './101/12' 2023-01-24T03:35:03.878 INFO:tasks.workunit.client.0.smithi112.stdout:removed './101/39' 2023-01-24T03:35:03.878 INFO:tasks.workunit.client.0.smithi112.stdout:removed './101/10' 2023-01-24T03:35:03.878 INFO:tasks.workunit.client.0.smithi112.stdout:removed './101/34' 2023-01-24T03:35:03.878 INFO:tasks.workunit.client.0.smithi112.stdout:removed './101/38' 2023-01-24T03:35:03.879 INFO:tasks.workunit.client.0.smithi112.stdout:removed './101/11' 2023-01-24T03:35:03.879 INFO:tasks.workunit.client.0.smithi112.stdout:removed './101/16' 2023-01-24T03:35:03.879 INFO:tasks.workunit.client.0.smithi112.stdout:removed './101/32' 2023-01-24T03:35:03.879 INFO:tasks.workunit.client.0.smithi112.stdout:removed './101/30' 2023-01-24T03:35:03.879 INFO:tasks.workunit.client.0.smithi112.stdout:removed './101/14' 2023-01-24T03:35:03.880 INFO:tasks.workunit.client.0.smithi112.stdout:removed './101/18' 2023-01-24T03:35:03.880 INFO:tasks.workunit.client.0.smithi112.stdout:removed './101/13' 2023-01-24T03:35:03.880 INFO:tasks.workunit.client.0.smithi112.stdout:removed './101/33' 2023-01-24T03:35:03.880 INFO:tasks.workunit.client.0.smithi112.stdout:removed './101/17' 2023-01-24T03:35:03.880 INFO:tasks.workunit.client.0.smithi112.stdout:removed './101/35' 2023-01-24T03:35:03.881 INFO:tasks.workunit.client.0.smithi112.stdout:removed './101/37' 2023-01-24T03:35:03.881 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './101' 2023-01-24T03:35:03.881 INFO:tasks.workunit.client.0.smithi112.stdout:removed './102/19' 2023-01-24T03:35:03.881 INFO:tasks.workunit.client.0.smithi112.stdout:removed './102/15' 2023-01-24T03:35:03.881 INFO:tasks.workunit.client.0.smithi112.stdout:removed './102/31' 2023-01-24T03:35:03.882 INFO:tasks.workunit.client.0.smithi112.stdout:removed './102/36' 2023-01-24T03:35:03.882 INFO:tasks.workunit.client.0.smithi112.stdout:removed './102/20' 2023-01-24T03:35:03.882 INFO:tasks.workunit.client.0.smithi112.stdout:removed './102/12' 2023-01-24T03:35:03.882 INFO:tasks.workunit.client.0.smithi112.stdout:removed './102/39' 2023-01-24T03:35:03.882 INFO:tasks.workunit.client.0.smithi112.stdout:removed './102/10' 2023-01-24T03:35:03.883 INFO:tasks.workunit.client.0.smithi112.stdout:removed './102/34' 2023-01-24T03:35:03.883 INFO:tasks.workunit.client.0.smithi112.stdout:removed './102/38' 2023-01-24T03:35:03.883 INFO:tasks.workunit.client.0.smithi112.stdout:removed './102/11' 2023-01-24T03:35:03.883 INFO:tasks.workunit.client.0.smithi112.stdout:removed './102/16' 2023-01-24T03:35:03.883 INFO:tasks.workunit.client.0.smithi112.stdout:removed './102/32' 2023-01-24T03:35:03.884 INFO:tasks.workunit.client.0.smithi112.stdout:removed './102/30' 2023-01-24T03:35:03.884 INFO:tasks.workunit.client.0.smithi112.stdout:removed './102/14' 2023-01-24T03:35:03.884 INFO:tasks.workunit.client.0.smithi112.stdout:removed './102/18' 2023-01-24T03:35:03.884 INFO:tasks.workunit.client.0.smithi112.stdout:removed './102/13' 2023-01-24T03:35:03.884 INFO:tasks.workunit.client.0.smithi112.stdout:removed './102/33' 2023-01-24T03:35:03.885 INFO:tasks.workunit.client.0.smithi112.stdout:removed './102/17' 2023-01-24T03:35:03.885 INFO:tasks.workunit.client.0.smithi112.stdout:removed './102/35' 2023-01-24T03:35:03.885 INFO:tasks.workunit.client.0.smithi112.stdout:removed './102/37' 2023-01-24T03:35:03.885 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './102' 2023-01-24T03:35:03.885 INFO:tasks.workunit.client.0.smithi112.stdout:removed './103/19' 2023-01-24T03:35:03.886 INFO:tasks.workunit.client.0.smithi112.stdout:removed './103/15' 2023-01-24T03:35:03.886 INFO:tasks.workunit.client.0.smithi112.stdout:removed './103/31' 2023-01-24T03:35:03.886 INFO:tasks.workunit.client.0.smithi112.stdout:removed './103/36' 2023-01-24T03:35:03.886 INFO:tasks.workunit.client.0.smithi112.stdout:removed './103/20' 2023-01-24T03:35:03.886 INFO:tasks.workunit.client.0.smithi112.stdout:removed './103/12' 2023-01-24T03:35:03.887 INFO:tasks.workunit.client.0.smithi112.stdout:removed './103/39' 2023-01-24T03:35:03.887 INFO:tasks.workunit.client.0.smithi112.stdout:removed './103/10' 2023-01-24T03:35:03.887 INFO:tasks.workunit.client.0.smithi112.stdout:removed './103/34' 2023-01-24T03:35:03.887 INFO:tasks.workunit.client.0.smithi112.stdout:removed './103/38' 2023-01-24T03:35:03.888 INFO:tasks.workunit.client.0.smithi112.stdout:removed './103/11' 2023-01-24T03:35:03.888 INFO:tasks.workunit.client.0.smithi112.stdout:removed './103/16' 2023-01-24T03:35:03.888 INFO:tasks.workunit.client.0.smithi112.stdout:removed './103/32' 2023-01-24T03:35:03.888 INFO:tasks.workunit.client.0.smithi112.stdout:removed './103/30' 2023-01-24T03:35:03.888 INFO:tasks.workunit.client.0.smithi112.stdout:removed './103/14' 2023-01-24T03:35:03.889 INFO:tasks.workunit.client.0.smithi112.stdout:removed './103/18' 2023-01-24T03:35:03.889 INFO:tasks.workunit.client.0.smithi112.stdout:removed './103/13' 2023-01-24T03:35:03.889 INFO:tasks.workunit.client.0.smithi112.stdout:removed './103/33' 2023-01-24T03:35:03.889 INFO:tasks.workunit.client.0.smithi112.stdout:removed './103/17' 2023-01-24T03:35:03.889 INFO:tasks.workunit.client.0.smithi112.stdout:removed './103/35' 2023-01-24T03:35:03.890 INFO:tasks.workunit.client.0.smithi112.stdout:removed './103/37' 2023-01-24T03:35:03.890 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './103' 2023-01-24T03:35:03.890 INFO:tasks.workunit.client.0.smithi112.stdout:removed './104/19' 2023-01-24T03:35:03.890 INFO:tasks.workunit.client.0.smithi112.stdout:removed './104/15' 2023-01-24T03:35:03.890 INFO:tasks.workunit.client.0.smithi112.stdout:removed './104/31' 2023-01-24T03:35:03.891 INFO:tasks.workunit.client.0.smithi112.stdout:removed './104/36' 2023-01-24T03:35:03.891 INFO:tasks.workunit.client.0.smithi112.stdout:removed './104/20' 2023-01-24T03:35:03.891 INFO:tasks.workunit.client.0.smithi112.stdout:removed './104/12' 2023-01-24T03:35:03.891 INFO:tasks.workunit.client.0.smithi112.stdout:removed './104/39' 2023-01-24T03:35:03.891 INFO:tasks.workunit.client.0.smithi112.stdout:removed './104/10' 2023-01-24T03:35:03.891 INFO:tasks.workunit.client.0.smithi112.stdout:removed './104/34' 2023-01-24T03:35:03.892 INFO:tasks.workunit.client.0.smithi112.stdout:removed './104/38' 2023-01-24T03:35:03.892 INFO:tasks.workunit.client.0.smithi112.stdout:removed './104/11' 2023-01-24T03:35:03.892 INFO:tasks.workunit.client.0.smithi112.stdout:removed './104/16' 2023-01-24T03:35:03.892 INFO:tasks.workunit.client.0.smithi112.stdout:removed './104/32' 2023-01-24T03:35:03.892 INFO:tasks.workunit.client.0.smithi112.stdout:removed './104/30' 2023-01-24T03:35:03.893 INFO:tasks.workunit.client.0.smithi112.stdout:removed './104/14' 2023-01-24T03:35:03.893 INFO:tasks.workunit.client.0.smithi112.stdout:removed './104/18' 2023-01-24T03:35:03.893 INFO:tasks.workunit.client.0.smithi112.stdout:removed './104/13' 2023-01-24T03:35:03.893 INFO:tasks.workunit.client.0.smithi112.stdout:removed './104/33' 2023-01-24T03:35:03.893 INFO:tasks.workunit.client.0.smithi112.stdout:removed './104/17' 2023-01-24T03:35:03.894 INFO:tasks.workunit.client.0.smithi112.stdout:removed './104/35' 2023-01-24T03:35:03.894 INFO:tasks.workunit.client.0.smithi112.stdout:removed './104/37' 2023-01-24T03:35:03.894 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './104' 2023-01-24T03:35:03.894 INFO:tasks.workunit.client.0.smithi112.stdout:removed './105/19' 2023-01-24T03:35:03.894 INFO:tasks.workunit.client.0.smithi112.stdout:removed './105/15' 2023-01-24T03:35:03.895 INFO:tasks.workunit.client.0.smithi112.stdout:removed './105/31' 2023-01-24T03:35:03.895 INFO:tasks.workunit.client.0.smithi112.stdout:removed './105/36' 2023-01-24T03:35:03.895 INFO:tasks.workunit.client.0.smithi112.stdout:removed './105/20' 2023-01-24T03:35:03.895 INFO:tasks.workunit.client.0.smithi112.stdout:removed './105/12' 2023-01-24T03:35:03.895 INFO:tasks.workunit.client.0.smithi112.stdout:removed './105/39' 2023-01-24T03:35:03.896 INFO:tasks.workunit.client.0.smithi112.stdout:removed './105/10' 2023-01-24T03:35:03.896 INFO:tasks.workunit.client.0.smithi112.stdout:removed './105/34' 2023-01-24T03:35:03.896 INFO:tasks.workunit.client.0.smithi112.stdout:removed './105/38' 2023-01-24T03:35:03.896 INFO:tasks.workunit.client.0.smithi112.stdout:removed './105/11' 2023-01-24T03:35:03.897 INFO:tasks.workunit.client.0.smithi112.stdout:removed './105/16' 2023-01-24T03:35:03.897 INFO:tasks.workunit.client.0.smithi112.stdout:removed './105/32' 2023-01-24T03:35:03.897 INFO:tasks.workunit.client.0.smithi112.stdout:removed './105/30' 2023-01-24T03:35:03.897 INFO:tasks.workunit.client.0.smithi112.stdout:removed './105/14' 2023-01-24T03:35:03.897 INFO:tasks.workunit.client.0.smithi112.stdout:removed './105/18' 2023-01-24T03:35:03.898 INFO:tasks.workunit.client.0.smithi112.stdout:removed './105/13' 2023-01-24T03:35:03.898 INFO:tasks.workunit.client.0.smithi112.stdout:removed './105/33' 2023-01-24T03:35:03.898 INFO:tasks.workunit.client.0.smithi112.stdout:removed './105/17' 2023-01-24T03:35:03.898 INFO:tasks.workunit.client.0.smithi112.stdout:removed './105/35' 2023-01-24T03:35:03.898 INFO:tasks.workunit.client.0.smithi112.stdout:removed './105/37' 2023-01-24T03:35:03.898 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './105' 2023-01-24T03:35:03.899 INFO:tasks.workunit.client.0.smithi112.stdout:removed './106/19' 2023-01-24T03:35:03.899 INFO:tasks.workunit.client.0.smithi112.stdout:removed './106/15' 2023-01-24T03:35:03.899 INFO:tasks.workunit.client.0.smithi112.stdout:removed './106/31' 2023-01-24T03:35:03.899 INFO:tasks.workunit.client.0.smithi112.stdout:removed './106/36' 2023-01-24T03:35:03.899 INFO:tasks.workunit.client.0.smithi112.stdout:removed './106/20' 2023-01-24T03:35:03.899 INFO:tasks.workunit.client.0.smithi112.stdout:removed './106/12' 2023-01-24T03:35:03.900 INFO:tasks.workunit.client.0.smithi112.stdout:removed './106/39' 2023-01-24T03:35:03.900 INFO:tasks.workunit.client.0.smithi112.stdout:removed './106/10' 2023-01-24T03:35:03.900 INFO:tasks.workunit.client.0.smithi112.stdout:removed './106/34' 2023-01-24T03:35:03.900 INFO:tasks.workunit.client.0.smithi112.stdout:removed './106/38' 2023-01-24T03:35:03.900 INFO:tasks.workunit.client.0.smithi112.stdout:removed './106/11' 2023-01-24T03:35:03.901 INFO:tasks.workunit.client.0.smithi112.stdout:removed './106/16' 2023-01-24T03:35:03.901 INFO:tasks.workunit.client.0.smithi112.stdout:removed './106/32' 2023-01-24T03:35:03.901 INFO:tasks.workunit.client.0.smithi112.stdout:removed './106/30' 2023-01-24T03:35:03.901 INFO:tasks.workunit.client.0.smithi112.stdout:removed './106/14' 2023-01-24T03:35:03.901 INFO:tasks.workunit.client.0.smithi112.stdout:removed './106/18' 2023-01-24T03:35:03.902 INFO:tasks.workunit.client.0.smithi112.stdout:removed './106/13' 2023-01-24T03:35:03.902 INFO:tasks.workunit.client.0.smithi112.stdout:removed './106/33' 2023-01-24T03:35:03.902 INFO:tasks.workunit.client.0.smithi112.stdout:removed './106/17' 2023-01-24T03:35:03.902 INFO:tasks.workunit.client.0.smithi112.stdout:removed './106/35' 2023-01-24T03:35:03.902 INFO:tasks.workunit.client.0.smithi112.stdout:removed './106/37' 2023-01-24T03:35:03.902 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './106' 2023-01-24T03:35:03.903 INFO:tasks.workunit.client.0.smithi112.stdout:removed './107/19' 2023-01-24T03:35:03.903 INFO:tasks.workunit.client.0.smithi112.stdout:removed './107/15' 2023-01-24T03:35:03.903 INFO:tasks.workunit.client.0.smithi112.stdout:removed './107/31' 2023-01-24T03:35:03.903 INFO:tasks.workunit.client.0.smithi112.stdout:removed './107/36' 2023-01-24T03:35:03.903 INFO:tasks.workunit.client.0.smithi112.stdout:removed './107/20' 2023-01-24T03:35:03.904 INFO:tasks.workunit.client.0.smithi112.stdout:removed './107/12' 2023-01-24T03:35:03.904 INFO:tasks.workunit.client.0.smithi112.stdout:removed './107/39' 2023-01-24T03:35:03.904 INFO:tasks.workunit.client.0.smithi112.stdout:removed './107/10' 2023-01-24T03:35:03.904 INFO:tasks.workunit.client.0.smithi112.stdout:removed './107/34' 2023-01-24T03:35:03.904 INFO:tasks.workunit.client.0.smithi112.stdout:removed './107/38' 2023-01-24T03:35:03.905 INFO:tasks.workunit.client.0.smithi112.stdout:removed './107/11' 2023-01-24T03:35:03.905 INFO:tasks.workunit.client.0.smithi112.stdout:removed './107/16' 2023-01-24T03:35:03.905 INFO:tasks.workunit.client.0.smithi112.stdout:removed './107/32' 2023-01-24T03:35:03.905 INFO:tasks.workunit.client.0.smithi112.stdout:removed './107/30' 2023-01-24T03:35:03.905 INFO:tasks.workunit.client.0.smithi112.stdout:removed './107/14' 2023-01-24T03:35:03.906 INFO:tasks.workunit.client.0.smithi112.stdout:removed './107/18' 2023-01-24T03:35:03.906 INFO:tasks.workunit.client.0.smithi112.stdout:removed './107/13' 2023-01-24T03:35:03.906 INFO:tasks.workunit.client.0.smithi112.stdout:removed './107/33' 2023-01-24T03:35:03.906 INFO:tasks.workunit.client.0.smithi112.stdout:removed './107/17' 2023-01-24T03:35:03.906 INFO:tasks.workunit.client.0.smithi112.stdout:removed './107/35' 2023-01-24T03:35:03.906 INFO:tasks.workunit.client.0.smithi112.stdout:removed './107/37' 2023-01-24T03:35:03.907 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './107' 2023-01-24T03:35:03.907 INFO:tasks.workunit.client.0.smithi112.stdout:removed './108/19' 2023-01-24T03:35:03.907 INFO:tasks.workunit.client.0.smithi112.stdout:removed './108/15' 2023-01-24T03:35:03.907 INFO:tasks.workunit.client.0.smithi112.stdout:removed './108/31' 2023-01-24T03:35:03.907 INFO:tasks.workunit.client.0.smithi112.stdout:removed './108/36' 2023-01-24T03:35:03.908 INFO:tasks.workunit.client.0.smithi112.stdout:removed './108/20' 2023-01-24T03:35:03.908 INFO:tasks.workunit.client.0.smithi112.stdout:removed './108/12' 2023-01-24T03:35:03.908 INFO:tasks.workunit.client.0.smithi112.stdout:removed './108/39' 2023-01-24T03:35:03.908 INFO:tasks.workunit.client.0.smithi112.stdout:removed './108/10' 2023-01-24T03:35:03.908 INFO:tasks.workunit.client.0.smithi112.stdout:removed './108/34' 2023-01-24T03:35:03.909 INFO:tasks.workunit.client.0.smithi112.stdout:removed './108/38' 2023-01-24T03:35:03.909 INFO:tasks.workunit.client.0.smithi112.stdout:removed './108/11' 2023-01-24T03:35:03.909 INFO:tasks.workunit.client.0.smithi112.stdout:removed './108/16' 2023-01-24T03:35:03.909 INFO:tasks.workunit.client.0.smithi112.stdout:removed './108/32' 2023-01-24T03:35:03.909 INFO:tasks.workunit.client.0.smithi112.stdout:removed './108/30' 2023-01-24T03:35:03.910 INFO:tasks.workunit.client.0.smithi112.stdout:removed './108/14' 2023-01-24T03:35:03.910 INFO:tasks.workunit.client.0.smithi112.stdout:removed './108/18' 2023-01-24T03:35:03.910 INFO:tasks.workunit.client.0.smithi112.stdout:removed './108/13' 2023-01-24T03:35:03.910 INFO:tasks.workunit.client.0.smithi112.stdout:removed './108/33' 2023-01-24T03:35:03.911 INFO:tasks.workunit.client.0.smithi112.stdout:removed './108/17' 2023-01-24T03:35:03.911 INFO:tasks.workunit.client.0.smithi112.stdout:removed './108/35' 2023-01-24T03:35:03.911 INFO:tasks.workunit.client.0.smithi112.stdout:removed './108/37' 2023-01-24T03:35:03.911 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './108' 2023-01-24T03:35:03.911 INFO:tasks.workunit.client.0.smithi112.stdout:removed './109/19' 2023-01-24T03:35:03.912 INFO:tasks.workunit.client.0.smithi112.stdout:removed './109/15' 2023-01-24T03:35:03.912 INFO:tasks.workunit.client.0.smithi112.stdout:removed './109/31' 2023-01-24T03:35:03.912 INFO:tasks.workunit.client.0.smithi112.stdout:removed './109/36' 2023-01-24T03:35:03.912 INFO:tasks.workunit.client.0.smithi112.stdout:removed './109/20' 2023-01-24T03:35:03.912 INFO:tasks.workunit.client.0.smithi112.stdout:removed './109/12' 2023-01-24T03:35:03.913 INFO:tasks.workunit.client.0.smithi112.stdout:removed './109/39' 2023-01-24T03:35:03.913 INFO:tasks.workunit.client.0.smithi112.stdout:removed './109/10' 2023-01-24T03:35:03.913 INFO:tasks.workunit.client.0.smithi112.stdout:removed './109/34' 2023-01-24T03:35:03.913 INFO:tasks.workunit.client.0.smithi112.stdout:removed './109/38' 2023-01-24T03:35:03.913 INFO:tasks.workunit.client.0.smithi112.stdout:removed './109/11' 2023-01-24T03:35:03.914 INFO:tasks.workunit.client.0.smithi112.stdout:removed './109/16' 2023-01-24T03:35:03.914 INFO:tasks.workunit.client.0.smithi112.stdout:removed './109/32' 2023-01-24T03:35:03.914 INFO:tasks.workunit.client.0.smithi112.stdout:removed './109/30' 2023-01-24T03:35:05.507 INFO:tasks.workunit.client.0.smithi112.stdout:removed './109/14' 2023-01-24T03:35:05.507 INFO:tasks.workunit.client.0.smithi112.stdout:removed './109/18' 2023-01-24T03:35:05.507 INFO:tasks.workunit.client.0.smithi112.stdout:removed './109/13' 2023-01-24T03:35:05.507 INFO:tasks.workunit.client.0.smithi112.stdout:removed './109/33' 2023-01-24T03:35:05.508 INFO:tasks.workunit.client.0.smithi112.stdout:removed './109/17' 2023-01-24T03:35:05.508 INFO:tasks.workunit.client.0.smithi112.stdout:removed './109/35' 2023-01-24T03:35:05.508 INFO:tasks.workunit.client.0.smithi112.stdout:removed './109/37' 2023-01-24T03:35:05.508 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './109' 2023-01-24T03:35:05.508 INFO:tasks.workunit.client.0.smithi112.stdout:removed './110/19' 2023-01-24T03:35:05.509 INFO:tasks.workunit.client.0.smithi112.stdout:removed './110/15' 2023-01-24T03:35:05.509 INFO:tasks.workunit.client.0.smithi112.stdout:removed './110/31' 2023-01-24T03:35:05.509 INFO:tasks.workunit.client.0.smithi112.stdout:removed './110/36' 2023-01-24T03:35:05.509 INFO:tasks.workunit.client.0.smithi112.stdout:removed './110/20' 2023-01-24T03:35:05.509 INFO:tasks.workunit.client.0.smithi112.stdout:removed './110/12' 2023-01-24T03:35:05.510 INFO:tasks.workunit.client.0.smithi112.stdout:removed './110/39' 2023-01-24T03:35:05.510 INFO:tasks.workunit.client.0.smithi112.stdout:removed './110/10' 2023-01-24T03:35:05.510 INFO:tasks.workunit.client.0.smithi112.stdout:removed './110/34' 2023-01-24T03:35:05.510 INFO:tasks.workunit.client.0.smithi112.stdout:removed './110/38' 2023-01-24T03:35:05.510 INFO:tasks.workunit.client.0.smithi112.stdout:removed './110/11' 2023-01-24T03:35:05.511 INFO:tasks.workunit.client.0.smithi112.stdout:removed './110/16' 2023-01-24T03:35:05.511 INFO:tasks.workunit.client.0.smithi112.stdout:removed './110/32' 2023-01-24T03:35:05.511 INFO:tasks.workunit.client.0.smithi112.stdout:removed './110/30' 2023-01-24T03:35:05.511 INFO:tasks.workunit.client.0.smithi112.stdout:removed './110/14' 2023-01-24T03:35:05.511 INFO:tasks.workunit.client.0.smithi112.stdout:removed './110/18' 2023-01-24T03:35:05.512 INFO:tasks.workunit.client.0.smithi112.stdout:removed './110/13' 2023-01-24T03:35:05.512 INFO:tasks.workunit.client.0.smithi112.stdout:removed './110/33' 2023-01-24T03:35:05.512 INFO:tasks.workunit.client.0.smithi112.stdout:removed './110/17' 2023-01-24T03:35:05.512 INFO:tasks.workunit.client.0.smithi112.stdout:removed './110/35' 2023-01-24T03:35:05.512 INFO:tasks.workunit.client.0.smithi112.stdout:removed './110/37' 2023-01-24T03:35:05.513 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './110' 2023-01-24T03:35:05.513 INFO:tasks.workunit.client.0.smithi112.stdout:removed './111/19' 2023-01-24T03:35:05.513 INFO:tasks.workunit.client.0.smithi112.stdout:removed './111/15' 2023-01-24T03:35:05.513 INFO:tasks.workunit.client.0.smithi112.stdout:removed './111/31' 2023-01-24T03:35:05.513 INFO:tasks.workunit.client.0.smithi112.stdout:removed './111/36' 2023-01-24T03:35:05.514 INFO:tasks.workunit.client.0.smithi112.stdout:removed './111/20' 2023-01-24T03:35:05.514 INFO:tasks.workunit.client.0.smithi112.stdout:removed './111/12' 2023-01-24T03:35:05.514 INFO:tasks.workunit.client.0.smithi112.stdout:removed './111/39' 2023-01-24T03:35:05.514 INFO:tasks.workunit.client.0.smithi112.stdout:removed './111/10' 2023-01-24T03:35:05.514 INFO:tasks.workunit.client.0.smithi112.stdout:removed './111/34' 2023-01-24T03:35:05.515 INFO:tasks.workunit.client.0.smithi112.stdout:removed './111/38' 2023-01-24T03:35:05.515 INFO:tasks.workunit.client.0.smithi112.stdout:removed './111/11' 2023-01-24T03:35:05.515 INFO:tasks.workunit.client.0.smithi112.stdout:removed './111/16' 2023-01-24T03:35:05.515 INFO:tasks.workunit.client.0.smithi112.stdout:removed './111/32' 2023-01-24T03:35:05.515 INFO:tasks.workunit.client.0.smithi112.stdout:removed './111/30' 2023-01-24T03:35:05.515 INFO:tasks.workunit.client.0.smithi112.stdout:removed './111/14' 2023-01-24T03:35:05.516 INFO:tasks.workunit.client.0.smithi112.stdout:removed './111/18' 2023-01-24T03:35:05.516 INFO:tasks.workunit.client.0.smithi112.stdout:removed './111/13' 2023-01-24T03:35:05.516 INFO:tasks.workunit.client.0.smithi112.stdout:removed './111/33' 2023-01-24T03:35:05.516 INFO:tasks.workunit.client.0.smithi112.stdout:removed './111/17' 2023-01-24T03:35:05.516 INFO:tasks.workunit.client.0.smithi112.stdout:removed './111/35' 2023-01-24T03:35:05.517 INFO:tasks.workunit.client.0.smithi112.stdout:removed './111/37' 2023-01-24T03:35:05.517 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './111' 2023-01-24T03:35:05.517 INFO:tasks.workunit.client.0.smithi112.stdout:removed './112/19' 2023-01-24T03:35:05.517 INFO:tasks.workunit.client.0.smithi112.stdout:removed './112/15' 2023-01-24T03:35:05.517 INFO:tasks.workunit.client.0.smithi112.stdout:removed './112/31' 2023-01-24T03:35:05.518 INFO:tasks.workunit.client.0.smithi112.stdout:removed './112/36' 2023-01-24T03:35:05.518 INFO:tasks.workunit.client.0.smithi112.stdout:removed './112/20' 2023-01-24T03:35:05.518 INFO:tasks.workunit.client.0.smithi112.stdout:removed './112/12' 2023-01-24T03:35:05.518 INFO:tasks.workunit.client.0.smithi112.stdout:removed './112/39' 2023-01-24T03:35:05.518 INFO:tasks.workunit.client.0.smithi112.stdout:removed './112/10' 2023-01-24T03:35:05.519 INFO:tasks.workunit.client.0.smithi112.stdout:removed './112/34' 2023-01-24T03:35:05.519 INFO:tasks.workunit.client.0.smithi112.stdout:removed './112/38' 2023-01-24T03:35:05.519 INFO:tasks.workunit.client.0.smithi112.stdout:removed './112/11' 2023-01-24T03:35:05.519 INFO:tasks.workunit.client.0.smithi112.stdout:removed './112/16' 2023-01-24T03:35:05.519 INFO:tasks.workunit.client.0.smithi112.stdout:removed './112/32' 2023-01-24T03:35:05.519 INFO:tasks.workunit.client.0.smithi112.stdout:removed './112/30' 2023-01-24T03:35:05.520 INFO:tasks.workunit.client.0.smithi112.stdout:removed './112/14' 2023-01-24T03:35:05.520 INFO:tasks.workunit.client.0.smithi112.stdout:removed './112/18' 2023-01-24T03:35:05.520 INFO:tasks.workunit.client.0.smithi112.stdout:removed './112/13' 2023-01-24T03:35:05.520 INFO:tasks.workunit.client.0.smithi112.stdout:removed './112/33' 2023-01-24T03:35:05.520 INFO:tasks.workunit.client.0.smithi112.stdout:removed './112/17' 2023-01-24T03:35:05.521 INFO:tasks.workunit.client.0.smithi112.stdout:removed './112/35' 2023-01-24T03:35:05.521 INFO:tasks.workunit.client.0.smithi112.stdout:removed './112/37' 2023-01-24T03:35:05.521 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './112' 2023-01-24T03:35:05.521 INFO:tasks.workunit.client.0.smithi112.stdout:removed './113/19' 2023-01-24T03:35:05.521 INFO:tasks.workunit.client.0.smithi112.stdout:removed './113/15' 2023-01-24T03:35:05.522 INFO:tasks.workunit.client.0.smithi112.stdout:removed './113/31' 2023-01-24T03:35:05.522 INFO:tasks.workunit.client.0.smithi112.stdout:removed './113/36' 2023-01-24T03:35:05.522 INFO:tasks.workunit.client.0.smithi112.stdout:removed './113/20' 2023-01-24T03:35:05.522 INFO:tasks.workunit.client.0.smithi112.stdout:removed './113/12' 2023-01-24T03:35:05.522 INFO:tasks.workunit.client.0.smithi112.stdout:removed './113/39' 2023-01-24T03:35:05.522 INFO:tasks.workunit.client.0.smithi112.stdout:removed './113/10' 2023-01-24T03:35:05.523 INFO:tasks.workunit.client.0.smithi112.stdout:removed './113/34' 2023-01-24T03:35:05.523 INFO:tasks.workunit.client.0.smithi112.stdout:removed './113/38' 2023-01-24T03:35:05.523 INFO:tasks.workunit.client.0.smithi112.stdout:removed './113/11' 2023-01-24T03:35:05.523 INFO:tasks.workunit.client.0.smithi112.stdout:removed './113/16' 2023-01-24T03:35:05.523 INFO:tasks.workunit.client.0.smithi112.stdout:removed './113/32' 2023-01-24T03:35:05.524 INFO:tasks.workunit.client.0.smithi112.stdout:removed './113/30' 2023-01-24T03:35:05.524 INFO:tasks.workunit.client.0.smithi112.stdout:removed './113/14' 2023-01-24T03:35:05.524 INFO:tasks.workunit.client.0.smithi112.stdout:removed './113/18' 2023-01-24T03:35:05.524 INFO:tasks.workunit.client.0.smithi112.stdout:removed './113/13' 2023-01-24T03:35:05.524 INFO:tasks.workunit.client.0.smithi112.stdout:removed './113/33' 2023-01-24T03:35:05.525 INFO:tasks.workunit.client.0.smithi112.stdout:removed './113/17' 2023-01-24T03:35:05.525 INFO:tasks.workunit.client.0.smithi112.stdout:removed './113/35' 2023-01-24T03:35:05.525 INFO:tasks.workunit.client.0.smithi112.stdout:removed './113/37' 2023-01-24T03:35:05.525 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './113' 2023-01-24T03:35:05.525 INFO:tasks.workunit.client.0.smithi112.stdout:removed './114/19' 2023-01-24T03:35:05.526 INFO:tasks.workunit.client.0.smithi112.stdout:removed './114/15' 2023-01-24T03:35:05.526 INFO:tasks.workunit.client.0.smithi112.stdout:removed './114/31' 2023-01-24T03:35:05.526 INFO:tasks.workunit.client.0.smithi112.stdout:removed './114/36' 2023-01-24T03:35:05.526 INFO:tasks.workunit.client.0.smithi112.stdout:removed './114/20' 2023-01-24T03:35:05.526 INFO:tasks.workunit.client.0.smithi112.stdout:removed './114/12' 2023-01-24T03:35:05.526 INFO:tasks.workunit.client.0.smithi112.stdout:removed './114/39' 2023-01-24T03:35:05.527 INFO:tasks.workunit.client.0.smithi112.stdout:removed './114/10' 2023-01-24T03:35:05.527 INFO:tasks.workunit.client.0.smithi112.stdout:removed './114/34' 2023-01-24T03:35:05.527 INFO:tasks.workunit.client.0.smithi112.stdout:removed './114/38' 2023-01-24T03:35:05.527 INFO:tasks.workunit.client.0.smithi112.stdout:removed './114/11' 2023-01-24T03:35:05.527 INFO:tasks.workunit.client.0.smithi112.stdout:removed './114/16' 2023-01-24T03:35:05.528 INFO:tasks.workunit.client.0.smithi112.stdout:removed './114/32' 2023-01-24T03:35:05.528 INFO:tasks.workunit.client.0.smithi112.stdout:removed './114/30' 2023-01-24T03:35:05.528 INFO:tasks.workunit.client.0.smithi112.stdout:removed './114/14' 2023-01-24T03:35:05.528 INFO:tasks.workunit.client.0.smithi112.stdout:removed './114/18' 2023-01-24T03:35:05.528 INFO:tasks.workunit.client.0.smithi112.stdout:removed './114/13' 2023-01-24T03:35:05.529 INFO:tasks.workunit.client.0.smithi112.stdout:removed './114/33' 2023-01-24T03:35:05.529 INFO:tasks.workunit.client.0.smithi112.stdout:removed './114/17' 2023-01-24T03:35:05.529 INFO:tasks.workunit.client.0.smithi112.stdout:removed './114/35' 2023-01-24T03:35:05.529 INFO:tasks.workunit.client.0.smithi112.stdout:removed './114/37' 2023-01-24T03:35:05.529 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './114' 2023-01-24T03:35:05.530 INFO:tasks.workunit.client.0.smithi112.stdout:removed './115/19' 2023-01-24T03:35:05.530 INFO:tasks.workunit.client.0.smithi112.stdout:removed './115/15' 2023-01-24T03:35:05.530 INFO:tasks.workunit.client.0.smithi112.stdout:removed './115/31' 2023-01-24T03:35:05.530 INFO:tasks.workunit.client.0.smithi112.stdout:removed './115/36' 2023-01-24T03:35:05.530 INFO:tasks.workunit.client.0.smithi112.stdout:removed './115/20' 2023-01-24T03:35:05.531 INFO:tasks.workunit.client.0.smithi112.stdout:removed './115/12' 2023-01-24T03:35:05.531 INFO:tasks.workunit.client.0.smithi112.stdout:removed './115/39' 2023-01-24T03:35:05.531 INFO:tasks.workunit.client.0.smithi112.stdout:removed './115/10' 2023-01-24T03:35:05.531 INFO:tasks.workunit.client.0.smithi112.stdout:removed './115/34' 2023-01-24T03:35:05.531 INFO:tasks.workunit.client.0.smithi112.stdout:removed './115/38' 2023-01-24T03:35:05.531 INFO:tasks.workunit.client.0.smithi112.stdout:removed './115/11' 2023-01-24T03:35:05.532 INFO:tasks.workunit.client.0.smithi112.stdout:removed './115/16' 2023-01-24T03:35:05.532 INFO:tasks.workunit.client.0.smithi112.stdout:removed './115/32' 2023-01-24T03:35:05.532 INFO:tasks.workunit.client.0.smithi112.stdout:removed './115/30' 2023-01-24T03:35:05.532 INFO:tasks.workunit.client.0.smithi112.stdout:removed './115/14' 2023-01-24T03:35:05.532 INFO:tasks.workunit.client.0.smithi112.stdout:removed './115/18' 2023-01-24T03:35:05.533 INFO:tasks.workunit.client.0.smithi112.stdout:removed './115/13' 2023-01-24T03:35:05.533 INFO:tasks.workunit.client.0.smithi112.stdout:removed './115/33' 2023-01-24T03:35:05.533 INFO:tasks.workunit.client.0.smithi112.stdout:removed './115/17' 2023-01-24T03:35:05.533 INFO:tasks.workunit.client.0.smithi112.stdout:removed './115/35' 2023-01-24T03:35:05.533 INFO:tasks.workunit.client.0.smithi112.stdout:removed './115/37' 2023-01-24T03:35:05.534 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './115' 2023-01-24T03:35:05.534 INFO:tasks.workunit.client.0.smithi112.stdout:removed './116/19' 2023-01-24T03:35:05.534 INFO:tasks.workunit.client.0.smithi112.stdout:removed './116/15' 2023-01-24T03:35:05.534 INFO:tasks.workunit.client.0.smithi112.stdout:removed './116/31' 2023-01-24T03:35:05.534 INFO:tasks.workunit.client.0.smithi112.stdout:removed './116/36' 2023-01-24T03:35:05.535 INFO:tasks.workunit.client.0.smithi112.stdout:removed './116/20' 2023-01-24T03:35:05.535 INFO:tasks.workunit.client.0.smithi112.stdout:removed './116/12' 2023-01-24T03:35:05.535 INFO:tasks.workunit.client.0.smithi112.stdout:removed './116/39' 2023-01-24T03:35:05.535 INFO:tasks.workunit.client.0.smithi112.stdout:removed './116/10' 2023-01-24T03:35:05.535 INFO:tasks.workunit.client.0.smithi112.stdout:removed './116/34' 2023-01-24T03:35:05.535 INFO:tasks.workunit.client.0.smithi112.stdout:removed './116/38' 2023-01-24T03:35:05.536 INFO:tasks.workunit.client.0.smithi112.stdout:removed './116/11' 2023-01-24T03:35:05.536 INFO:tasks.workunit.client.0.smithi112.stdout:removed './116/16' 2023-01-24T03:35:05.536 INFO:tasks.workunit.client.0.smithi112.stdout:removed './116/32' 2023-01-24T03:35:05.536 INFO:tasks.workunit.client.0.smithi112.stdout:removed './116/30' 2023-01-24T03:35:05.536 INFO:tasks.workunit.client.0.smithi112.stdout:removed './116/14' 2023-01-24T03:35:05.537 INFO:tasks.workunit.client.0.smithi112.stdout:removed './116/18' 2023-01-24T03:35:05.537 INFO:tasks.workunit.client.0.smithi112.stdout:removed './116/13' 2023-01-24T03:35:05.537 INFO:tasks.workunit.client.0.smithi112.stdout:removed './116/33' 2023-01-24T03:35:05.537 INFO:tasks.workunit.client.0.smithi112.stdout:removed './116/17' 2023-01-24T03:35:05.537 INFO:tasks.workunit.client.0.smithi112.stdout:removed './116/35' 2023-01-24T03:35:05.538 INFO:tasks.workunit.client.0.smithi112.stdout:removed './116/37' 2023-01-24T03:35:05.538 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './116' 2023-01-24T03:35:05.538 INFO:tasks.workunit.client.0.smithi112.stdout:removed './117/19' 2023-01-24T03:35:05.538 INFO:tasks.workunit.client.0.smithi112.stdout:removed './117/15' 2023-01-24T03:35:05.538 INFO:tasks.workunit.client.0.smithi112.stdout:removed './117/31' 2023-01-24T03:35:05.539 INFO:tasks.workunit.client.0.smithi112.stdout:removed './117/36' 2023-01-24T03:35:05.539 INFO:tasks.workunit.client.0.smithi112.stdout:removed './117/20' 2023-01-24T03:35:05.539 INFO:tasks.workunit.client.0.smithi112.stdout:removed './117/12' 2023-01-24T03:35:05.539 INFO:tasks.workunit.client.0.smithi112.stdout:removed './117/39' 2023-01-24T03:35:05.539 INFO:tasks.workunit.client.0.smithi112.stdout:removed './117/10' 2023-01-24T03:35:05.540 INFO:tasks.workunit.client.0.smithi112.stdout:removed './117/34' 2023-01-24T03:35:05.540 INFO:tasks.workunit.client.0.smithi112.stdout:removed './117/38' 2023-01-24T03:35:05.540 INFO:tasks.workunit.client.0.smithi112.stdout:removed './117/11' 2023-01-24T03:35:05.540 INFO:tasks.workunit.client.0.smithi112.stdout:removed './117/16' 2023-01-24T03:35:05.540 INFO:tasks.workunit.client.0.smithi112.stdout:removed './117/32' 2023-01-24T03:35:05.541 INFO:tasks.workunit.client.0.smithi112.stdout:removed './117/30' 2023-01-24T03:35:05.541 INFO:tasks.workunit.client.0.smithi112.stdout:removed './117/14' 2023-01-24T03:35:05.541 INFO:tasks.workunit.client.0.smithi112.stdout:removed './117/18' 2023-01-24T03:35:05.541 INFO:tasks.workunit.client.0.smithi112.stdout:removed './117/13' 2023-01-24T03:35:05.541 INFO:tasks.workunit.client.0.smithi112.stdout:removed './117/33' 2023-01-24T03:35:05.541 INFO:tasks.workunit.client.0.smithi112.stdout:removed './117/17' 2023-01-24T03:35:05.542 INFO:tasks.workunit.client.0.smithi112.stdout:removed './117/35' 2023-01-24T03:35:05.542 INFO:tasks.workunit.client.0.smithi112.stdout:removed './117/37' 2023-01-24T03:35:05.542 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './117' 2023-01-24T03:35:05.542 INFO:tasks.workunit.client.0.smithi112.stdout:removed './118/19' 2023-01-24T03:35:05.542 INFO:tasks.workunit.client.0.smithi112.stdout:removed './118/15' 2023-01-24T03:35:05.543 INFO:tasks.workunit.client.0.smithi112.stdout:removed './118/31' 2023-01-24T03:35:05.543 INFO:tasks.workunit.client.0.smithi112.stdout:removed './118/36' 2023-01-24T03:35:05.543 INFO:tasks.workunit.client.0.smithi112.stdout:removed './118/20' 2023-01-24T03:35:05.543 INFO:tasks.workunit.client.0.smithi112.stdout:removed './118/12' 2023-01-24T03:35:05.543 INFO:tasks.workunit.client.0.smithi112.stdout:removed './118/39' 2023-01-24T03:35:05.544 INFO:tasks.workunit.client.0.smithi112.stdout:removed './118/10' 2023-01-24T03:35:05.544 INFO:tasks.workunit.client.0.smithi112.stdout:removed './118/34' 2023-01-24T03:35:05.544 INFO:tasks.workunit.client.0.smithi112.stdout:removed './118/38' 2023-01-24T03:35:05.544 INFO:tasks.workunit.client.0.smithi112.stdout:removed './118/11' 2023-01-24T03:35:05.544 INFO:tasks.workunit.client.0.smithi112.stdout:removed './118/16' 2023-01-24T03:35:05.545 INFO:tasks.workunit.client.0.smithi112.stdout:removed './118/32' 2023-01-24T03:35:05.545 INFO:tasks.workunit.client.0.smithi112.stdout:removed './118/30' 2023-01-24T03:35:05.545 INFO:tasks.workunit.client.0.smithi112.stdout:removed './118/14' 2023-01-24T03:35:05.545 INFO:tasks.workunit.client.0.smithi112.stdout:removed './118/18' 2023-01-24T03:35:05.545 INFO:tasks.workunit.client.0.smithi112.stdout:removed './118/13' 2023-01-24T03:35:05.545 INFO:tasks.workunit.client.0.smithi112.stdout:removed './118/33' 2023-01-24T03:35:05.546 INFO:tasks.workunit.client.0.smithi112.stdout:removed './118/17' 2023-01-24T03:35:05.546 INFO:tasks.workunit.client.0.smithi112.stdout:removed './118/35' 2023-01-24T03:35:05.546 INFO:tasks.workunit.client.0.smithi112.stdout:removed './118/37' 2023-01-24T03:35:05.546 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './118' 2023-01-24T03:35:05.546 INFO:tasks.workunit.client.0.smithi112.stdout:removed './119/19' 2023-01-24T03:35:05.547 INFO:tasks.workunit.client.0.smithi112.stdout:removed './119/15' 2023-01-24T03:35:05.547 INFO:tasks.workunit.client.0.smithi112.stdout:removed './119/31' 2023-01-24T03:35:05.547 INFO:tasks.workunit.client.0.smithi112.stdout:removed './119/36' 2023-01-24T03:35:05.547 INFO:tasks.workunit.client.0.smithi112.stdout:removed './119/20' 2023-01-24T03:35:05.547 INFO:tasks.workunit.client.0.smithi112.stdout:removed './119/12' 2023-01-24T03:35:06.291 INFO:tasks.workunit.client.0.smithi112.stdout:removed './119/39' 2023-01-24T03:35:06.291 INFO:tasks.workunit.client.0.smithi112.stdout:removed './119/10' 2023-01-24T03:35:06.291 INFO:tasks.workunit.client.0.smithi112.stdout:removed './119/34' 2023-01-24T03:35:06.291 INFO:tasks.workunit.client.0.smithi112.stdout:removed './119/38' 2023-01-24T03:35:06.292 INFO:tasks.workunit.client.0.smithi112.stdout:removed './119/11' 2023-01-24T03:35:06.292 INFO:tasks.workunit.client.0.smithi112.stdout:removed './119/16' 2023-01-24T03:35:06.292 INFO:tasks.workunit.client.0.smithi112.stdout:removed './119/32' 2023-01-24T03:35:06.292 INFO:tasks.workunit.client.0.smithi112.stdout:removed './119/30' 2023-01-24T03:35:06.293 INFO:tasks.workunit.client.0.smithi112.stdout:removed './119/14' 2023-01-24T03:35:06.293 INFO:tasks.workunit.client.0.smithi112.stdout:removed './119/18' 2023-01-24T03:35:06.293 INFO:tasks.workunit.client.0.smithi112.stdout:removed './119/13' 2023-01-24T03:35:06.293 INFO:tasks.workunit.client.0.smithi112.stdout:removed './119/33' 2023-01-24T03:35:06.293 INFO:tasks.workunit.client.0.smithi112.stdout:removed './119/17' 2023-01-24T03:35:06.294 INFO:tasks.workunit.client.0.smithi112.stdout:removed './119/35' 2023-01-24T03:35:06.294 INFO:tasks.workunit.client.0.smithi112.stdout:removed './119/37' 2023-01-24T03:35:06.294 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './119' 2023-01-24T03:35:06.294 INFO:tasks.workunit.client.0.smithi112.stdout:removed './120/19' 2023-01-24T03:35:06.294 INFO:tasks.workunit.client.0.smithi112.stdout:removed './120/15' 2023-01-24T03:35:06.295 INFO:tasks.workunit.client.0.smithi112.stdout:removed './120/31' 2023-01-24T03:35:06.295 INFO:tasks.workunit.client.0.smithi112.stdout:removed './120/36' 2023-01-24T03:35:06.295 INFO:tasks.workunit.client.0.smithi112.stdout:removed './120/20' 2023-01-24T03:35:06.295 INFO:tasks.workunit.client.0.smithi112.stdout:removed './120/12' 2023-01-24T03:35:06.295 INFO:tasks.workunit.client.0.smithi112.stdout:removed './120/39' 2023-01-24T03:35:06.296 INFO:tasks.workunit.client.0.smithi112.stdout:removed './120/10' 2023-01-24T03:35:06.296 INFO:tasks.workunit.client.0.smithi112.stdout:removed './120/34' 2023-01-24T03:35:06.296 INFO:tasks.workunit.client.0.smithi112.stdout:removed './120/38' 2023-01-24T03:35:06.296 INFO:tasks.workunit.client.0.smithi112.stdout:removed './120/11' 2023-01-24T03:35:06.296 INFO:tasks.workunit.client.0.smithi112.stdout:removed './120/16' 2023-01-24T03:35:06.297 INFO:tasks.workunit.client.0.smithi112.stdout:removed './120/32' 2023-01-24T03:35:06.297 INFO:tasks.workunit.client.0.smithi112.stdout:removed './120/30' 2023-01-24T03:35:06.297 INFO:tasks.workunit.client.0.smithi112.stdout:removed './120/14' 2023-01-24T03:35:06.297 INFO:tasks.workunit.client.0.smithi112.stdout:removed './120/18' 2023-01-24T03:35:06.297 INFO:tasks.workunit.client.0.smithi112.stdout:removed './120/13' 2023-01-24T03:35:06.297 INFO:tasks.workunit.client.0.smithi112.stdout:removed './120/33' 2023-01-24T03:35:06.298 INFO:tasks.workunit.client.0.smithi112.stdout:removed './120/17' 2023-01-24T03:35:06.298 INFO:tasks.workunit.client.0.smithi112.stdout:removed './120/35' 2023-01-24T03:35:06.298 INFO:tasks.workunit.client.0.smithi112.stdout:removed './120/37' 2023-01-24T03:35:06.298 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './120' 2023-01-24T03:35:06.298 INFO:tasks.workunit.client.0.smithi112.stdout:removed './121/19' 2023-01-24T03:35:06.299 INFO:tasks.workunit.client.0.smithi112.stdout:removed './121/15' 2023-01-24T03:35:06.299 INFO:tasks.workunit.client.0.smithi112.stdout:removed './121/31' 2023-01-24T03:35:06.299 INFO:tasks.workunit.client.0.smithi112.stdout:removed './121/36' 2023-01-24T03:35:06.299 INFO:tasks.workunit.client.0.smithi112.stdout:removed './121/20' 2023-01-24T03:35:06.299 INFO:tasks.workunit.client.0.smithi112.stdout:removed './121/12' 2023-01-24T03:35:06.299 INFO:tasks.workunit.client.0.smithi112.stdout:removed './121/39' 2023-01-24T03:35:06.300 INFO:tasks.workunit.client.0.smithi112.stdout:removed './121/10' 2023-01-24T03:35:06.300 INFO:tasks.workunit.client.0.smithi112.stdout:removed './121/34' 2023-01-24T03:35:06.300 INFO:tasks.workunit.client.0.smithi112.stdout:removed './121/38' 2023-01-24T03:35:06.300 INFO:tasks.workunit.client.0.smithi112.stdout:removed './121/11' 2023-01-24T03:35:06.300 INFO:tasks.workunit.client.0.smithi112.stdout:removed './121/16' 2023-01-24T03:35:06.301 INFO:tasks.workunit.client.0.smithi112.stdout:removed './121/32' 2023-01-24T03:35:06.301 INFO:tasks.workunit.client.0.smithi112.stdout:removed './121/30' 2023-01-24T03:35:06.301 INFO:tasks.workunit.client.0.smithi112.stdout:removed './121/14' 2023-01-24T03:35:06.301 INFO:tasks.workunit.client.0.smithi112.stdout:removed './121/18' 2023-01-24T03:35:06.301 INFO:tasks.workunit.client.0.smithi112.stdout:removed './121/13' 2023-01-24T03:35:06.302 INFO:tasks.workunit.client.0.smithi112.stdout:removed './121/33' 2023-01-24T03:35:06.302 INFO:tasks.workunit.client.0.smithi112.stdout:removed './121/17' 2023-01-24T03:35:06.302 INFO:tasks.workunit.client.0.smithi112.stdout:removed './121/35' 2023-01-24T03:35:06.302 INFO:tasks.workunit.client.0.smithi112.stdout:removed './121/37' 2023-01-24T03:35:06.302 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './121' 2023-01-24T03:35:06.303 INFO:tasks.workunit.client.0.smithi112.stdout:removed './122/19' 2023-01-24T03:35:06.303 INFO:tasks.workunit.client.0.smithi112.stdout:removed './122/15' 2023-01-24T03:35:06.303 INFO:tasks.workunit.client.0.smithi112.stdout:removed './122/31' 2023-01-24T03:35:06.303 INFO:tasks.workunit.client.0.smithi112.stdout:removed './122/36' 2023-01-24T03:35:06.303 INFO:tasks.workunit.client.0.smithi112.stdout:removed './122/20' 2023-01-24T03:35:06.304 INFO:tasks.workunit.client.0.smithi112.stdout:removed './122/12' 2023-01-24T03:35:06.304 INFO:tasks.workunit.client.0.smithi112.stdout:removed './122/39' 2023-01-24T03:35:06.304 INFO:tasks.workunit.client.0.smithi112.stdout:removed './122/10' 2023-01-24T03:35:06.304 INFO:tasks.workunit.client.0.smithi112.stdout:removed './122/34' 2023-01-24T03:35:06.304 INFO:tasks.workunit.client.0.smithi112.stdout:removed './122/38' 2023-01-24T03:35:06.304 INFO:tasks.workunit.client.0.smithi112.stdout:removed './122/11' 2023-01-24T03:35:06.305 INFO:tasks.workunit.client.0.smithi112.stdout:removed './122/16' 2023-01-24T03:35:06.305 INFO:tasks.workunit.client.0.smithi112.stdout:removed './122/32' 2023-01-24T03:35:06.305 INFO:tasks.workunit.client.0.smithi112.stdout:removed './122/30' 2023-01-24T03:35:06.305 INFO:tasks.workunit.client.0.smithi112.stdout:removed './122/14' 2023-01-24T03:35:06.305 INFO:tasks.workunit.client.0.smithi112.stdout:removed './122/18' 2023-01-24T03:35:06.306 INFO:tasks.workunit.client.0.smithi112.stdout:removed './122/13' 2023-01-24T03:35:06.306 INFO:tasks.workunit.client.0.smithi112.stdout:removed './122/33' 2023-01-24T03:35:06.306 INFO:tasks.workunit.client.0.smithi112.stdout:removed './122/17' 2023-01-24T03:35:06.306 INFO:tasks.workunit.client.0.smithi112.stdout:removed './122/35' 2023-01-24T03:35:06.306 INFO:tasks.workunit.client.0.smithi112.stdout:removed './122/37' 2023-01-24T03:35:06.307 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './122' 2023-01-24T03:35:06.307 INFO:tasks.workunit.client.0.smithi112.stdout:removed './123/19' 2023-01-24T03:35:06.307 INFO:tasks.workunit.client.0.smithi112.stdout:removed './123/15' 2023-01-24T03:35:06.307 INFO:tasks.workunit.client.0.smithi112.stdout:removed './123/31' 2023-01-24T03:35:06.307 INFO:tasks.workunit.client.0.smithi112.stdout:removed './123/36' 2023-01-24T03:35:06.308 INFO:tasks.workunit.client.0.smithi112.stdout:removed './123/20' 2023-01-24T03:35:06.308 INFO:tasks.workunit.client.0.smithi112.stdout:removed './123/12' 2023-01-24T03:35:06.308 INFO:tasks.workunit.client.0.smithi112.stdout:removed './123/39' 2023-01-24T03:35:06.308 INFO:tasks.workunit.client.0.smithi112.stdout:removed './123/10' 2023-01-24T03:35:06.308 INFO:tasks.workunit.client.0.smithi112.stdout:removed './123/34' 2023-01-24T03:35:06.309 INFO:tasks.workunit.client.0.smithi112.stdout:removed './123/38' 2023-01-24T03:35:06.309 INFO:tasks.workunit.client.0.smithi112.stdout:removed './123/11' 2023-01-24T03:35:06.309 INFO:tasks.workunit.client.0.smithi112.stdout:removed './123/16' 2023-01-24T03:35:06.309 INFO:tasks.workunit.client.0.smithi112.stdout:removed './123/32' 2023-01-24T03:35:06.309 INFO:tasks.workunit.client.0.smithi112.stdout:removed './123/30' 2023-01-24T03:35:06.309 INFO:tasks.workunit.client.0.smithi112.stdout:removed './123/14' 2023-01-24T03:35:06.310 INFO:tasks.workunit.client.0.smithi112.stdout:removed './123/18' 2023-01-24T03:35:06.310 INFO:tasks.workunit.client.0.smithi112.stdout:removed './123/13' 2023-01-24T03:35:06.310 INFO:tasks.workunit.client.0.smithi112.stdout:removed './123/33' 2023-01-24T03:35:06.310 INFO:tasks.workunit.client.0.smithi112.stdout:removed './123/17' 2023-01-24T03:35:06.310 INFO:tasks.workunit.client.0.smithi112.stdout:removed './123/35' 2023-01-24T03:35:06.311 INFO:tasks.workunit.client.0.smithi112.stdout:removed './123/37' 2023-01-24T03:35:06.311 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './123' 2023-01-24T03:35:06.311 INFO:tasks.workunit.client.0.smithi112.stdout:removed './124/19' 2023-01-24T03:35:06.311 INFO:tasks.workunit.client.0.smithi112.stdout:removed './124/15' 2023-01-24T03:35:06.311 INFO:tasks.workunit.client.0.smithi112.stdout:removed './124/31' 2023-01-24T03:35:06.312 INFO:tasks.workunit.client.0.smithi112.stdout:removed './124/36' 2023-01-24T03:35:06.312 INFO:tasks.workunit.client.0.smithi112.stdout:removed './124/20' 2023-01-24T03:35:06.312 INFO:tasks.workunit.client.0.smithi112.stdout:removed './124/12' 2023-01-24T03:35:06.312 INFO:tasks.workunit.client.0.smithi112.stdout:removed './124/39' 2023-01-24T03:35:06.312 INFO:tasks.workunit.client.0.smithi112.stdout:removed './124/10' 2023-01-24T03:35:06.313 INFO:tasks.workunit.client.0.smithi112.stdout:removed './124/34' 2023-01-24T03:35:06.313 INFO:tasks.workunit.client.0.smithi112.stdout:removed './124/38' 2023-01-24T03:35:06.313 INFO:tasks.workunit.client.0.smithi112.stdout:removed './124/11' 2023-01-24T03:35:06.313 INFO:tasks.workunit.client.0.smithi112.stdout:removed './124/16' 2023-01-24T03:35:06.313 INFO:tasks.workunit.client.0.smithi112.stdout:removed './124/32' 2023-01-24T03:35:06.314 INFO:tasks.workunit.client.0.smithi112.stdout:removed './124/30' 2023-01-24T03:35:06.314 INFO:tasks.workunit.client.0.smithi112.stdout:removed './124/14' 2023-01-24T03:35:06.314 INFO:tasks.workunit.client.0.smithi112.stdout:removed './124/18' 2023-01-24T03:35:06.314 INFO:tasks.workunit.client.0.smithi112.stdout:removed './124/13' 2023-01-24T03:35:06.314 INFO:tasks.workunit.client.0.smithi112.stdout:removed './124/33' 2023-01-24T03:35:06.315 INFO:tasks.workunit.client.0.smithi112.stdout:removed './124/17' 2023-01-24T03:35:06.315 INFO:tasks.workunit.client.0.smithi112.stdout:removed './124/35' 2023-01-24T03:35:06.315 INFO:tasks.workunit.client.0.smithi112.stdout:removed './124/37' 2023-01-24T03:35:06.315 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './124' 2023-01-24T03:35:06.315 INFO:tasks.workunit.client.0.smithi112.stdout:removed './125/19' 2023-01-24T03:35:06.315 INFO:tasks.workunit.client.0.smithi112.stdout:removed './125/15' 2023-01-24T03:35:06.316 INFO:tasks.workunit.client.0.smithi112.stdout:removed './125/31' 2023-01-24T03:35:06.316 INFO:tasks.workunit.client.0.smithi112.stdout:removed './125/36' 2023-01-24T03:35:06.316 INFO:tasks.workunit.client.0.smithi112.stdout:removed './125/20' 2023-01-24T03:35:06.316 INFO:tasks.workunit.client.0.smithi112.stdout:removed './125/12' 2023-01-24T03:35:06.316 INFO:tasks.workunit.client.0.smithi112.stdout:removed './125/39' 2023-01-24T03:35:06.317 INFO:tasks.workunit.client.0.smithi112.stdout:removed './125/10' 2023-01-24T03:35:06.317 INFO:tasks.workunit.client.0.smithi112.stdout:removed './125/34' 2023-01-24T03:35:06.317 INFO:tasks.workunit.client.0.smithi112.stdout:removed './125/38' 2023-01-24T03:35:06.317 INFO:tasks.workunit.client.0.smithi112.stdout:removed './125/11' 2023-01-24T03:35:06.317 INFO:tasks.workunit.client.0.smithi112.stdout:removed './125/16' 2023-01-24T03:35:06.318 INFO:tasks.workunit.client.0.smithi112.stdout:removed './125/32' 2023-01-24T03:35:06.318 INFO:tasks.workunit.client.0.smithi112.stdout:removed './125/30' 2023-01-24T03:35:06.318 INFO:tasks.workunit.client.0.smithi112.stdout:removed './125/14' 2023-01-24T03:35:06.318 INFO:tasks.workunit.client.0.smithi112.stdout:removed './125/18' 2023-01-24T03:35:06.318 INFO:tasks.workunit.client.0.smithi112.stdout:removed './125/13' 2023-01-24T03:35:06.319 INFO:tasks.workunit.client.0.smithi112.stdout:removed './125/33' 2023-01-24T03:35:06.319 INFO:tasks.workunit.client.0.smithi112.stdout:removed './125/17' 2023-01-24T03:35:06.319 INFO:tasks.workunit.client.0.smithi112.stdout:removed './125/35' 2023-01-24T03:35:06.319 INFO:tasks.workunit.client.0.smithi112.stdout:removed './125/37' 2023-01-24T03:35:06.319 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './125' 2023-01-24T03:35:06.320 INFO:tasks.workunit.client.0.smithi112.stdout:removed './126/19' 2023-01-24T03:35:06.320 INFO:tasks.workunit.client.0.smithi112.stdout:removed './126/15' 2023-01-24T03:35:06.320 INFO:tasks.workunit.client.0.smithi112.stdout:removed './126/31' 2023-01-24T03:35:06.320 INFO:tasks.workunit.client.0.smithi112.stdout:removed './126/36' 2023-01-24T03:35:06.320 INFO:tasks.workunit.client.0.smithi112.stdout:removed './126/20' 2023-01-24T03:35:06.321 INFO:tasks.workunit.client.0.smithi112.stdout:removed './126/12' 2023-01-24T03:35:06.321 INFO:tasks.workunit.client.0.smithi112.stdout:removed './126/39' 2023-01-24T03:35:06.321 INFO:tasks.workunit.client.0.smithi112.stdout:removed './126/10' 2023-01-24T03:35:06.321 INFO:tasks.workunit.client.0.smithi112.stdout:removed './126/34' 2023-01-24T03:35:06.321 INFO:tasks.workunit.client.0.smithi112.stdout:removed './126/38' 2023-01-24T03:35:06.322 INFO:tasks.workunit.client.0.smithi112.stdout:removed './126/11' 2023-01-24T03:35:06.322 INFO:tasks.workunit.client.0.smithi112.stdout:removed './126/16' 2023-01-24T03:35:06.322 INFO:tasks.workunit.client.0.smithi112.stdout:removed './126/32' 2023-01-24T03:35:06.322 INFO:tasks.workunit.client.0.smithi112.stdout:removed './126/30' 2023-01-24T03:35:06.322 INFO:tasks.workunit.client.0.smithi112.stdout:removed './126/14' 2023-01-24T03:35:06.323 INFO:tasks.workunit.client.0.smithi112.stdout:removed './126/18' 2023-01-24T03:35:06.323 INFO:tasks.workunit.client.0.smithi112.stdout:removed './126/13' 2023-01-24T03:35:06.323 INFO:tasks.workunit.client.0.smithi112.stdout:removed './126/33' 2023-01-24T03:35:06.323 INFO:tasks.workunit.client.0.smithi112.stdout:removed './126/17' 2023-01-24T03:35:06.323 INFO:tasks.workunit.client.0.smithi112.stdout:removed './126/35' 2023-01-24T03:35:06.324 INFO:tasks.workunit.client.0.smithi112.stdout:removed './126/37' 2023-01-24T03:35:06.324 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './126' 2023-01-24T03:35:06.324 INFO:tasks.workunit.client.0.smithi112.stdout:removed './127/19' 2023-01-24T03:35:06.324 INFO:tasks.workunit.client.0.smithi112.stdout:removed './127/15' 2023-01-24T03:35:06.324 INFO:tasks.workunit.client.0.smithi112.stdout:removed './127/31' 2023-01-24T03:35:06.325 INFO:tasks.workunit.client.0.smithi112.stdout:removed './127/36' 2023-01-24T03:35:06.325 INFO:tasks.workunit.client.0.smithi112.stdout:removed './127/20' 2023-01-24T03:35:06.325 INFO:tasks.workunit.client.0.smithi112.stdout:removed './127/12' 2023-01-24T03:35:06.325 INFO:tasks.workunit.client.0.smithi112.stdout:removed './127/39' 2023-01-24T03:35:06.325 INFO:tasks.workunit.client.0.smithi112.stdout:removed './127/10' 2023-01-24T03:35:06.325 INFO:tasks.workunit.client.0.smithi112.stdout:removed './127/34' 2023-01-24T03:35:06.326 INFO:tasks.workunit.client.0.smithi112.stdout:removed './127/38' 2023-01-24T03:35:06.326 INFO:tasks.workunit.client.0.smithi112.stdout:removed './127/11' 2023-01-24T03:35:06.326 INFO:tasks.workunit.client.0.smithi112.stdout:removed './127/16' 2023-01-24T03:35:06.326 INFO:tasks.workunit.client.0.smithi112.stdout:removed './127/32' 2023-01-24T03:35:06.326 INFO:tasks.workunit.client.0.smithi112.stdout:removed './127/30' 2023-01-24T03:35:06.327 INFO:tasks.workunit.client.0.smithi112.stdout:removed './127/14' 2023-01-24T03:35:06.327 INFO:tasks.workunit.client.0.smithi112.stdout:removed './127/18' 2023-01-24T03:35:06.327 INFO:tasks.workunit.client.0.smithi112.stdout:removed './127/13' 2023-01-24T03:35:06.327 INFO:tasks.workunit.client.0.smithi112.stdout:removed './127/33' 2023-01-24T03:35:06.327 INFO:tasks.workunit.client.0.smithi112.stdout:removed './127/17' 2023-01-24T03:35:06.328 INFO:tasks.workunit.client.0.smithi112.stdout:removed './127/35' 2023-01-24T03:35:06.328 INFO:tasks.workunit.client.0.smithi112.stdout:removed './127/37' 2023-01-24T03:35:06.328 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './127' 2023-01-24T03:35:06.328 INFO:tasks.workunit.client.0.smithi112.stdout:removed './128/19' 2023-01-24T03:35:06.328 INFO:tasks.workunit.client.0.smithi112.stdout:removed './128/15' 2023-01-24T03:35:06.329 INFO:tasks.workunit.client.0.smithi112.stdout:removed './128/31' 2023-01-24T03:35:06.329 INFO:tasks.workunit.client.0.smithi112.stdout:removed './128/36' 2023-01-24T03:35:06.329 INFO:tasks.workunit.client.0.smithi112.stdout:removed './128/20' 2023-01-24T03:35:06.329 INFO:tasks.workunit.client.0.smithi112.stdout:removed './128/12' 2023-01-24T03:35:06.329 INFO:tasks.workunit.client.0.smithi112.stdout:removed './128/39' 2023-01-24T03:35:06.330 INFO:tasks.workunit.client.0.smithi112.stdout:removed './128/10' 2023-01-24T03:35:06.330 INFO:tasks.workunit.client.0.smithi112.stdout:removed './128/34' 2023-01-24T03:35:06.330 INFO:tasks.workunit.client.0.smithi112.stdout:removed './128/38' 2023-01-24T03:35:06.330 INFO:tasks.workunit.client.0.smithi112.stdout:removed './128/11' 2023-01-24T03:35:06.330 INFO:tasks.workunit.client.0.smithi112.stdout:removed './128/16' 2023-01-24T03:35:06.330 INFO:tasks.workunit.client.0.smithi112.stdout:removed './128/32' 2023-01-24T03:35:06.331 INFO:tasks.workunit.client.0.smithi112.stdout:removed './128/30' 2023-01-24T03:35:06.331 INFO:tasks.workunit.client.0.smithi112.stdout:removed './128/14' 2023-01-24T03:35:06.331 INFO:tasks.workunit.client.0.smithi112.stdout:removed './128/18' 2023-01-24T03:35:06.331 INFO:tasks.workunit.client.0.smithi112.stdout:removed './128/13' 2023-01-24T03:35:06.331 INFO:tasks.workunit.client.0.smithi112.stdout:removed './128/33' 2023-01-24T03:35:06.332 INFO:tasks.workunit.client.0.smithi112.stdout:removed './128/17' 2023-01-24T03:35:06.332 INFO:tasks.workunit.client.0.smithi112.stdout:removed './128/35' 2023-01-24T03:35:06.676 INFO:tasks.workunit.client.0.smithi112.stdout:removed './128/37' 2023-01-24T03:35:06.677 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './128' 2023-01-24T03:35:06.677 INFO:tasks.workunit.client.0.smithi112.stdout:removed './129/19' 2023-01-24T03:35:06.677 INFO:tasks.workunit.client.0.smithi112.stdout:removed './129/15' 2023-01-24T03:35:06.677 INFO:tasks.workunit.client.0.smithi112.stdout:removed './129/31' 2023-01-24T03:35:06.677 INFO:tasks.workunit.client.0.smithi112.stdout:removed './129/36' 2023-01-24T03:35:06.678 INFO:tasks.workunit.client.0.smithi112.stdout:removed './129/20' 2023-01-24T03:35:06.678 INFO:tasks.workunit.client.0.smithi112.stdout:removed './129/12' 2023-01-24T03:35:06.678 INFO:tasks.workunit.client.0.smithi112.stdout:removed './129/39' 2023-01-24T03:35:06.678 INFO:tasks.workunit.client.0.smithi112.stdout:removed './129/10' 2023-01-24T03:35:06.678 INFO:tasks.workunit.client.0.smithi112.stdout:removed './129/34' 2023-01-24T03:35:06.678 INFO:tasks.workunit.client.0.smithi112.stdout:removed './129/38' 2023-01-24T03:35:06.679 INFO:tasks.workunit.client.0.smithi112.stdout:removed './129/11' 2023-01-24T03:35:06.679 INFO:tasks.workunit.client.0.smithi112.stdout:removed './129/16' 2023-01-24T03:35:06.679 INFO:tasks.workunit.client.0.smithi112.stdout:removed './129/32' 2023-01-24T03:35:06.679 INFO:tasks.workunit.client.0.smithi112.stdout:removed './129/30' 2023-01-24T03:35:06.679 INFO:tasks.workunit.client.0.smithi112.stdout:removed './129/14' 2023-01-24T03:35:06.680 INFO:tasks.workunit.client.0.smithi112.stdout:removed './129/18' 2023-01-24T03:35:06.680 INFO:tasks.workunit.client.0.smithi112.stdout:removed './129/13' 2023-01-24T03:35:06.680 INFO:tasks.workunit.client.0.smithi112.stdout:removed './129/33' 2023-01-24T03:35:06.680 INFO:tasks.workunit.client.0.smithi112.stdout:removed './129/17' 2023-01-24T03:35:06.680 INFO:tasks.workunit.client.0.smithi112.stdout:removed './129/35' 2023-01-24T03:35:06.681 INFO:tasks.workunit.client.0.smithi112.stdout:removed './129/37' 2023-01-24T03:35:06.681 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './129' 2023-01-24T03:35:06.681 INFO:tasks.workunit.client.0.smithi112.stdout:removed './130/19' 2023-01-24T03:35:06.681 INFO:tasks.workunit.client.0.smithi112.stdout:removed './130/15' 2023-01-24T03:35:06.681 INFO:tasks.workunit.client.0.smithi112.stdout:removed './130/31' 2023-01-24T03:35:06.682 INFO:tasks.workunit.client.0.smithi112.stdout:removed './130/36' 2023-01-24T03:35:06.682 INFO:tasks.workunit.client.0.smithi112.stdout:removed './130/20' 2023-01-24T03:35:06.682 INFO:tasks.workunit.client.0.smithi112.stdout:removed './130/12' 2023-01-24T03:35:06.682 INFO:tasks.workunit.client.0.smithi112.stdout:removed './130/39' 2023-01-24T03:35:06.682 INFO:tasks.workunit.client.0.smithi112.stdout:removed './130/10' 2023-01-24T03:35:06.683 INFO:tasks.workunit.client.0.smithi112.stdout:removed './130/34' 2023-01-24T03:35:06.683 INFO:tasks.workunit.client.0.smithi112.stdout:removed './130/38' 2023-01-24T03:35:06.683 INFO:tasks.workunit.client.0.smithi112.stdout:removed './130/11' 2023-01-24T03:35:06.683 INFO:tasks.workunit.client.0.smithi112.stdout:removed './130/16' 2023-01-24T03:35:06.683 INFO:tasks.workunit.client.0.smithi112.stdout:removed './130/32' 2023-01-24T03:35:06.684 INFO:tasks.workunit.client.0.smithi112.stdout:removed './130/30' 2023-01-24T03:35:06.684 INFO:tasks.workunit.client.0.smithi112.stdout:removed './130/14' 2023-01-24T03:35:06.684 INFO:tasks.workunit.client.0.smithi112.stdout:removed './130/18' 2023-01-24T03:35:06.684 INFO:tasks.workunit.client.0.smithi112.stdout:removed './130/13' 2023-01-24T03:35:06.684 INFO:tasks.workunit.client.0.smithi112.stdout:removed './130/33' 2023-01-24T03:35:06.685 INFO:tasks.workunit.client.0.smithi112.stdout:removed './130/17' 2023-01-24T03:35:06.685 INFO:tasks.workunit.client.0.smithi112.stdout:removed './130/35' 2023-01-24T03:35:06.685 INFO:tasks.workunit.client.0.smithi112.stdout:removed './130/37' 2023-01-24T03:35:06.685 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './130' 2023-01-24T03:35:06.685 INFO:tasks.workunit.client.0.smithi112.stdout:removed './131/19' 2023-01-24T03:35:06.685 INFO:tasks.workunit.client.0.smithi112.stdout:removed './131/15' 2023-01-24T03:35:06.686 INFO:tasks.workunit.client.0.smithi112.stdout:removed './131/31' 2023-01-24T03:35:06.686 INFO:tasks.workunit.client.0.smithi112.stdout:removed './131/36' 2023-01-24T03:35:06.686 INFO:tasks.workunit.client.0.smithi112.stdout:removed './131/20' 2023-01-24T03:35:06.686 INFO:tasks.workunit.client.0.smithi112.stdout:removed './131/12' 2023-01-24T03:35:06.686 INFO:tasks.workunit.client.0.smithi112.stdout:removed './131/39' 2023-01-24T03:35:06.687 INFO:tasks.workunit.client.0.smithi112.stdout:removed './131/10' 2023-01-24T03:35:06.687 INFO:tasks.workunit.client.0.smithi112.stdout:removed './131/34' 2023-01-24T03:35:06.687 INFO:tasks.workunit.client.0.smithi112.stdout:removed './131/38' 2023-01-24T03:35:06.687 INFO:tasks.workunit.client.0.smithi112.stdout:removed './131/11' 2023-01-24T03:35:06.687 INFO:tasks.workunit.client.0.smithi112.stdout:removed './131/16' 2023-01-24T03:35:06.688 INFO:tasks.workunit.client.0.smithi112.stdout:removed './131/32' 2023-01-24T03:35:06.688 INFO:tasks.workunit.client.0.smithi112.stdout:removed './131/30' 2023-01-24T03:35:06.688 INFO:tasks.workunit.client.0.smithi112.stdout:removed './131/14' 2023-01-24T03:35:06.688 INFO:tasks.workunit.client.0.smithi112.stdout:removed './131/18' 2023-01-24T03:35:06.688 INFO:tasks.workunit.client.0.smithi112.stdout:removed './131/13' 2023-01-24T03:35:06.688 INFO:tasks.workunit.client.0.smithi112.stdout:removed './131/33' 2023-01-24T03:35:06.689 INFO:tasks.workunit.client.0.smithi112.stdout:removed './131/17' 2023-01-24T03:35:06.689 INFO:tasks.workunit.client.0.smithi112.stdout:removed './131/35' 2023-01-24T03:35:06.689 INFO:tasks.workunit.client.0.smithi112.stdout:removed './131/37' 2023-01-24T03:35:06.689 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './131' 2023-01-24T03:35:06.690 INFO:tasks.workunit.client.0.smithi112.stdout:removed './132/19' 2023-01-24T03:35:06.690 INFO:tasks.workunit.client.0.smithi112.stdout:removed './132/15' 2023-01-24T03:35:06.690 INFO:tasks.workunit.client.0.smithi112.stdout:removed './132/31' 2023-01-24T03:35:06.690 INFO:tasks.workunit.client.0.smithi112.stdout:removed './132/36' 2023-01-24T03:35:06.690 INFO:tasks.workunit.client.0.smithi112.stdout:removed './132/20' 2023-01-24T03:35:06.691 INFO:tasks.workunit.client.0.smithi112.stdout:removed './132/12' 2023-01-24T03:35:06.691 INFO:tasks.workunit.client.0.smithi112.stdout:removed './132/39' 2023-01-24T03:35:06.691 INFO:tasks.workunit.client.0.smithi112.stdout:removed './132/10' 2023-01-24T03:35:06.691 INFO:tasks.workunit.client.0.smithi112.stdout:removed './132/34' 2023-01-24T03:35:06.691 INFO:tasks.workunit.client.0.smithi112.stdout:removed './132/38' 2023-01-24T03:35:06.692 INFO:tasks.workunit.client.0.smithi112.stdout:removed './132/11' 2023-01-24T03:35:06.692 INFO:tasks.workunit.client.0.smithi112.stdout:removed './132/16' 2023-01-24T03:35:06.692 INFO:tasks.workunit.client.0.smithi112.stdout:removed './132/32' 2023-01-24T03:35:06.692 INFO:tasks.workunit.client.0.smithi112.stdout:removed './132/30' 2023-01-24T03:35:06.692 INFO:tasks.workunit.client.0.smithi112.stdout:removed './132/14' 2023-01-24T03:35:06.693 INFO:tasks.workunit.client.0.smithi112.stdout:removed './132/18' 2023-01-24T03:35:06.693 INFO:tasks.workunit.client.0.smithi112.stdout:removed './132/13' 2023-01-24T03:35:06.693 INFO:tasks.workunit.client.0.smithi112.stdout:removed './132/33' 2023-01-24T03:35:06.693 INFO:tasks.workunit.client.0.smithi112.stdout:removed './132/17' 2023-01-24T03:35:06.693 INFO:tasks.workunit.client.0.smithi112.stdout:removed './132/35' 2023-01-24T03:35:06.693 INFO:tasks.workunit.client.0.smithi112.stdout:removed './132/37' 2023-01-24T03:35:06.694 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './132' 2023-01-24T03:35:06.694 INFO:tasks.workunit.client.0.smithi112.stdout:removed './133/19' 2023-01-24T03:35:06.694 INFO:tasks.workunit.client.0.smithi112.stdout:removed './133/15' 2023-01-24T03:35:06.694 INFO:tasks.workunit.client.0.smithi112.stdout:removed './133/31' 2023-01-24T03:35:06.694 INFO:tasks.workunit.client.0.smithi112.stdout:removed './133/36' 2023-01-24T03:35:06.695 INFO:tasks.workunit.client.0.smithi112.stdout:removed './133/20' 2023-01-24T03:35:06.695 INFO:tasks.workunit.client.0.smithi112.stdout:removed './133/12' 2023-01-24T03:35:06.695 INFO:tasks.workunit.client.0.smithi112.stdout:removed './133/39' 2023-01-24T03:35:06.695 INFO:tasks.workunit.client.0.smithi112.stdout:removed './133/10' 2023-01-24T03:35:06.695 INFO:tasks.workunit.client.0.smithi112.stdout:removed './133/34' 2023-01-24T03:35:06.696 INFO:tasks.workunit.client.0.smithi112.stdout:removed './133/38' 2023-01-24T03:35:06.696 INFO:tasks.workunit.client.0.smithi112.stdout:removed './133/11' 2023-01-24T03:35:06.696 INFO:tasks.workunit.client.0.smithi112.stdout:removed './133/16' 2023-01-24T03:35:06.696 INFO:tasks.workunit.client.0.smithi112.stdout:removed './133/32' 2023-01-24T03:35:06.696 INFO:tasks.workunit.client.0.smithi112.stdout:removed './133/30' 2023-01-24T03:35:06.697 INFO:tasks.workunit.client.0.smithi112.stdout:removed './133/14' 2023-01-24T03:35:06.697 INFO:tasks.workunit.client.0.smithi112.stdout:removed './133/18' 2023-01-24T03:35:06.697 INFO:tasks.workunit.client.0.smithi112.stdout:removed './133/13' 2023-01-24T03:35:06.697 INFO:tasks.workunit.client.0.smithi112.stdout:removed './133/33' 2023-01-24T03:35:06.697 INFO:tasks.workunit.client.0.smithi112.stdout:removed './133/17' 2023-01-24T03:35:06.698 INFO:tasks.workunit.client.0.smithi112.stdout:removed './133/35' 2023-01-24T03:35:06.698 INFO:tasks.workunit.client.0.smithi112.stdout:removed './133/37' 2023-01-24T03:35:06.698 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './133' 2023-01-24T03:35:06.698 INFO:tasks.workunit.client.0.smithi112.stdout:removed './134/19' 2023-01-24T03:35:06.698 INFO:tasks.workunit.client.0.smithi112.stdout:removed './134/15' 2023-01-24T03:35:06.699 INFO:tasks.workunit.client.0.smithi112.stdout:removed './134/31' 2023-01-24T03:35:06.699 INFO:tasks.workunit.client.0.smithi112.stdout:removed './134/36' 2023-01-24T03:35:06.699 INFO:tasks.workunit.client.0.smithi112.stdout:removed './134/20' 2023-01-24T03:35:06.699 INFO:tasks.workunit.client.0.smithi112.stdout:removed './134/12' 2023-01-24T03:35:06.699 INFO:tasks.workunit.client.0.smithi112.stdout:removed './134/39' 2023-01-24T03:35:06.700 INFO:tasks.workunit.client.0.smithi112.stdout:removed './134/10' 2023-01-24T03:35:06.700 INFO:tasks.workunit.client.0.smithi112.stdout:removed './134/34' 2023-01-24T03:35:06.700 INFO:tasks.workunit.client.0.smithi112.stdout:removed './134/38' 2023-01-24T03:35:06.700 INFO:tasks.workunit.client.0.smithi112.stdout:removed './134/11' 2023-01-24T03:35:06.700 INFO:tasks.workunit.client.0.smithi112.stdout:removed './134/16' 2023-01-24T03:35:06.701 INFO:tasks.workunit.client.0.smithi112.stdout:removed './134/32' 2023-01-24T03:35:06.701 INFO:tasks.workunit.client.0.smithi112.stdout:removed './134/30' 2023-01-24T03:35:06.701 INFO:tasks.workunit.client.0.smithi112.stdout:removed './134/14' 2023-01-24T03:35:06.701 INFO:tasks.workunit.client.0.smithi112.stdout:removed './134/18' 2023-01-24T03:35:06.701 INFO:tasks.workunit.client.0.smithi112.stdout:removed './134/13' 2023-01-24T03:35:06.701 INFO:tasks.workunit.client.0.smithi112.stdout:removed './134/33' 2023-01-24T03:35:06.702 INFO:tasks.workunit.client.0.smithi112.stdout:removed './134/17' 2023-01-24T03:35:06.702 INFO:tasks.workunit.client.0.smithi112.stdout:removed './134/35' 2023-01-24T03:35:06.702 INFO:tasks.workunit.client.0.smithi112.stdout:removed './134/37' 2023-01-24T03:35:06.702 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './134' 2023-01-24T03:35:06.702 INFO:tasks.workunit.client.0.smithi112.stdout:removed './135/19' 2023-01-24T03:35:06.703 INFO:tasks.workunit.client.0.smithi112.stdout:removed './135/15' 2023-01-24T03:35:06.703 INFO:tasks.workunit.client.0.smithi112.stdout:removed './135/31' 2023-01-24T03:35:06.703 INFO:tasks.workunit.client.0.smithi112.stdout:removed './135/36' 2023-01-24T03:35:06.703 INFO:tasks.workunit.client.0.smithi112.stdout:removed './135/20' 2023-01-24T03:35:06.703 INFO:tasks.workunit.client.0.smithi112.stdout:removed './135/12' 2023-01-24T03:35:06.704 INFO:tasks.workunit.client.0.smithi112.stdout:removed './135/39' 2023-01-24T03:35:06.704 INFO:tasks.workunit.client.0.smithi112.stdout:removed './135/10' 2023-01-24T03:35:06.704 INFO:tasks.workunit.client.0.smithi112.stdout:removed './135/34' 2023-01-24T03:35:06.704 INFO:tasks.workunit.client.0.smithi112.stdout:removed './135/38' 2023-01-24T03:35:06.704 INFO:tasks.workunit.client.0.smithi112.stdout:removed './135/11' 2023-01-24T03:35:06.705 INFO:tasks.workunit.client.0.smithi112.stdout:removed './135/16' 2023-01-24T03:35:06.705 INFO:tasks.workunit.client.0.smithi112.stdout:removed './135/32' 2023-01-24T03:35:06.705 INFO:tasks.workunit.client.0.smithi112.stdout:removed './135/30' 2023-01-24T03:35:06.705 INFO:tasks.workunit.client.0.smithi112.stdout:removed './135/14' 2023-01-24T03:35:06.705 INFO:tasks.workunit.client.0.smithi112.stdout:removed './135/18' 2023-01-24T03:35:06.706 INFO:tasks.workunit.client.0.smithi112.stdout:removed './135/13' 2023-01-24T03:35:06.706 INFO:tasks.workunit.client.0.smithi112.stdout:removed './135/33' 2023-01-24T03:35:06.706 INFO:tasks.workunit.client.0.smithi112.stdout:removed './135/17' 2023-01-24T03:35:06.706 INFO:tasks.workunit.client.0.smithi112.stdout:removed './135/35' 2023-01-24T03:35:06.706 INFO:tasks.workunit.client.0.smithi112.stdout:removed './135/37' 2023-01-24T03:35:06.707 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './135' 2023-01-24T03:35:06.707 INFO:tasks.workunit.client.0.smithi112.stdout:removed './136/19' 2023-01-24T03:35:06.707 INFO:tasks.workunit.client.0.smithi112.stdout:removed './136/15' 2023-01-24T03:35:06.707 INFO:tasks.workunit.client.0.smithi112.stdout:removed './136/31' 2023-01-24T03:35:06.707 INFO:tasks.workunit.client.0.smithi112.stdout:removed './136/36' 2023-01-24T03:35:06.708 INFO:tasks.workunit.client.0.smithi112.stdout:removed './136/20' 2023-01-24T03:35:06.708 INFO:tasks.workunit.client.0.smithi112.stdout:removed './136/12' 2023-01-24T03:35:06.708 INFO:tasks.workunit.client.0.smithi112.stdout:removed './136/39' 2023-01-24T03:35:06.708 INFO:tasks.workunit.client.0.smithi112.stdout:removed './136/10' 2023-01-24T03:35:06.708 INFO:tasks.workunit.client.0.smithi112.stdout:removed './136/34' 2023-01-24T03:35:06.709 INFO:tasks.workunit.client.0.smithi112.stdout:removed './136/38' 2023-01-24T03:35:06.709 INFO:tasks.workunit.client.0.smithi112.stdout:removed './136/11' 2023-01-24T03:35:06.709 INFO:tasks.workunit.client.0.smithi112.stdout:removed './136/16' 2023-01-24T03:35:06.709 INFO:tasks.workunit.client.0.smithi112.stdout:removed './136/32' 2023-01-24T03:35:06.709 INFO:tasks.workunit.client.0.smithi112.stdout:removed './136/30' 2023-01-24T03:35:06.709 INFO:tasks.workunit.client.0.smithi112.stdout:removed './136/14' 2023-01-24T03:35:06.710 INFO:tasks.workunit.client.0.smithi112.stdout:removed './136/18' 2023-01-24T03:35:06.710 INFO:tasks.workunit.client.0.smithi112.stdout:removed './136/13' 2023-01-24T03:35:06.710 INFO:tasks.workunit.client.0.smithi112.stdout:removed './136/33' 2023-01-24T03:35:06.710 INFO:tasks.workunit.client.0.smithi112.stdout:removed './136/17' 2023-01-24T03:35:06.710 INFO:tasks.workunit.client.0.smithi112.stdout:removed './136/35' 2023-01-24T03:35:06.711 INFO:tasks.workunit.client.0.smithi112.stdout:removed './136/37' 2023-01-24T03:35:06.711 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './136' 2023-01-24T03:35:06.711 INFO:tasks.workunit.client.0.smithi112.stdout:removed './137/19' 2023-01-24T03:35:06.711 INFO:tasks.workunit.client.0.smithi112.stdout:removed './137/15' 2023-01-24T03:35:06.711 INFO:tasks.workunit.client.0.smithi112.stdout:removed './137/31' 2023-01-24T03:35:06.711 INFO:tasks.workunit.client.0.smithi112.stdout:removed './137/36' 2023-01-24T03:35:06.712 INFO:tasks.workunit.client.0.smithi112.stdout:removed './137/20' 2023-01-24T03:35:06.712 INFO:tasks.workunit.client.0.smithi112.stdout:removed './137/12' 2023-01-24T03:35:06.712 INFO:tasks.workunit.client.0.smithi112.stdout:removed './137/39' 2023-01-24T03:35:06.712 INFO:tasks.workunit.client.0.smithi112.stdout:removed './137/10' 2023-01-24T03:35:06.712 INFO:tasks.workunit.client.0.smithi112.stdout:removed './137/34' 2023-01-24T03:35:06.713 INFO:tasks.workunit.client.0.smithi112.stdout:removed './137/38' 2023-01-24T03:35:06.713 INFO:tasks.workunit.client.0.smithi112.stdout:removed './137/11' 2023-01-24T03:35:06.713 INFO:tasks.workunit.client.0.smithi112.stdout:removed './137/16' 2023-01-24T03:35:06.713 INFO:tasks.workunit.client.0.smithi112.stdout:removed './137/32' 2023-01-24T03:35:06.713 INFO:tasks.workunit.client.0.smithi112.stdout:removed './137/30' 2023-01-24T03:35:06.714 INFO:tasks.workunit.client.0.smithi112.stdout:removed './137/14' 2023-01-24T03:35:06.714 INFO:tasks.workunit.client.0.smithi112.stdout:removed './137/18' 2023-01-24T03:35:06.714 INFO:tasks.workunit.client.0.smithi112.stdout:removed './137/13' 2023-01-24T03:35:06.714 INFO:tasks.workunit.client.0.smithi112.stdout:removed './137/33' 2023-01-24T03:35:06.714 INFO:tasks.workunit.client.0.smithi112.stdout:removed './137/17' 2023-01-24T03:35:06.715 INFO:tasks.workunit.client.0.smithi112.stdout:removed './137/35' 2023-01-24T03:35:06.715 INFO:tasks.workunit.client.0.smithi112.stdout:removed './137/37' 2023-01-24T03:35:06.715 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './137' 2023-01-24T03:35:06.715 INFO:tasks.workunit.client.0.smithi112.stdout:removed './138/19' 2023-01-24T03:35:06.715 INFO:tasks.workunit.client.0.smithi112.stdout:removed './138/15' 2023-01-24T03:35:06.716 INFO:tasks.workunit.client.0.smithi112.stdout:removed './138/31' 2023-01-24T03:35:06.716 INFO:tasks.workunit.client.0.smithi112.stdout:removed './138/36' 2023-01-24T03:35:06.716 INFO:tasks.workunit.client.0.smithi112.stdout:removed './138/20' 2023-01-24T03:35:06.716 INFO:tasks.workunit.client.0.smithi112.stdout:removed './138/12' 2023-01-24T03:35:06.716 INFO:tasks.workunit.client.0.smithi112.stdout:removed './138/39' 2023-01-24T03:35:06.717 INFO:tasks.workunit.client.0.smithi112.stdout:removed './138/10' 2023-01-24T03:35:06.717 INFO:tasks.workunit.client.0.smithi112.stdout:removed './138/34' 2023-01-24T03:35:06.717 INFO:tasks.workunit.client.0.smithi112.stdout:removed './138/38' 2023-01-24T03:35:06.717 INFO:tasks.workunit.client.0.smithi112.stdout:removed './138/11' 2023-01-24T03:35:06.717 INFO:tasks.workunit.client.0.smithi112.stdout:removed './138/16' 2023-01-24T03:35:07.786 INFO:tasks.workunit.client.0.smithi112.stdout:removed './138/32' 2023-01-24T03:35:07.787 INFO:tasks.workunit.client.0.smithi112.stdout:removed './138/30' 2023-01-24T03:35:07.787 INFO:tasks.workunit.client.0.smithi112.stdout:removed './138/14' 2023-01-24T03:35:07.787 INFO:tasks.workunit.client.0.smithi112.stdout:removed './138/18' 2023-01-24T03:35:07.787 INFO:tasks.workunit.client.0.smithi112.stdout:removed './138/13' 2023-01-24T03:35:07.788 INFO:tasks.workunit.client.0.smithi112.stdout:removed './138/33' 2023-01-24T03:35:07.788 INFO:tasks.workunit.client.0.smithi112.stdout:removed './138/17' 2023-01-24T03:35:07.788 INFO:tasks.workunit.client.0.smithi112.stdout:removed './138/35' 2023-01-24T03:35:07.788 INFO:tasks.workunit.client.0.smithi112.stdout:removed './138/37' 2023-01-24T03:35:07.788 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './138' 2023-01-24T03:35:07.789 INFO:tasks.workunit.client.0.smithi112.stdout:removed './139/19' 2023-01-24T03:35:07.789 INFO:tasks.workunit.client.0.smithi112.stdout:removed './139/15' 2023-01-24T03:35:07.789 INFO:tasks.workunit.client.0.smithi112.stdout:removed './139/31' 2023-01-24T03:35:07.789 INFO:tasks.workunit.client.0.smithi112.stdout:removed './139/36' 2023-01-24T03:35:07.789 INFO:tasks.workunit.client.0.smithi112.stdout:removed './139/20' 2023-01-24T03:35:07.790 INFO:tasks.workunit.client.0.smithi112.stdout:removed './139/12' 2023-01-24T03:35:07.790 INFO:tasks.workunit.client.0.smithi112.stdout:removed './139/39' 2023-01-24T03:35:07.790 INFO:tasks.workunit.client.0.smithi112.stdout:removed './139/10' 2023-01-24T03:35:07.790 INFO:tasks.workunit.client.0.smithi112.stdout:removed './139/34' 2023-01-24T03:35:07.790 INFO:tasks.workunit.client.0.smithi112.stdout:removed './139/38' 2023-01-24T03:35:07.791 INFO:tasks.workunit.client.0.smithi112.stdout:removed './139/11' 2023-01-24T03:35:07.791 INFO:tasks.workunit.client.0.smithi112.stdout:removed './139/16' 2023-01-24T03:35:07.791 INFO:tasks.workunit.client.0.smithi112.stdout:removed './139/32' 2023-01-24T03:35:07.791 INFO:tasks.workunit.client.0.smithi112.stdout:removed './139/30' 2023-01-24T03:35:07.791 INFO:tasks.workunit.client.0.smithi112.stdout:removed './139/14' 2023-01-24T03:35:07.792 INFO:tasks.workunit.client.0.smithi112.stdout:removed './139/18' 2023-01-24T03:35:07.792 INFO:tasks.workunit.client.0.smithi112.stdout:removed './139/13' 2023-01-24T03:35:07.792 INFO:tasks.workunit.client.0.smithi112.stdout:removed './139/33' 2023-01-24T03:35:07.792 INFO:tasks.workunit.client.0.smithi112.stdout:removed './139/17' 2023-01-24T03:35:07.792 INFO:tasks.workunit.client.0.smithi112.stdout:removed './139/35' 2023-01-24T03:35:07.793 INFO:tasks.workunit.client.0.smithi112.stdout:removed './139/37' 2023-01-24T03:35:07.793 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './139' 2023-01-24T03:35:07.793 INFO:tasks.workunit.client.0.smithi112.stdout:removed './140/19' 2023-01-24T03:35:07.793 INFO:tasks.workunit.client.0.smithi112.stdout:removed './140/15' 2023-01-24T03:35:07.793 INFO:tasks.workunit.client.0.smithi112.stdout:removed './140/31' 2023-01-24T03:35:07.793 INFO:tasks.workunit.client.0.smithi112.stdout:removed './140/36' 2023-01-24T03:35:07.794 INFO:tasks.workunit.client.0.smithi112.stdout:removed './140/20' 2023-01-24T03:35:07.794 INFO:tasks.workunit.client.0.smithi112.stdout:removed './140/12' 2023-01-24T03:35:07.794 INFO:tasks.workunit.client.0.smithi112.stdout:removed './140/39' 2023-01-24T03:35:07.794 INFO:tasks.workunit.client.0.smithi112.stdout:removed './140/10' 2023-01-24T03:35:07.795 INFO:tasks.workunit.client.0.smithi112.stdout:removed './140/34' 2023-01-24T03:35:07.795 INFO:tasks.workunit.client.0.smithi112.stdout:removed './140/38' 2023-01-24T03:35:07.795 INFO:tasks.workunit.client.0.smithi112.stdout:removed './140/11' 2023-01-24T03:35:07.795 INFO:tasks.workunit.client.0.smithi112.stdout:removed './140/16' 2023-01-24T03:35:07.795 INFO:tasks.workunit.client.0.smithi112.stdout:removed './140/32' 2023-01-24T03:35:07.795 INFO:tasks.workunit.client.0.smithi112.stdout:removed './140/30' 2023-01-24T03:35:07.796 INFO:tasks.workunit.client.0.smithi112.stdout:removed './140/14' 2023-01-24T03:35:07.796 INFO:tasks.workunit.client.0.smithi112.stdout:removed './140/18' 2023-01-24T03:35:07.796 INFO:tasks.workunit.client.0.smithi112.stdout:removed './140/13' 2023-01-24T03:35:07.796 INFO:tasks.workunit.client.0.smithi112.stdout:removed './140/33' 2023-01-24T03:35:07.796 INFO:tasks.workunit.client.0.smithi112.stdout:removed './140/17' 2023-01-24T03:35:07.797 INFO:tasks.workunit.client.0.smithi112.stdout:removed './140/35' 2023-01-24T03:35:07.797 INFO:tasks.workunit.client.0.smithi112.stdout:removed './140/37' 2023-01-24T03:35:07.797 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './140' 2023-01-24T03:35:07.797 INFO:tasks.workunit.client.0.smithi112.stdout:removed './141/19' 2023-01-24T03:35:07.797 INFO:tasks.workunit.client.0.smithi112.stdout:removed './141/15' 2023-01-24T03:35:07.798 INFO:tasks.workunit.client.0.smithi112.stdout:removed './141/31' 2023-01-24T03:35:07.798 INFO:tasks.workunit.client.0.smithi112.stdout:removed './141/36' 2023-01-24T03:35:07.798 INFO:tasks.workunit.client.0.smithi112.stdout:removed './141/20' 2023-01-24T03:35:07.798 INFO:tasks.workunit.client.0.smithi112.stdout:removed './141/12' 2023-01-24T03:35:07.798 INFO:tasks.workunit.client.0.smithi112.stdout:removed './141/39' 2023-01-24T03:35:07.798 INFO:tasks.workunit.client.0.smithi112.stdout:removed './141/10' 2023-01-24T03:35:07.799 INFO:tasks.workunit.client.0.smithi112.stdout:removed './141/34' 2023-01-24T03:35:07.799 INFO:tasks.workunit.client.0.smithi112.stdout:removed './141/38' 2023-01-24T03:35:07.799 INFO:tasks.workunit.client.0.smithi112.stdout:removed './141/11' 2023-01-24T03:35:07.799 INFO:tasks.workunit.client.0.smithi112.stdout:removed './141/16' 2023-01-24T03:35:07.799 INFO:tasks.workunit.client.0.smithi112.stdout:removed './141/32' 2023-01-24T03:35:07.800 INFO:tasks.workunit.client.0.smithi112.stdout:removed './141/30' 2023-01-24T03:35:07.800 INFO:tasks.workunit.client.0.smithi112.stdout:removed './141/14' 2023-01-24T03:35:07.800 INFO:tasks.workunit.client.0.smithi112.stdout:removed './141/18' 2023-01-24T03:35:07.800 INFO:tasks.workunit.client.0.smithi112.stdout:removed './141/13' 2023-01-24T03:35:07.800 INFO:tasks.workunit.client.0.smithi112.stdout:removed './141/33' 2023-01-24T03:35:07.801 INFO:tasks.workunit.client.0.smithi112.stdout:removed './141/17' 2023-01-24T03:35:07.801 INFO:tasks.workunit.client.0.smithi112.stdout:removed './141/35' 2023-01-24T03:35:07.801 INFO:tasks.workunit.client.0.smithi112.stdout:removed './141/37' 2023-01-24T03:35:07.801 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './141' 2023-01-24T03:35:07.801 INFO:tasks.workunit.client.0.smithi112.stdout:removed './142/19' 2023-01-24T03:35:07.802 INFO:tasks.workunit.client.0.smithi112.stdout:removed './142/15' 2023-01-24T03:35:07.802 INFO:tasks.workunit.client.0.smithi112.stdout:removed './142/31' 2023-01-24T03:35:07.802 INFO:tasks.workunit.client.0.smithi112.stdout:removed './142/36' 2023-01-24T03:35:07.802 INFO:tasks.workunit.client.0.smithi112.stdout:removed './142/20' 2023-01-24T03:35:07.802 INFO:tasks.workunit.client.0.smithi112.stdout:removed './142/12' 2023-01-24T03:35:07.803 INFO:tasks.workunit.client.0.smithi112.stdout:removed './142/39' 2023-01-24T03:35:07.803 INFO:tasks.workunit.client.0.smithi112.stdout:removed './142/10' 2023-01-24T03:35:07.803 INFO:tasks.workunit.client.0.smithi112.stdout:removed './142/34' 2023-01-24T03:35:07.803 INFO:tasks.workunit.client.0.smithi112.stdout:removed './142/38' 2023-01-24T03:35:07.803 INFO:tasks.workunit.client.0.smithi112.stdout:removed './142/11' 2023-01-24T03:35:07.804 INFO:tasks.workunit.client.0.smithi112.stdout:removed './142/16' 2023-01-24T03:35:07.804 INFO:tasks.workunit.client.0.smithi112.stdout:removed './142/32' 2023-01-24T03:35:07.804 INFO:tasks.workunit.client.0.smithi112.stdout:removed './142/30' 2023-01-24T03:35:07.804 INFO:tasks.workunit.client.0.smithi112.stdout:removed './142/14' 2023-01-24T03:35:07.804 INFO:tasks.workunit.client.0.smithi112.stdout:removed './142/18' 2023-01-24T03:35:07.805 INFO:tasks.workunit.client.0.smithi112.stdout:removed './142/13' 2023-01-24T03:35:07.805 INFO:tasks.workunit.client.0.smithi112.stdout:removed './142/33' 2023-01-24T03:35:07.805 INFO:tasks.workunit.client.0.smithi112.stdout:removed './142/17' 2023-01-24T03:35:07.805 INFO:tasks.workunit.client.0.smithi112.stdout:removed './142/35' 2023-01-24T03:35:07.805 INFO:tasks.workunit.client.0.smithi112.stdout:removed './142/37' 2023-01-24T03:35:07.805 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './142' 2023-01-24T03:35:07.806 INFO:tasks.workunit.client.0.smithi112.stdout:removed './143/19' 2023-01-24T03:35:07.806 INFO:tasks.workunit.client.0.smithi112.stdout:removed './143/15' 2023-01-24T03:35:07.806 INFO:tasks.workunit.client.0.smithi112.stdout:removed './143/31' 2023-01-24T03:35:07.806 INFO:tasks.workunit.client.0.smithi112.stdout:removed './143/36' 2023-01-24T03:35:07.806 INFO:tasks.workunit.client.0.smithi112.stdout:removed './143/20' 2023-01-24T03:35:07.807 INFO:tasks.workunit.client.0.smithi112.stdout:removed './143/12' 2023-01-24T03:35:07.807 INFO:tasks.workunit.client.0.smithi112.stdout:removed './143/39' 2023-01-24T03:35:07.807 INFO:tasks.workunit.client.0.smithi112.stdout:removed './143/10' 2023-01-24T03:35:07.807 INFO:tasks.workunit.client.0.smithi112.stdout:removed './143/34' 2023-01-24T03:35:07.808 INFO:tasks.workunit.client.0.smithi112.stdout:removed './143/38' 2023-01-24T03:35:07.808 INFO:tasks.workunit.client.0.smithi112.stdout:removed './143/11' 2023-01-24T03:35:07.808 INFO:tasks.workunit.client.0.smithi112.stdout:removed './143/16' 2023-01-24T03:35:07.808 INFO:tasks.workunit.client.0.smithi112.stdout:removed './143/32' 2023-01-24T03:35:07.808 INFO:tasks.workunit.client.0.smithi112.stdout:removed './143/30' 2023-01-24T03:35:07.809 INFO:tasks.workunit.client.0.smithi112.stdout:removed './143/14' 2023-01-24T03:35:07.809 INFO:tasks.workunit.client.0.smithi112.stdout:removed './143/18' 2023-01-24T03:35:07.809 INFO:tasks.workunit.client.0.smithi112.stdout:removed './143/13' 2023-01-24T03:35:07.809 INFO:tasks.workunit.client.0.smithi112.stdout:removed './143/33' 2023-01-24T03:35:07.809 INFO:tasks.workunit.client.0.smithi112.stdout:removed './143/17' 2023-01-24T03:35:07.810 INFO:tasks.workunit.client.0.smithi112.stdout:removed './143/35' 2023-01-24T03:35:07.810 INFO:tasks.workunit.client.0.smithi112.stdout:removed './143/37' 2023-01-24T03:35:07.810 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './143' 2023-01-24T03:35:07.810 INFO:tasks.workunit.client.0.smithi112.stdout:removed './144/19' 2023-01-24T03:35:07.810 INFO:tasks.workunit.client.0.smithi112.stdout:removed './144/15' 2023-01-24T03:35:07.810 INFO:tasks.workunit.client.0.smithi112.stdout:removed './144/31' 2023-01-24T03:35:07.811 INFO:tasks.workunit.client.0.smithi112.stdout:removed './144/36' 2023-01-24T03:35:07.811 INFO:tasks.workunit.client.0.smithi112.stdout:removed './144/20' 2023-01-24T03:35:07.811 INFO:tasks.workunit.client.0.smithi112.stdout:removed './144/12' 2023-01-24T03:35:07.811 INFO:tasks.workunit.client.0.smithi112.stdout:removed './144/39' 2023-01-24T03:35:07.811 INFO:tasks.workunit.client.0.smithi112.stdout:removed './144/10' 2023-01-24T03:35:07.812 INFO:tasks.workunit.client.0.smithi112.stdout:removed './144/34' 2023-01-24T03:35:07.812 INFO:tasks.workunit.client.0.smithi112.stdout:removed './144/38' 2023-01-24T03:35:07.812 INFO:tasks.workunit.client.0.smithi112.stdout:removed './144/11' 2023-01-24T03:35:07.812 INFO:tasks.workunit.client.0.smithi112.stdout:removed './144/16' 2023-01-24T03:35:07.812 INFO:tasks.workunit.client.0.smithi112.stdout:removed './144/32' 2023-01-24T03:35:07.813 INFO:tasks.workunit.client.0.smithi112.stdout:removed './144/30' 2023-01-24T03:35:07.813 INFO:tasks.workunit.client.0.smithi112.stdout:removed './144/14' 2023-01-24T03:35:07.813 INFO:tasks.workunit.client.0.smithi112.stdout:removed './144/18' 2023-01-24T03:35:07.813 INFO:tasks.workunit.client.0.smithi112.stdout:removed './144/13' 2023-01-24T03:35:07.813 INFO:tasks.workunit.client.0.smithi112.stdout:removed './144/33' 2023-01-24T03:35:07.814 INFO:tasks.workunit.client.0.smithi112.stdout:removed './144/17' 2023-01-24T03:35:07.814 INFO:tasks.workunit.client.0.smithi112.stdout:removed './144/35' 2023-01-24T03:35:07.814 INFO:tasks.workunit.client.0.smithi112.stdout:removed './144/37' 2023-01-24T03:35:07.814 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './144' 2023-01-24T03:35:07.814 INFO:tasks.workunit.client.0.smithi112.stdout:removed './145/19' 2023-01-24T03:35:07.814 INFO:tasks.workunit.client.0.smithi112.stdout:removed './145/15' 2023-01-24T03:35:07.815 INFO:tasks.workunit.client.0.smithi112.stdout:removed './145/31' 2023-01-24T03:35:07.815 INFO:tasks.workunit.client.0.smithi112.stdout:removed './145/36' 2023-01-24T03:35:07.815 INFO:tasks.workunit.client.0.smithi112.stdout:removed './145/20' 2023-01-24T03:35:07.815 INFO:tasks.workunit.client.0.smithi112.stdout:removed './145/12' 2023-01-24T03:35:07.815 INFO:tasks.workunit.client.0.smithi112.stdout:removed './145/39' 2023-01-24T03:35:07.816 INFO:tasks.workunit.client.0.smithi112.stdout:removed './145/10' 2023-01-24T03:35:07.816 INFO:tasks.workunit.client.0.smithi112.stdout:removed './145/34' 2023-01-24T03:35:07.816 INFO:tasks.workunit.client.0.smithi112.stdout:removed './145/38' 2023-01-24T03:35:07.816 INFO:tasks.workunit.client.0.smithi112.stdout:removed './145/11' 2023-01-24T03:35:07.816 INFO:tasks.workunit.client.0.smithi112.stdout:removed './145/16' 2023-01-24T03:35:07.817 INFO:tasks.workunit.client.0.smithi112.stdout:removed './145/32' 2023-01-24T03:35:07.817 INFO:tasks.workunit.client.0.smithi112.stdout:removed './145/30' 2023-01-24T03:35:07.817 INFO:tasks.workunit.client.0.smithi112.stdout:removed './145/14' 2023-01-24T03:35:07.817 INFO:tasks.workunit.client.0.smithi112.stdout:removed './145/18' 2023-01-24T03:35:07.817 INFO:tasks.workunit.client.0.smithi112.stdout:removed './145/13' 2023-01-24T03:35:07.817 INFO:tasks.workunit.client.0.smithi112.stdout:removed './145/33' 2023-01-24T03:35:07.818 INFO:tasks.workunit.client.0.smithi112.stdout:removed './145/17' 2023-01-24T03:35:07.818 INFO:tasks.workunit.client.0.smithi112.stdout:removed './145/35' 2023-01-24T03:35:07.818 INFO:tasks.workunit.client.0.smithi112.stdout:removed './145/37' 2023-01-24T03:35:07.818 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './145' 2023-01-24T03:35:07.818 INFO:tasks.workunit.client.0.smithi112.stdout:removed './146/19' 2023-01-24T03:35:07.819 INFO:tasks.workunit.client.0.smithi112.stdout:removed './146/15' 2023-01-24T03:35:07.819 INFO:tasks.workunit.client.0.smithi112.stdout:removed './146/31' 2023-01-24T03:35:07.819 INFO:tasks.workunit.client.0.smithi112.stdout:removed './146/36' 2023-01-24T03:35:07.819 INFO:tasks.workunit.client.0.smithi112.stdout:removed './146/20' 2023-01-24T03:35:07.819 INFO:tasks.workunit.client.0.smithi112.stdout:removed './146/12' 2023-01-24T03:35:07.820 INFO:tasks.workunit.client.0.smithi112.stdout:removed './146/39' 2023-01-24T03:35:07.820 INFO:tasks.workunit.client.0.smithi112.stdout:removed './146/10' 2023-01-24T03:35:07.820 INFO:tasks.workunit.client.0.smithi112.stdout:removed './146/34' 2023-01-24T03:35:07.820 INFO:tasks.workunit.client.0.smithi112.stdout:removed './146/38' 2023-01-24T03:35:07.820 INFO:tasks.workunit.client.0.smithi112.stdout:removed './146/11' 2023-01-24T03:35:07.821 INFO:tasks.workunit.client.0.smithi112.stdout:removed './146/16' 2023-01-24T03:35:07.821 INFO:tasks.workunit.client.0.smithi112.stdout:removed './146/32' 2023-01-24T03:35:07.821 INFO:tasks.workunit.client.0.smithi112.stdout:removed './146/30' 2023-01-24T03:35:07.821 INFO:tasks.workunit.client.0.smithi112.stdout:removed './146/14' 2023-01-24T03:35:07.821 INFO:tasks.workunit.client.0.smithi112.stdout:removed './146/18' 2023-01-24T03:35:07.822 INFO:tasks.workunit.client.0.smithi112.stdout:removed './146/13' 2023-01-24T03:35:07.822 INFO:tasks.workunit.client.0.smithi112.stdout:removed './146/33' 2023-01-24T03:35:07.822 INFO:tasks.workunit.client.0.smithi112.stdout:removed './146/17' 2023-01-24T03:35:07.822 INFO:tasks.workunit.client.0.smithi112.stdout:removed './146/35' 2023-01-24T03:35:07.822 INFO:tasks.workunit.client.0.smithi112.stdout:removed './146/37' 2023-01-24T03:35:07.822 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './146' 2023-01-24T03:35:07.823 INFO:tasks.workunit.client.0.smithi112.stdout:removed './147/19' 2023-01-24T03:35:07.823 INFO:tasks.workunit.client.0.smithi112.stdout:removed './147/15' 2023-01-24T03:35:07.823 INFO:tasks.workunit.client.0.smithi112.stdout:removed './147/31' 2023-01-24T03:35:07.823 INFO:tasks.workunit.client.0.smithi112.stdout:removed './147/36' 2023-01-24T03:35:07.824 INFO:tasks.workunit.client.0.smithi112.stdout:removed './147/20' 2023-01-24T03:35:07.824 INFO:tasks.workunit.client.0.smithi112.stdout:removed './147/12' 2023-01-24T03:35:07.824 INFO:tasks.workunit.client.0.smithi112.stdout:removed './147/39' 2023-01-24T03:35:07.824 INFO:tasks.workunit.client.0.smithi112.stdout:removed './147/10' 2023-01-24T03:35:07.824 INFO:tasks.workunit.client.0.smithi112.stdout:removed './147/34' 2023-01-24T03:35:07.825 INFO:tasks.workunit.client.0.smithi112.stdout:removed './147/38' 2023-01-24T03:35:07.825 INFO:tasks.workunit.client.0.smithi112.stdout:removed './147/11' 2023-01-24T03:35:07.825 INFO:tasks.workunit.client.0.smithi112.stdout:removed './147/16' 2023-01-24T03:35:07.825 INFO:tasks.workunit.client.0.smithi112.stdout:removed './147/32' 2023-01-24T03:35:07.825 INFO:tasks.workunit.client.0.smithi112.stdout:removed './147/30' 2023-01-24T03:35:07.825 INFO:tasks.workunit.client.0.smithi112.stdout:removed './147/14' 2023-01-24T03:35:07.826 INFO:tasks.workunit.client.0.smithi112.stdout:removed './147/18' 2023-01-24T03:35:07.826 INFO:tasks.workunit.client.0.smithi112.stdout:removed './147/13' 2023-01-24T03:35:07.826 INFO:tasks.workunit.client.0.smithi112.stdout:removed './147/33' 2023-01-24T03:35:07.826 INFO:tasks.workunit.client.0.smithi112.stdout:removed './147/17' 2023-01-24T03:35:07.826 INFO:tasks.workunit.client.0.smithi112.stdout:removed './147/35' 2023-01-24T03:35:07.827 INFO:tasks.workunit.client.0.smithi112.stdout:removed './147/37' 2023-01-24T03:35:07.827 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './147' 2023-01-24T03:35:07.827 INFO:tasks.workunit.client.0.smithi112.stdout:removed './148/19' 2023-01-24T03:35:07.827 INFO:tasks.workunit.client.0.smithi112.stdout:removed './148/15' 2023-01-24T03:35:07.828 INFO:tasks.workunit.client.0.smithi112.stdout:removed './148/31' 2023-01-24T03:35:07.828 INFO:tasks.workunit.client.0.smithi112.stdout:removed './148/36' 2023-01-24T03:35:08.219 INFO:tasks.workunit.client.0.smithi112.stdout:removed './148/20' 2023-01-24T03:35:08.219 INFO:tasks.workunit.client.0.smithi112.stdout:removed './148/12' 2023-01-24T03:35:08.219 INFO:tasks.workunit.client.0.smithi112.stdout:removed './148/39' 2023-01-24T03:35:08.220 INFO:tasks.workunit.client.0.smithi112.stdout:removed './148/10' 2023-01-24T03:35:08.220 INFO:tasks.workunit.client.0.smithi112.stdout:removed './148/34' 2023-01-24T03:35:08.220 INFO:tasks.workunit.client.0.smithi112.stdout:removed './148/38' 2023-01-24T03:35:08.220 INFO:tasks.workunit.client.0.smithi112.stdout:removed './148/11' 2023-01-24T03:35:08.220 INFO:tasks.workunit.client.0.smithi112.stdout:removed './148/16' 2023-01-24T03:35:08.221 INFO:tasks.workunit.client.0.smithi112.stdout:removed './148/32' 2023-01-24T03:35:08.221 INFO:tasks.workunit.client.0.smithi112.stdout:removed './148/30' 2023-01-24T03:35:08.221 INFO:tasks.workunit.client.0.smithi112.stdout:removed './148/14' 2023-01-24T03:35:08.221 INFO:tasks.workunit.client.0.smithi112.stdout:removed './148/18' 2023-01-24T03:35:08.221 INFO:tasks.workunit.client.0.smithi112.stdout:removed './148/13' 2023-01-24T03:35:08.222 INFO:tasks.workunit.client.0.smithi112.stdout:removed './148/33' 2023-01-24T03:35:08.222 INFO:tasks.workunit.client.0.smithi112.stdout:removed './148/17' 2023-01-24T03:35:08.222 INFO:tasks.workunit.client.0.smithi112.stdout:removed './148/35' 2023-01-24T03:35:08.222 INFO:tasks.workunit.client.0.smithi112.stdout:removed './148/37' 2023-01-24T03:35:08.222 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './148' 2023-01-24T03:35:08.222 INFO:tasks.workunit.client.0.smithi112.stdout:removed './149/19' 2023-01-24T03:35:08.223 INFO:tasks.workunit.client.0.smithi112.stdout:removed './149/15' 2023-01-24T03:35:08.223 INFO:tasks.workunit.client.0.smithi112.stdout:removed './149/31' 2023-01-24T03:35:08.223 INFO:tasks.workunit.client.0.smithi112.stdout:removed './149/36' 2023-01-24T03:35:08.223 INFO:tasks.workunit.client.0.smithi112.stdout:removed './149/20' 2023-01-24T03:35:08.223 INFO:tasks.workunit.client.0.smithi112.stdout:removed './149/12' 2023-01-24T03:35:08.224 INFO:tasks.workunit.client.0.smithi112.stdout:removed './149/39' 2023-01-24T03:35:08.224 INFO:tasks.workunit.client.0.smithi112.stdout:removed './149/10' 2023-01-24T03:35:08.224 INFO:tasks.workunit.client.0.smithi112.stdout:removed './149/34' 2023-01-24T03:35:08.224 INFO:tasks.workunit.client.0.smithi112.stdout:removed './149/38' 2023-01-24T03:35:08.224 INFO:tasks.workunit.client.0.smithi112.stdout:removed './149/11' 2023-01-24T03:35:08.224 INFO:tasks.workunit.client.0.smithi112.stdout:removed './149/16' 2023-01-24T03:35:08.225 INFO:tasks.workunit.client.0.smithi112.stdout:removed './149/32' 2023-01-24T03:35:08.225 INFO:tasks.workunit.client.0.smithi112.stdout:removed './149/30' 2023-01-24T03:35:08.225 INFO:tasks.workunit.client.0.smithi112.stdout:removed './149/14' 2023-01-24T03:35:08.225 INFO:tasks.workunit.client.0.smithi112.stdout:removed './149/18' 2023-01-24T03:35:08.225 INFO:tasks.workunit.client.0.smithi112.stdout:removed './149/13' 2023-01-24T03:35:08.226 INFO:tasks.workunit.client.0.smithi112.stdout:removed './149/33' 2023-01-24T03:35:08.226 INFO:tasks.workunit.client.0.smithi112.stdout:removed './149/17' 2023-01-24T03:35:08.226 INFO:tasks.workunit.client.0.smithi112.stdout:removed './149/35' 2023-01-24T03:35:08.226 INFO:tasks.workunit.client.0.smithi112.stdout:removed './149/37' 2023-01-24T03:35:08.226 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './149' 2023-01-24T03:35:08.227 INFO:tasks.workunit.client.0.smithi112.stdout:removed './150/19' 2023-01-24T03:35:08.227 INFO:tasks.workunit.client.0.smithi112.stdout:removed './150/15' 2023-01-24T03:35:08.227 INFO:tasks.workunit.client.0.smithi112.stdout:removed './150/31' 2023-01-24T03:35:08.227 INFO:tasks.workunit.client.0.smithi112.stdout:removed './150/36' 2023-01-24T03:35:08.227 INFO:tasks.workunit.client.0.smithi112.stdout:removed './150/20' 2023-01-24T03:35:08.227 INFO:tasks.workunit.client.0.smithi112.stdout:removed './150/12' 2023-01-24T03:35:08.228 INFO:tasks.workunit.client.0.smithi112.stdout:removed './150/39' 2023-01-24T03:35:08.228 INFO:tasks.workunit.client.0.smithi112.stdout:removed './150/10' 2023-01-24T03:35:08.228 INFO:tasks.workunit.client.0.smithi112.stdout:removed './150/34' 2023-01-24T03:35:08.228 INFO:tasks.workunit.client.0.smithi112.stdout:removed './150/38' 2023-01-24T03:35:08.228 INFO:tasks.workunit.client.0.smithi112.stdout:removed './150/11' 2023-01-24T03:35:08.229 INFO:tasks.workunit.client.0.smithi112.stdout:removed './150/16' 2023-01-24T03:35:08.229 INFO:tasks.workunit.client.0.smithi112.stdout:removed './150/32' 2023-01-24T03:35:08.229 INFO:tasks.workunit.client.0.smithi112.stdout:removed './150/30' 2023-01-24T03:35:08.229 INFO:tasks.workunit.client.0.smithi112.stdout:removed './150/14' 2023-01-24T03:35:08.229 INFO:tasks.workunit.client.0.smithi112.stdout:removed './150/18' 2023-01-24T03:35:08.230 INFO:tasks.workunit.client.0.smithi112.stdout:removed './150/13' 2023-01-24T03:35:08.230 INFO:tasks.workunit.client.0.smithi112.stdout:removed './150/33' 2023-01-24T03:35:08.230 INFO:tasks.workunit.client.0.smithi112.stdout:removed './150/17' 2023-01-24T03:35:08.230 INFO:tasks.workunit.client.0.smithi112.stdout:removed './150/35' 2023-01-24T03:35:08.230 INFO:tasks.workunit.client.0.smithi112.stdout:removed './150/37' 2023-01-24T03:35:08.231 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './150' 2023-01-24T03:35:08.231 INFO:tasks.workunit.client.0.smithi112.stdout:removed './151/19' 2023-01-24T03:35:08.231 INFO:tasks.workunit.client.0.smithi112.stdout:removed './151/15' 2023-01-24T03:35:08.231 INFO:tasks.workunit.client.0.smithi112.stdout:removed './151/31' 2023-01-24T03:35:08.231 INFO:tasks.workunit.client.0.smithi112.stdout:removed './151/36' 2023-01-24T03:35:08.232 INFO:tasks.workunit.client.0.smithi112.stdout:removed './151/20' 2023-01-24T03:35:08.232 INFO:tasks.workunit.client.0.smithi112.stdout:removed './151/12' 2023-01-24T03:35:08.232 INFO:tasks.workunit.client.0.smithi112.stdout:removed './151/39' 2023-01-24T03:35:08.232 INFO:tasks.workunit.client.0.smithi112.stdout:removed './151/10' 2023-01-24T03:35:08.232 INFO:tasks.workunit.client.0.smithi112.stdout:removed './151/34' 2023-01-24T03:35:08.233 INFO:tasks.workunit.client.0.smithi112.stdout:removed './151/38' 2023-01-24T03:35:08.233 INFO:tasks.workunit.client.0.smithi112.stdout:removed './151/11' 2023-01-24T03:35:08.233 INFO:tasks.workunit.client.0.smithi112.stdout:removed './151/16' 2023-01-24T03:35:08.233 INFO:tasks.workunit.client.0.smithi112.stdout:removed './151/32' 2023-01-24T03:35:08.233 INFO:tasks.workunit.client.0.smithi112.stdout:removed './151/30' 2023-01-24T03:35:08.233 INFO:tasks.workunit.client.0.smithi112.stdout:removed './151/14' 2023-01-24T03:35:08.234 INFO:tasks.workunit.client.0.smithi112.stdout:removed './151/18' 2023-01-24T03:35:08.234 INFO:tasks.workunit.client.0.smithi112.stdout:removed './151/13' 2023-01-24T03:35:08.234 INFO:tasks.workunit.client.0.smithi112.stdout:removed './151/33' 2023-01-24T03:35:08.234 INFO:tasks.workunit.client.0.smithi112.stdout:removed './151/17' 2023-01-24T03:35:08.234 INFO:tasks.workunit.client.0.smithi112.stdout:removed './151/35' 2023-01-24T03:35:08.235 INFO:tasks.workunit.client.0.smithi112.stdout:removed './151/37' 2023-01-24T03:35:08.235 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './151' 2023-01-24T03:35:08.235 INFO:tasks.workunit.client.0.smithi112.stdout:removed './152/19' 2023-01-24T03:35:08.235 INFO:tasks.workunit.client.0.smithi112.stdout:removed './152/15' 2023-01-24T03:35:08.235 INFO:tasks.workunit.client.0.smithi112.stdout:removed './152/31' 2023-01-24T03:35:08.235 INFO:tasks.workunit.client.0.smithi112.stdout:removed './152/36' 2023-01-24T03:35:08.236 INFO:tasks.workunit.client.0.smithi112.stdout:removed './152/20' 2023-01-24T03:35:08.236 INFO:tasks.workunit.client.0.smithi112.stdout:removed './152/12' 2023-01-24T03:35:08.236 INFO:tasks.workunit.client.0.smithi112.stdout:removed './152/39' 2023-01-24T03:35:08.236 INFO:tasks.workunit.client.0.smithi112.stdout:removed './152/10' 2023-01-24T03:35:08.236 INFO:tasks.workunit.client.0.smithi112.stdout:removed './152/34' 2023-01-24T03:35:08.237 INFO:tasks.workunit.client.0.smithi112.stdout:removed './152/38' 2023-01-24T03:35:08.237 INFO:tasks.workunit.client.0.smithi112.stdout:removed './152/11' 2023-01-24T03:35:08.237 INFO:tasks.workunit.client.0.smithi112.stdout:removed './152/16' 2023-01-24T03:35:08.237 INFO:tasks.workunit.client.0.smithi112.stdout:removed './152/32' 2023-01-24T03:35:08.237 INFO:tasks.workunit.client.0.smithi112.stdout:removed './152/30' 2023-01-24T03:35:08.238 INFO:tasks.workunit.client.0.smithi112.stdout:removed './152/14' 2023-01-24T03:35:08.238 INFO:tasks.workunit.client.0.smithi112.stdout:removed './152/18' 2023-01-24T03:35:08.238 INFO:tasks.workunit.client.0.smithi112.stdout:removed './152/13' 2023-01-24T03:35:08.238 INFO:tasks.workunit.client.0.smithi112.stdout:removed './152/33' 2023-01-24T03:35:08.238 INFO:tasks.workunit.client.0.smithi112.stdout:removed './152/17' 2023-01-24T03:35:08.239 INFO:tasks.workunit.client.0.smithi112.stdout:removed './152/35' 2023-01-24T03:35:08.239 INFO:tasks.workunit.client.0.smithi112.stdout:removed './152/37' 2023-01-24T03:35:08.239 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './152' 2023-01-24T03:35:08.239 INFO:tasks.workunit.client.0.smithi112.stdout:removed './153/19' 2023-01-24T03:35:08.239 INFO:tasks.workunit.client.0.smithi112.stdout:removed './153/15' 2023-01-24T03:35:08.240 INFO:tasks.workunit.client.0.smithi112.stdout:removed './153/31' 2023-01-24T03:35:08.240 INFO:tasks.workunit.client.0.smithi112.stdout:removed './153/36' 2023-01-24T03:35:08.240 INFO:tasks.workunit.client.0.smithi112.stdout:removed './153/20' 2023-01-24T03:35:08.240 INFO:tasks.workunit.client.0.smithi112.stdout:removed './153/12' 2023-01-24T03:35:08.240 INFO:tasks.workunit.client.0.smithi112.stdout:removed './153/39' 2023-01-24T03:35:08.240 INFO:tasks.workunit.client.0.smithi112.stdout:removed './153/10' 2023-01-24T03:35:08.241 INFO:tasks.workunit.client.0.smithi112.stdout:removed './153/34' 2023-01-24T03:35:08.241 INFO:tasks.workunit.client.0.smithi112.stdout:removed './153/38' 2023-01-24T03:35:08.241 INFO:tasks.workunit.client.0.smithi112.stdout:removed './153/11' 2023-01-24T03:35:08.241 INFO:tasks.workunit.client.0.smithi112.stdout:removed './153/16' 2023-01-24T03:35:08.241 INFO:tasks.workunit.client.0.smithi112.stdout:removed './153/32' 2023-01-24T03:35:08.242 INFO:tasks.workunit.client.0.smithi112.stdout:removed './153/30' 2023-01-24T03:35:08.242 INFO:tasks.workunit.client.0.smithi112.stdout:removed './153/14' 2023-01-24T03:35:08.242 INFO:tasks.workunit.client.0.smithi112.stdout:removed './153/18' 2023-01-24T03:35:08.242 INFO:tasks.workunit.client.0.smithi112.stdout:removed './153/13' 2023-01-24T03:35:08.242 INFO:tasks.workunit.client.0.smithi112.stdout:removed './153/33' 2023-01-24T03:35:08.243 INFO:tasks.workunit.client.0.smithi112.stdout:removed './153/17' 2023-01-24T03:35:08.243 INFO:tasks.workunit.client.0.smithi112.stdout:removed './153/35' 2023-01-24T03:35:08.243 INFO:tasks.workunit.client.0.smithi112.stdout:removed './153/37' 2023-01-24T03:35:08.243 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './153' 2023-01-24T03:35:08.243 INFO:tasks.workunit.client.0.smithi112.stdout:removed './154/19' 2023-01-24T03:35:08.244 INFO:tasks.workunit.client.0.smithi112.stdout:removed './154/15' 2023-01-24T03:35:08.244 INFO:tasks.workunit.client.0.smithi112.stdout:removed './154/31' 2023-01-24T03:35:08.244 INFO:tasks.workunit.client.0.smithi112.stdout:removed './154/36' 2023-01-24T03:35:08.244 INFO:tasks.workunit.client.0.smithi112.stdout:removed './154/20' 2023-01-24T03:35:08.244 INFO:tasks.workunit.client.0.smithi112.stdout:removed './154/12' 2023-01-24T03:35:08.245 INFO:tasks.workunit.client.0.smithi112.stdout:removed './154/39' 2023-01-24T03:35:08.245 INFO:tasks.workunit.client.0.smithi112.stdout:removed './154/10' 2023-01-24T03:35:08.245 INFO:tasks.workunit.client.0.smithi112.stdout:removed './154/34' 2023-01-24T03:35:08.245 INFO:tasks.workunit.client.0.smithi112.stdout:removed './154/38' 2023-01-24T03:35:08.245 INFO:tasks.workunit.client.0.smithi112.stdout:removed './154/11' 2023-01-24T03:35:08.245 INFO:tasks.workunit.client.0.smithi112.stdout:removed './154/16' 2023-01-24T03:35:08.246 INFO:tasks.workunit.client.0.smithi112.stdout:removed './154/32' 2023-01-24T03:35:08.246 INFO:tasks.workunit.client.0.smithi112.stdout:removed './154/30' 2023-01-24T03:35:08.246 INFO:tasks.workunit.client.0.smithi112.stdout:removed './154/14' 2023-01-24T03:35:08.246 INFO:tasks.workunit.client.0.smithi112.stdout:removed './154/18' 2023-01-24T03:35:08.246 INFO:tasks.workunit.client.0.smithi112.stdout:removed './154/13' 2023-01-24T03:35:08.247 INFO:tasks.workunit.client.0.smithi112.stdout:removed './154/33' 2023-01-24T03:35:08.247 INFO:tasks.workunit.client.0.smithi112.stdout:removed './154/17' 2023-01-24T03:35:08.247 INFO:tasks.workunit.client.0.smithi112.stdout:removed './154/35' 2023-01-24T03:35:08.247 INFO:tasks.workunit.client.0.smithi112.stdout:removed './154/37' 2023-01-24T03:35:08.247 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './154' 2023-01-24T03:35:08.248 INFO:tasks.workunit.client.0.smithi112.stdout:removed './155/19' 2023-01-24T03:35:08.248 INFO:tasks.workunit.client.0.smithi112.stdout:removed './155/15' 2023-01-24T03:35:08.248 INFO:tasks.workunit.client.0.smithi112.stdout:removed './155/31' 2023-01-24T03:35:08.248 INFO:tasks.workunit.client.0.smithi112.stdout:removed './155/36' 2023-01-24T03:35:08.248 INFO:tasks.workunit.client.0.smithi112.stdout:removed './155/20' 2023-01-24T03:35:08.248 INFO:tasks.workunit.client.0.smithi112.stdout:removed './155/12' 2023-01-24T03:35:08.249 INFO:tasks.workunit.client.0.smithi112.stdout:removed './155/39' 2023-01-24T03:35:08.249 INFO:tasks.workunit.client.0.smithi112.stdout:removed './155/10' 2023-01-24T03:35:08.249 INFO:tasks.workunit.client.0.smithi112.stdout:removed './155/34' 2023-01-24T03:35:08.249 INFO:tasks.workunit.client.0.smithi112.stdout:removed './155/38' 2023-01-24T03:35:08.249 INFO:tasks.workunit.client.0.smithi112.stdout:removed './155/11' 2023-01-24T03:35:08.250 INFO:tasks.workunit.client.0.smithi112.stdout:removed './155/16' 2023-01-24T03:35:08.250 INFO:tasks.workunit.client.0.smithi112.stdout:removed './155/32' 2023-01-24T03:35:08.250 INFO:tasks.workunit.client.0.smithi112.stdout:removed './155/30' 2023-01-24T03:35:08.250 INFO:tasks.workunit.client.0.smithi112.stdout:removed './155/14' 2023-01-24T03:35:08.250 INFO:tasks.workunit.client.0.smithi112.stdout:removed './155/18' 2023-01-24T03:35:08.251 INFO:tasks.workunit.client.0.smithi112.stdout:removed './155/13' 2023-01-24T03:35:08.251 INFO:tasks.workunit.client.0.smithi112.stdout:removed './155/33' 2023-01-24T03:35:08.251 INFO:tasks.workunit.client.0.smithi112.stdout:removed './155/17' 2023-01-24T03:35:08.251 INFO:tasks.workunit.client.0.smithi112.stdout:removed './155/35' 2023-01-24T03:35:08.251 INFO:tasks.workunit.client.0.smithi112.stdout:removed './155/37' 2023-01-24T03:35:08.252 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './155' 2023-01-24T03:35:08.252 INFO:tasks.workunit.client.0.smithi112.stdout:removed './156/19' 2023-01-24T03:35:08.252 INFO:tasks.workunit.client.0.smithi112.stdout:removed './156/15' 2023-01-24T03:35:08.252 INFO:tasks.workunit.client.0.smithi112.stdout:removed './156/31' 2023-01-24T03:35:08.252 INFO:tasks.workunit.client.0.smithi112.stdout:removed './156/36' 2023-01-24T03:35:08.253 INFO:tasks.workunit.client.0.smithi112.stdout:removed './156/20' 2023-01-24T03:35:08.253 INFO:tasks.workunit.client.0.smithi112.stdout:removed './156/12' 2023-01-24T03:35:08.253 INFO:tasks.workunit.client.0.smithi112.stdout:removed './156/39' 2023-01-24T03:35:08.253 INFO:tasks.workunit.client.0.smithi112.stdout:removed './156/10' 2023-01-24T03:35:08.253 INFO:tasks.workunit.client.0.smithi112.stdout:removed './156/34' 2023-01-24T03:35:08.253 INFO:tasks.workunit.client.0.smithi112.stdout:removed './156/38' 2023-01-24T03:35:08.254 INFO:tasks.workunit.client.0.smithi112.stdout:removed './156/11' 2023-01-24T03:35:08.254 INFO:tasks.workunit.client.0.smithi112.stdout:removed './156/16' 2023-01-24T03:35:08.254 INFO:tasks.workunit.client.0.smithi112.stdout:removed './156/32' 2023-01-24T03:35:08.254 INFO:tasks.workunit.client.0.smithi112.stdout:removed './156/30' 2023-01-24T03:35:08.254 INFO:tasks.workunit.client.0.smithi112.stdout:removed './156/14' 2023-01-24T03:35:08.254 INFO:tasks.workunit.client.0.smithi112.stdout:removed './156/18' 2023-01-24T03:35:08.255 INFO:tasks.workunit.client.0.smithi112.stdout:removed './156/13' 2023-01-24T03:35:08.255 INFO:tasks.workunit.client.0.smithi112.stdout:removed './156/33' 2023-01-24T03:35:08.255 INFO:tasks.workunit.client.0.smithi112.stdout:removed './156/17' 2023-01-24T03:35:08.255 INFO:tasks.workunit.client.0.smithi112.stdout:removed './156/35' 2023-01-24T03:35:08.255 INFO:tasks.workunit.client.0.smithi112.stdout:removed './156/37' 2023-01-24T03:35:08.256 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './156' 2023-01-24T03:35:08.256 INFO:tasks.workunit.client.0.smithi112.stdout:removed './157/19' 2023-01-24T03:35:08.256 INFO:tasks.workunit.client.0.smithi112.stdout:removed './157/15' 2023-01-24T03:35:08.256 INFO:tasks.workunit.client.0.smithi112.stdout:removed './157/31' 2023-01-24T03:35:08.256 INFO:tasks.workunit.client.0.smithi112.stdout:removed './157/36' 2023-01-24T03:35:08.256 INFO:tasks.workunit.client.0.smithi112.stdout:removed './157/20' 2023-01-24T03:35:08.257 INFO:tasks.workunit.client.0.smithi112.stdout:removed './157/12' 2023-01-24T03:35:08.257 INFO:tasks.workunit.client.0.smithi112.stdout:removed './157/39' 2023-01-24T03:35:08.257 INFO:tasks.workunit.client.0.smithi112.stdout:removed './157/10' 2023-01-24T03:35:08.257 INFO:tasks.workunit.client.0.smithi112.stdout:removed './157/34' 2023-01-24T03:35:08.257 INFO:tasks.workunit.client.0.smithi112.stdout:removed './157/38' 2023-01-24T03:35:08.258 INFO:tasks.workunit.client.0.smithi112.stdout:removed './157/11' 2023-01-24T03:35:08.258 INFO:tasks.workunit.client.0.smithi112.stdout:removed './157/16' 2023-01-24T03:35:08.258 INFO:tasks.workunit.client.0.smithi112.stdout:removed './157/32' 2023-01-24T03:35:08.258 INFO:tasks.workunit.client.0.smithi112.stdout:removed './157/30' 2023-01-24T03:35:08.258 INFO:tasks.workunit.client.0.smithi112.stdout:removed './157/14' 2023-01-24T03:35:08.259 INFO:tasks.workunit.client.0.smithi112.stdout:removed './157/18' 2023-01-24T03:35:08.259 INFO:tasks.workunit.client.0.smithi112.stdout:removed './157/13' 2023-01-24T03:35:08.259 INFO:tasks.workunit.client.0.smithi112.stdout:removed './157/33' 2023-01-24T03:35:09.298 INFO:tasks.workunit.client.0.smithi112.stdout:removed './157/17' 2023-01-24T03:35:09.298 INFO:tasks.workunit.client.0.smithi112.stdout:removed './157/35' 2023-01-24T03:35:09.298 INFO:tasks.workunit.client.0.smithi112.stdout:removed './157/37' 2023-01-24T03:35:09.299 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './157' 2023-01-24T03:35:09.299 INFO:tasks.workunit.client.0.smithi112.stdout:removed './158/19' 2023-01-24T03:35:09.299 INFO:tasks.workunit.client.0.smithi112.stdout:removed './158/15' 2023-01-24T03:35:09.299 INFO:tasks.workunit.client.0.smithi112.stdout:removed './158/31' 2023-01-24T03:35:09.299 INFO:tasks.workunit.client.0.smithi112.stdout:removed './158/36' 2023-01-24T03:35:09.300 INFO:tasks.workunit.client.0.smithi112.stdout:removed './158/20' 2023-01-24T03:35:09.300 INFO:tasks.workunit.client.0.smithi112.stdout:removed './158/12' 2023-01-24T03:35:09.300 INFO:tasks.workunit.client.0.smithi112.stdout:removed './158/39' 2023-01-24T03:35:09.300 INFO:tasks.workunit.client.0.smithi112.stdout:removed './158/10' 2023-01-24T03:35:09.300 INFO:tasks.workunit.client.0.smithi112.stdout:removed './158/34' 2023-01-24T03:35:09.301 INFO:tasks.workunit.client.0.smithi112.stdout:removed './158/38' 2023-01-24T03:35:09.301 INFO:tasks.workunit.client.0.smithi112.stdout:removed './158/11' 2023-01-24T03:35:09.301 INFO:tasks.workunit.client.0.smithi112.stdout:removed './158/16' 2023-01-24T03:35:09.301 INFO:tasks.workunit.client.0.smithi112.stdout:removed './158/32' 2023-01-24T03:35:09.301 INFO:tasks.workunit.client.0.smithi112.stdout:removed './158/30' 2023-01-24T03:35:09.301 INFO:tasks.workunit.client.0.smithi112.stdout:removed './158/14' 2023-01-24T03:35:09.302 INFO:tasks.workunit.client.0.smithi112.stdout:removed './158/18' 2023-01-24T03:35:09.302 INFO:tasks.workunit.client.0.smithi112.stdout:removed './158/13' 2023-01-24T03:35:09.302 INFO:tasks.workunit.client.0.smithi112.stdout:removed './158/33' 2023-01-24T03:35:09.302 INFO:tasks.workunit.client.0.smithi112.stdout:removed './158/17' 2023-01-24T03:35:09.302 INFO:tasks.workunit.client.0.smithi112.stdout:removed './158/35' 2023-01-24T03:35:09.302 INFO:tasks.workunit.client.0.smithi112.stdout:removed './158/37' 2023-01-24T03:35:09.303 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './158' 2023-01-24T03:35:09.303 INFO:tasks.workunit.client.0.smithi112.stdout:removed './159/19' 2023-01-24T03:35:09.303 INFO:tasks.workunit.client.0.smithi112.stdout:removed './159/15' 2023-01-24T03:35:09.303 INFO:tasks.workunit.client.0.smithi112.stdout:removed './159/31' 2023-01-24T03:35:09.303 INFO:tasks.workunit.client.0.smithi112.stdout:removed './159/36' 2023-01-24T03:35:09.304 INFO:tasks.workunit.client.0.smithi112.stdout:removed './159/20' 2023-01-24T03:35:09.304 INFO:tasks.workunit.client.0.smithi112.stdout:removed './159/12' 2023-01-24T03:35:09.304 INFO:tasks.workunit.client.0.smithi112.stdout:removed './159/39' 2023-01-24T03:35:09.304 INFO:tasks.workunit.client.0.smithi112.stdout:removed './159/10' 2023-01-24T03:35:09.304 INFO:tasks.workunit.client.0.smithi112.stdout:removed './159/34' 2023-01-24T03:35:09.305 INFO:tasks.workunit.client.0.smithi112.stdout:removed './159/38' 2023-01-24T03:35:09.305 INFO:tasks.workunit.client.0.smithi112.stdout:removed './159/11' 2023-01-24T03:35:09.305 INFO:tasks.workunit.client.0.smithi112.stdout:removed './159/16' 2023-01-24T03:35:09.305 INFO:tasks.workunit.client.0.smithi112.stdout:removed './159/32' 2023-01-24T03:35:09.305 INFO:tasks.workunit.client.0.smithi112.stdout:removed './159/30' 2023-01-24T03:35:09.306 INFO:tasks.workunit.client.0.smithi112.stdout:removed './159/14' 2023-01-24T03:35:09.306 INFO:tasks.workunit.client.0.smithi112.stdout:removed './159/18' 2023-01-24T03:35:09.306 INFO:tasks.workunit.client.0.smithi112.stdout:removed './159/13' 2023-01-24T03:35:09.306 INFO:tasks.workunit.client.0.smithi112.stdout:removed './159/33' 2023-01-24T03:35:09.306 INFO:tasks.workunit.client.0.smithi112.stdout:removed './159/17' 2023-01-24T03:35:09.307 INFO:tasks.workunit.client.0.smithi112.stdout:removed './159/35' 2023-01-24T03:35:09.307 INFO:tasks.workunit.client.0.smithi112.stdout:removed './159/37' 2023-01-24T03:35:09.307 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './159' 2023-01-24T03:35:09.307 INFO:tasks.workunit.client.0.smithi112.stdout:removed './160/19' 2023-01-24T03:35:09.307 INFO:tasks.workunit.client.0.smithi112.stdout:removed './160/15' 2023-01-24T03:35:09.308 INFO:tasks.workunit.client.0.smithi112.stdout:removed './160/31' 2023-01-24T03:35:09.308 INFO:tasks.workunit.client.0.smithi112.stdout:removed './160/36' 2023-01-24T03:35:09.308 INFO:tasks.workunit.client.0.smithi112.stdout:removed './160/20' 2023-01-24T03:35:09.308 INFO:tasks.workunit.client.0.smithi112.stdout:removed './160/12' 2023-01-24T03:35:09.308 INFO:tasks.workunit.client.0.smithi112.stdout:removed './160/39' 2023-01-24T03:35:09.309 INFO:tasks.workunit.client.0.smithi112.stdout:removed './160/10' 2023-01-24T03:35:09.309 INFO:tasks.workunit.client.0.smithi112.stdout:removed './160/34' 2023-01-24T03:35:09.309 INFO:tasks.workunit.client.0.smithi112.stdout:removed './160/38' 2023-01-24T03:35:09.309 INFO:tasks.workunit.client.0.smithi112.stdout:removed './160/11' 2023-01-24T03:35:09.309 INFO:tasks.workunit.client.0.smithi112.stdout:removed './160/16' 2023-01-24T03:35:09.310 INFO:tasks.workunit.client.0.smithi112.stdout:removed './160/32' 2023-01-24T03:35:09.310 INFO:tasks.workunit.client.0.smithi112.stdout:removed './160/30' 2023-01-24T03:35:09.310 INFO:tasks.workunit.client.0.smithi112.stdout:removed './160/14' 2023-01-24T03:35:09.310 INFO:tasks.workunit.client.0.smithi112.stdout:removed './160/18' 2023-01-24T03:35:09.310 INFO:tasks.workunit.client.0.smithi112.stdout:removed './160/13' 2023-01-24T03:35:09.310 INFO:tasks.workunit.client.0.smithi112.stdout:removed './160/33' 2023-01-24T03:35:09.311 INFO:tasks.workunit.client.0.smithi112.stdout:removed './160/17' 2023-01-24T03:35:09.311 INFO:tasks.workunit.client.0.smithi112.stdout:removed './160/35' 2023-01-24T03:35:09.311 INFO:tasks.workunit.client.0.smithi112.stdout:removed './160/37' 2023-01-24T03:35:09.311 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './160' 2023-01-24T03:35:09.311 INFO:tasks.workunit.client.0.smithi112.stdout:removed './161/19' 2023-01-24T03:35:09.312 INFO:tasks.workunit.client.0.smithi112.stdout:removed './161/15' 2023-01-24T03:35:09.312 INFO:tasks.workunit.client.0.smithi112.stdout:removed './161/31' 2023-01-24T03:35:09.312 INFO:tasks.workunit.client.0.smithi112.stdout:removed './161/36' 2023-01-24T03:35:09.312 INFO:tasks.workunit.client.0.smithi112.stdout:removed './161/20' 2023-01-24T03:35:09.312 INFO:tasks.workunit.client.0.smithi112.stdout:removed './161/12' 2023-01-24T03:35:09.313 INFO:tasks.workunit.client.0.smithi112.stdout:removed './161/39' 2023-01-24T03:35:09.313 INFO:tasks.workunit.client.0.smithi112.stdout:removed './161/10' 2023-01-24T03:35:09.313 INFO:tasks.workunit.client.0.smithi112.stdout:removed './161/34' 2023-01-24T03:35:09.313 INFO:tasks.workunit.client.0.smithi112.stdout:removed './161/38' 2023-01-24T03:35:09.313 INFO:tasks.workunit.client.0.smithi112.stdout:removed './161/11' 2023-01-24T03:35:09.314 INFO:tasks.workunit.client.0.smithi112.stdout:removed './161/16' 2023-01-24T03:35:09.314 INFO:tasks.workunit.client.0.smithi112.stdout:removed './161/32' 2023-01-24T03:35:09.314 INFO:tasks.workunit.client.0.smithi112.stdout:removed './161/30' 2023-01-24T03:35:09.314 INFO:tasks.workunit.client.0.smithi112.stdout:removed './161/14' 2023-01-24T03:35:09.314 INFO:tasks.workunit.client.0.smithi112.stdout:removed './161/18' 2023-01-24T03:35:09.315 INFO:tasks.workunit.client.0.smithi112.stdout:removed './161/13' 2023-01-24T03:35:09.315 INFO:tasks.workunit.client.0.smithi112.stdout:removed './161/33' 2023-01-24T03:35:09.315 INFO:tasks.workunit.client.0.smithi112.stdout:removed './161/17' 2023-01-24T03:35:09.315 INFO:tasks.workunit.client.0.smithi112.stdout:removed './161/35' 2023-01-24T03:35:09.315 INFO:tasks.workunit.client.0.smithi112.stdout:removed './161/37' 2023-01-24T03:35:09.316 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './161' 2023-01-24T03:35:09.316 INFO:tasks.workunit.client.0.smithi112.stdout:removed './162/19' 2023-01-24T03:35:09.316 INFO:tasks.workunit.client.0.smithi112.stdout:removed './162/15' 2023-01-24T03:35:09.316 INFO:tasks.workunit.client.0.smithi112.stdout:removed './162/31' 2023-01-24T03:35:09.316 INFO:tasks.workunit.client.0.smithi112.stdout:removed './162/36' 2023-01-24T03:35:09.316 INFO:tasks.workunit.client.0.smithi112.stdout:removed './162/20' 2023-01-24T03:35:09.317 INFO:tasks.workunit.client.0.smithi112.stdout:removed './162/12' 2023-01-24T03:35:09.317 INFO:tasks.workunit.client.0.smithi112.stdout:removed './162/39' 2023-01-24T03:35:09.317 INFO:tasks.workunit.client.0.smithi112.stdout:removed './162/10' 2023-01-24T03:35:09.317 INFO:tasks.workunit.client.0.smithi112.stdout:removed './162/34' 2023-01-24T03:35:09.317 INFO:tasks.workunit.client.0.smithi112.stdout:removed './162/38' 2023-01-24T03:35:09.318 INFO:tasks.workunit.client.0.smithi112.stdout:removed './162/11' 2023-01-24T03:35:09.318 INFO:tasks.workunit.client.0.smithi112.stdout:removed './162/16' 2023-01-24T03:35:09.318 INFO:tasks.workunit.client.0.smithi112.stdout:removed './162/32' 2023-01-24T03:35:09.318 INFO:tasks.workunit.client.0.smithi112.stdout:removed './162/30' 2023-01-24T03:35:09.318 INFO:tasks.workunit.client.0.smithi112.stdout:removed './162/14' 2023-01-24T03:35:09.318 INFO:tasks.workunit.client.0.smithi112.stdout:removed './162/18' 2023-01-24T03:35:09.319 INFO:tasks.workunit.client.0.smithi112.stdout:removed './162/13' 2023-01-24T03:35:09.319 INFO:tasks.workunit.client.0.smithi112.stdout:removed './162/33' 2023-01-24T03:35:09.319 INFO:tasks.workunit.client.0.smithi112.stdout:removed './162/17' 2023-01-24T03:35:09.319 INFO:tasks.workunit.client.0.smithi112.stdout:removed './162/35' 2023-01-24T03:35:09.319 INFO:tasks.workunit.client.0.smithi112.stdout:removed './162/37' 2023-01-24T03:35:09.320 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './162' 2023-01-24T03:35:09.320 INFO:tasks.workunit.client.0.smithi112.stdout:removed './163/19' 2023-01-24T03:35:09.320 INFO:tasks.workunit.client.0.smithi112.stdout:removed './163/15' 2023-01-24T03:35:09.320 INFO:tasks.workunit.client.0.smithi112.stdout:removed './163/31' 2023-01-24T03:35:09.320 INFO:tasks.workunit.client.0.smithi112.stdout:removed './163/36' 2023-01-24T03:35:09.320 INFO:tasks.workunit.client.0.smithi112.stdout:removed './163/20' 2023-01-24T03:35:09.321 INFO:tasks.workunit.client.0.smithi112.stdout:removed './163/12' 2023-01-24T03:35:09.321 INFO:tasks.workunit.client.0.smithi112.stdout:removed './163/39' 2023-01-24T03:35:09.321 INFO:tasks.workunit.client.0.smithi112.stdout:removed './163/10' 2023-01-24T03:35:09.321 INFO:tasks.workunit.client.0.smithi112.stdout:removed './163/34' 2023-01-24T03:35:09.321 INFO:tasks.workunit.client.0.smithi112.stdout:removed './163/38' 2023-01-24T03:35:09.322 INFO:tasks.workunit.client.0.smithi112.stdout:removed './163/11' 2023-01-24T03:35:09.322 INFO:tasks.workunit.client.0.smithi112.stdout:removed './163/16' 2023-01-24T03:35:09.322 INFO:tasks.workunit.client.0.smithi112.stdout:removed './163/32' 2023-01-24T03:35:09.322 INFO:tasks.workunit.client.0.smithi112.stdout:removed './163/30' 2023-01-24T03:35:09.322 INFO:tasks.workunit.client.0.smithi112.stdout:removed './163/14' 2023-01-24T03:35:09.322 INFO:tasks.workunit.client.0.smithi112.stdout:removed './163/18' 2023-01-24T03:35:09.323 INFO:tasks.workunit.client.0.smithi112.stdout:removed './163/13' 2023-01-24T03:35:09.323 INFO:tasks.workunit.client.0.smithi112.stdout:removed './163/33' 2023-01-24T03:35:09.323 INFO:tasks.workunit.client.0.smithi112.stdout:removed './163/17' 2023-01-24T03:35:09.323 INFO:tasks.workunit.client.0.smithi112.stdout:removed './163/35' 2023-01-24T03:35:09.323 INFO:tasks.workunit.client.0.smithi112.stdout:removed './163/37' 2023-01-24T03:35:09.324 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './163' 2023-01-24T03:35:09.324 INFO:tasks.workunit.client.0.smithi112.stdout:removed './164/19' 2023-01-24T03:35:09.324 INFO:tasks.workunit.client.0.smithi112.stdout:removed './164/15' 2023-01-24T03:35:09.324 INFO:tasks.workunit.client.0.smithi112.stdout:removed './164/31' 2023-01-24T03:35:09.324 INFO:tasks.workunit.client.0.smithi112.stdout:removed './164/36' 2023-01-24T03:35:09.324 INFO:tasks.workunit.client.0.smithi112.stdout:removed './164/20' 2023-01-24T03:35:09.325 INFO:tasks.workunit.client.0.smithi112.stdout:removed './164/12' 2023-01-24T03:35:09.325 INFO:tasks.workunit.client.0.smithi112.stdout:removed './164/39' 2023-01-24T03:35:09.325 INFO:tasks.workunit.client.0.smithi112.stdout:removed './164/10' 2023-01-24T03:35:09.325 INFO:tasks.workunit.client.0.smithi112.stdout:removed './164/34' 2023-01-24T03:35:09.325 INFO:tasks.workunit.client.0.smithi112.stdout:removed './164/38' 2023-01-24T03:35:09.326 INFO:tasks.workunit.client.0.smithi112.stdout:removed './164/11' 2023-01-24T03:35:09.326 INFO:tasks.workunit.client.0.smithi112.stdout:removed './164/16' 2023-01-24T03:35:09.326 INFO:tasks.workunit.client.0.smithi112.stdout:removed './164/32' 2023-01-24T03:35:09.326 INFO:tasks.workunit.client.0.smithi112.stdout:removed './164/30' 2023-01-24T03:35:09.326 INFO:tasks.workunit.client.0.smithi112.stdout:removed './164/14' 2023-01-24T03:35:09.327 INFO:tasks.workunit.client.0.smithi112.stdout:removed './164/18' 2023-01-24T03:35:09.327 INFO:tasks.workunit.client.0.smithi112.stdout:removed './164/13' 2023-01-24T03:35:09.327 INFO:tasks.workunit.client.0.smithi112.stdout:removed './164/33' 2023-01-24T03:35:09.327 INFO:tasks.workunit.client.0.smithi112.stdout:removed './164/17' 2023-01-24T03:35:09.327 INFO:tasks.workunit.client.0.smithi112.stdout:removed './164/35' 2023-01-24T03:35:09.327 INFO:tasks.workunit.client.0.smithi112.stdout:removed './164/37' 2023-01-24T03:35:09.328 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './164' 2023-01-24T03:35:09.328 INFO:tasks.workunit.client.0.smithi112.stdout:removed './165/19' 2023-01-24T03:35:09.328 INFO:tasks.workunit.client.0.smithi112.stdout:removed './165/15' 2023-01-24T03:35:09.328 INFO:tasks.workunit.client.0.smithi112.stdout:removed './165/31' 2023-01-24T03:35:09.328 INFO:tasks.workunit.client.0.smithi112.stdout:removed './165/36' 2023-01-24T03:35:09.328 INFO:tasks.workunit.client.0.smithi112.stdout:removed './165/20' 2023-01-24T03:35:09.329 INFO:tasks.workunit.client.0.smithi112.stdout:removed './165/12' 2023-01-24T03:35:09.329 INFO:tasks.workunit.client.0.smithi112.stdout:removed './165/39' 2023-01-24T03:35:09.329 INFO:tasks.workunit.client.0.smithi112.stdout:removed './165/10' 2023-01-24T03:35:09.329 INFO:tasks.workunit.client.0.smithi112.stdout:removed './165/34' 2023-01-24T03:35:09.329 INFO:tasks.workunit.client.0.smithi112.stdout:removed './165/38' 2023-01-24T03:35:09.330 INFO:tasks.workunit.client.0.smithi112.stdout:removed './165/11' 2023-01-24T03:35:09.330 INFO:tasks.workunit.client.0.smithi112.stdout:removed './165/16' 2023-01-24T03:35:09.330 INFO:tasks.workunit.client.0.smithi112.stdout:removed './165/32' 2023-01-24T03:35:09.330 INFO:tasks.workunit.client.0.smithi112.stdout:removed './165/30' 2023-01-24T03:35:09.330 INFO:tasks.workunit.client.0.smithi112.stdout:removed './165/14' 2023-01-24T03:35:09.330 INFO:tasks.workunit.client.0.smithi112.stdout:removed './165/18' 2023-01-24T03:35:09.331 INFO:tasks.workunit.client.0.smithi112.stdout:removed './165/13' 2023-01-24T03:35:09.331 INFO:tasks.workunit.client.0.smithi112.stdout:removed './165/33' 2023-01-24T03:35:09.331 INFO:tasks.workunit.client.0.smithi112.stdout:removed './165/17' 2023-01-24T03:35:09.331 INFO:tasks.workunit.client.0.smithi112.stdout:removed './165/35' 2023-01-24T03:35:09.331 INFO:tasks.workunit.client.0.smithi112.stdout:removed './165/37' 2023-01-24T03:35:09.332 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './165' 2023-01-24T03:35:09.332 INFO:tasks.workunit.client.0.smithi112.stdout:removed './166/19' 2023-01-24T03:35:09.332 INFO:tasks.workunit.client.0.smithi112.stdout:removed './166/15' 2023-01-24T03:35:09.332 INFO:tasks.workunit.client.0.smithi112.stdout:removed './166/31' 2023-01-24T03:35:09.333 INFO:tasks.workunit.client.0.smithi112.stdout:removed './166/36' 2023-01-24T03:35:09.333 INFO:tasks.workunit.client.0.smithi112.stdout:removed './166/20' 2023-01-24T03:35:09.333 INFO:tasks.workunit.client.0.smithi112.stdout:removed './166/12' 2023-01-24T03:35:09.333 INFO:tasks.workunit.client.0.smithi112.stdout:removed './166/39' 2023-01-24T03:35:09.333 INFO:tasks.workunit.client.0.smithi112.stdout:removed './166/10' 2023-01-24T03:35:09.333 INFO:tasks.workunit.client.0.smithi112.stdout:removed './166/34' 2023-01-24T03:35:09.334 INFO:tasks.workunit.client.0.smithi112.stdout:removed './166/38' 2023-01-24T03:35:09.334 INFO:tasks.workunit.client.0.smithi112.stdout:removed './166/11' 2023-01-24T03:35:09.334 INFO:tasks.workunit.client.0.smithi112.stdout:removed './166/16' 2023-01-24T03:35:09.334 INFO:tasks.workunit.client.0.smithi112.stdout:removed './166/32' 2023-01-24T03:35:09.334 INFO:tasks.workunit.client.0.smithi112.stdout:removed './166/30' 2023-01-24T03:35:09.335 INFO:tasks.workunit.client.0.smithi112.stdout:removed './166/14' 2023-01-24T03:35:09.335 INFO:tasks.workunit.client.0.smithi112.stdout:removed './166/18' 2023-01-24T03:35:09.335 INFO:tasks.workunit.client.0.smithi112.stdout:removed './166/13' 2023-01-24T03:35:09.335 INFO:tasks.workunit.client.0.smithi112.stdout:removed './166/33' 2023-01-24T03:35:09.335 INFO:tasks.workunit.client.0.smithi112.stdout:removed './166/17' 2023-01-24T03:35:09.335 INFO:tasks.workunit.client.0.smithi112.stdout:removed './166/35' 2023-01-24T03:35:09.336 INFO:tasks.workunit.client.0.smithi112.stdout:removed './166/37' 2023-01-24T03:35:09.336 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './166' 2023-01-24T03:35:09.336 INFO:tasks.workunit.client.0.smithi112.stdout:removed './167/19' 2023-01-24T03:35:09.336 INFO:tasks.workunit.client.0.smithi112.stdout:removed './167/15' 2023-01-24T03:35:09.336 INFO:tasks.workunit.client.0.smithi112.stdout:removed './167/31' 2023-01-24T03:35:09.337 INFO:tasks.workunit.client.0.smithi112.stdout:removed './167/36' 2023-01-24T03:35:09.337 INFO:tasks.workunit.client.0.smithi112.stdout:removed './167/20' 2023-01-24T03:35:09.337 INFO:tasks.workunit.client.0.smithi112.stdout:removed './167/12' 2023-01-24T03:35:09.337 INFO:tasks.workunit.client.0.smithi112.stdout:removed './167/39' 2023-01-24T03:35:09.337 INFO:tasks.workunit.client.0.smithi112.stdout:removed './167/10' 2023-01-24T03:35:09.337 INFO:tasks.workunit.client.0.smithi112.stdout:removed './167/34' 2023-01-24T03:35:09.338 INFO:tasks.workunit.client.0.smithi112.stdout:removed './167/38' 2023-01-24T03:35:10.727 INFO:tasks.workunit.client.0.smithi112.stdout:removed './167/11' 2023-01-24T03:35:10.728 INFO:tasks.workunit.client.0.smithi112.stdout:removed './167/16' 2023-01-24T03:35:10.728 INFO:tasks.workunit.client.0.smithi112.stdout:removed './167/32' 2023-01-24T03:35:10.728 INFO:tasks.workunit.client.0.smithi112.stdout:removed './167/30' 2023-01-24T03:35:10.729 INFO:tasks.workunit.client.0.smithi112.stdout:removed './167/14' 2023-01-24T03:35:10.729 INFO:tasks.workunit.client.0.smithi112.stdout:removed './167/18' 2023-01-24T03:35:10.729 INFO:tasks.workunit.client.0.smithi112.stdout:removed './167/13' 2023-01-24T03:35:10.729 INFO:tasks.workunit.client.0.smithi112.stdout:removed './167/33' 2023-01-24T03:35:10.729 INFO:tasks.workunit.client.0.smithi112.stdout:removed './167/17' 2023-01-24T03:35:10.729 INFO:tasks.workunit.client.0.smithi112.stdout:removed './167/35' 2023-01-24T03:35:10.730 INFO:tasks.workunit.client.0.smithi112.stdout:removed './167/37' 2023-01-24T03:35:10.730 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './167' 2023-01-24T03:35:10.730 INFO:tasks.workunit.client.0.smithi112.stdout:removed './168/19' 2023-01-24T03:35:10.730 INFO:tasks.workunit.client.0.smithi112.stdout:removed './168/15' 2023-01-24T03:35:10.730 INFO:tasks.workunit.client.0.smithi112.stdout:removed './168/31' 2023-01-24T03:35:10.731 INFO:tasks.workunit.client.0.smithi112.stdout:removed './168/36' 2023-01-24T03:35:10.731 INFO:tasks.workunit.client.0.smithi112.stdout:removed './168/20' 2023-01-24T03:35:10.731 INFO:tasks.workunit.client.0.smithi112.stdout:removed './168/12' 2023-01-24T03:35:10.731 INFO:tasks.workunit.client.0.smithi112.stdout:removed './168/39' 2023-01-24T03:35:10.731 INFO:tasks.workunit.client.0.smithi112.stdout:removed './168/10' 2023-01-24T03:35:10.732 INFO:tasks.workunit.client.0.smithi112.stdout:removed './168/34' 2023-01-24T03:35:10.732 INFO:tasks.workunit.client.0.smithi112.stdout:removed './168/38' 2023-01-24T03:35:10.732 INFO:tasks.workunit.client.0.smithi112.stdout:removed './168/11' 2023-01-24T03:35:10.732 INFO:tasks.workunit.client.0.smithi112.stdout:removed './168/16' 2023-01-24T03:35:10.733 INFO:tasks.workunit.client.0.smithi112.stdout:removed './168/32' 2023-01-24T03:35:10.733 INFO:tasks.workunit.client.0.smithi112.stdout:removed './168/30' 2023-01-24T03:35:10.733 INFO:tasks.workunit.client.0.smithi112.stdout:removed './168/14' 2023-01-24T03:35:10.733 INFO:tasks.workunit.client.0.smithi112.stdout:removed './168/18' 2023-01-24T03:35:10.733 INFO:tasks.workunit.client.0.smithi112.stdout:removed './168/13' 2023-01-24T03:35:10.733 INFO:tasks.workunit.client.0.smithi112.stdout:removed './168/33' 2023-01-24T03:35:10.734 INFO:tasks.workunit.client.0.smithi112.stdout:removed './168/17' 2023-01-24T03:35:10.734 INFO:tasks.workunit.client.0.smithi112.stdout:removed './168/35' 2023-01-24T03:35:10.734 INFO:tasks.workunit.client.0.smithi112.stdout:removed './168/37' 2023-01-24T03:35:10.734 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './168' 2023-01-24T03:35:10.735 INFO:tasks.workunit.client.0.smithi112.stdout:removed './169/19' 2023-01-24T03:35:10.735 INFO:tasks.workunit.client.0.smithi112.stdout:removed './169/15' 2023-01-24T03:35:10.735 INFO:tasks.workunit.client.0.smithi112.stdout:removed './169/31' 2023-01-24T03:35:10.735 INFO:tasks.workunit.client.0.smithi112.stdout:removed './169/36' 2023-01-24T03:35:10.735 INFO:tasks.workunit.client.0.smithi112.stdout:removed './169/20' 2023-01-24T03:35:10.736 INFO:tasks.workunit.client.0.smithi112.stdout:removed './169/12' 2023-01-24T03:35:10.736 INFO:tasks.workunit.client.0.smithi112.stdout:removed './169/39' 2023-01-24T03:35:10.736 INFO:tasks.workunit.client.0.smithi112.stdout:removed './169/10' 2023-01-24T03:35:10.736 INFO:tasks.workunit.client.0.smithi112.stdout:removed './169/34' 2023-01-24T03:35:10.736 INFO:tasks.workunit.client.0.smithi112.stdout:removed './169/38' 2023-01-24T03:35:10.737 INFO:tasks.workunit.client.0.smithi112.stdout:removed './169/11' 2023-01-24T03:35:10.737 INFO:tasks.workunit.client.0.smithi112.stdout:removed './169/16' 2023-01-24T03:35:10.737 INFO:tasks.workunit.client.0.smithi112.stdout:removed './169/32' 2023-01-24T03:35:10.737 INFO:tasks.workunit.client.0.smithi112.stdout:removed './169/30' 2023-01-24T03:35:10.737 INFO:tasks.workunit.client.0.smithi112.stdout:removed './169/14' 2023-01-24T03:35:10.738 INFO:tasks.workunit.client.0.smithi112.stdout:removed './169/18' 2023-01-24T03:35:10.738 INFO:tasks.workunit.client.0.smithi112.stdout:removed './169/13' 2023-01-24T03:35:10.738 INFO:tasks.workunit.client.0.smithi112.stdout:removed './169/33' 2023-01-24T03:35:10.738 INFO:tasks.workunit.client.0.smithi112.stdout:removed './169/17' 2023-01-24T03:35:10.738 INFO:tasks.workunit.client.0.smithi112.stdout:removed './169/35' 2023-01-24T03:35:10.739 INFO:tasks.workunit.client.0.smithi112.stdout:removed './169/37' 2023-01-24T03:35:10.739 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './169' 2023-01-24T03:35:10.739 INFO:tasks.workunit.client.0.smithi112.stdout:removed './170/19' 2023-01-24T03:35:10.739 INFO:tasks.workunit.client.0.smithi112.stdout:removed './170/15' 2023-01-24T03:35:10.739 INFO:tasks.workunit.client.0.smithi112.stdout:removed './170/31' 2023-01-24T03:35:10.740 INFO:tasks.workunit.client.0.smithi112.stdout:removed './170/36' 2023-01-24T03:35:10.740 INFO:tasks.workunit.client.0.smithi112.stdout:removed './170/20' 2023-01-24T03:35:10.740 INFO:tasks.workunit.client.0.smithi112.stdout:removed './170/12' 2023-01-24T03:35:10.740 INFO:tasks.workunit.client.0.smithi112.stdout:removed './170/39' 2023-01-24T03:35:10.741 INFO:tasks.workunit.client.0.smithi112.stdout:removed './170/10' 2023-01-24T03:35:10.741 INFO:tasks.workunit.client.0.smithi112.stdout:removed './170/34' 2023-01-24T03:35:10.741 INFO:tasks.workunit.client.0.smithi112.stdout:removed './170/38' 2023-01-24T03:35:10.741 INFO:tasks.workunit.client.0.smithi112.stdout:removed './170/11' 2023-01-24T03:35:10.741 INFO:tasks.workunit.client.0.smithi112.stdout:removed './170/16' 2023-01-24T03:35:10.742 INFO:tasks.workunit.client.0.smithi112.stdout:removed './170/32' 2023-01-24T03:35:10.742 INFO:tasks.workunit.client.0.smithi112.stdout:removed './170/30' 2023-01-24T03:35:10.742 INFO:tasks.workunit.client.0.smithi112.stdout:removed './170/14' 2023-01-24T03:35:10.742 INFO:tasks.workunit.client.0.smithi112.stdout:removed './170/18' 2023-01-24T03:35:10.742 INFO:tasks.workunit.client.0.smithi112.stdout:removed './170/13' 2023-01-24T03:35:10.743 INFO:tasks.workunit.client.0.smithi112.stdout:removed './170/33' 2023-01-24T03:35:10.743 INFO:tasks.workunit.client.0.smithi112.stdout:removed './170/17' 2023-01-24T03:35:10.743 INFO:tasks.workunit.client.0.smithi112.stdout:removed './170/35' 2023-01-24T03:35:10.743 INFO:tasks.workunit.client.0.smithi112.stdout:removed './170/37' 2023-01-24T03:35:10.743 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './170' 2023-01-24T03:35:10.743 INFO:tasks.workunit.client.0.smithi112.stdout:removed './171/19' 2023-01-24T03:35:10.744 INFO:tasks.workunit.client.0.smithi112.stdout:removed './171/15' 2023-01-24T03:35:10.744 INFO:tasks.workunit.client.0.smithi112.stdout:removed './171/31' 2023-01-24T03:35:10.744 INFO:tasks.workunit.client.0.smithi112.stdout:removed './171/36' 2023-01-24T03:35:10.744 INFO:tasks.workunit.client.0.smithi112.stdout:removed './171/20' 2023-01-24T03:35:10.744 INFO:tasks.workunit.client.0.smithi112.stdout:removed './171/12' 2023-01-24T03:35:10.745 INFO:tasks.workunit.client.0.smithi112.stdout:removed './171/39' 2023-01-24T03:35:10.745 INFO:tasks.workunit.client.0.smithi112.stdout:removed './171/10' 2023-01-24T03:35:10.745 INFO:tasks.workunit.client.0.smithi112.stdout:removed './171/34' 2023-01-24T03:35:10.745 INFO:tasks.workunit.client.0.smithi112.stdout:removed './171/38' 2023-01-24T03:35:10.745 INFO:tasks.workunit.client.0.smithi112.stdout:removed './171/11' 2023-01-24T03:35:10.746 INFO:tasks.workunit.client.0.smithi112.stdout:removed './171/16' 2023-01-24T03:35:10.746 INFO:tasks.workunit.client.0.smithi112.stdout:removed './171/32' 2023-01-24T03:35:10.746 INFO:tasks.workunit.client.0.smithi112.stdout:removed './171/30' 2023-01-24T03:35:10.746 INFO:tasks.workunit.client.0.smithi112.stdout:removed './171/14' 2023-01-24T03:35:10.746 INFO:tasks.workunit.client.0.smithi112.stdout:removed './171/18' 2023-01-24T03:35:10.747 INFO:tasks.workunit.client.0.smithi112.stdout:removed './171/13' 2023-01-24T03:35:10.747 INFO:tasks.workunit.client.0.smithi112.stdout:removed './171/33' 2023-01-24T03:35:10.747 INFO:tasks.workunit.client.0.smithi112.stdout:removed './171/17' 2023-01-24T03:35:10.747 INFO:tasks.workunit.client.0.smithi112.stdout:removed './171/35' 2023-01-24T03:35:10.747 INFO:tasks.workunit.client.0.smithi112.stdout:removed './171/37' 2023-01-24T03:35:10.748 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './171' 2023-01-24T03:35:10.748 INFO:tasks.workunit.client.0.smithi112.stdout:removed './172/19' 2023-01-24T03:35:10.748 INFO:tasks.workunit.client.0.smithi112.stdout:removed './172/15' 2023-01-24T03:35:10.748 INFO:tasks.workunit.client.0.smithi112.stdout:removed './172/31' 2023-01-24T03:35:10.748 INFO:tasks.workunit.client.0.smithi112.stdout:removed './172/36' 2023-01-24T03:35:10.749 INFO:tasks.workunit.client.0.smithi112.stdout:removed './172/20' 2023-01-24T03:35:10.749 INFO:tasks.workunit.client.0.smithi112.stdout:removed './172/12' 2023-01-24T03:35:10.749 INFO:tasks.workunit.client.0.smithi112.stdout:removed './172/39' 2023-01-24T03:35:10.749 INFO:tasks.workunit.client.0.smithi112.stdout:removed './172/10' 2023-01-24T03:35:10.749 INFO:tasks.workunit.client.0.smithi112.stdout:removed './172/34' 2023-01-24T03:35:10.750 INFO:tasks.workunit.client.0.smithi112.stdout:removed './172/38' 2023-01-24T03:35:10.750 INFO:tasks.workunit.client.0.smithi112.stdout:removed './172/11' 2023-01-24T03:35:10.750 INFO:tasks.workunit.client.0.smithi112.stdout:removed './172/16' 2023-01-24T03:35:10.750 INFO:tasks.workunit.client.0.smithi112.stdout:removed './172/32' 2023-01-24T03:35:10.750 INFO:tasks.workunit.client.0.smithi112.stdout:removed './172/30' 2023-01-24T03:35:10.750 INFO:tasks.workunit.client.0.smithi112.stdout:removed './172/14' 2023-01-24T03:35:10.751 INFO:tasks.workunit.client.0.smithi112.stdout:removed './172/18' 2023-01-24T03:35:10.751 INFO:tasks.workunit.client.0.smithi112.stdout:removed './172/13' 2023-01-24T03:35:10.751 INFO:tasks.workunit.client.0.smithi112.stdout:removed './172/33' 2023-01-24T03:35:10.751 INFO:tasks.workunit.client.0.smithi112.stdout:removed './172/17' 2023-01-24T03:35:10.751 INFO:tasks.workunit.client.0.smithi112.stdout:removed './172/35' 2023-01-24T03:35:10.752 INFO:tasks.workunit.client.0.smithi112.stdout:removed './172/37' 2023-01-24T03:35:10.752 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './172' 2023-01-24T03:35:10.752 INFO:tasks.workunit.client.0.smithi112.stdout:removed './173/19' 2023-01-24T03:35:10.752 INFO:tasks.workunit.client.0.smithi112.stdout:removed './173/15' 2023-01-24T03:35:10.752 INFO:tasks.workunit.client.0.smithi112.stdout:removed './173/31' 2023-01-24T03:35:10.753 INFO:tasks.workunit.client.0.smithi112.stdout:removed './173/36' 2023-01-24T03:35:10.753 INFO:tasks.workunit.client.0.smithi112.stdout:removed './173/20' 2023-01-24T03:35:10.753 INFO:tasks.workunit.client.0.smithi112.stdout:removed './173/12' 2023-01-24T03:35:10.753 INFO:tasks.workunit.client.0.smithi112.stdout:removed './173/39' 2023-01-24T03:35:10.753 INFO:tasks.workunit.client.0.smithi112.stdout:removed './173/10' 2023-01-24T03:35:10.754 INFO:tasks.workunit.client.0.smithi112.stdout:removed './173/34' 2023-01-24T03:35:10.754 INFO:tasks.workunit.client.0.smithi112.stdout:removed './173/38' 2023-01-24T03:35:10.754 INFO:tasks.workunit.client.0.smithi112.stdout:removed './173/11' 2023-01-24T03:35:10.754 INFO:tasks.workunit.client.0.smithi112.stdout:removed './173/16' 2023-01-24T03:35:10.754 INFO:tasks.workunit.client.0.smithi112.stdout:removed './173/32' 2023-01-24T03:35:10.755 INFO:tasks.workunit.client.0.smithi112.stdout:removed './173/30' 2023-01-24T03:35:10.755 INFO:tasks.workunit.client.0.smithi112.stdout:removed './173/14' 2023-01-24T03:35:10.755 INFO:tasks.workunit.client.0.smithi112.stdout:removed './173/18' 2023-01-24T03:35:10.755 INFO:tasks.workunit.client.0.smithi112.stdout:removed './173/13' 2023-01-24T03:35:10.755 INFO:tasks.workunit.client.0.smithi112.stdout:removed './173/33' 2023-01-24T03:35:10.756 INFO:tasks.workunit.client.0.smithi112.stdout:removed './173/17' 2023-01-24T03:35:10.756 INFO:tasks.workunit.client.0.smithi112.stdout:removed './173/35' 2023-01-24T03:35:10.756 INFO:tasks.workunit.client.0.smithi112.stdout:removed './173/37' 2023-01-24T03:35:10.756 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './173' 2023-01-24T03:35:10.756 INFO:tasks.workunit.client.0.smithi112.stdout:removed './174/19' 2023-01-24T03:35:10.757 INFO:tasks.workunit.client.0.smithi112.stdout:removed './174/15' 2023-01-24T03:35:10.757 INFO:tasks.workunit.client.0.smithi112.stdout:removed './174/31' 2023-01-24T03:35:10.757 INFO:tasks.workunit.client.0.smithi112.stdout:removed './174/36' 2023-01-24T03:35:10.757 INFO:tasks.workunit.client.0.smithi112.stdout:removed './174/20' 2023-01-24T03:35:10.757 INFO:tasks.workunit.client.0.smithi112.stdout:removed './174/12' 2023-01-24T03:35:10.757 INFO:tasks.workunit.client.0.smithi112.stdout:removed './174/39' 2023-01-24T03:35:10.758 INFO:tasks.workunit.client.0.smithi112.stdout:removed './174/10' 2023-01-24T03:35:10.758 INFO:tasks.workunit.client.0.smithi112.stdout:removed './174/34' 2023-01-24T03:35:10.758 INFO:tasks.workunit.client.0.smithi112.stdout:removed './174/38' 2023-01-24T03:35:10.758 INFO:tasks.workunit.client.0.smithi112.stdout:removed './174/11' 2023-01-24T03:35:10.758 INFO:tasks.workunit.client.0.smithi112.stdout:removed './174/16' 2023-01-24T03:35:10.759 INFO:tasks.workunit.client.0.smithi112.stdout:removed './174/32' 2023-01-24T03:35:10.759 INFO:tasks.workunit.client.0.smithi112.stdout:removed './174/30' 2023-01-24T03:35:10.759 INFO:tasks.workunit.client.0.smithi112.stdout:removed './174/14' 2023-01-24T03:35:10.759 INFO:tasks.workunit.client.0.smithi112.stdout:removed './174/18' 2023-01-24T03:35:10.759 INFO:tasks.workunit.client.0.smithi112.stdout:removed './174/13' 2023-01-24T03:35:10.760 INFO:tasks.workunit.client.0.smithi112.stdout:removed './174/33' 2023-01-24T03:35:10.760 INFO:tasks.workunit.client.0.smithi112.stdout:removed './174/17' 2023-01-24T03:35:10.760 INFO:tasks.workunit.client.0.smithi112.stdout:removed './174/35' 2023-01-24T03:35:10.760 INFO:tasks.workunit.client.0.smithi112.stdout:removed './174/37' 2023-01-24T03:35:10.760 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './174' 2023-01-24T03:35:10.761 INFO:tasks.workunit.client.0.smithi112.stdout:removed './175/19' 2023-01-24T03:35:10.761 INFO:tasks.workunit.client.0.smithi112.stdout:removed './175/15' 2023-01-24T03:35:10.761 INFO:tasks.workunit.client.0.smithi112.stdout:removed './175/31' 2023-01-24T03:35:10.761 INFO:tasks.workunit.client.0.smithi112.stdout:removed './175/36' 2023-01-24T03:35:10.761 INFO:tasks.workunit.client.0.smithi112.stdout:removed './175/20' 2023-01-24T03:35:10.762 INFO:tasks.workunit.client.0.smithi112.stdout:removed './175/12' 2023-01-24T03:35:10.762 INFO:tasks.workunit.client.0.smithi112.stdout:removed './175/39' 2023-01-24T03:35:10.762 INFO:tasks.workunit.client.0.smithi112.stdout:removed './175/10' 2023-01-24T03:35:10.762 INFO:tasks.workunit.client.0.smithi112.stdout:removed './175/34' 2023-01-24T03:35:10.763 INFO:tasks.workunit.client.0.smithi112.stdout:removed './175/38' 2023-01-24T03:35:10.763 INFO:tasks.workunit.client.0.smithi112.stdout:removed './175/11' 2023-01-24T03:35:10.763 INFO:tasks.workunit.client.0.smithi112.stdout:removed './175/16' 2023-01-24T03:35:10.763 INFO:tasks.workunit.client.0.smithi112.stdout:removed './175/32' 2023-01-24T03:35:10.763 INFO:tasks.workunit.client.0.smithi112.stdout:removed './175/30' 2023-01-24T03:35:10.763 INFO:tasks.workunit.client.0.smithi112.stdout:removed './175/14' 2023-01-24T03:35:10.764 INFO:tasks.workunit.client.0.smithi112.stdout:removed './175/18' 2023-01-24T03:35:10.764 INFO:tasks.workunit.client.0.smithi112.stdout:removed './175/13' 2023-01-24T03:35:10.764 INFO:tasks.workunit.client.0.smithi112.stdout:removed './175/33' 2023-01-24T03:35:10.764 INFO:tasks.workunit.client.0.smithi112.stdout:removed './175/17' 2023-01-24T03:35:10.764 INFO:tasks.workunit.client.0.smithi112.stdout:removed './175/35' 2023-01-24T03:35:10.765 INFO:tasks.workunit.client.0.smithi112.stdout:removed './175/37' 2023-01-24T03:35:10.765 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './175' 2023-01-24T03:35:10.765 INFO:tasks.workunit.client.0.smithi112.stdout:removed './176/19' 2023-01-24T03:35:10.765 INFO:tasks.workunit.client.0.smithi112.stdout:removed './176/15' 2023-01-24T03:35:10.765 INFO:tasks.workunit.client.0.smithi112.stdout:removed './176/31' 2023-01-24T03:35:10.766 INFO:tasks.workunit.client.0.smithi112.stdout:removed './176/36' 2023-01-24T03:35:10.766 INFO:tasks.workunit.client.0.smithi112.stdout:removed './176/20' 2023-01-24T03:35:10.766 INFO:tasks.workunit.client.0.smithi112.stdout:removed './176/12' 2023-01-24T03:35:10.766 INFO:tasks.workunit.client.0.smithi112.stdout:removed './176/39' 2023-01-24T03:35:10.766 INFO:tasks.workunit.client.0.smithi112.stdout:removed './176/10' 2023-01-24T03:35:10.767 INFO:tasks.workunit.client.0.smithi112.stdout:removed './176/34' 2023-01-24T03:35:10.767 INFO:tasks.workunit.client.0.smithi112.stdout:removed './176/38' 2023-01-24T03:35:10.767 INFO:tasks.workunit.client.0.smithi112.stdout:removed './176/11' 2023-01-24T03:35:10.767 INFO:tasks.workunit.client.0.smithi112.stdout:removed './176/16' 2023-01-24T03:35:10.767 INFO:tasks.workunit.client.0.smithi112.stdout:removed './176/32' 2023-01-24T03:35:10.768 INFO:tasks.workunit.client.0.smithi112.stdout:removed './176/30' 2023-01-24T03:35:10.768 INFO:tasks.workunit.client.0.smithi112.stdout:removed './176/14' 2023-01-24T03:35:10.768 INFO:tasks.workunit.client.0.smithi112.stdout:removed './176/18' 2023-01-24T03:35:10.768 INFO:tasks.workunit.client.0.smithi112.stdout:removed './176/13' 2023-01-24T03:35:10.768 INFO:tasks.workunit.client.0.smithi112.stdout:removed './176/33' 2023-01-24T03:35:10.769 INFO:tasks.workunit.client.0.smithi112.stdout:removed './176/17' 2023-01-24T03:35:10.769 INFO:tasks.workunit.client.0.smithi112.stdout:removed './176/35' 2023-01-24T03:35:10.769 INFO:tasks.workunit.client.0.smithi112.stdout:removed './176/37' 2023-01-24T03:35:10.769 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './176' 2023-01-24T03:35:10.769 INFO:tasks.workunit.client.0.smithi112.stdout:removed './177/19' 2023-01-24T03:35:10.770 INFO:tasks.workunit.client.0.smithi112.stdout:removed './177/15' 2023-01-24T03:35:11.417 INFO:tasks.workunit.client.0.smithi112.stdout:removed './177/31' 2023-01-24T03:35:11.418 INFO:tasks.workunit.client.0.smithi112.stdout:removed './177/36' 2023-01-24T03:35:11.418 INFO:tasks.workunit.client.0.smithi112.stdout:removed './177/20' 2023-01-24T03:35:11.419 INFO:tasks.workunit.client.0.smithi112.stdout:removed './177/12' 2023-01-24T03:35:11.419 INFO:tasks.workunit.client.0.smithi112.stdout:removed './177/39' 2023-01-24T03:35:11.419 INFO:tasks.workunit.client.0.smithi112.stdout:removed './177/10' 2023-01-24T03:35:11.419 INFO:tasks.workunit.client.0.smithi112.stdout:removed './177/34' 2023-01-24T03:35:11.419 INFO:tasks.workunit.client.0.smithi112.stdout:removed './177/38' 2023-01-24T03:35:11.420 INFO:tasks.workunit.client.0.smithi112.stdout:removed './177/11' 2023-01-24T03:35:11.420 INFO:tasks.workunit.client.0.smithi112.stdout:removed './177/16' 2023-01-24T03:35:11.420 INFO:tasks.workunit.client.0.smithi112.stdout:removed './177/32' 2023-01-24T03:35:11.420 INFO:tasks.workunit.client.0.smithi112.stdout:removed './177/30' 2023-01-24T03:35:11.420 INFO:tasks.workunit.client.0.smithi112.stdout:removed './177/14' 2023-01-24T03:35:11.421 INFO:tasks.workunit.client.0.smithi112.stdout:removed './177/18' 2023-01-24T03:35:11.421 INFO:tasks.workunit.client.0.smithi112.stdout:removed './177/13' 2023-01-24T03:35:11.421 INFO:tasks.workunit.client.0.smithi112.stdout:removed './177/33' 2023-01-24T03:35:11.421 INFO:tasks.workunit.client.0.smithi112.stdout:removed './177/17' 2023-01-24T03:35:11.421 INFO:tasks.workunit.client.0.smithi112.stdout:removed './177/35' 2023-01-24T03:35:11.422 INFO:tasks.workunit.client.0.smithi112.stdout:removed './177/37' 2023-01-24T03:35:11.422 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './177' 2023-01-24T03:35:11.422 INFO:tasks.workunit.client.0.smithi112.stdout:removed './178/19' 2023-01-24T03:35:11.422 INFO:tasks.workunit.client.0.smithi112.stdout:removed './178/15' 2023-01-24T03:35:11.422 INFO:tasks.workunit.client.0.smithi112.stdout:removed './178/31' 2023-01-24T03:35:11.423 INFO:tasks.workunit.client.0.smithi112.stdout:removed './178/36' 2023-01-24T03:35:11.423 INFO:tasks.workunit.client.0.smithi112.stdout:removed './178/20' 2023-01-24T03:35:11.423 INFO:tasks.workunit.client.0.smithi112.stdout:removed './178/12' 2023-01-24T03:35:11.423 INFO:tasks.workunit.client.0.smithi112.stdout:removed './178/39' 2023-01-24T03:35:11.424 INFO:tasks.workunit.client.0.smithi112.stdout:removed './178/10' 2023-01-24T03:35:11.424 INFO:tasks.workunit.client.0.smithi112.stdout:removed './178/34' 2023-01-24T03:35:11.424 INFO:tasks.workunit.client.0.smithi112.stdout:removed './178/38' 2023-01-24T03:35:11.424 INFO:tasks.workunit.client.0.smithi112.stdout:removed './178/11' 2023-01-24T03:35:11.424 INFO:tasks.workunit.client.0.smithi112.stdout:removed './178/16' 2023-01-24T03:35:11.425 INFO:tasks.workunit.client.0.smithi112.stdout:removed './178/32' 2023-01-24T03:35:11.425 INFO:tasks.workunit.client.0.smithi112.stdout:removed './178/30' 2023-01-24T03:35:11.425 INFO:tasks.workunit.client.0.smithi112.stdout:removed './178/14' 2023-01-24T03:35:11.425 INFO:tasks.workunit.client.0.smithi112.stdout:removed './178/18' 2023-01-24T03:35:11.425 INFO:tasks.workunit.client.0.smithi112.stdout:removed './178/13' 2023-01-24T03:35:11.426 INFO:tasks.workunit.client.0.smithi112.stdout:removed './178/33' 2023-01-24T03:35:11.426 INFO:tasks.workunit.client.0.smithi112.stdout:removed './178/17' 2023-01-24T03:35:11.426 INFO:tasks.workunit.client.0.smithi112.stdout:removed './178/35' 2023-01-24T03:35:11.426 INFO:tasks.workunit.client.0.smithi112.stdout:removed './178/37' 2023-01-24T03:35:11.426 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './178' 2023-01-24T03:35:11.427 INFO:tasks.workunit.client.0.smithi112.stdout:removed './179/19' 2023-01-24T03:35:11.427 INFO:tasks.workunit.client.0.smithi112.stdout:removed './179/15' 2023-01-24T03:35:11.427 INFO:tasks.workunit.client.0.smithi112.stdout:removed './179/31' 2023-01-24T03:35:11.427 INFO:tasks.workunit.client.0.smithi112.stdout:removed './179/36' 2023-01-24T03:35:11.427 INFO:tasks.workunit.client.0.smithi112.stdout:removed './179/20' 2023-01-24T03:35:11.428 INFO:tasks.workunit.client.0.smithi112.stdout:removed './179/12' 2023-01-24T03:35:11.428 INFO:tasks.workunit.client.0.smithi112.stdout:removed './179/39' 2023-01-24T03:35:11.428 INFO:tasks.workunit.client.0.smithi112.stdout:removed './179/10' 2023-01-24T03:35:11.428 INFO:tasks.workunit.client.0.smithi112.stdout:removed './179/34' 2023-01-24T03:35:11.428 INFO:tasks.workunit.client.0.smithi112.stdout:removed './179/38' 2023-01-24T03:35:11.428 INFO:tasks.workunit.client.0.smithi112.stdout:removed './179/11' 2023-01-24T03:35:11.429 INFO:tasks.workunit.client.0.smithi112.stdout:removed './179/16' 2023-01-24T03:35:11.429 INFO:tasks.workunit.client.0.smithi112.stdout:removed './179/32' 2023-01-24T03:35:11.429 INFO:tasks.workunit.client.0.smithi112.stdout:removed './179/30' 2023-01-24T03:35:11.429 INFO:tasks.workunit.client.0.smithi112.stdout:removed './179/14' 2023-01-24T03:35:11.429 INFO:tasks.workunit.client.0.smithi112.stdout:removed './179/18' 2023-01-24T03:35:11.430 INFO:tasks.workunit.client.0.smithi112.stdout:removed './179/13' 2023-01-24T03:35:11.430 INFO:tasks.workunit.client.0.smithi112.stdout:removed './179/33' 2023-01-24T03:35:11.430 INFO:tasks.workunit.client.0.smithi112.stdout:removed './179/17' 2023-01-24T03:35:11.430 INFO:tasks.workunit.client.0.smithi112.stdout:removed './179/35' 2023-01-24T03:35:11.430 INFO:tasks.workunit.client.0.smithi112.stdout:removed './179/37' 2023-01-24T03:35:11.431 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './179' 2023-01-24T03:35:11.431 INFO:tasks.workunit.client.0.smithi112.stdout:removed './180/19' 2023-01-24T03:35:11.431 INFO:tasks.workunit.client.0.smithi112.stdout:removed './180/15' 2023-01-24T03:35:11.431 INFO:tasks.workunit.client.0.smithi112.stdout:removed './180/31' 2023-01-24T03:35:11.431 INFO:tasks.workunit.client.0.smithi112.stdout:removed './180/36' 2023-01-24T03:35:11.431 INFO:tasks.workunit.client.0.smithi112.stdout:removed './180/20' 2023-01-24T03:35:11.432 INFO:tasks.workunit.client.0.smithi112.stdout:removed './180/12' 2023-01-24T03:35:11.432 INFO:tasks.workunit.client.0.smithi112.stdout:removed './180/39' 2023-01-24T03:35:11.432 INFO:tasks.workunit.client.0.smithi112.stdout:removed './180/10' 2023-01-24T03:35:11.432 INFO:tasks.workunit.client.0.smithi112.stdout:removed './180/34' 2023-01-24T03:35:11.432 INFO:tasks.workunit.client.0.smithi112.stdout:removed './180/38' 2023-01-24T03:35:11.433 INFO:tasks.workunit.client.0.smithi112.stdout:removed './180/11' 2023-01-24T03:35:11.433 INFO:tasks.workunit.client.0.smithi112.stdout:removed './180/16' 2023-01-24T03:35:11.433 INFO:tasks.workunit.client.0.smithi112.stdout:removed './180/32' 2023-01-24T03:35:11.433 INFO:tasks.workunit.client.0.smithi112.stdout:removed './180/30' 2023-01-24T03:35:11.433 INFO:tasks.workunit.client.0.smithi112.stdout:removed './180/14' 2023-01-24T03:35:11.433 INFO:tasks.workunit.client.0.smithi112.stdout:removed './180/18' 2023-01-24T03:35:11.434 INFO:tasks.workunit.client.0.smithi112.stdout:removed './180/13' 2023-01-24T03:35:11.434 INFO:tasks.workunit.client.0.smithi112.stdout:removed './180/33' 2023-01-24T03:35:11.434 INFO:tasks.workunit.client.0.smithi112.stdout:removed './180/17' 2023-01-24T03:35:11.434 INFO:tasks.workunit.client.0.smithi112.stdout:removed './180/35' 2023-01-24T03:35:11.434 INFO:tasks.workunit.client.0.smithi112.stdout:removed './180/37' 2023-01-24T03:35:11.435 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './180' 2023-01-24T03:35:11.435 INFO:tasks.workunit.client.0.smithi112.stdout:removed './181/19' 2023-01-24T03:35:11.435 INFO:tasks.workunit.client.0.smithi112.stdout:removed './181/15' 2023-01-24T03:35:11.435 INFO:tasks.workunit.client.0.smithi112.stdout:removed './181/31' 2023-01-24T03:35:11.435 INFO:tasks.workunit.client.0.smithi112.stdout:removed './181/36' 2023-01-24T03:35:11.435 INFO:tasks.workunit.client.0.smithi112.stdout:removed './181/20' 2023-01-24T03:35:11.436 INFO:tasks.workunit.client.0.smithi112.stdout:removed './181/12' 2023-01-24T03:35:11.436 INFO:tasks.workunit.client.0.smithi112.stdout:removed './181/39' 2023-01-24T03:35:11.436 INFO:tasks.workunit.client.0.smithi112.stdout:removed './181/10' 2023-01-24T03:35:11.436 INFO:tasks.workunit.client.0.smithi112.stdout:removed './181/34' 2023-01-24T03:35:11.436 INFO:tasks.workunit.client.0.smithi112.stdout:removed './181/38' 2023-01-24T03:35:11.437 INFO:tasks.workunit.client.0.smithi112.stdout:removed './181/11' 2023-01-24T03:35:11.437 INFO:tasks.workunit.client.0.smithi112.stdout:removed './181/16' 2023-01-24T03:35:11.437 INFO:tasks.workunit.client.0.smithi112.stdout:removed './181/32' 2023-01-24T03:35:11.437 INFO:tasks.workunit.client.0.smithi112.stdout:removed './181/30' 2023-01-24T03:35:11.437 INFO:tasks.workunit.client.0.smithi112.stdout:removed './181/14' 2023-01-24T03:35:11.437 INFO:tasks.workunit.client.0.smithi112.stdout:removed './181/18' 2023-01-24T03:35:11.438 INFO:tasks.workunit.client.0.smithi112.stdout:removed './181/13' 2023-01-24T03:35:11.438 INFO:tasks.workunit.client.0.smithi112.stdout:removed './181/33' 2023-01-24T03:35:11.438 INFO:tasks.workunit.client.0.smithi112.stdout:removed './181/17' 2023-01-24T03:35:11.438 INFO:tasks.workunit.client.0.smithi112.stdout:removed './181/35' 2023-01-24T03:35:11.438 INFO:tasks.workunit.client.0.smithi112.stdout:removed './181/37' 2023-01-24T03:35:11.439 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './181' 2023-01-24T03:35:11.439 INFO:tasks.workunit.client.0.smithi112.stdout:removed './182/19' 2023-01-24T03:35:11.439 INFO:tasks.workunit.client.0.smithi112.stdout:removed './182/15' 2023-01-24T03:35:11.439 INFO:tasks.workunit.client.0.smithi112.stdout:removed './182/31' 2023-01-24T03:35:11.439 INFO:tasks.workunit.client.0.smithi112.stdout:removed './182/36' 2023-01-24T03:35:11.440 INFO:tasks.workunit.client.0.smithi112.stdout:removed './182/20' 2023-01-24T03:35:11.440 INFO:tasks.workunit.client.0.smithi112.stdout:removed './182/12' 2023-01-24T03:35:11.440 INFO:tasks.workunit.client.0.smithi112.stdout:removed './182/39' 2023-01-24T03:35:11.440 INFO:tasks.workunit.client.0.smithi112.stdout:removed './182/10' 2023-01-24T03:35:11.440 INFO:tasks.workunit.client.0.smithi112.stdout:removed './182/34' 2023-01-24T03:35:11.441 INFO:tasks.workunit.client.0.smithi112.stdout:removed './182/38' 2023-01-24T03:35:11.441 INFO:tasks.workunit.client.0.smithi112.stdout:removed './182/11' 2023-01-24T03:35:11.441 INFO:tasks.workunit.client.0.smithi112.stdout:removed './182/16' 2023-01-24T03:35:11.441 INFO:tasks.workunit.client.0.smithi112.stdout:removed './182/32' 2023-01-24T03:35:11.441 INFO:tasks.workunit.client.0.smithi112.stdout:removed './182/30' 2023-01-24T03:35:11.441 INFO:tasks.workunit.client.0.smithi112.stdout:removed './182/14' 2023-01-24T03:35:11.442 INFO:tasks.workunit.client.0.smithi112.stdout:removed './182/18' 2023-01-24T03:35:11.442 INFO:tasks.workunit.client.0.smithi112.stdout:removed './182/13' 2023-01-24T03:35:11.442 INFO:tasks.workunit.client.0.smithi112.stdout:removed './182/33' 2023-01-24T03:35:11.442 INFO:tasks.workunit.client.0.smithi112.stdout:removed './182/17' 2023-01-24T03:35:11.442 INFO:tasks.workunit.client.0.smithi112.stdout:removed './182/35' 2023-01-24T03:35:11.443 INFO:tasks.workunit.client.0.smithi112.stdout:removed './182/37' 2023-01-24T03:35:11.443 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './182' 2023-01-24T03:35:11.443 INFO:tasks.workunit.client.0.smithi112.stdout:removed './183/19' 2023-01-24T03:35:11.443 INFO:tasks.workunit.client.0.smithi112.stdout:removed './183/15' 2023-01-24T03:35:11.443 INFO:tasks.workunit.client.0.smithi112.stdout:removed './183/31' 2023-01-24T03:35:11.443 INFO:tasks.workunit.client.0.smithi112.stdout:removed './183/36' 2023-01-24T03:35:11.444 INFO:tasks.workunit.client.0.smithi112.stdout:removed './183/20' 2023-01-24T03:35:11.444 INFO:tasks.workunit.client.0.smithi112.stdout:removed './183/12' 2023-01-24T03:35:11.444 INFO:tasks.workunit.client.0.smithi112.stdout:removed './183/39' 2023-01-24T03:35:11.444 INFO:tasks.workunit.client.0.smithi112.stdout:removed './183/10' 2023-01-24T03:35:11.444 INFO:tasks.workunit.client.0.smithi112.stdout:removed './183/34' 2023-01-24T03:35:11.445 INFO:tasks.workunit.client.0.smithi112.stdout:removed './183/38' 2023-01-24T03:35:11.445 INFO:tasks.workunit.client.0.smithi112.stdout:removed './183/11' 2023-01-24T03:35:11.445 INFO:tasks.workunit.client.0.smithi112.stdout:removed './183/16' 2023-01-24T03:35:11.445 INFO:tasks.workunit.client.0.smithi112.stdout:removed './183/32' 2023-01-24T03:35:11.445 INFO:tasks.workunit.client.0.smithi112.stdout:removed './183/30' 2023-01-24T03:35:11.446 INFO:tasks.workunit.client.0.smithi112.stdout:removed './183/14' 2023-01-24T03:35:11.446 INFO:tasks.workunit.client.0.smithi112.stdout:removed './183/18' 2023-01-24T03:35:11.446 INFO:tasks.workunit.client.0.smithi112.stdout:removed './183/13' 2023-01-24T03:35:11.446 INFO:tasks.workunit.client.0.smithi112.stdout:removed './183/33' 2023-01-24T03:35:11.446 INFO:tasks.workunit.client.0.smithi112.stdout:removed './183/17' 2023-01-24T03:35:11.446 INFO:tasks.workunit.client.0.smithi112.stdout:removed './183/35' 2023-01-24T03:35:11.447 INFO:tasks.workunit.client.0.smithi112.stdout:removed './183/37' 2023-01-24T03:35:11.447 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './183' 2023-01-24T03:35:11.447 INFO:tasks.workunit.client.0.smithi112.stdout:removed './184/19' 2023-01-24T03:35:11.447 INFO:tasks.workunit.client.0.smithi112.stdout:removed './184/15' 2023-01-24T03:35:11.447 INFO:tasks.workunit.client.0.smithi112.stdout:removed './184/31' 2023-01-24T03:35:11.448 INFO:tasks.workunit.client.0.smithi112.stdout:removed './184/36' 2023-01-24T03:35:11.448 INFO:tasks.workunit.client.0.smithi112.stdout:removed './184/20' 2023-01-24T03:35:11.448 INFO:tasks.workunit.client.0.smithi112.stdout:removed './184/12' 2023-01-24T03:35:11.448 INFO:tasks.workunit.client.0.smithi112.stdout:removed './184/39' 2023-01-24T03:35:11.448 INFO:tasks.workunit.client.0.smithi112.stdout:removed './184/10' 2023-01-24T03:35:11.449 INFO:tasks.workunit.client.0.smithi112.stdout:removed './184/34' 2023-01-24T03:35:11.449 INFO:tasks.workunit.client.0.smithi112.stdout:removed './184/38' 2023-01-24T03:35:11.449 INFO:tasks.workunit.client.0.smithi112.stdout:removed './184/11' 2023-01-24T03:35:11.449 INFO:tasks.workunit.client.0.smithi112.stdout:removed './184/16' 2023-01-24T03:35:11.450 INFO:tasks.workunit.client.0.smithi112.stdout:removed './184/32' 2023-01-24T03:35:11.450 INFO:tasks.workunit.client.0.smithi112.stdout:removed './184/30' 2023-01-24T03:35:11.450 INFO:tasks.workunit.client.0.smithi112.stdout:removed './184/14' 2023-01-24T03:35:11.450 INFO:tasks.workunit.client.0.smithi112.stdout:removed './184/18' 2023-01-24T03:35:11.450 INFO:tasks.workunit.client.0.smithi112.stdout:removed './184/13' 2023-01-24T03:35:11.450 INFO:tasks.workunit.client.0.smithi112.stdout:removed './184/33' 2023-01-24T03:35:11.451 INFO:tasks.workunit.client.0.smithi112.stdout:removed './184/17' 2023-01-24T03:35:11.451 INFO:tasks.workunit.client.0.smithi112.stdout:removed './184/35' 2023-01-24T03:35:11.451 INFO:tasks.workunit.client.0.smithi112.stdout:removed './184/37' 2023-01-24T03:35:11.451 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './184' 2023-01-24T03:35:11.451 INFO:tasks.workunit.client.0.smithi112.stdout:removed './185/19' 2023-01-24T03:35:11.451 INFO:tasks.workunit.client.0.smithi112.stdout:removed './185/15' 2023-01-24T03:35:11.452 INFO:tasks.workunit.client.0.smithi112.stdout:removed './185/31' 2023-01-24T03:35:11.452 INFO:tasks.workunit.client.0.smithi112.stdout:removed './185/36' 2023-01-24T03:35:11.452 INFO:tasks.workunit.client.0.smithi112.stdout:removed './185/20' 2023-01-24T03:35:11.452 INFO:tasks.workunit.client.0.smithi112.stdout:removed './185/12' 2023-01-24T03:35:11.452 INFO:tasks.workunit.client.0.smithi112.stdout:removed './185/39' 2023-01-24T03:35:11.453 INFO:tasks.workunit.client.0.smithi112.stdout:removed './185/10' 2023-01-24T03:35:11.453 INFO:tasks.workunit.client.0.smithi112.stdout:removed './185/34' 2023-01-24T03:35:11.453 INFO:tasks.workunit.client.0.smithi112.stdout:removed './185/38' 2023-01-24T03:35:11.453 INFO:tasks.workunit.client.0.smithi112.stdout:removed './185/11' 2023-01-24T03:35:11.453 INFO:tasks.workunit.client.0.smithi112.stdout:removed './185/16' 2023-01-24T03:35:11.453 INFO:tasks.workunit.client.0.smithi112.stdout:removed './185/32' 2023-01-24T03:35:11.454 INFO:tasks.workunit.client.0.smithi112.stdout:removed './185/30' 2023-01-24T03:35:11.454 INFO:tasks.workunit.client.0.smithi112.stdout:removed './185/14' 2023-01-24T03:35:11.454 INFO:tasks.workunit.client.0.smithi112.stdout:removed './185/18' 2023-01-24T03:35:11.454 INFO:tasks.workunit.client.0.smithi112.stdout:removed './185/13' 2023-01-24T03:35:11.454 INFO:tasks.workunit.client.0.smithi112.stdout:removed './185/33' 2023-01-24T03:35:11.454 INFO:tasks.workunit.client.0.smithi112.stdout:removed './185/17' 2023-01-24T03:35:11.455 INFO:tasks.workunit.client.0.smithi112.stdout:removed './185/35' 2023-01-24T03:35:11.455 INFO:tasks.workunit.client.0.smithi112.stdout:removed './185/37' 2023-01-24T03:35:11.455 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './185' 2023-01-24T03:35:11.455 INFO:tasks.workunit.client.0.smithi112.stdout:removed './186/19' 2023-01-24T03:35:11.455 INFO:tasks.workunit.client.0.smithi112.stdout:removed './186/15' 2023-01-24T03:35:11.456 INFO:tasks.workunit.client.0.smithi112.stdout:removed './186/31' 2023-01-24T03:35:11.456 INFO:tasks.workunit.client.0.smithi112.stdout:removed './186/36' 2023-01-24T03:35:11.456 INFO:tasks.workunit.client.0.smithi112.stdout:removed './186/20' 2023-01-24T03:35:11.456 INFO:tasks.workunit.client.0.smithi112.stdout:removed './186/12' 2023-01-24T03:35:11.456 INFO:tasks.workunit.client.0.smithi112.stdout:removed './186/39' 2023-01-24T03:35:11.456 INFO:tasks.workunit.client.0.smithi112.stdout:removed './186/10' 2023-01-24T03:35:11.457 INFO:tasks.workunit.client.0.smithi112.stdout:removed './186/34' 2023-01-24T03:35:11.457 INFO:tasks.workunit.client.0.smithi112.stdout:removed './186/38' 2023-01-24T03:35:11.457 INFO:tasks.workunit.client.0.smithi112.stdout:removed './186/11' 2023-01-24T03:35:11.457 INFO:tasks.workunit.client.0.smithi112.stdout:removed './186/16' 2023-01-24T03:35:11.457 INFO:tasks.workunit.client.0.smithi112.stdout:removed './186/32' 2023-01-24T03:35:11.457 INFO:tasks.workunit.client.0.smithi112.stdout:removed './186/30' 2023-01-24T03:35:11.458 INFO:tasks.workunit.client.0.smithi112.stdout:removed './186/14' 2023-01-24T03:35:11.458 INFO:tasks.workunit.client.0.smithi112.stdout:removed './186/18' 2023-01-24T03:35:14.308 INFO:tasks.workunit.client.0.smithi112.stdout:removed './186/13' 2023-01-24T03:35:14.309 INFO:tasks.workunit.client.0.smithi112.stdout:removed './186/33' 2023-01-24T03:35:14.309 INFO:tasks.workunit.client.0.smithi112.stdout:removed './186/17' 2023-01-24T03:35:14.309 INFO:tasks.workunit.client.0.smithi112.stdout:removed './186/35' 2023-01-24T03:35:14.309 INFO:tasks.workunit.client.0.smithi112.stdout:removed './186/37' 2023-01-24T03:35:14.309 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './186' 2023-01-24T03:35:14.310 INFO:tasks.workunit.client.0.smithi112.stdout:removed './187/19' 2023-01-24T03:35:14.310 INFO:tasks.workunit.client.0.smithi112.stdout:removed './187/15' 2023-01-24T03:35:14.310 INFO:tasks.workunit.client.0.smithi112.stdout:removed './187/31' 2023-01-24T03:35:14.310 INFO:tasks.workunit.client.0.smithi112.stdout:removed './187/36' 2023-01-24T03:35:14.310 INFO:tasks.workunit.client.0.smithi112.stdout:removed './187/20' 2023-01-24T03:35:14.311 INFO:tasks.workunit.client.0.smithi112.stdout:removed './187/12' 2023-01-24T03:35:14.311 INFO:tasks.workunit.client.0.smithi112.stdout:removed './187/39' 2023-01-24T03:35:14.311 INFO:tasks.workunit.client.0.smithi112.stdout:removed './187/10' 2023-01-24T03:35:14.311 INFO:tasks.workunit.client.0.smithi112.stdout:removed './187/34' 2023-01-24T03:35:14.311 INFO:tasks.workunit.client.0.smithi112.stdout:removed './187/38' 2023-01-24T03:35:14.312 INFO:tasks.workunit.client.0.smithi112.stdout:removed './187/11' 2023-01-24T03:35:14.312 INFO:tasks.workunit.client.0.smithi112.stdout:removed './187/16' 2023-01-24T03:35:14.312 INFO:tasks.workunit.client.0.smithi112.stdout:removed './187/32' 2023-01-24T03:35:14.312 INFO:tasks.workunit.client.0.smithi112.stdout:removed './187/30' 2023-01-24T03:35:14.312 INFO:tasks.workunit.client.0.smithi112.stdout:removed './187/14' 2023-01-24T03:35:14.313 INFO:tasks.workunit.client.0.smithi112.stdout:removed './187/18' 2023-01-24T03:35:14.313 INFO:tasks.workunit.client.0.smithi112.stdout:removed './187/13' 2023-01-24T03:35:14.313 INFO:tasks.workunit.client.0.smithi112.stdout:removed './187/33' 2023-01-24T03:35:14.313 INFO:tasks.workunit.client.0.smithi112.stdout:removed './187/17' 2023-01-24T03:35:14.313 INFO:tasks.workunit.client.0.smithi112.stdout:removed './187/35' 2023-01-24T03:35:14.314 INFO:tasks.workunit.client.0.smithi112.stdout:removed './187/37' 2023-01-24T03:35:14.314 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './187' 2023-01-24T03:35:14.314 INFO:tasks.workunit.client.0.smithi112.stdout:removed './188/19' 2023-01-24T03:35:14.314 INFO:tasks.workunit.client.0.smithi112.stdout:removed './188/15' 2023-01-24T03:35:14.314 INFO:tasks.workunit.client.0.smithi112.stdout:removed './188/31' 2023-01-24T03:35:14.315 INFO:tasks.workunit.client.0.smithi112.stdout:removed './188/36' 2023-01-24T03:35:14.315 INFO:tasks.workunit.client.0.smithi112.stdout:removed './188/20' 2023-01-24T03:35:14.316 INFO:tasks.workunit.client.0.smithi112.stdout:removed './188/12' 2023-01-24T03:35:14.316 INFO:tasks.workunit.client.0.smithi112.stdout:removed './188/39' 2023-01-24T03:35:14.316 INFO:tasks.workunit.client.0.smithi112.stdout:removed './188/10' 2023-01-24T03:35:14.316 INFO:tasks.workunit.client.0.smithi112.stdout:removed './188/34' 2023-01-24T03:35:14.316 INFO:tasks.workunit.client.0.smithi112.stdout:removed './188/38' 2023-01-24T03:35:14.317 INFO:tasks.workunit.client.0.smithi112.stdout:removed './188/11' 2023-01-24T03:35:14.317 INFO:tasks.workunit.client.0.smithi112.stdout:removed './188/16' 2023-01-24T03:35:14.317 INFO:tasks.workunit.client.0.smithi112.stdout:removed './188/32' 2023-01-24T03:35:14.317 INFO:tasks.workunit.client.0.smithi112.stdout:removed './188/30' 2023-01-24T03:35:14.317 INFO:tasks.workunit.client.0.smithi112.stdout:removed './188/14' 2023-01-24T03:35:14.318 INFO:tasks.workunit.client.0.smithi112.stdout:removed './188/18' 2023-01-24T03:35:14.318 INFO:tasks.workunit.client.0.smithi112.stdout:removed './188/13' 2023-01-24T03:35:14.318 INFO:tasks.workunit.client.0.smithi112.stdout:removed './188/33' 2023-01-24T03:35:14.318 INFO:tasks.workunit.client.0.smithi112.stdout:removed './188/17' 2023-01-24T03:35:14.318 INFO:tasks.workunit.client.0.smithi112.stdout:removed './188/35' 2023-01-24T03:35:14.319 INFO:tasks.workunit.client.0.smithi112.stdout:removed './188/37' 2023-01-24T03:35:14.319 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './188' 2023-01-24T03:35:14.319 INFO:tasks.workunit.client.0.smithi112.stdout:removed './189/19' 2023-01-24T03:35:14.319 INFO:tasks.workunit.client.0.smithi112.stdout:removed './189/15' 2023-01-24T03:35:14.319 INFO:tasks.workunit.client.0.smithi112.stdout:removed './189/31' 2023-01-24T03:35:14.320 INFO:tasks.workunit.client.0.smithi112.stdout:removed './189/36' 2023-01-24T03:35:14.320 INFO:tasks.workunit.client.0.smithi112.stdout:removed './189/20' 2023-01-24T03:35:14.320 INFO:tasks.workunit.client.0.smithi112.stdout:removed './189/12' 2023-01-24T03:35:14.320 INFO:tasks.workunit.client.0.smithi112.stdout:removed './189/39' 2023-01-24T03:35:14.320 INFO:tasks.workunit.client.0.smithi112.stdout:removed './189/10' 2023-01-24T03:35:14.321 INFO:tasks.workunit.client.0.smithi112.stdout:removed './189/34' 2023-01-24T03:35:14.321 INFO:tasks.workunit.client.0.smithi112.stdout:removed './189/38' 2023-01-24T03:35:14.321 INFO:tasks.workunit.client.0.smithi112.stdout:removed './189/11' 2023-01-24T03:35:14.321 INFO:tasks.workunit.client.0.smithi112.stdout:removed './189/16' 2023-01-24T03:35:14.321 INFO:tasks.workunit.client.0.smithi112.stdout:removed './189/32' 2023-01-24T03:35:14.322 INFO:tasks.workunit.client.0.smithi112.stdout:removed './189/30' 2023-01-24T03:35:14.322 INFO:tasks.workunit.client.0.smithi112.stdout:removed './189/14' 2023-01-24T03:35:14.322 INFO:tasks.workunit.client.0.smithi112.stdout:removed './189/18' 2023-01-24T03:35:14.322 INFO:tasks.workunit.client.0.smithi112.stdout:removed './189/13' 2023-01-24T03:35:14.322 INFO:tasks.workunit.client.0.smithi112.stdout:removed './189/33' 2023-01-24T03:35:14.323 INFO:tasks.workunit.client.0.smithi112.stdout:removed './189/17' 2023-01-24T03:35:14.323 INFO:tasks.workunit.client.0.smithi112.stdout:removed './189/35' 2023-01-24T03:35:14.323 INFO:tasks.workunit.client.0.smithi112.stdout:removed './189/37' 2023-01-24T03:35:14.323 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './189' 2023-01-24T03:35:14.323 INFO:tasks.workunit.client.0.smithi112.stdout:removed './190/19' 2023-01-24T03:35:14.324 INFO:tasks.workunit.client.0.smithi112.stdout:removed './190/15' 2023-01-24T03:35:14.324 INFO:tasks.workunit.client.0.smithi112.stdout:removed './190/31' 2023-01-24T03:35:14.324 INFO:tasks.workunit.client.0.smithi112.stdout:removed './190/36' 2023-01-24T03:35:14.324 INFO:tasks.workunit.client.0.smithi112.stdout:removed './190/20' 2023-01-24T03:35:14.324 INFO:tasks.workunit.client.0.smithi112.stdout:removed './190/12' 2023-01-24T03:35:14.325 INFO:tasks.workunit.client.0.smithi112.stdout:removed './190/39' 2023-01-24T03:35:14.325 INFO:tasks.workunit.client.0.smithi112.stdout:removed './190/10' 2023-01-24T03:35:14.326 INFO:tasks.workunit.client.0.smithi112.stdout:removed './190/34' 2023-01-24T03:35:14.326 INFO:tasks.workunit.client.0.smithi112.stdout:removed './190/38' 2023-01-24T03:35:14.326 INFO:tasks.workunit.client.0.smithi112.stdout:removed './190/11' 2023-01-24T03:35:14.326 INFO:tasks.workunit.client.0.smithi112.stdout:removed './190/16' 2023-01-24T03:35:14.326 INFO:tasks.workunit.client.0.smithi112.stdout:removed './190/32' 2023-01-24T03:35:14.327 INFO:tasks.workunit.client.0.smithi112.stdout:removed './190/30' 2023-01-24T03:35:14.327 INFO:tasks.workunit.client.0.smithi112.stdout:removed './190/14' 2023-01-24T03:35:14.327 INFO:tasks.workunit.client.0.smithi112.stdout:removed './190/18' 2023-01-24T03:35:14.327 INFO:tasks.workunit.client.0.smithi112.stdout:removed './190/13' 2023-01-24T03:35:14.327 INFO:tasks.workunit.client.0.smithi112.stdout:removed './190/33' 2023-01-24T03:35:14.328 INFO:tasks.workunit.client.0.smithi112.stdout:removed './190/17' 2023-01-24T03:35:14.328 INFO:tasks.workunit.client.0.smithi112.stdout:removed './190/35' 2023-01-24T03:35:14.328 INFO:tasks.workunit.client.0.smithi112.stdout:removed './190/37' 2023-01-24T03:35:14.328 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './190' 2023-01-24T03:35:14.328 INFO:tasks.workunit.client.0.smithi112.stdout:removed './191/19' 2023-01-24T03:35:14.329 INFO:tasks.workunit.client.0.smithi112.stdout:removed './191/15' 2023-01-24T03:35:14.329 INFO:tasks.workunit.client.0.smithi112.stdout:removed './191/31' 2023-01-24T03:35:14.329 INFO:tasks.workunit.client.0.smithi112.stdout:removed './191/36' 2023-01-24T03:35:14.329 INFO:tasks.workunit.client.0.smithi112.stdout:removed './191/20' 2023-01-24T03:35:14.329 INFO:tasks.workunit.client.0.smithi112.stdout:removed './191/12' 2023-01-24T03:35:14.330 INFO:tasks.workunit.client.0.smithi112.stdout:removed './191/39' 2023-01-24T03:35:14.330 INFO:tasks.workunit.client.0.smithi112.stdout:removed './191/10' 2023-01-24T03:35:14.330 INFO:tasks.workunit.client.0.smithi112.stdout:removed './191/34' 2023-01-24T03:35:14.330 INFO:tasks.workunit.client.0.smithi112.stdout:removed './191/38' 2023-01-24T03:35:14.330 INFO:tasks.workunit.client.0.smithi112.stdout:removed './191/11' 2023-01-24T03:35:14.330 INFO:tasks.workunit.client.0.smithi112.stdout:removed './191/16' 2023-01-24T03:35:14.331 INFO:tasks.workunit.client.0.smithi112.stdout:removed './191/32' 2023-01-24T03:35:14.331 INFO:tasks.workunit.client.0.smithi112.stdout:removed './191/30' 2023-01-24T03:35:14.331 INFO:tasks.workunit.client.0.smithi112.stdout:removed './191/14' 2023-01-24T03:35:14.331 INFO:tasks.workunit.client.0.smithi112.stdout:removed './191/18' 2023-01-24T03:35:14.331 INFO:tasks.workunit.client.0.smithi112.stdout:removed './191/13' 2023-01-24T03:35:14.332 INFO:tasks.workunit.client.0.smithi112.stdout:removed './191/33' 2023-01-24T03:35:14.332 INFO:tasks.workunit.client.0.smithi112.stdout:removed './191/17' 2023-01-24T03:35:14.332 INFO:tasks.workunit.client.0.smithi112.stdout:removed './191/35' 2023-01-24T03:35:14.332 INFO:tasks.workunit.client.0.smithi112.stdout:removed './191/37' 2023-01-24T03:35:14.333 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './191' 2023-01-24T03:35:14.333 INFO:tasks.workunit.client.0.smithi112.stdout:removed './192/19' 2023-01-24T03:35:14.333 INFO:tasks.workunit.client.0.smithi112.stdout:removed './192/15' 2023-01-24T03:35:14.333 INFO:tasks.workunit.client.0.smithi112.stdout:removed './192/31' 2023-01-24T03:35:14.333 INFO:tasks.workunit.client.0.smithi112.stdout:removed './192/36' 2023-01-24T03:35:14.334 INFO:tasks.workunit.client.0.smithi112.stdout:removed './192/20' 2023-01-24T03:35:14.334 INFO:tasks.workunit.client.0.smithi112.stdout:removed './192/12' 2023-01-24T03:35:14.334 INFO:tasks.workunit.client.0.smithi112.stdout:removed './192/39' 2023-01-24T03:35:14.334 INFO:tasks.workunit.client.0.smithi112.stdout:removed './192/10' 2023-01-24T03:35:14.334 INFO:tasks.workunit.client.0.smithi112.stdout:removed './192/34' 2023-01-24T03:35:14.335 INFO:tasks.workunit.client.0.smithi112.stdout:removed './192/38' 2023-01-24T03:35:14.335 INFO:tasks.workunit.client.0.smithi112.stdout:removed './192/11' 2023-01-24T03:35:14.335 INFO:tasks.workunit.client.0.smithi112.stdout:removed './192/16' 2023-01-24T03:35:14.335 INFO:tasks.workunit.client.0.smithi112.stdout:removed './192/32' 2023-01-24T03:35:14.335 INFO:tasks.workunit.client.0.smithi112.stdout:removed './192/30' 2023-01-24T03:35:14.336 INFO:tasks.workunit.client.0.smithi112.stdout:removed './192/14' 2023-01-24T03:35:14.336 INFO:tasks.workunit.client.0.smithi112.stdout:removed './192/18' 2023-01-24T03:35:14.336 INFO:tasks.workunit.client.0.smithi112.stdout:removed './192/13' 2023-01-24T03:35:14.337 INFO:tasks.workunit.client.0.smithi112.stdout:removed './192/33' 2023-01-24T03:35:14.337 INFO:tasks.workunit.client.0.smithi112.stdout:removed './192/17' 2023-01-24T03:35:14.337 INFO:tasks.workunit.client.0.smithi112.stdout:removed './192/35' 2023-01-24T03:35:14.337 INFO:tasks.workunit.client.0.smithi112.stdout:removed './192/37' 2023-01-24T03:35:14.337 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './192' 2023-01-24T03:35:14.338 INFO:tasks.workunit.client.0.smithi112.stdout:removed './193/19' 2023-01-24T03:35:14.338 INFO:tasks.workunit.client.0.smithi112.stdout:removed './193/15' 2023-01-24T03:35:14.338 INFO:tasks.workunit.client.0.smithi112.stdout:removed './193/31' 2023-01-24T03:35:14.338 INFO:tasks.workunit.client.0.smithi112.stdout:removed './193/36' 2023-01-24T03:35:14.338 INFO:tasks.workunit.client.0.smithi112.stdout:removed './193/20' 2023-01-24T03:35:14.339 INFO:tasks.workunit.client.0.smithi112.stdout:removed './193/12' 2023-01-24T03:35:14.339 INFO:tasks.workunit.client.0.smithi112.stdout:removed './193/39' 2023-01-24T03:35:14.339 INFO:tasks.workunit.client.0.smithi112.stdout:removed './193/10' 2023-01-24T03:35:14.339 INFO:tasks.workunit.client.0.smithi112.stdout:removed './193/34' 2023-01-24T03:35:14.339 INFO:tasks.workunit.client.0.smithi112.stdout:removed './193/38' 2023-01-24T03:35:14.340 INFO:tasks.workunit.client.0.smithi112.stdout:removed './193/11' 2023-01-24T03:35:14.340 INFO:tasks.workunit.client.0.smithi112.stdout:removed './193/16' 2023-01-24T03:35:14.340 INFO:tasks.workunit.client.0.smithi112.stdout:removed './193/32' 2023-01-24T03:35:14.340 INFO:tasks.workunit.client.0.smithi112.stdout:removed './193/30' 2023-01-24T03:35:14.340 INFO:tasks.workunit.client.0.smithi112.stdout:removed './193/14' 2023-01-24T03:35:14.341 INFO:tasks.workunit.client.0.smithi112.stdout:removed './193/18' 2023-01-24T03:35:14.341 INFO:tasks.workunit.client.0.smithi112.stdout:removed './193/13' 2023-01-24T03:35:14.341 INFO:tasks.workunit.client.0.smithi112.stdout:removed './193/33' 2023-01-24T03:35:14.341 INFO:tasks.workunit.client.0.smithi112.stdout:removed './193/17' 2023-01-24T03:35:14.341 INFO:tasks.workunit.client.0.smithi112.stdout:removed './193/35' 2023-01-24T03:35:14.342 INFO:tasks.workunit.client.0.smithi112.stdout:removed './193/37' 2023-01-24T03:35:14.342 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './193' 2023-01-24T03:35:14.342 INFO:tasks.workunit.client.0.smithi112.stdout:removed './194/19' 2023-01-24T03:35:14.342 INFO:tasks.workunit.client.0.smithi112.stdout:removed './194/15' 2023-01-24T03:35:14.342 INFO:tasks.workunit.client.0.smithi112.stdout:removed './194/31' 2023-01-24T03:35:14.343 INFO:tasks.workunit.client.0.smithi112.stdout:removed './194/36' 2023-01-24T03:35:14.343 INFO:tasks.workunit.client.0.smithi112.stdout:removed './194/20' 2023-01-24T03:35:14.343 INFO:tasks.workunit.client.0.smithi112.stdout:removed './194/12' 2023-01-24T03:35:14.343 INFO:tasks.workunit.client.0.smithi112.stdout:removed './194/39' 2023-01-24T03:35:14.343 INFO:tasks.workunit.client.0.smithi112.stdout:removed './194/10' 2023-01-24T03:35:14.344 INFO:tasks.workunit.client.0.smithi112.stdout:removed './194/34' 2023-01-24T03:35:14.344 INFO:tasks.workunit.client.0.smithi112.stdout:removed './194/38' 2023-01-24T03:35:14.344 INFO:tasks.workunit.client.0.smithi112.stdout:removed './194/11' 2023-01-24T03:35:14.344 INFO:tasks.workunit.client.0.smithi112.stdout:removed './194/16' 2023-01-24T03:35:14.345 INFO:tasks.workunit.client.0.smithi112.stdout:removed './194/32' 2023-01-24T03:35:14.345 INFO:tasks.workunit.client.0.smithi112.stdout:removed './194/30' 2023-01-24T03:35:14.345 INFO:tasks.workunit.client.0.smithi112.stdout:removed './194/14' 2023-01-24T03:35:14.345 INFO:tasks.workunit.client.0.smithi112.stdout:removed './194/18' 2023-01-24T03:35:14.345 INFO:tasks.workunit.client.0.smithi112.stdout:removed './194/13' 2023-01-24T03:35:14.346 INFO:tasks.workunit.client.0.smithi112.stdout:removed './194/33' 2023-01-24T03:35:14.346 INFO:tasks.workunit.client.0.smithi112.stdout:removed './194/17' 2023-01-24T03:35:14.346 INFO:tasks.workunit.client.0.smithi112.stdout:removed './194/35' 2023-01-24T03:35:14.347 INFO:tasks.workunit.client.0.smithi112.stdout:removed './194/37' 2023-01-24T03:35:14.347 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './194' 2023-01-24T03:35:14.347 INFO:tasks.workunit.client.0.smithi112.stdout:removed './195/19' 2023-01-24T03:35:14.347 INFO:tasks.workunit.client.0.smithi112.stdout:removed './195/15' 2023-01-24T03:35:14.347 INFO:tasks.workunit.client.0.smithi112.stdout:removed './195/31' 2023-01-24T03:35:14.348 INFO:tasks.workunit.client.0.smithi112.stdout:removed './195/36' 2023-01-24T03:35:14.348 INFO:tasks.workunit.client.0.smithi112.stdout:removed './195/20' 2023-01-24T03:35:14.348 INFO:tasks.workunit.client.0.smithi112.stdout:removed './195/12' 2023-01-24T03:35:14.348 INFO:tasks.workunit.client.0.smithi112.stdout:removed './195/39' 2023-01-24T03:35:14.348 INFO:tasks.workunit.client.0.smithi112.stdout:removed './195/10' 2023-01-24T03:35:14.349 INFO:tasks.workunit.client.0.smithi112.stdout:removed './195/34' 2023-01-24T03:35:14.349 INFO:tasks.workunit.client.0.smithi112.stdout:removed './195/38' 2023-01-24T03:35:14.349 INFO:tasks.workunit.client.0.smithi112.stdout:removed './195/11' 2023-01-24T03:35:14.350 INFO:tasks.workunit.client.0.smithi112.stdout:removed './195/16' 2023-01-24T03:35:14.350 INFO:tasks.workunit.client.0.smithi112.stdout:removed './195/32' 2023-01-24T03:35:14.350 INFO:tasks.workunit.client.0.smithi112.stdout:removed './195/30' 2023-01-24T03:35:14.350 INFO:tasks.workunit.client.0.smithi112.stdout:removed './195/14' 2023-01-24T03:35:14.350 INFO:tasks.workunit.client.0.smithi112.stdout:removed './195/18' 2023-01-24T03:35:14.351 INFO:tasks.workunit.client.0.smithi112.stdout:removed './195/13' 2023-01-24T03:35:14.351 INFO:tasks.workunit.client.0.smithi112.stdout:removed './195/33' 2023-01-24T03:35:14.351 INFO:tasks.workunit.client.0.smithi112.stdout:removed './195/17' 2023-01-24T03:35:14.351 INFO:tasks.workunit.client.0.smithi112.stdout:removed './195/35' 2023-01-24T03:35:14.351 INFO:tasks.workunit.client.0.smithi112.stdout:removed './195/37' 2023-01-24T03:35:14.351 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './195' 2023-01-24T03:35:14.352 INFO:tasks.workunit.client.0.smithi112.stdout:removed './196/19' 2023-01-24T03:35:14.352 INFO:tasks.workunit.client.0.smithi112.stdout:removed './196/15' 2023-01-24T03:35:14.352 INFO:tasks.workunit.client.0.smithi112.stdout:removed './196/31' 2023-01-24T03:35:14.352 INFO:tasks.workunit.client.0.smithi112.stdout:removed './196/36' 2023-01-24T03:35:14.353 INFO:tasks.workunit.client.0.smithi112.stdout:removed './196/20' 2023-01-24T03:35:14.353 INFO:tasks.workunit.client.0.smithi112.stdout:removed './196/12' 2023-01-24T03:35:14.353 INFO:tasks.workunit.client.0.smithi112.stdout:removed './196/39' 2023-01-24T03:35:14.353 INFO:tasks.workunit.client.0.smithi112.stdout:removed './196/10' 2023-01-24T03:35:16.006 INFO:tasks.workunit.client.0.smithi112.stdout:removed './196/34' 2023-01-24T03:35:16.006 INFO:tasks.workunit.client.0.smithi112.stdout:removed './196/38' 2023-01-24T03:35:16.006 INFO:tasks.workunit.client.0.smithi112.stdout:removed './196/11' 2023-01-24T03:35:16.007 INFO:tasks.workunit.client.0.smithi112.stdout:removed './196/16' 2023-01-24T03:35:16.007 INFO:tasks.workunit.client.0.smithi112.stdout:removed './196/32' 2023-01-24T03:35:16.007 INFO:tasks.workunit.client.0.smithi112.stdout:removed './196/30' 2023-01-24T03:35:16.007 INFO:tasks.workunit.client.0.smithi112.stdout:removed './196/14' 2023-01-24T03:35:16.007 INFO:tasks.workunit.client.0.smithi112.stdout:removed './196/18' 2023-01-24T03:35:16.008 INFO:tasks.workunit.client.0.smithi112.stdout:removed './196/13' 2023-01-24T03:35:16.008 INFO:tasks.workunit.client.0.smithi112.stdout:removed './196/33' 2023-01-24T03:35:16.008 INFO:tasks.workunit.client.0.smithi112.stdout:removed './196/17' 2023-01-24T03:35:16.008 INFO:tasks.workunit.client.0.smithi112.stdout:removed './196/35' 2023-01-24T03:35:16.008 INFO:tasks.workunit.client.0.smithi112.stdout:removed './196/37' 2023-01-24T03:35:16.009 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './196' 2023-01-24T03:35:16.009 INFO:tasks.workunit.client.0.smithi112.stdout:removed './197/19' 2023-01-24T03:35:16.009 INFO:tasks.workunit.client.0.smithi112.stdout:removed './197/15' 2023-01-24T03:35:16.009 INFO:tasks.workunit.client.0.smithi112.stdout:removed './197/31' 2023-01-24T03:35:16.009 INFO:tasks.workunit.client.0.smithi112.stdout:removed './197/36' 2023-01-24T03:35:16.010 INFO:tasks.workunit.client.0.smithi112.stdout:removed './197/20' 2023-01-24T03:35:16.010 INFO:tasks.workunit.client.0.smithi112.stdout:removed './197/12' 2023-01-24T03:35:16.010 INFO:tasks.workunit.client.0.smithi112.stdout:removed './197/39' 2023-01-24T03:35:16.010 INFO:tasks.workunit.client.0.smithi112.stdout:removed './197/10' 2023-01-24T03:35:16.010 INFO:tasks.workunit.client.0.smithi112.stdout:removed './197/34' 2023-01-24T03:35:16.011 INFO:tasks.workunit.client.0.smithi112.stdout:removed './197/38' 2023-01-24T03:35:16.011 INFO:tasks.workunit.client.0.smithi112.stdout:removed './197/11' 2023-01-24T03:35:16.011 INFO:tasks.workunit.client.0.smithi112.stdout:removed './197/16' 2023-01-24T03:35:16.011 INFO:tasks.workunit.client.0.smithi112.stdout:removed './197/32' 2023-01-24T03:35:16.011 INFO:tasks.workunit.client.0.smithi112.stdout:removed './197/30' 2023-01-24T03:35:16.012 INFO:tasks.workunit.client.0.smithi112.stdout:removed './197/14' 2023-01-24T03:35:16.012 INFO:tasks.workunit.client.0.smithi112.stdout:removed './197/18' 2023-01-24T03:35:16.012 INFO:tasks.workunit.client.0.smithi112.stdout:removed './197/13' 2023-01-24T03:35:16.012 INFO:tasks.workunit.client.0.smithi112.stdout:removed './197/33' 2023-01-24T03:35:16.012 INFO:tasks.workunit.client.0.smithi112.stdout:removed './197/17' 2023-01-24T03:35:16.013 INFO:tasks.workunit.client.0.smithi112.stdout:removed './197/35' 2023-01-24T03:35:16.013 INFO:tasks.workunit.client.0.smithi112.stdout:removed './197/37' 2023-01-24T03:35:16.013 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './197' 2023-01-24T03:35:16.013 INFO:tasks.workunit.client.0.smithi112.stdout:removed './198/19' 2023-01-24T03:35:16.013 INFO:tasks.workunit.client.0.smithi112.stdout:removed './198/15' 2023-01-24T03:35:16.013 INFO:tasks.workunit.client.0.smithi112.stdout:removed './198/31' 2023-01-24T03:35:16.014 INFO:tasks.workunit.client.0.smithi112.stdout:removed './198/36' 2023-01-24T03:35:16.014 INFO:tasks.workunit.client.0.smithi112.stdout:removed './198/20' 2023-01-24T03:35:16.014 INFO:tasks.workunit.client.0.smithi112.stdout:removed './198/12' 2023-01-24T03:35:16.014 INFO:tasks.workunit.client.0.smithi112.stdout:removed './198/39' 2023-01-24T03:35:16.014 INFO:tasks.workunit.client.0.smithi112.stdout:removed './198/10' 2023-01-24T03:35:16.015 INFO:tasks.workunit.client.0.smithi112.stdout:removed './198/34' 2023-01-24T03:35:16.015 INFO:tasks.workunit.client.0.smithi112.stdout:removed './198/38' 2023-01-24T03:35:16.015 INFO:tasks.workunit.client.0.smithi112.stdout:removed './198/11' 2023-01-24T03:35:16.015 INFO:tasks.workunit.client.0.smithi112.stdout:removed './198/16' 2023-01-24T03:35:16.015 INFO:tasks.workunit.client.0.smithi112.stdout:removed './198/32' 2023-01-24T03:35:16.016 INFO:tasks.workunit.client.0.smithi112.stdout:removed './198/30' 2023-01-24T03:35:16.016 INFO:tasks.workunit.client.0.smithi112.stdout:removed './198/14' 2023-01-24T03:35:16.016 INFO:tasks.workunit.client.0.smithi112.stdout:removed './198/18' 2023-01-24T03:35:16.016 INFO:tasks.workunit.client.0.smithi112.stdout:removed './198/13' 2023-01-24T03:35:16.016 INFO:tasks.workunit.client.0.smithi112.stdout:removed './198/33' 2023-01-24T03:35:16.017 INFO:tasks.workunit.client.0.smithi112.stdout:removed './198/17' 2023-01-24T03:35:16.017 INFO:tasks.workunit.client.0.smithi112.stdout:removed './198/35' 2023-01-24T03:35:16.017 INFO:tasks.workunit.client.0.smithi112.stdout:removed './198/37' 2023-01-24T03:35:16.017 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './198' 2023-01-24T03:35:16.017 INFO:tasks.workunit.client.0.smithi112.stdout:removed './199/19' 2023-01-24T03:35:16.018 INFO:tasks.workunit.client.0.smithi112.stdout:removed './199/15' 2023-01-24T03:35:16.018 INFO:tasks.workunit.client.0.smithi112.stdout:removed './199/31' 2023-01-24T03:35:16.018 INFO:tasks.workunit.client.0.smithi112.stdout:removed './199/36' 2023-01-24T03:35:16.018 INFO:tasks.workunit.client.0.smithi112.stdout:removed './199/20' 2023-01-24T03:35:16.018 INFO:tasks.workunit.client.0.smithi112.stdout:removed './199/12' 2023-01-24T03:35:16.019 INFO:tasks.workunit.client.0.smithi112.stdout:removed './199/39' 2023-01-24T03:35:16.019 INFO:tasks.workunit.client.0.smithi112.stdout:removed './199/10' 2023-01-24T03:35:16.019 INFO:tasks.workunit.client.0.smithi112.stdout:removed './199/34' 2023-01-24T03:35:16.019 INFO:tasks.workunit.client.0.smithi112.stdout:removed './199/38' 2023-01-24T03:35:16.019 INFO:tasks.workunit.client.0.smithi112.stdout:removed './199/11' 2023-01-24T03:35:16.020 INFO:tasks.workunit.client.0.smithi112.stdout:removed './199/16' 2023-01-24T03:35:16.020 INFO:tasks.workunit.client.0.smithi112.stdout:removed './199/32' 2023-01-24T03:35:16.020 INFO:tasks.workunit.client.0.smithi112.stdout:removed './199/30' 2023-01-24T03:35:16.020 INFO:tasks.workunit.client.0.smithi112.stdout:removed './199/14' 2023-01-24T03:35:16.020 INFO:tasks.workunit.client.0.smithi112.stdout:removed './199/18' 2023-01-24T03:35:16.021 INFO:tasks.workunit.client.0.smithi112.stdout:removed './199/13' 2023-01-24T03:35:16.021 INFO:tasks.workunit.client.0.smithi112.stdout:removed './199/33' 2023-01-24T03:35:16.021 INFO:tasks.workunit.client.0.smithi112.stdout:removed './199/17' 2023-01-24T03:35:16.021 INFO:tasks.workunit.client.0.smithi112.stdout:removed './199/35' 2023-01-24T03:35:16.021 INFO:tasks.workunit.client.0.smithi112.stdout:removed './199/37' 2023-01-24T03:35:16.022 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './199' 2023-01-24T03:35:16.022 INFO:tasks.workunit.client.0.smithi112.stdout:removed './200/21' 2023-01-24T03:35:16.022 INFO:tasks.workunit.client.0.smithi112.stdout:removed './200/27' 2023-01-24T03:35:16.022 INFO:tasks.workunit.client.0.smithi112.stdout:removed './200/31' 2023-01-24T03:35:16.022 INFO:tasks.workunit.client.0.smithi112.stdout:removed './200/36' 2023-01-24T03:35:16.023 INFO:tasks.workunit.client.0.smithi112.stdout:removed './200/20' 2023-01-24T03:35:16.023 INFO:tasks.workunit.client.0.smithi112.stdout:removed './200/39' 2023-01-24T03:35:16.023 INFO:tasks.workunit.client.0.smithi112.stdout:removed './200/34' 2023-01-24T03:35:16.023 INFO:tasks.workunit.client.0.smithi112.stdout:removed './200/38' 2023-01-24T03:35:16.023 INFO:tasks.workunit.client.0.smithi112.stdout:removed './200/28' 2023-01-24T03:35:16.024 INFO:tasks.workunit.client.0.smithi112.stdout:removed './200/24' 2023-01-24T03:35:16.024 INFO:tasks.workunit.client.0.smithi112.stdout:removed './200/29' 2023-01-24T03:35:16.024 INFO:tasks.workunit.client.0.smithi112.stdout:removed './200/32' 2023-01-24T03:35:16.024 INFO:tasks.workunit.client.0.smithi112.stdout:removed './200/22' 2023-01-24T03:35:16.024 INFO:tasks.workunit.client.0.smithi112.stdout:removed './200/25' 2023-01-24T03:35:16.025 INFO:tasks.workunit.client.0.smithi112.stdout:removed './200/30' 2023-01-24T03:35:16.025 INFO:tasks.workunit.client.0.smithi112.stdout:removed './200/33' 2023-01-24T03:35:16.025 INFO:tasks.workunit.client.0.smithi112.stdout:removed './200/23' 2023-01-24T03:35:16.025 INFO:tasks.workunit.client.0.smithi112.stdout:removed './200/35' 2023-01-24T03:35:16.025 INFO:tasks.workunit.client.0.smithi112.stdout:removed './200/26' 2023-01-24T03:35:16.026 INFO:tasks.workunit.client.0.smithi112.stdout:removed './200/37' 2023-01-24T03:35:16.026 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './200' 2023-01-24T03:35:16.026 INFO:tasks.workunit.client.0.smithi112.stdout:removed './201/21' 2023-01-24T03:35:16.026 INFO:tasks.workunit.client.0.smithi112.stdout:removed './201/27' 2023-01-24T03:35:16.026 INFO:tasks.workunit.client.0.smithi112.stdout:removed './201/31' 2023-01-24T03:35:16.027 INFO:tasks.workunit.client.0.smithi112.stdout:removed './201/36' 2023-01-24T03:35:16.027 INFO:tasks.workunit.client.0.smithi112.stdout:removed './201/20' 2023-01-24T03:35:16.027 INFO:tasks.workunit.client.0.smithi112.stdout:removed './201/39' 2023-01-24T03:35:16.027 INFO:tasks.workunit.client.0.smithi112.stdout:removed './201/34' 2023-01-24T03:35:16.027 INFO:tasks.workunit.client.0.smithi112.stdout:removed './201/38' 2023-01-24T03:35:16.028 INFO:tasks.workunit.client.0.smithi112.stdout:removed './201/28' 2023-01-24T03:35:16.028 INFO:tasks.workunit.client.0.smithi112.stdout:removed './201/24' 2023-01-24T03:35:16.028 INFO:tasks.workunit.client.0.smithi112.stdout:removed './201/29' 2023-01-24T03:35:16.028 INFO:tasks.workunit.client.0.smithi112.stdout:removed './201/32' 2023-01-24T03:35:16.028 INFO:tasks.workunit.client.0.smithi112.stdout:removed './201/22' 2023-01-24T03:35:16.029 INFO:tasks.workunit.client.0.smithi112.stdout:removed './201/25' 2023-01-24T03:35:16.029 INFO:tasks.workunit.client.0.smithi112.stdout:removed './201/30' 2023-01-24T03:35:16.029 INFO:tasks.workunit.client.0.smithi112.stdout:removed './201/33' 2023-01-24T03:35:16.029 INFO:tasks.workunit.client.0.smithi112.stdout:removed './201/23' 2023-01-24T03:35:16.029 INFO:tasks.workunit.client.0.smithi112.stdout:removed './201/35' 2023-01-24T03:35:16.029 INFO:tasks.workunit.client.0.smithi112.stdout:removed './201/26' 2023-01-24T03:35:16.030 INFO:tasks.workunit.client.0.smithi112.stdout:removed './201/37' 2023-01-24T03:35:16.030 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './201' 2023-01-24T03:35:16.030 INFO:tasks.workunit.client.0.smithi112.stdout:removed './202/21' 2023-01-24T03:35:16.030 INFO:tasks.workunit.client.0.smithi112.stdout:removed './202/27' 2023-01-24T03:35:16.030 INFO:tasks.workunit.client.0.smithi112.stdout:removed './202/31' 2023-01-24T03:35:16.031 INFO:tasks.workunit.client.0.smithi112.stdout:removed './202/36' 2023-01-24T03:35:16.031 INFO:tasks.workunit.client.0.smithi112.stdout:removed './202/20' 2023-01-24T03:35:16.031 INFO:tasks.workunit.client.0.smithi112.stdout:removed './202/39' 2023-01-24T03:35:16.031 INFO:tasks.workunit.client.0.smithi112.stdout:removed './202/34' 2023-01-24T03:35:16.031 INFO:tasks.workunit.client.0.smithi112.stdout:removed './202/38' 2023-01-24T03:35:16.032 INFO:tasks.workunit.client.0.smithi112.stdout:removed './202/28' 2023-01-24T03:35:16.032 INFO:tasks.workunit.client.0.smithi112.stdout:removed './202/24' 2023-01-24T03:35:16.032 INFO:tasks.workunit.client.0.smithi112.stdout:removed './202/29' 2023-01-24T03:35:16.032 INFO:tasks.workunit.client.0.smithi112.stdout:removed './202/32' 2023-01-24T03:35:16.032 INFO:tasks.workunit.client.0.smithi112.stdout:removed './202/22' 2023-01-24T03:35:16.032 INFO:tasks.workunit.client.0.smithi112.stdout:removed './202/25' 2023-01-24T03:35:16.033 INFO:tasks.workunit.client.0.smithi112.stdout:removed './202/30' 2023-01-24T03:35:16.033 INFO:tasks.workunit.client.0.smithi112.stdout:removed './202/33' 2023-01-24T03:35:16.033 INFO:tasks.workunit.client.0.smithi112.stdout:removed './202/23' 2023-01-24T03:35:16.033 INFO:tasks.workunit.client.0.smithi112.stdout:removed './202/35' 2023-01-24T03:35:16.033 INFO:tasks.workunit.client.0.smithi112.stdout:removed './202/26' 2023-01-24T03:35:16.034 INFO:tasks.workunit.client.0.smithi112.stdout:removed './202/37' 2023-01-24T03:35:16.034 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './202' 2023-01-24T03:35:16.034 INFO:tasks.workunit.client.0.smithi112.stdout:removed './203/21' 2023-01-24T03:35:16.034 INFO:tasks.workunit.client.0.smithi112.stdout:removed './203/27' 2023-01-24T03:35:16.034 INFO:tasks.workunit.client.0.smithi112.stdout:removed './203/31' 2023-01-24T03:35:16.035 INFO:tasks.workunit.client.0.smithi112.stdout:removed './203/36' 2023-01-24T03:35:16.035 INFO:tasks.workunit.client.0.smithi112.stdout:removed './203/20' 2023-01-24T03:35:16.035 INFO:tasks.workunit.client.0.smithi112.stdout:removed './203/39' 2023-01-24T03:35:16.035 INFO:tasks.workunit.client.0.smithi112.stdout:removed './203/34' 2023-01-24T03:35:16.035 INFO:tasks.workunit.client.0.smithi112.stdout:removed './203/38' 2023-01-24T03:35:16.036 INFO:tasks.workunit.client.0.smithi112.stdout:removed './203/28' 2023-01-24T03:35:16.036 INFO:tasks.workunit.client.0.smithi112.stdout:removed './203/24' 2023-01-24T03:35:16.036 INFO:tasks.workunit.client.0.smithi112.stdout:removed './203/29' 2023-01-24T03:35:16.036 INFO:tasks.workunit.client.0.smithi112.stdout:removed './203/32' 2023-01-24T03:35:16.036 INFO:tasks.workunit.client.0.smithi112.stdout:removed './203/22' 2023-01-24T03:35:16.037 INFO:tasks.workunit.client.0.smithi112.stdout:removed './203/25' 2023-01-24T03:35:16.037 INFO:tasks.workunit.client.0.smithi112.stdout:removed './203/30' 2023-01-24T03:35:16.037 INFO:tasks.workunit.client.0.smithi112.stdout:removed './203/33' 2023-01-24T03:35:16.037 INFO:tasks.workunit.client.0.smithi112.stdout:removed './203/23' 2023-01-24T03:35:16.037 INFO:tasks.workunit.client.0.smithi112.stdout:removed './203/35' 2023-01-24T03:35:16.038 INFO:tasks.workunit.client.0.smithi112.stdout:removed './203/26' 2023-01-24T03:35:16.038 INFO:tasks.workunit.client.0.smithi112.stdout:removed './203/37' 2023-01-24T03:35:16.038 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './203' 2023-01-24T03:35:16.038 INFO:tasks.workunit.client.0.smithi112.stdout:removed './204/21' 2023-01-24T03:35:16.038 INFO:tasks.workunit.client.0.smithi112.stdout:removed './204/27' 2023-01-24T03:35:16.039 INFO:tasks.workunit.client.0.smithi112.stdout:removed './204/31' 2023-01-24T03:35:16.039 INFO:tasks.workunit.client.0.smithi112.stdout:removed './204/36' 2023-01-24T03:35:16.039 INFO:tasks.workunit.client.0.smithi112.stdout:removed './204/20' 2023-01-24T03:35:16.039 INFO:tasks.workunit.client.0.smithi112.stdout:removed './204/39' 2023-01-24T03:35:16.039 INFO:tasks.workunit.client.0.smithi112.stdout:removed './204/34' 2023-01-24T03:35:16.039 INFO:tasks.workunit.client.0.smithi112.stdout:removed './204/38' 2023-01-24T03:35:16.040 INFO:tasks.workunit.client.0.smithi112.stdout:removed './204/28' 2023-01-24T03:35:16.040 INFO:tasks.workunit.client.0.smithi112.stdout:removed './204/24' 2023-01-24T03:35:16.040 INFO:tasks.workunit.client.0.smithi112.stdout:removed './204/29' 2023-01-24T03:35:16.040 INFO:tasks.workunit.client.0.smithi112.stdout:removed './204/32' 2023-01-24T03:35:16.040 INFO:tasks.workunit.client.0.smithi112.stdout:removed './204/22' 2023-01-24T03:35:16.041 INFO:tasks.workunit.client.0.smithi112.stdout:removed './204/25' 2023-01-24T03:35:16.041 INFO:tasks.workunit.client.0.smithi112.stdout:removed './204/30' 2023-01-24T03:35:16.041 INFO:tasks.workunit.client.0.smithi112.stdout:removed './204/33' 2023-01-24T03:35:16.041 INFO:tasks.workunit.client.0.smithi112.stdout:removed './204/23' 2023-01-24T03:35:16.041 INFO:tasks.workunit.client.0.smithi112.stdout:removed './204/35' 2023-01-24T03:35:16.042 INFO:tasks.workunit.client.0.smithi112.stdout:removed './204/26' 2023-01-24T03:35:16.042 INFO:tasks.workunit.client.0.smithi112.stdout:removed './204/37' 2023-01-24T03:35:16.042 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './204' 2023-01-24T03:35:16.042 INFO:tasks.workunit.client.0.smithi112.stdout:removed './205/21' 2023-01-24T03:35:16.042 INFO:tasks.workunit.client.0.smithi112.stdout:removed './205/27' 2023-01-24T03:35:16.043 INFO:tasks.workunit.client.0.smithi112.stdout:removed './205/31' 2023-01-24T03:35:16.043 INFO:tasks.workunit.client.0.smithi112.stdout:removed './205/36' 2023-01-24T03:35:16.043 INFO:tasks.workunit.client.0.smithi112.stdout:removed './205/20' 2023-01-24T03:35:16.043 INFO:tasks.workunit.client.0.smithi112.stdout:removed './205/39' 2023-01-24T03:35:16.043 INFO:tasks.workunit.client.0.smithi112.stdout:removed './205/34' 2023-01-24T03:35:16.043 INFO:tasks.workunit.client.0.smithi112.stdout:removed './205/38' 2023-01-24T03:35:16.044 INFO:tasks.workunit.client.0.smithi112.stdout:removed './205/28' 2023-01-24T03:35:16.044 INFO:tasks.workunit.client.0.smithi112.stdout:removed './205/24' 2023-01-24T03:35:16.044 INFO:tasks.workunit.client.0.smithi112.stdout:removed './205/29' 2023-01-24T03:35:16.044 INFO:tasks.workunit.client.0.smithi112.stdout:removed './205/32' 2023-01-24T03:35:16.044 INFO:tasks.workunit.client.0.smithi112.stdout:removed './205/22' 2023-01-24T03:35:16.045 INFO:tasks.workunit.client.0.smithi112.stdout:removed './205/25' 2023-01-24T03:35:16.045 INFO:tasks.workunit.client.0.smithi112.stdout:removed './205/30' 2023-01-24T03:35:16.045 INFO:tasks.workunit.client.0.smithi112.stdout:removed './205/33' 2023-01-24T03:35:16.045 INFO:tasks.workunit.client.0.smithi112.stdout:removed './205/23' 2023-01-24T03:35:16.045 INFO:tasks.workunit.client.0.smithi112.stdout:removed './205/35' 2023-01-24T03:35:16.046 INFO:tasks.workunit.client.0.smithi112.stdout:removed './205/26' 2023-01-24T03:35:16.046 INFO:tasks.workunit.client.0.smithi112.stdout:removed './205/37' 2023-01-24T03:35:16.046 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './205' 2023-01-24T03:35:16.046 INFO:tasks.workunit.client.0.smithi112.stdout:removed './206/21' 2023-01-24T03:35:16.046 INFO:tasks.workunit.client.0.smithi112.stdout:removed './206/27' 2023-01-24T03:35:16.047 INFO:tasks.workunit.client.0.smithi112.stdout:removed './206/31' 2023-01-24T03:35:16.047 INFO:tasks.workunit.client.0.smithi112.stdout:removed './206/36' 2023-01-24T03:35:16.047 INFO:tasks.workunit.client.0.smithi112.stdout:removed './206/20' 2023-01-24T03:35:16.047 INFO:tasks.workunit.client.0.smithi112.stdout:removed './206/39' 2023-01-24T03:35:17.031 INFO:tasks.workunit.client.0.smithi112.stdout:removed './206/34' 2023-01-24T03:35:17.031 INFO:tasks.workunit.client.0.smithi112.stdout:removed './206/38' 2023-01-24T03:35:17.031 INFO:tasks.workunit.client.0.smithi112.stdout:removed './206/28' 2023-01-24T03:35:17.031 INFO:tasks.workunit.client.0.smithi112.stdout:removed './206/24' 2023-01-24T03:35:17.032 INFO:tasks.workunit.client.0.smithi112.stdout:removed './206/29' 2023-01-24T03:35:17.032 INFO:tasks.workunit.client.0.smithi112.stdout:removed './206/32' 2023-01-24T03:35:17.032 INFO:tasks.workunit.client.0.smithi112.stdout:removed './206/22' 2023-01-24T03:35:17.032 INFO:tasks.workunit.client.0.smithi112.stdout:removed './206/25' 2023-01-24T03:35:17.033 INFO:tasks.workunit.client.0.smithi112.stdout:removed './206/30' 2023-01-24T03:35:17.033 INFO:tasks.workunit.client.0.smithi112.stdout:removed './206/33' 2023-01-24T03:35:17.033 INFO:tasks.workunit.client.0.smithi112.stdout:removed './206/23' 2023-01-24T03:35:17.033 INFO:tasks.workunit.client.0.smithi112.stdout:removed './206/35' 2023-01-24T03:35:17.033 INFO:tasks.workunit.client.0.smithi112.stdout:removed './206/26' 2023-01-24T03:35:17.033 INFO:tasks.workunit.client.0.smithi112.stdout:removed './206/37' 2023-01-24T03:35:17.034 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './206' 2023-01-24T03:35:17.034 INFO:tasks.workunit.client.0.smithi112.stdout:removed './207/21' 2023-01-24T03:35:17.034 INFO:tasks.workunit.client.0.smithi112.stdout:removed './207/27' 2023-01-24T03:35:17.034 INFO:tasks.workunit.client.0.smithi112.stdout:removed './207/31' 2023-01-24T03:35:17.034 INFO:tasks.workunit.client.0.smithi112.stdout:removed './207/36' 2023-01-24T03:35:17.034 INFO:tasks.workunit.client.0.smithi112.stdout:removed './207/20' 2023-01-24T03:35:17.035 INFO:tasks.workunit.client.0.smithi112.stdout:removed './207/39' 2023-01-24T03:35:17.035 INFO:tasks.workunit.client.0.smithi112.stdout:removed './207/34' 2023-01-24T03:35:17.035 INFO:tasks.workunit.client.0.smithi112.stdout:removed './207/38' 2023-01-24T03:35:17.035 INFO:tasks.workunit.client.0.smithi112.stdout:removed './207/28' 2023-01-24T03:35:17.035 INFO:tasks.workunit.client.0.smithi112.stdout:removed './207/24' 2023-01-24T03:35:17.036 INFO:tasks.workunit.client.0.smithi112.stdout:removed './207/29' 2023-01-24T03:35:17.036 INFO:tasks.workunit.client.0.smithi112.stdout:removed './207/32' 2023-01-24T03:35:17.036 INFO:tasks.workunit.client.0.smithi112.stdout:removed './207/22' 2023-01-24T03:35:17.036 INFO:tasks.workunit.client.0.smithi112.stdout:removed './207/25' 2023-01-24T03:35:17.036 INFO:tasks.workunit.client.0.smithi112.stdout:removed './207/30' 2023-01-24T03:35:17.037 INFO:tasks.workunit.client.0.smithi112.stdout:removed './207/33' 2023-01-24T03:35:17.037 INFO:tasks.workunit.client.0.smithi112.stdout:removed './207/23' 2023-01-24T03:35:17.037 INFO:tasks.workunit.client.0.smithi112.stdout:removed './207/35' 2023-01-24T03:35:17.037 INFO:tasks.workunit.client.0.smithi112.stdout:removed './207/26' 2023-01-24T03:35:17.037 INFO:tasks.workunit.client.0.smithi112.stdout:removed './207/37' 2023-01-24T03:35:17.038 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './207' 2023-01-24T03:35:17.038 INFO:tasks.workunit.client.0.smithi112.stdout:removed './208/21' 2023-01-24T03:35:17.038 INFO:tasks.workunit.client.0.smithi112.stdout:removed './208/27' 2023-01-24T03:35:17.038 INFO:tasks.workunit.client.0.smithi112.stdout:removed './208/31' 2023-01-24T03:35:17.038 INFO:tasks.workunit.client.0.smithi112.stdout:removed './208/36' 2023-01-24T03:35:17.039 INFO:tasks.workunit.client.0.smithi112.stdout:removed './208/20' 2023-01-24T03:35:17.039 INFO:tasks.workunit.client.0.smithi112.stdout:removed './208/39' 2023-01-24T03:35:17.039 INFO:tasks.workunit.client.0.smithi112.stdout:removed './208/34' 2023-01-24T03:35:17.039 INFO:tasks.workunit.client.0.smithi112.stdout:removed './208/38' 2023-01-24T03:35:17.040 INFO:tasks.workunit.client.0.smithi112.stdout:removed './208/28' 2023-01-24T03:35:17.040 INFO:tasks.workunit.client.0.smithi112.stdout:removed './208/24' 2023-01-24T03:35:17.040 INFO:tasks.workunit.client.0.smithi112.stdout:removed './208/29' 2023-01-24T03:35:17.040 INFO:tasks.workunit.client.0.smithi112.stdout:removed './208/32' 2023-01-24T03:35:17.041 INFO:tasks.workunit.client.0.smithi112.stdout:removed './208/22' 2023-01-24T03:35:17.041 INFO:tasks.workunit.client.0.smithi112.stdout:removed './208/25' 2023-01-24T03:35:17.041 INFO:tasks.workunit.client.0.smithi112.stdout:removed './208/30' 2023-01-24T03:35:17.041 INFO:tasks.workunit.client.0.smithi112.stdout:removed './208/33' 2023-01-24T03:35:17.041 INFO:tasks.workunit.client.0.smithi112.stdout:removed './208/23' 2023-01-24T03:35:17.042 INFO:tasks.workunit.client.0.smithi112.stdout:removed './208/35' 2023-01-24T03:35:17.042 INFO:tasks.workunit.client.0.smithi112.stdout:removed './208/26' 2023-01-24T03:35:17.042 INFO:tasks.workunit.client.0.smithi112.stdout:removed './208/37' 2023-01-24T03:35:17.042 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './208' 2023-01-24T03:35:17.042 INFO:tasks.workunit.client.0.smithi112.stdout:removed './209/21' 2023-01-24T03:35:17.043 INFO:tasks.workunit.client.0.smithi112.stdout:removed './209/27' 2023-01-24T03:35:17.043 INFO:tasks.workunit.client.0.smithi112.stdout:removed './209/31' 2023-01-24T03:35:17.043 INFO:tasks.workunit.client.0.smithi112.stdout:removed './209/36' 2023-01-24T03:35:17.043 INFO:tasks.workunit.client.0.smithi112.stdout:removed './209/20' 2023-01-24T03:35:17.043 INFO:tasks.workunit.client.0.smithi112.stdout:removed './209/39' 2023-01-24T03:35:17.043 INFO:tasks.workunit.client.0.smithi112.stdout:removed './209/34' 2023-01-24T03:35:17.044 INFO:tasks.workunit.client.0.smithi112.stdout:removed './209/38' 2023-01-24T03:35:17.044 INFO:tasks.workunit.client.0.smithi112.stdout:removed './209/28' 2023-01-24T03:35:17.044 INFO:tasks.workunit.client.0.smithi112.stdout:removed './209/24' 2023-01-24T03:35:17.044 INFO:tasks.workunit.client.0.smithi112.stdout:removed './209/29' 2023-01-24T03:35:17.044 INFO:tasks.workunit.client.0.smithi112.stdout:removed './209/32' 2023-01-24T03:35:17.045 INFO:tasks.workunit.client.0.smithi112.stdout:removed './209/22' 2023-01-24T03:35:17.045 INFO:tasks.workunit.client.0.smithi112.stdout:removed './209/25' 2023-01-24T03:35:17.045 INFO:tasks.workunit.client.0.smithi112.stdout:removed './209/30' 2023-01-24T03:35:17.045 INFO:tasks.workunit.client.0.smithi112.stdout:removed './209/33' 2023-01-24T03:35:17.045 INFO:tasks.workunit.client.0.smithi112.stdout:removed './209/23' 2023-01-24T03:35:17.046 INFO:tasks.workunit.client.0.smithi112.stdout:removed './209/35' 2023-01-24T03:35:17.046 INFO:tasks.workunit.client.0.smithi112.stdout:removed './209/26' 2023-01-24T03:35:17.046 INFO:tasks.workunit.client.0.smithi112.stdout:removed './209/37' 2023-01-24T03:35:17.046 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './209' 2023-01-24T03:35:17.046 INFO:tasks.workunit.client.0.smithi112.stdout:removed './210/21' 2023-01-24T03:35:17.047 INFO:tasks.workunit.client.0.smithi112.stdout:removed './210/27' 2023-01-24T03:35:17.047 INFO:tasks.workunit.client.0.smithi112.stdout:removed './210/31' 2023-01-24T03:35:17.047 INFO:tasks.workunit.client.0.smithi112.stdout:removed './210/36' 2023-01-24T03:35:17.047 INFO:tasks.workunit.client.0.smithi112.stdout:removed './210/20' 2023-01-24T03:35:17.047 INFO:tasks.workunit.client.0.smithi112.stdout:removed './210/39' 2023-01-24T03:35:17.047 INFO:tasks.workunit.client.0.smithi112.stdout:removed './210/34' 2023-01-24T03:35:17.048 INFO:tasks.workunit.client.0.smithi112.stdout:removed './210/38' 2023-01-24T03:35:17.048 INFO:tasks.workunit.client.0.smithi112.stdout:removed './210/28' 2023-01-24T03:35:17.048 INFO:tasks.workunit.client.0.smithi112.stdout:removed './210/24' 2023-01-24T03:35:17.049 INFO:tasks.workunit.client.0.smithi112.stdout:removed './210/29' 2023-01-24T03:35:17.049 INFO:tasks.workunit.client.0.smithi112.stdout:removed './210/32' 2023-01-24T03:35:17.049 INFO:tasks.workunit.client.0.smithi112.stdout:removed './210/22' 2023-01-24T03:35:17.049 INFO:tasks.workunit.client.0.smithi112.stdout:removed './210/25' 2023-01-24T03:35:17.050 INFO:tasks.workunit.client.0.smithi112.stdout:removed './210/30' 2023-01-24T03:35:17.050 INFO:tasks.workunit.client.0.smithi112.stdout:removed './210/33' 2023-01-24T03:35:17.050 INFO:tasks.workunit.client.0.smithi112.stdout:removed './210/23' 2023-01-24T03:35:17.050 INFO:tasks.workunit.client.0.smithi112.stdout:removed './210/35' 2023-01-24T03:35:17.051 INFO:tasks.workunit.client.0.smithi112.stdout:removed './210/26' 2023-01-24T03:35:17.051 INFO:tasks.workunit.client.0.smithi112.stdout:removed './210/37' 2023-01-24T03:35:17.051 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './210' 2023-01-24T03:35:17.051 INFO:tasks.workunit.client.0.smithi112.stdout:removed './211/21' 2023-01-24T03:35:17.051 INFO:tasks.workunit.client.0.smithi112.stdout:removed './211/27' 2023-01-24T03:35:17.052 INFO:tasks.workunit.client.0.smithi112.stdout:removed './211/31' 2023-01-24T03:35:17.052 INFO:tasks.workunit.client.0.smithi112.stdout:removed './211/36' 2023-01-24T03:35:17.052 INFO:tasks.workunit.client.0.smithi112.stdout:removed './211/20' 2023-01-24T03:35:17.052 INFO:tasks.workunit.client.0.smithi112.stdout:removed './211/39' 2023-01-24T03:35:17.052 INFO:tasks.workunit.client.0.smithi112.stdout:removed './211/34' 2023-01-24T03:35:17.053 INFO:tasks.workunit.client.0.smithi112.stdout:removed './211/38' 2023-01-24T03:35:17.053 INFO:tasks.workunit.client.0.smithi112.stdout:removed './211/28' 2023-01-24T03:35:17.053 INFO:tasks.workunit.client.0.smithi112.stdout:removed './211/24' 2023-01-24T03:35:17.053 INFO:tasks.workunit.client.0.smithi112.stdout:removed './211/29' 2023-01-24T03:35:17.053 INFO:tasks.workunit.client.0.smithi112.stdout:removed './211/32' 2023-01-24T03:35:17.053 INFO:tasks.workunit.client.0.smithi112.stdout:removed './211/22' 2023-01-24T03:35:17.054 INFO:tasks.workunit.client.0.smithi112.stdout:removed './211/25' 2023-01-24T03:35:17.054 INFO:tasks.workunit.client.0.smithi112.stdout:removed './211/30' 2023-01-24T03:35:17.054 INFO:tasks.workunit.client.0.smithi112.stdout:removed './211/33' 2023-01-24T03:35:17.054 INFO:tasks.workunit.client.0.smithi112.stdout:removed './211/23' 2023-01-24T03:35:17.054 INFO:tasks.workunit.client.0.smithi112.stdout:removed './211/35' 2023-01-24T03:35:17.055 INFO:tasks.workunit.client.0.smithi112.stdout:removed './211/26' 2023-01-24T03:35:17.055 INFO:tasks.workunit.client.0.smithi112.stdout:removed './211/37' 2023-01-24T03:35:17.055 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './211' 2023-01-24T03:35:17.055 INFO:tasks.workunit.client.0.smithi112.stdout:removed './212/21' 2023-01-24T03:35:17.055 INFO:tasks.workunit.client.0.smithi112.stdout:removed './212/27' 2023-01-24T03:35:17.056 INFO:tasks.workunit.client.0.smithi112.stdout:removed './212/31' 2023-01-24T03:35:17.056 INFO:tasks.workunit.client.0.smithi112.stdout:removed './212/36' 2023-01-24T03:35:17.056 INFO:tasks.workunit.client.0.smithi112.stdout:removed './212/20' 2023-01-24T03:35:17.056 INFO:tasks.workunit.client.0.smithi112.stdout:removed './212/39' 2023-01-24T03:35:17.056 INFO:tasks.workunit.client.0.smithi112.stdout:removed './212/34' 2023-01-24T03:35:17.056 INFO:tasks.workunit.client.0.smithi112.stdout:removed './212/38' 2023-01-24T03:35:17.057 INFO:tasks.workunit.client.0.smithi112.stdout:removed './212/28' 2023-01-24T03:35:17.057 INFO:tasks.workunit.client.0.smithi112.stdout:removed './212/24' 2023-01-24T03:35:17.057 INFO:tasks.workunit.client.0.smithi112.stdout:removed './212/29' 2023-01-24T03:35:17.057 INFO:tasks.workunit.client.0.smithi112.stdout:removed './212/32' 2023-01-24T03:35:17.057 INFO:tasks.workunit.client.0.smithi112.stdout:removed './212/22' 2023-01-24T03:35:17.058 INFO:tasks.workunit.client.0.smithi112.stdout:removed './212/25' 2023-01-24T03:35:17.058 INFO:tasks.workunit.client.0.smithi112.stdout:removed './212/30' 2023-01-24T03:35:17.058 INFO:tasks.workunit.client.0.smithi112.stdout:removed './212/33' 2023-01-24T03:35:17.058 INFO:tasks.workunit.client.0.smithi112.stdout:removed './212/23' 2023-01-24T03:35:17.058 INFO:tasks.workunit.client.0.smithi112.stdout:removed './212/35' 2023-01-24T03:35:17.058 INFO:tasks.workunit.client.0.smithi112.stdout:removed './212/26' 2023-01-24T03:35:17.059 INFO:tasks.workunit.client.0.smithi112.stdout:removed './212/37' 2023-01-24T03:35:17.059 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './212' 2023-01-24T03:35:17.059 INFO:tasks.workunit.client.0.smithi112.stdout:removed './213/21' 2023-01-24T03:35:17.059 INFO:tasks.workunit.client.0.smithi112.stdout:removed './213/27' 2023-01-24T03:35:17.059 INFO:tasks.workunit.client.0.smithi112.stdout:removed './213/31' 2023-01-24T03:35:17.060 INFO:tasks.workunit.client.0.smithi112.stdout:removed './213/36' 2023-01-24T03:35:17.060 INFO:tasks.workunit.client.0.smithi112.stdout:removed './213/20' 2023-01-24T03:35:17.060 INFO:tasks.workunit.client.0.smithi112.stdout:removed './213/39' 2023-01-24T03:35:17.060 INFO:tasks.workunit.client.0.smithi112.stdout:removed './213/34' 2023-01-24T03:35:17.060 INFO:tasks.workunit.client.0.smithi112.stdout:removed './213/38' 2023-01-24T03:35:17.061 INFO:tasks.workunit.client.0.smithi112.stdout:removed './213/28' 2023-01-24T03:35:17.061 INFO:tasks.workunit.client.0.smithi112.stdout:removed './213/24' 2023-01-24T03:35:17.061 INFO:tasks.workunit.client.0.smithi112.stdout:removed './213/29' 2023-01-24T03:35:17.061 INFO:tasks.workunit.client.0.smithi112.stdout:removed './213/32' 2023-01-24T03:35:17.062 INFO:tasks.workunit.client.0.smithi112.stdout:removed './213/22' 2023-01-24T03:35:17.062 INFO:tasks.workunit.client.0.smithi112.stdout:removed './213/25' 2023-01-24T03:35:17.062 INFO:tasks.workunit.client.0.smithi112.stdout:removed './213/30' 2023-01-24T03:35:17.062 INFO:tasks.workunit.client.0.smithi112.stdout:removed './213/33' 2023-01-24T03:35:17.062 INFO:tasks.workunit.client.0.smithi112.stdout:removed './213/23' 2023-01-24T03:35:17.063 INFO:tasks.workunit.client.0.smithi112.stdout:removed './213/35' 2023-01-24T03:35:17.063 INFO:tasks.workunit.client.0.smithi112.stdout:removed './213/26' 2023-01-24T03:35:17.063 INFO:tasks.workunit.client.0.smithi112.stdout:removed './213/37' 2023-01-24T03:35:17.063 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './213' 2023-01-24T03:35:17.063 INFO:tasks.workunit.client.0.smithi112.stdout:removed './214/21' 2023-01-24T03:35:17.063 INFO:tasks.workunit.client.0.smithi112.stdout:removed './214/27' 2023-01-24T03:35:17.064 INFO:tasks.workunit.client.0.smithi112.stdout:removed './214/31' 2023-01-24T03:35:17.064 INFO:tasks.workunit.client.0.smithi112.stdout:removed './214/36' 2023-01-24T03:35:17.064 INFO:tasks.workunit.client.0.smithi112.stdout:removed './214/20' 2023-01-24T03:35:17.064 INFO:tasks.workunit.client.0.smithi112.stdout:removed './214/39' 2023-01-24T03:35:17.064 INFO:tasks.workunit.client.0.smithi112.stdout:removed './214/34' 2023-01-24T03:35:17.065 INFO:tasks.workunit.client.0.smithi112.stdout:removed './214/38' 2023-01-24T03:35:17.065 INFO:tasks.workunit.client.0.smithi112.stdout:removed './214/28' 2023-01-24T03:35:17.065 INFO:tasks.workunit.client.0.smithi112.stdout:removed './214/24' 2023-01-24T03:35:17.065 INFO:tasks.workunit.client.0.smithi112.stdout:removed './214/29' 2023-01-24T03:35:17.065 INFO:tasks.workunit.client.0.smithi112.stdout:removed './214/32' 2023-01-24T03:35:17.065 INFO:tasks.workunit.client.0.smithi112.stdout:removed './214/22' 2023-01-24T03:35:17.066 INFO:tasks.workunit.client.0.smithi112.stdout:removed './214/25' 2023-01-24T03:35:17.066 INFO:tasks.workunit.client.0.smithi112.stdout:removed './214/30' 2023-01-24T03:35:17.066 INFO:tasks.workunit.client.0.smithi112.stdout:removed './214/33' 2023-01-24T03:35:17.066 INFO:tasks.workunit.client.0.smithi112.stdout:removed './214/23' 2023-01-24T03:35:17.066 INFO:tasks.workunit.client.0.smithi112.stdout:removed './214/35' 2023-01-24T03:35:17.066 INFO:tasks.workunit.client.0.smithi112.stdout:removed './214/26' 2023-01-24T03:35:17.067 INFO:tasks.workunit.client.0.smithi112.stdout:removed './214/37' 2023-01-24T03:35:17.067 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './214' 2023-01-24T03:35:17.067 INFO:tasks.workunit.client.0.smithi112.stdout:removed './215/21' 2023-01-24T03:35:17.067 INFO:tasks.workunit.client.0.smithi112.stdout:removed './215/27' 2023-01-24T03:35:17.067 INFO:tasks.workunit.client.0.smithi112.stdout:removed './215/31' 2023-01-24T03:35:17.067 INFO:tasks.workunit.client.0.smithi112.stdout:removed './215/36' 2023-01-24T03:35:17.068 INFO:tasks.workunit.client.0.smithi112.stdout:removed './215/20' 2023-01-24T03:35:17.068 INFO:tasks.workunit.client.0.smithi112.stdout:removed './215/39' 2023-01-24T03:35:17.068 INFO:tasks.workunit.client.0.smithi112.stdout:removed './215/34' 2023-01-24T03:35:17.068 INFO:tasks.workunit.client.0.smithi112.stdout:removed './215/38' 2023-01-24T03:35:17.068 INFO:tasks.workunit.client.0.smithi112.stdout:removed './215/28' 2023-01-24T03:35:17.068 INFO:tasks.workunit.client.0.smithi112.stdout:removed './215/24' 2023-01-24T03:35:17.069 INFO:tasks.workunit.client.0.smithi112.stdout:removed './215/29' 2023-01-24T03:35:17.069 INFO:tasks.workunit.client.0.smithi112.stdout:removed './215/32' 2023-01-24T03:35:17.069 INFO:tasks.workunit.client.0.smithi112.stdout:removed './215/22' 2023-01-24T03:35:17.069 INFO:tasks.workunit.client.0.smithi112.stdout:removed './215/25' 2023-01-24T03:35:17.069 INFO:tasks.workunit.client.0.smithi112.stdout:removed './215/30' 2023-01-24T03:35:17.070 INFO:tasks.workunit.client.0.smithi112.stdout:removed './215/33' 2023-01-24T03:35:17.070 INFO:tasks.workunit.client.0.smithi112.stdout:removed './215/23' 2023-01-24T03:35:17.070 INFO:tasks.workunit.client.0.smithi112.stdout:removed './215/35' 2023-01-24T03:35:17.070 INFO:tasks.workunit.client.0.smithi112.stdout:removed './215/26' 2023-01-24T03:35:17.070 INFO:tasks.workunit.client.0.smithi112.stdout:removed './215/37' 2023-01-24T03:35:17.071 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './215' 2023-01-24T03:35:17.071 INFO:tasks.workunit.client.0.smithi112.stdout:removed './216/21' 2023-01-24T03:35:17.071 INFO:tasks.workunit.client.0.smithi112.stdout:removed './216/27' 2023-01-24T03:35:17.071 INFO:tasks.workunit.client.0.smithi112.stdout:removed './216/31' 2023-01-24T03:35:17.071 INFO:tasks.workunit.client.0.smithi112.stdout:removed './216/36' 2023-01-24T03:35:17.072 INFO:tasks.workunit.client.0.smithi112.stdout:removed './216/20' 2023-01-24T03:35:17.072 INFO:tasks.workunit.client.0.smithi112.stdout:removed './216/39' 2023-01-24T03:35:17.072 INFO:tasks.workunit.client.0.smithi112.stdout:removed './216/34' 2023-01-24T03:35:17.072 INFO:tasks.workunit.client.0.smithi112.stdout:removed './216/38' 2023-01-24T03:35:18.327 INFO:tasks.workunit.client.0.smithi112.stdout:removed './216/28' 2023-01-24T03:35:18.327 INFO:tasks.workunit.client.0.smithi112.stdout:removed './216/24' 2023-01-24T03:35:18.327 INFO:tasks.workunit.client.0.smithi112.stdout:removed './216/29' 2023-01-24T03:35:18.328 INFO:tasks.workunit.client.0.smithi112.stdout:removed './216/32' 2023-01-24T03:35:18.328 INFO:tasks.workunit.client.0.smithi112.stdout:removed './216/22' 2023-01-24T03:35:18.328 INFO:tasks.workunit.client.0.smithi112.stdout:removed './216/25' 2023-01-24T03:35:18.328 INFO:tasks.workunit.client.0.smithi112.stdout:removed './216/30' 2023-01-24T03:35:18.328 INFO:tasks.workunit.client.0.smithi112.stdout:removed './216/33' 2023-01-24T03:35:18.329 INFO:tasks.workunit.client.0.smithi112.stdout:removed './216/23' 2023-01-24T03:35:18.329 INFO:tasks.workunit.client.0.smithi112.stdout:removed './216/35' 2023-01-24T03:35:18.329 INFO:tasks.workunit.client.0.smithi112.stdout:removed './216/26' 2023-01-24T03:35:18.329 INFO:tasks.workunit.client.0.smithi112.stdout:removed './216/37' 2023-01-24T03:35:18.329 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './216' 2023-01-24T03:35:18.329 INFO:tasks.workunit.client.0.smithi112.stdout:removed './217/21' 2023-01-24T03:35:18.330 INFO:tasks.workunit.client.0.smithi112.stdout:removed './217/27' 2023-01-24T03:35:18.330 INFO:tasks.workunit.client.0.smithi112.stdout:removed './217/31' 2023-01-24T03:35:18.330 INFO:tasks.workunit.client.0.smithi112.stdout:removed './217/36' 2023-01-24T03:35:18.330 INFO:tasks.workunit.client.0.smithi112.stdout:removed './217/20' 2023-01-24T03:35:18.330 INFO:tasks.workunit.client.0.smithi112.stdout:removed './217/39' 2023-01-24T03:35:18.331 INFO:tasks.workunit.client.0.smithi112.stdout:removed './217/34' 2023-01-24T03:35:18.331 INFO:tasks.workunit.client.0.smithi112.stdout:removed './217/38' 2023-01-24T03:35:18.331 INFO:tasks.workunit.client.0.smithi112.stdout:removed './217/28' 2023-01-24T03:35:18.331 INFO:tasks.workunit.client.0.smithi112.stdout:removed './217/24' 2023-01-24T03:35:18.332 INFO:tasks.workunit.client.0.smithi112.stdout:removed './217/29' 2023-01-24T03:35:18.332 INFO:tasks.workunit.client.0.smithi112.stdout:removed './217/32' 2023-01-24T03:35:18.332 INFO:tasks.workunit.client.0.smithi112.stdout:removed './217/22' 2023-01-24T03:35:18.332 INFO:tasks.workunit.client.0.smithi112.stdout:removed './217/25' 2023-01-24T03:35:18.332 INFO:tasks.workunit.client.0.smithi112.stdout:removed './217/30' 2023-01-24T03:35:18.333 INFO:tasks.workunit.client.0.smithi112.stdout:removed './217/33' 2023-01-24T03:35:18.333 INFO:tasks.workunit.client.0.smithi112.stdout:removed './217/23' 2023-01-24T03:35:18.333 INFO:tasks.workunit.client.0.smithi112.stdout:removed './217/35' 2023-01-24T03:35:18.333 INFO:tasks.workunit.client.0.smithi112.stdout:removed './217/26' 2023-01-24T03:35:18.333 INFO:tasks.workunit.client.0.smithi112.stdout:removed './217/37' 2023-01-24T03:35:18.334 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './217' 2023-01-24T03:35:18.334 INFO:tasks.workunit.client.0.smithi112.stdout:removed './218/21' 2023-01-24T03:35:18.334 INFO:tasks.workunit.client.0.smithi112.stdout:removed './218/27' 2023-01-24T03:35:18.334 INFO:tasks.workunit.client.0.smithi112.stdout:removed './218/31' 2023-01-24T03:35:18.335 INFO:tasks.workunit.client.0.smithi112.stdout:removed './218/36' 2023-01-24T03:35:18.335 INFO:tasks.workunit.client.0.smithi112.stdout:removed './218/20' 2023-01-24T03:35:18.335 INFO:tasks.workunit.client.0.smithi112.stdout:removed './218/39' 2023-01-24T03:35:18.335 INFO:tasks.workunit.client.0.smithi112.stdout:removed './218/34' 2023-01-24T03:35:18.335 INFO:tasks.workunit.client.0.smithi112.stdout:removed './218/38' 2023-01-24T03:35:18.336 INFO:tasks.workunit.client.0.smithi112.stdout:removed './218/28' 2023-01-24T03:35:18.336 INFO:tasks.workunit.client.0.smithi112.stdout:removed './218/24' 2023-01-24T03:35:18.336 INFO:tasks.workunit.client.0.smithi112.stdout:removed './218/29' 2023-01-24T03:35:18.336 INFO:tasks.workunit.client.0.smithi112.stdout:removed './218/32' 2023-01-24T03:35:18.336 INFO:tasks.workunit.client.0.smithi112.stdout:removed './218/22' 2023-01-24T03:35:18.337 INFO:tasks.workunit.client.0.smithi112.stdout:removed './218/25' 2023-01-24T03:35:18.337 INFO:tasks.workunit.client.0.smithi112.stdout:removed './218/30' 2023-01-24T03:35:18.337 INFO:tasks.workunit.client.0.smithi112.stdout:removed './218/33' 2023-01-24T03:35:18.337 INFO:tasks.workunit.client.0.smithi112.stdout:removed './218/23' 2023-01-24T03:35:18.337 INFO:tasks.workunit.client.0.smithi112.stdout:removed './218/35' 2023-01-24T03:35:18.338 INFO:tasks.workunit.client.0.smithi112.stdout:removed './218/26' 2023-01-24T03:35:18.338 INFO:tasks.workunit.client.0.smithi112.stdout:removed './218/37' 2023-01-24T03:35:18.338 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './218' 2023-01-24T03:35:18.338 INFO:tasks.workunit.client.0.smithi112.stdout:removed './219/21' 2023-01-24T03:35:18.338 INFO:tasks.workunit.client.0.smithi112.stdout:removed './219/27' 2023-01-24T03:35:18.339 INFO:tasks.workunit.client.0.smithi112.stdout:removed './219/31' 2023-01-24T03:35:18.339 INFO:tasks.workunit.client.0.smithi112.stdout:removed './219/36' 2023-01-24T03:35:18.339 INFO:tasks.workunit.client.0.smithi112.stdout:removed './219/20' 2023-01-24T03:35:18.339 INFO:tasks.workunit.client.0.smithi112.stdout:removed './219/39' 2023-01-24T03:35:18.339 INFO:tasks.workunit.client.0.smithi112.stdout:removed './219/34' 2023-01-24T03:35:18.339 INFO:tasks.workunit.client.0.smithi112.stdout:removed './219/38' 2023-01-24T03:35:18.340 INFO:tasks.workunit.client.0.smithi112.stdout:removed './219/28' 2023-01-24T03:35:18.340 INFO:tasks.workunit.client.0.smithi112.stdout:removed './219/24' 2023-01-24T03:35:18.340 INFO:tasks.workunit.client.0.smithi112.stdout:removed './219/29' 2023-01-24T03:35:18.340 INFO:tasks.workunit.client.0.smithi112.stdout:removed './219/32' 2023-01-24T03:35:18.340 INFO:tasks.workunit.client.0.smithi112.stdout:removed './219/22' 2023-01-24T03:35:18.341 INFO:tasks.workunit.client.0.smithi112.stdout:removed './219/25' 2023-01-24T03:35:18.341 INFO:tasks.workunit.client.0.smithi112.stdout:removed './219/30' 2023-01-24T03:35:18.341 INFO:tasks.workunit.client.0.smithi112.stdout:removed './219/33' 2023-01-24T03:35:18.341 INFO:tasks.workunit.client.0.smithi112.stdout:removed './219/23' 2023-01-24T03:35:18.341 INFO:tasks.workunit.client.0.smithi112.stdout:removed './219/35' 2023-01-24T03:35:18.342 INFO:tasks.workunit.client.0.smithi112.stdout:removed './219/26' 2023-01-24T03:35:18.342 INFO:tasks.workunit.client.0.smithi112.stdout:removed './219/37' 2023-01-24T03:35:18.342 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './219' 2023-01-24T03:35:18.342 INFO:tasks.workunit.client.0.smithi112.stdout:removed './220/21' 2023-01-24T03:35:18.342 INFO:tasks.workunit.client.0.smithi112.stdout:removed './220/27' 2023-01-24T03:35:18.343 INFO:tasks.workunit.client.0.smithi112.stdout:removed './220/31' 2023-01-24T03:35:18.343 INFO:tasks.workunit.client.0.smithi112.stdout:removed './220/36' 2023-01-24T03:35:18.343 INFO:tasks.workunit.client.0.smithi112.stdout:removed './220/20' 2023-01-24T03:35:18.343 INFO:tasks.workunit.client.0.smithi112.stdout:removed './220/39' 2023-01-24T03:35:18.344 INFO:tasks.workunit.client.0.smithi112.stdout:removed './220/34' 2023-01-24T03:35:18.344 INFO:tasks.workunit.client.0.smithi112.stdout:removed './220/38' 2023-01-24T03:35:18.344 INFO:tasks.workunit.client.0.smithi112.stdout:removed './220/28' 2023-01-24T03:35:18.344 INFO:tasks.workunit.client.0.smithi112.stdout:removed './220/24' 2023-01-24T03:35:18.344 INFO:tasks.workunit.client.0.smithi112.stdout:removed './220/29' 2023-01-24T03:35:18.345 INFO:tasks.workunit.client.0.smithi112.stdout:removed './220/32' 2023-01-24T03:35:18.345 INFO:tasks.workunit.client.0.smithi112.stdout:removed './220/22' 2023-01-24T03:35:18.345 INFO:tasks.workunit.client.0.smithi112.stdout:removed './220/25' 2023-01-24T03:35:18.345 INFO:tasks.workunit.client.0.smithi112.stdout:removed './220/30' 2023-01-24T03:35:18.345 INFO:tasks.workunit.client.0.smithi112.stdout:removed './220/33' 2023-01-24T03:35:18.346 INFO:tasks.workunit.client.0.smithi112.stdout:removed './220/23' 2023-01-24T03:35:18.346 INFO:tasks.workunit.client.0.smithi112.stdout:removed './220/35' 2023-01-24T03:35:18.346 INFO:tasks.workunit.client.0.smithi112.stdout:removed './220/26' 2023-01-24T03:35:18.346 INFO:tasks.workunit.client.0.smithi112.stdout:removed './220/37' 2023-01-24T03:35:18.346 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './220' 2023-01-24T03:35:18.347 INFO:tasks.workunit.client.0.smithi112.stdout:removed './221/21' 2023-01-24T03:35:18.347 INFO:tasks.workunit.client.0.smithi112.stdout:removed './221/27' 2023-01-24T03:35:18.347 INFO:tasks.workunit.client.0.smithi112.stdout:removed './221/31' 2023-01-24T03:35:18.347 INFO:tasks.workunit.client.0.smithi112.stdout:removed './221/36' 2023-01-24T03:35:18.347 INFO:tasks.workunit.client.0.smithi112.stdout:removed './221/20' 2023-01-24T03:35:18.347 INFO:tasks.workunit.client.0.smithi112.stdout:removed './221/39' 2023-01-24T03:35:18.348 INFO:tasks.workunit.client.0.smithi112.stdout:removed './221/34' 2023-01-24T03:35:18.348 INFO:tasks.workunit.client.0.smithi112.stdout:removed './221/38' 2023-01-24T03:35:18.348 INFO:tasks.workunit.client.0.smithi112.stdout:removed './221/28' 2023-01-24T03:35:18.348 INFO:tasks.workunit.client.0.smithi112.stdout:removed './221/24' 2023-01-24T03:35:18.348 INFO:tasks.workunit.client.0.smithi112.stdout:removed './221/29' 2023-01-24T03:35:18.349 INFO:tasks.workunit.client.0.smithi112.stdout:removed './221/32' 2023-01-24T03:35:18.349 INFO:tasks.workunit.client.0.smithi112.stdout:removed './221/22' 2023-01-24T03:35:18.349 INFO:tasks.workunit.client.0.smithi112.stdout:removed './221/25' 2023-01-24T03:35:18.349 INFO:tasks.workunit.client.0.smithi112.stdout:removed './221/30' 2023-01-24T03:35:18.349 INFO:tasks.workunit.client.0.smithi112.stdout:removed './221/33' 2023-01-24T03:35:18.350 INFO:tasks.workunit.client.0.smithi112.stdout:removed './221/23' 2023-01-24T03:35:18.350 INFO:tasks.workunit.client.0.smithi112.stdout:removed './221/35' 2023-01-24T03:35:18.350 INFO:tasks.workunit.client.0.smithi112.stdout:removed './221/26' 2023-01-24T03:35:18.350 INFO:tasks.workunit.client.0.smithi112.stdout:removed './221/37' 2023-01-24T03:35:18.350 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './221' 2023-01-24T03:35:18.351 INFO:tasks.workunit.client.0.smithi112.stdout:removed './222/21' 2023-01-24T03:35:18.351 INFO:tasks.workunit.client.0.smithi112.stdout:removed './222/27' 2023-01-24T03:35:18.351 INFO:tasks.workunit.client.0.smithi112.stdout:removed './222/31' 2023-01-24T03:35:18.351 INFO:tasks.workunit.client.0.smithi112.stdout:removed './222/36' 2023-01-24T03:35:18.351 INFO:tasks.workunit.client.0.smithi112.stdout:removed './222/20' 2023-01-24T03:35:18.352 INFO:tasks.workunit.client.0.smithi112.stdout:removed './222/39' 2023-01-24T03:35:18.352 INFO:tasks.workunit.client.0.smithi112.stdout:removed './222/34' 2023-01-24T03:35:18.352 INFO:tasks.workunit.client.0.smithi112.stdout:removed './222/38' 2023-01-24T03:35:18.352 INFO:tasks.workunit.client.0.smithi112.stdout:removed './222/28' 2023-01-24T03:35:18.352 INFO:tasks.workunit.client.0.smithi112.stdout:removed './222/24' 2023-01-24T03:35:18.352 INFO:tasks.workunit.client.0.smithi112.stdout:removed './222/29' 2023-01-24T03:35:18.353 INFO:tasks.workunit.client.0.smithi112.stdout:removed './222/32' 2023-01-24T03:35:18.353 INFO:tasks.workunit.client.0.smithi112.stdout:removed './222/22' 2023-01-24T03:35:18.353 INFO:tasks.workunit.client.0.smithi112.stdout:removed './222/25' 2023-01-24T03:35:18.353 INFO:tasks.workunit.client.0.smithi112.stdout:removed './222/30' 2023-01-24T03:35:18.353 INFO:tasks.workunit.client.0.smithi112.stdout:removed './222/33' 2023-01-24T03:35:18.354 INFO:tasks.workunit.client.0.smithi112.stdout:removed './222/23' 2023-01-24T03:35:18.354 INFO:tasks.workunit.client.0.smithi112.stdout:removed './222/35' 2023-01-24T03:35:18.354 INFO:tasks.workunit.client.0.smithi112.stdout:removed './222/26' 2023-01-24T03:35:18.354 INFO:tasks.workunit.client.0.smithi112.stdout:removed './222/37' 2023-01-24T03:35:18.354 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './222' 2023-01-24T03:35:18.355 INFO:tasks.workunit.client.0.smithi112.stdout:removed './223/21' 2023-01-24T03:35:18.355 INFO:tasks.workunit.client.0.smithi112.stdout:removed './223/27' 2023-01-24T03:35:18.355 INFO:tasks.workunit.client.0.smithi112.stdout:removed './223/31' 2023-01-24T03:35:18.355 INFO:tasks.workunit.client.0.smithi112.stdout:removed './223/36' 2023-01-24T03:35:18.355 INFO:tasks.workunit.client.0.smithi112.stdout:removed './223/20' 2023-01-24T03:35:18.356 INFO:tasks.workunit.client.0.smithi112.stdout:removed './223/39' 2023-01-24T03:35:18.356 INFO:tasks.workunit.client.0.smithi112.stdout:removed './223/34' 2023-01-24T03:35:18.356 INFO:tasks.workunit.client.0.smithi112.stdout:removed './223/38' 2023-01-24T03:35:18.356 INFO:tasks.workunit.client.0.smithi112.stdout:removed './223/28' 2023-01-24T03:35:18.356 INFO:tasks.workunit.client.0.smithi112.stdout:removed './223/24' 2023-01-24T03:35:18.357 INFO:tasks.workunit.client.0.smithi112.stdout:removed './223/29' 2023-01-24T03:35:18.357 INFO:tasks.workunit.client.0.smithi112.stdout:removed './223/32' 2023-01-24T03:35:18.357 INFO:tasks.workunit.client.0.smithi112.stdout:removed './223/22' 2023-01-24T03:35:18.357 INFO:tasks.workunit.client.0.smithi112.stdout:removed './223/25' 2023-01-24T03:35:18.357 INFO:tasks.workunit.client.0.smithi112.stdout:removed './223/30' 2023-01-24T03:35:18.358 INFO:tasks.workunit.client.0.smithi112.stdout:removed './223/33' 2023-01-24T03:35:18.358 INFO:tasks.workunit.client.0.smithi112.stdout:removed './223/23' 2023-01-24T03:35:18.358 INFO:tasks.workunit.client.0.smithi112.stdout:removed './223/35' 2023-01-24T03:35:18.358 INFO:tasks.workunit.client.0.smithi112.stdout:removed './223/26' 2023-01-24T03:35:18.358 INFO:tasks.workunit.client.0.smithi112.stdout:removed './223/37' 2023-01-24T03:35:18.359 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './223' 2023-01-24T03:35:18.359 INFO:tasks.workunit.client.0.smithi112.stdout:removed './224/21' 2023-01-24T03:35:18.359 INFO:tasks.workunit.client.0.smithi112.stdout:removed './224/27' 2023-01-24T03:35:18.359 INFO:tasks.workunit.client.0.smithi112.stdout:removed './224/31' 2023-01-24T03:35:18.359 INFO:tasks.workunit.client.0.smithi112.stdout:removed './224/36' 2023-01-24T03:35:18.360 INFO:tasks.workunit.client.0.smithi112.stdout:removed './224/20' 2023-01-24T03:35:18.360 INFO:tasks.workunit.client.0.smithi112.stdout:removed './224/39' 2023-01-24T03:35:18.360 INFO:tasks.workunit.client.0.smithi112.stdout:removed './224/34' 2023-01-24T03:35:18.360 INFO:tasks.workunit.client.0.smithi112.stdout:removed './224/38' 2023-01-24T03:35:18.360 INFO:tasks.workunit.client.0.smithi112.stdout:removed './224/28' 2023-01-24T03:35:18.361 INFO:tasks.workunit.client.0.smithi112.stdout:removed './224/24' 2023-01-24T03:35:18.361 INFO:tasks.workunit.client.0.smithi112.stdout:removed './224/29' 2023-01-24T03:35:18.361 INFO:tasks.workunit.client.0.smithi112.stdout:removed './224/32' 2023-01-24T03:35:18.361 INFO:tasks.workunit.client.0.smithi112.stdout:removed './224/22' 2023-01-24T03:35:18.361 INFO:tasks.workunit.client.0.smithi112.stdout:removed './224/25' 2023-01-24T03:35:18.362 INFO:tasks.workunit.client.0.smithi112.stdout:removed './224/30' 2023-01-24T03:35:18.362 INFO:tasks.workunit.client.0.smithi112.stdout:removed './224/33' 2023-01-24T03:35:18.362 INFO:tasks.workunit.client.0.smithi112.stdout:removed './224/23' 2023-01-24T03:35:18.362 INFO:tasks.workunit.client.0.smithi112.stdout:removed './224/35' 2023-01-24T03:35:18.362 INFO:tasks.workunit.client.0.smithi112.stdout:removed './224/26' 2023-01-24T03:35:18.362 INFO:tasks.workunit.client.0.smithi112.stdout:removed './224/37' 2023-01-24T03:35:18.363 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './224' 2023-01-24T03:35:18.363 INFO:tasks.workunit.client.0.smithi112.stdout:removed './225/21' 2023-01-24T03:35:18.363 INFO:tasks.workunit.client.0.smithi112.stdout:removed './225/27' 2023-01-24T03:35:18.363 INFO:tasks.workunit.client.0.smithi112.stdout:removed './225/31' 2023-01-24T03:35:18.363 INFO:tasks.workunit.client.0.smithi112.stdout:removed './225/36' 2023-01-24T03:35:18.364 INFO:tasks.workunit.client.0.smithi112.stdout:removed './225/20' 2023-01-24T03:35:18.364 INFO:tasks.workunit.client.0.smithi112.stdout:removed './225/39' 2023-01-24T03:35:18.364 INFO:tasks.workunit.client.0.smithi112.stdout:removed './225/34' 2023-01-24T03:35:18.365 INFO:tasks.workunit.client.0.smithi112.stdout:removed './225/38' 2023-01-24T03:35:18.365 INFO:tasks.workunit.client.0.smithi112.stdout:removed './225/28' 2023-01-24T03:35:18.365 INFO:tasks.workunit.client.0.smithi112.stdout:removed './225/24' 2023-01-24T03:35:18.365 INFO:tasks.workunit.client.0.smithi112.stdout:removed './225/29' 2023-01-24T03:35:18.366 INFO:tasks.workunit.client.0.smithi112.stdout:removed './225/32' 2023-01-24T03:35:18.366 INFO:tasks.workunit.client.0.smithi112.stdout:removed './225/22' 2023-01-24T03:35:18.366 INFO:tasks.workunit.client.0.smithi112.stdout:removed './225/25' 2023-01-24T03:35:18.366 INFO:tasks.workunit.client.0.smithi112.stdout:removed './225/30' 2023-01-24T03:35:18.366 INFO:tasks.workunit.client.0.smithi112.stdout:removed './225/33' 2023-01-24T03:35:18.367 INFO:tasks.workunit.client.0.smithi112.stdout:removed './225/23' 2023-01-24T03:35:18.367 INFO:tasks.workunit.client.0.smithi112.stdout:removed './225/35' 2023-01-24T03:35:18.367 INFO:tasks.workunit.client.0.smithi112.stdout:removed './225/26' 2023-01-24T03:35:18.367 INFO:tasks.workunit.client.0.smithi112.stdout:removed './225/37' 2023-01-24T03:35:18.367 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './225' 2023-01-24T03:35:18.368 INFO:tasks.workunit.client.0.smithi112.stdout:removed './226/21' 2023-01-24T03:35:18.368 INFO:tasks.workunit.client.0.smithi112.stdout:removed './226/27' 2023-01-24T03:35:18.368 INFO:tasks.workunit.client.0.smithi112.stdout:removed './226/31' 2023-01-24T03:35:18.368 INFO:tasks.workunit.client.0.smithi112.stdout:removed './226/36' 2023-01-24T03:35:18.368 INFO:tasks.workunit.client.0.smithi112.stdout:removed './226/20' 2023-01-24T03:35:18.369 INFO:tasks.workunit.client.0.smithi112.stdout:removed './226/39' 2023-01-24T03:35:18.369 INFO:tasks.workunit.client.0.smithi112.stdout:removed './226/34' 2023-01-24T03:35:18.369 INFO:tasks.workunit.client.0.smithi112.stdout:removed './226/38' 2023-01-24T03:35:18.369 INFO:tasks.workunit.client.0.smithi112.stdout:removed './226/28' 2023-01-24T03:35:18.369 INFO:tasks.workunit.client.0.smithi112.stdout:removed './226/24' 2023-01-24T03:35:18.976 INFO:tasks.workunit.client.0.smithi112.stdout:removed './226/29' 2023-01-24T03:35:18.976 INFO:tasks.workunit.client.0.smithi112.stdout:removed './226/32' 2023-01-24T03:35:18.977 INFO:tasks.workunit.client.0.smithi112.stdout:removed './226/22' 2023-01-24T03:35:18.977 INFO:tasks.workunit.client.0.smithi112.stdout:removed './226/25' 2023-01-24T03:35:18.977 INFO:tasks.workunit.client.0.smithi112.stdout:removed './226/30' 2023-01-24T03:35:18.977 INFO:tasks.workunit.client.0.smithi112.stdout:removed './226/33' 2023-01-24T03:35:18.978 INFO:tasks.workunit.client.0.smithi112.stdout:removed './226/23' 2023-01-24T03:35:18.978 INFO:tasks.workunit.client.0.smithi112.stdout:removed './226/35' 2023-01-24T03:35:18.978 INFO:tasks.workunit.client.0.smithi112.stdout:removed './226/26' 2023-01-24T03:35:18.978 INFO:tasks.workunit.client.0.smithi112.stdout:removed './226/37' 2023-01-24T03:35:18.978 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './226' 2023-01-24T03:35:18.979 INFO:tasks.workunit.client.0.smithi112.stdout:removed './227/21' 2023-01-24T03:35:18.979 INFO:tasks.workunit.client.0.smithi112.stdout:removed './227/27' 2023-01-24T03:35:18.979 INFO:tasks.workunit.client.0.smithi112.stdout:removed './227/31' 2023-01-24T03:35:18.979 INFO:tasks.workunit.client.0.smithi112.stdout:removed './227/36' 2023-01-24T03:35:18.979 INFO:tasks.workunit.client.0.smithi112.stdout:removed './227/20' 2023-01-24T03:35:18.980 INFO:tasks.workunit.client.0.smithi112.stdout:removed './227/39' 2023-01-24T03:35:18.980 INFO:tasks.workunit.client.0.smithi112.stdout:removed './227/34' 2023-01-24T03:35:18.980 INFO:tasks.workunit.client.0.smithi112.stdout:removed './227/38' 2023-01-24T03:35:18.980 INFO:tasks.workunit.client.0.smithi112.stdout:removed './227/28' 2023-01-24T03:35:18.980 INFO:tasks.workunit.client.0.smithi112.stdout:removed './227/24' 2023-01-24T03:35:18.981 INFO:tasks.workunit.client.0.smithi112.stdout:removed './227/29' 2023-01-24T03:35:18.981 INFO:tasks.workunit.client.0.smithi112.stdout:removed './227/32' 2023-01-24T03:35:18.981 INFO:tasks.workunit.client.0.smithi112.stdout:removed './227/22' 2023-01-24T03:35:18.981 INFO:tasks.workunit.client.0.smithi112.stdout:removed './227/25' 2023-01-24T03:35:18.981 INFO:tasks.workunit.client.0.smithi112.stdout:removed './227/30' 2023-01-24T03:35:18.982 INFO:tasks.workunit.client.0.smithi112.stdout:removed './227/33' 2023-01-24T03:35:18.982 INFO:tasks.workunit.client.0.smithi112.stdout:removed './227/23' 2023-01-24T03:35:18.982 INFO:tasks.workunit.client.0.smithi112.stdout:removed './227/35' 2023-01-24T03:35:18.982 INFO:tasks.workunit.client.0.smithi112.stdout:removed './227/26' 2023-01-24T03:35:18.982 INFO:tasks.workunit.client.0.smithi112.stdout:removed './227/37' 2023-01-24T03:35:18.983 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './227' 2023-01-24T03:35:18.983 INFO:tasks.workunit.client.0.smithi112.stdout:removed './228/21' 2023-01-24T03:35:18.983 INFO:tasks.workunit.client.0.smithi112.stdout:removed './228/27' 2023-01-24T03:35:18.983 INFO:tasks.workunit.client.0.smithi112.stdout:removed './228/31' 2023-01-24T03:35:18.983 INFO:tasks.workunit.client.0.smithi112.stdout:removed './228/36' 2023-01-24T03:35:18.984 INFO:tasks.workunit.client.0.smithi112.stdout:removed './228/20' 2023-01-24T03:35:18.984 INFO:tasks.workunit.client.0.smithi112.stdout:removed './228/39' 2023-01-24T03:35:18.984 INFO:tasks.workunit.client.0.smithi112.stdout:removed './228/34' 2023-01-24T03:35:18.984 INFO:tasks.workunit.client.0.smithi112.stdout:removed './228/38' 2023-01-24T03:35:18.984 INFO:tasks.workunit.client.0.smithi112.stdout:removed './228/28' 2023-01-24T03:35:18.985 INFO:tasks.workunit.client.0.smithi112.stdout:removed './228/24' 2023-01-24T03:35:18.985 INFO:tasks.workunit.client.0.smithi112.stdout:removed './228/29' 2023-01-24T03:35:18.985 INFO:tasks.workunit.client.0.smithi112.stdout:removed './228/32' 2023-01-24T03:35:18.985 INFO:tasks.workunit.client.0.smithi112.stdout:removed './228/22' 2023-01-24T03:35:18.985 INFO:tasks.workunit.client.0.smithi112.stdout:removed './228/25' 2023-01-24T03:35:18.985 INFO:tasks.workunit.client.0.smithi112.stdout:removed './228/30' 2023-01-24T03:35:18.986 INFO:tasks.workunit.client.0.smithi112.stdout:removed './228/33' 2023-01-24T03:35:18.986 INFO:tasks.workunit.client.0.smithi112.stdout:removed './228/23' 2023-01-24T03:35:18.986 INFO:tasks.workunit.client.0.smithi112.stdout:removed './228/35' 2023-01-24T03:35:18.986 INFO:tasks.workunit.client.0.smithi112.stdout:removed './228/26' 2023-01-24T03:35:18.986 INFO:tasks.workunit.client.0.smithi112.stdout:removed './228/37' 2023-01-24T03:35:18.986 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './228' 2023-01-24T03:35:18.987 INFO:tasks.workunit.client.0.smithi112.stdout:removed './229/21' 2023-01-24T03:35:18.987 INFO:tasks.workunit.client.0.smithi112.stdout:removed './229/27' 2023-01-24T03:35:18.987 INFO:tasks.workunit.client.0.smithi112.stdout:removed './229/31' 2023-01-24T03:35:18.987 INFO:tasks.workunit.client.0.smithi112.stdout:removed './229/36' 2023-01-24T03:35:18.987 INFO:tasks.workunit.client.0.smithi112.stdout:removed './229/20' 2023-01-24T03:35:18.988 INFO:tasks.workunit.client.0.smithi112.stdout:removed './229/39' 2023-01-24T03:35:18.988 INFO:tasks.workunit.client.0.smithi112.stdout:removed './229/34' 2023-01-24T03:35:18.988 INFO:tasks.workunit.client.0.smithi112.stdout:removed './229/38' 2023-01-24T03:35:18.988 INFO:tasks.workunit.client.0.smithi112.stdout:removed './229/28' 2023-01-24T03:35:18.989 INFO:tasks.workunit.client.0.smithi112.stdout:removed './229/24' 2023-01-24T03:35:18.989 INFO:tasks.workunit.client.0.smithi112.stdout:removed './229/29' 2023-01-24T03:35:18.989 INFO:tasks.workunit.client.0.smithi112.stdout:removed './229/32' 2023-01-24T03:35:18.989 INFO:tasks.workunit.client.0.smithi112.stdout:removed './229/22' 2023-01-24T03:35:18.989 INFO:tasks.workunit.client.0.smithi112.stdout:removed './229/25' 2023-01-24T03:35:18.989 INFO:tasks.workunit.client.0.smithi112.stdout:removed './229/30' 2023-01-24T03:35:18.990 INFO:tasks.workunit.client.0.smithi112.stdout:removed './229/33' 2023-01-24T03:35:18.990 INFO:tasks.workunit.client.0.smithi112.stdout:removed './229/23' 2023-01-24T03:35:18.990 INFO:tasks.workunit.client.0.smithi112.stdout:removed './229/35' 2023-01-24T03:35:18.990 INFO:tasks.workunit.client.0.smithi112.stdout:removed './229/26' 2023-01-24T03:35:18.990 INFO:tasks.workunit.client.0.smithi112.stdout:removed './229/37' 2023-01-24T03:35:18.991 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './229' 2023-01-24T03:35:18.991 INFO:tasks.workunit.client.0.smithi112.stdout:removed './230/21' 2023-01-24T03:35:18.991 INFO:tasks.workunit.client.0.smithi112.stdout:removed './230/27' 2023-01-24T03:35:18.991 INFO:tasks.workunit.client.0.smithi112.stdout:removed './230/31' 2023-01-24T03:35:18.991 INFO:tasks.workunit.client.0.smithi112.stdout:removed './230/36' 2023-01-24T03:35:18.992 INFO:tasks.workunit.client.0.smithi112.stdout:removed './230/20' 2023-01-24T03:35:18.992 INFO:tasks.workunit.client.0.smithi112.stdout:removed './230/39' 2023-01-24T03:35:18.992 INFO:tasks.workunit.client.0.smithi112.stdout:removed './230/34' 2023-01-24T03:35:18.992 INFO:tasks.workunit.client.0.smithi112.stdout:removed './230/38' 2023-01-24T03:35:18.992 INFO:tasks.workunit.client.0.smithi112.stdout:removed './230/28' 2023-01-24T03:35:18.993 INFO:tasks.workunit.client.0.smithi112.stdout:removed './230/24' 2023-01-24T03:35:18.993 INFO:tasks.workunit.client.0.smithi112.stdout:removed './230/29' 2023-01-24T03:35:18.993 INFO:tasks.workunit.client.0.smithi112.stdout:removed './230/32' 2023-01-24T03:35:18.993 INFO:tasks.workunit.client.0.smithi112.stdout:removed './230/22' 2023-01-24T03:35:18.993 INFO:tasks.workunit.client.0.smithi112.stdout:removed './230/25' 2023-01-24T03:35:18.994 INFO:tasks.workunit.client.0.smithi112.stdout:removed './230/30' 2023-01-24T03:35:18.994 INFO:tasks.workunit.client.0.smithi112.stdout:removed './230/33' 2023-01-24T03:35:18.994 INFO:tasks.workunit.client.0.smithi112.stdout:removed './230/23' 2023-01-24T03:35:18.994 INFO:tasks.workunit.client.0.smithi112.stdout:removed './230/35' 2023-01-24T03:35:18.994 INFO:tasks.workunit.client.0.smithi112.stdout:removed './230/26' 2023-01-24T03:35:18.995 INFO:tasks.workunit.client.0.smithi112.stdout:removed './230/37' 2023-01-24T03:35:18.995 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './230' 2023-01-24T03:35:18.995 INFO:tasks.workunit.client.0.smithi112.stdout:removed './231/21' 2023-01-24T03:35:18.995 INFO:tasks.workunit.client.0.smithi112.stdout:removed './231/27' 2023-01-24T03:35:18.995 INFO:tasks.workunit.client.0.smithi112.stdout:removed './231/31' 2023-01-24T03:35:18.995 INFO:tasks.workunit.client.0.smithi112.stdout:removed './231/36' 2023-01-24T03:35:18.996 INFO:tasks.workunit.client.0.smithi112.stdout:removed './231/20' 2023-01-24T03:35:18.996 INFO:tasks.workunit.client.0.smithi112.stdout:removed './231/39' 2023-01-24T03:35:18.996 INFO:tasks.workunit.client.0.smithi112.stdout:removed './231/34' 2023-01-24T03:35:18.996 INFO:tasks.workunit.client.0.smithi112.stdout:removed './231/38' 2023-01-24T03:35:18.996 INFO:tasks.workunit.client.0.smithi112.stdout:removed './231/28' 2023-01-24T03:35:18.997 INFO:tasks.workunit.client.0.smithi112.stdout:removed './231/24' 2023-01-24T03:35:18.997 INFO:tasks.workunit.client.0.smithi112.stdout:removed './231/29' 2023-01-24T03:35:18.997 INFO:tasks.workunit.client.0.smithi112.stdout:removed './231/32' 2023-01-24T03:35:18.997 INFO:tasks.workunit.client.0.smithi112.stdout:removed './231/22' 2023-01-24T03:35:18.997 INFO:tasks.workunit.client.0.smithi112.stdout:removed './231/25' 2023-01-24T03:35:18.998 INFO:tasks.workunit.client.0.smithi112.stdout:removed './231/30' 2023-01-24T03:35:18.998 INFO:tasks.workunit.client.0.smithi112.stdout:removed './231/33' 2023-01-24T03:35:18.998 INFO:tasks.workunit.client.0.smithi112.stdout:removed './231/23' 2023-01-24T03:35:18.998 INFO:tasks.workunit.client.0.smithi112.stdout:removed './231/35' 2023-01-24T03:35:18.998 INFO:tasks.workunit.client.0.smithi112.stdout:removed './231/26' 2023-01-24T03:35:18.999 INFO:tasks.workunit.client.0.smithi112.stdout:removed './231/37' 2023-01-24T03:35:18.999 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './231' 2023-01-24T03:35:18.999 INFO:tasks.workunit.client.0.smithi112.stdout:removed './232/21' 2023-01-24T03:35:18.999 INFO:tasks.workunit.client.0.smithi112.stdout:removed './232/27' 2023-01-24T03:35:18.999 INFO:tasks.workunit.client.0.smithi112.stdout:removed './232/31' 2023-01-24T03:35:19.000 INFO:tasks.workunit.client.0.smithi112.stdout:removed './232/36' 2023-01-24T03:35:19.000 INFO:tasks.workunit.client.0.smithi112.stdout:removed './232/20' 2023-01-24T03:35:19.000 INFO:tasks.workunit.client.0.smithi112.stdout:removed './232/39' 2023-01-24T03:35:19.000 INFO:tasks.workunit.client.0.smithi112.stdout:removed './232/34' 2023-01-24T03:35:19.000 INFO:tasks.workunit.client.0.smithi112.stdout:removed './232/38' 2023-01-24T03:35:19.001 INFO:tasks.workunit.client.0.smithi112.stdout:removed './232/28' 2023-01-24T03:35:19.001 INFO:tasks.workunit.client.0.smithi112.stdout:removed './232/24' 2023-01-24T03:35:19.001 INFO:tasks.workunit.client.0.smithi112.stdout:removed './232/29' 2023-01-24T03:35:19.001 INFO:tasks.workunit.client.0.smithi112.stdout:removed './232/32' 2023-01-24T03:35:19.001 INFO:tasks.workunit.client.0.smithi112.stdout:removed './232/22' 2023-01-24T03:35:19.002 INFO:tasks.workunit.client.0.smithi112.stdout:removed './232/25' 2023-01-24T03:35:19.002 INFO:tasks.workunit.client.0.smithi112.stdout:removed './232/30' 2023-01-24T03:35:19.002 INFO:tasks.workunit.client.0.smithi112.stdout:removed './232/33' 2023-01-24T03:35:19.002 INFO:tasks.workunit.client.0.smithi112.stdout:removed './232/23' 2023-01-24T03:35:19.002 INFO:tasks.workunit.client.0.smithi112.stdout:removed './232/35' 2023-01-24T03:35:19.002 INFO:tasks.workunit.client.0.smithi112.stdout:removed './232/26' 2023-01-24T03:35:19.003 INFO:tasks.workunit.client.0.smithi112.stdout:removed './232/37' 2023-01-24T03:35:19.003 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './232' 2023-01-24T03:35:19.003 INFO:tasks.workunit.client.0.smithi112.stdout:removed './233/21' 2023-01-24T03:35:19.003 INFO:tasks.workunit.client.0.smithi112.stdout:removed './233/27' 2023-01-24T03:35:19.003 INFO:tasks.workunit.client.0.smithi112.stdout:removed './233/31' 2023-01-24T03:35:19.004 INFO:tasks.workunit.client.0.smithi112.stdout:removed './233/36' 2023-01-24T03:35:19.004 INFO:tasks.workunit.client.0.smithi112.stdout:removed './233/20' 2023-01-24T03:35:19.004 INFO:tasks.workunit.client.0.smithi112.stdout:removed './233/39' 2023-01-24T03:35:19.004 INFO:tasks.workunit.client.0.smithi112.stdout:removed './233/34' 2023-01-24T03:35:19.004 INFO:tasks.workunit.client.0.smithi112.stdout:removed './233/38' 2023-01-24T03:35:19.005 INFO:tasks.workunit.client.0.smithi112.stdout:removed './233/28' 2023-01-24T03:35:19.005 INFO:tasks.workunit.client.0.smithi112.stdout:removed './233/24' 2023-01-24T03:35:19.005 INFO:tasks.workunit.client.0.smithi112.stdout:removed './233/29' 2023-01-24T03:35:19.005 INFO:tasks.workunit.client.0.smithi112.stdout:removed './233/32' 2023-01-24T03:35:19.005 INFO:tasks.workunit.client.0.smithi112.stdout:removed './233/22' 2023-01-24T03:35:19.006 INFO:tasks.workunit.client.0.smithi112.stdout:removed './233/25' 2023-01-24T03:35:19.006 INFO:tasks.workunit.client.0.smithi112.stdout:removed './233/30' 2023-01-24T03:35:19.006 INFO:tasks.workunit.client.0.smithi112.stdout:removed './233/33' 2023-01-24T03:35:19.006 INFO:tasks.workunit.client.0.smithi112.stdout:removed './233/23' 2023-01-24T03:35:19.006 INFO:tasks.workunit.client.0.smithi112.stdout:removed './233/35' 2023-01-24T03:35:19.007 INFO:tasks.workunit.client.0.smithi112.stdout:removed './233/26' 2023-01-24T03:35:19.007 INFO:tasks.workunit.client.0.smithi112.stdout:removed './233/37' 2023-01-24T03:35:19.007 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './233' 2023-01-24T03:35:19.007 INFO:tasks.workunit.client.0.smithi112.stdout:removed './234/21' 2023-01-24T03:35:19.007 INFO:tasks.workunit.client.0.smithi112.stdout:removed './234/27' 2023-01-24T03:35:19.008 INFO:tasks.workunit.client.0.smithi112.stdout:removed './234/31' 2023-01-24T03:35:19.008 INFO:tasks.workunit.client.0.smithi112.stdout:removed './234/36' 2023-01-24T03:35:19.008 INFO:tasks.workunit.client.0.smithi112.stdout:removed './234/20' 2023-01-24T03:35:19.008 INFO:tasks.workunit.client.0.smithi112.stdout:removed './234/39' 2023-01-24T03:35:19.008 INFO:tasks.workunit.client.0.smithi112.stdout:removed './234/34' 2023-01-24T03:35:19.009 INFO:tasks.workunit.client.0.smithi112.stdout:removed './234/38' 2023-01-24T03:35:19.009 INFO:tasks.workunit.client.0.smithi112.stdout:removed './234/28' 2023-01-24T03:35:19.009 INFO:tasks.workunit.client.0.smithi112.stdout:removed './234/24' 2023-01-24T03:35:19.009 INFO:tasks.workunit.client.0.smithi112.stdout:removed './234/29' 2023-01-24T03:35:19.009 INFO:tasks.workunit.client.0.smithi112.stdout:removed './234/32' 2023-01-24T03:35:19.010 INFO:tasks.workunit.client.0.smithi112.stdout:removed './234/22' 2023-01-24T03:35:19.010 INFO:tasks.workunit.client.0.smithi112.stdout:removed './234/25' 2023-01-24T03:35:19.010 INFO:tasks.workunit.client.0.smithi112.stdout:removed './234/30' 2023-01-24T03:35:19.010 INFO:tasks.workunit.client.0.smithi112.stdout:removed './234/33' 2023-01-24T03:35:19.010 INFO:tasks.workunit.client.0.smithi112.stdout:removed './234/23' 2023-01-24T03:35:19.011 INFO:tasks.workunit.client.0.smithi112.stdout:removed './234/35' 2023-01-24T03:35:19.011 INFO:tasks.workunit.client.0.smithi112.stdout:removed './234/26' 2023-01-24T03:35:19.011 INFO:tasks.workunit.client.0.smithi112.stdout:removed './234/37' 2023-01-24T03:35:19.011 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './234' 2023-01-24T03:35:19.011 INFO:tasks.workunit.client.0.smithi112.stdout:removed './235/21' 2023-01-24T03:35:19.012 INFO:tasks.workunit.client.0.smithi112.stdout:removed './235/27' 2023-01-24T03:35:19.012 INFO:tasks.workunit.client.0.smithi112.stdout:removed './235/31' 2023-01-24T03:35:19.012 INFO:tasks.workunit.client.0.smithi112.stdout:removed './235/36' 2023-01-24T03:35:19.012 INFO:tasks.workunit.client.0.smithi112.stdout:removed './235/20' 2023-01-24T03:35:19.013 INFO:tasks.workunit.client.0.smithi112.stdout:removed './235/39' 2023-01-24T03:35:19.013 INFO:tasks.workunit.client.0.smithi112.stdout:removed './235/34' 2023-01-24T03:35:19.013 INFO:tasks.workunit.client.0.smithi112.stdout:removed './235/38' 2023-01-24T03:35:19.013 INFO:tasks.workunit.client.0.smithi112.stdout:removed './235/28' 2023-01-24T03:35:19.013 INFO:tasks.workunit.client.0.smithi112.stdout:removed './235/24' 2023-01-24T03:35:19.013 INFO:tasks.workunit.client.0.smithi112.stdout:removed './235/29' 2023-01-24T03:35:19.014 INFO:tasks.workunit.client.0.smithi112.stdout:removed './235/32' 2023-01-24T03:35:19.014 INFO:tasks.workunit.client.0.smithi112.stdout:removed './235/22' 2023-01-24T03:35:19.014 INFO:tasks.workunit.client.0.smithi112.stdout:removed './235/25' 2023-01-24T03:35:19.014 INFO:tasks.workunit.client.0.smithi112.stdout:removed './235/30' 2023-01-24T03:35:19.014 INFO:tasks.workunit.client.0.smithi112.stdout:removed './235/33' 2023-01-24T03:35:19.014 INFO:tasks.workunit.client.0.smithi112.stdout:removed './235/23' 2023-01-24T03:35:19.015 INFO:tasks.workunit.client.0.smithi112.stdout:removed './235/35' 2023-01-24T03:35:19.015 INFO:tasks.workunit.client.0.smithi112.stdout:removed './235/26' 2023-01-24T03:35:19.015 INFO:tasks.workunit.client.0.smithi112.stdout:removed './235/37' 2023-01-24T03:35:19.015 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './235' 2023-01-24T03:35:19.015 INFO:tasks.workunit.client.0.smithi112.stdout:removed './236/21' 2023-01-24T03:35:19.016 INFO:tasks.workunit.client.0.smithi112.stdout:removed './236/27' 2023-01-24T03:35:19.016 INFO:tasks.workunit.client.0.smithi112.stdout:removed './236/31' 2023-01-24T03:35:19.016 INFO:tasks.workunit.client.0.smithi112.stdout:removed './236/36' 2023-01-24T03:35:19.016 INFO:tasks.workunit.client.0.smithi112.stdout:removed './236/20' 2023-01-24T03:35:19.017 INFO:tasks.workunit.client.0.smithi112.stdout:removed './236/39' 2023-01-24T03:35:19.017 INFO:tasks.workunit.client.0.smithi112.stdout:removed './236/34' 2023-01-24T03:35:19.017 INFO:tasks.workunit.client.0.smithi112.stdout:removed './236/38' 2023-01-24T03:35:19.017 INFO:tasks.workunit.client.0.smithi112.stdout:removed './236/28' 2023-01-24T03:35:19.017 INFO:tasks.workunit.client.0.smithi112.stdout:removed './236/24' 2023-01-24T03:35:19.017 INFO:tasks.workunit.client.0.smithi112.stdout:removed './236/29' 2023-01-24T03:35:19.018 INFO:tasks.workunit.client.0.smithi112.stdout:removed './236/32' 2023-01-24T03:35:20.383 INFO:tasks.workunit.client.0.smithi112.stdout:removed './236/22' 2023-01-24T03:35:20.384 INFO:tasks.workunit.client.0.smithi112.stdout:removed './236/25' 2023-01-24T03:35:20.384 INFO:tasks.workunit.client.0.smithi112.stdout:removed './236/30' 2023-01-24T03:35:20.384 INFO:tasks.workunit.client.0.smithi112.stdout:removed './236/33' 2023-01-24T03:35:20.384 INFO:tasks.workunit.client.0.smithi112.stdout:removed './236/23' 2023-01-24T03:35:20.385 INFO:tasks.workunit.client.0.smithi112.stdout:removed './236/35' 2023-01-24T03:35:20.385 INFO:tasks.workunit.client.0.smithi112.stdout:removed './236/26' 2023-01-24T03:35:20.385 INFO:tasks.workunit.client.0.smithi112.stdout:removed './236/37' 2023-01-24T03:35:20.385 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './236' 2023-01-24T03:35:20.385 INFO:tasks.workunit.client.0.smithi112.stdout:removed './237/21' 2023-01-24T03:35:20.386 INFO:tasks.workunit.client.0.smithi112.stdout:removed './237/27' 2023-01-24T03:35:20.386 INFO:tasks.workunit.client.0.smithi112.stdout:removed './237/31' 2023-01-24T03:35:20.386 INFO:tasks.workunit.client.0.smithi112.stdout:removed './237/36' 2023-01-24T03:35:20.386 INFO:tasks.workunit.client.0.smithi112.stdout:removed './237/20' 2023-01-24T03:35:20.386 INFO:tasks.workunit.client.0.smithi112.stdout:removed './237/39' 2023-01-24T03:35:20.387 INFO:tasks.workunit.client.0.smithi112.stdout:removed './237/34' 2023-01-24T03:35:20.387 INFO:tasks.workunit.client.0.smithi112.stdout:removed './237/38' 2023-01-24T03:35:20.387 INFO:tasks.workunit.client.0.smithi112.stdout:removed './237/28' 2023-01-24T03:35:20.387 INFO:tasks.workunit.client.0.smithi112.stdout:removed './237/24' 2023-01-24T03:35:20.387 INFO:tasks.workunit.client.0.smithi112.stdout:removed './237/29' 2023-01-24T03:35:20.388 INFO:tasks.workunit.client.0.smithi112.stdout:removed './237/32' 2023-01-24T03:35:20.388 INFO:tasks.workunit.client.0.smithi112.stdout:removed './237/22' 2023-01-24T03:35:20.388 INFO:tasks.workunit.client.0.smithi112.stdout:removed './237/25' 2023-01-24T03:35:20.388 INFO:tasks.workunit.client.0.smithi112.stdout:removed './237/30' 2023-01-24T03:35:20.388 INFO:tasks.workunit.client.0.smithi112.stdout:removed './237/33' 2023-01-24T03:35:20.389 INFO:tasks.workunit.client.0.smithi112.stdout:removed './237/23' 2023-01-24T03:35:20.389 INFO:tasks.workunit.client.0.smithi112.stdout:removed './237/35' 2023-01-24T03:35:20.389 INFO:tasks.workunit.client.0.smithi112.stdout:removed './237/26' 2023-01-24T03:35:20.389 INFO:tasks.workunit.client.0.smithi112.stdout:removed './237/37' 2023-01-24T03:35:20.389 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './237' 2023-01-24T03:35:20.390 INFO:tasks.workunit.client.0.smithi112.stdout:removed './238/21' 2023-01-24T03:35:20.390 INFO:tasks.workunit.client.0.smithi112.stdout:removed './238/27' 2023-01-24T03:35:20.390 INFO:tasks.workunit.client.0.smithi112.stdout:removed './238/31' 2023-01-24T03:35:20.390 INFO:tasks.workunit.client.0.smithi112.stdout:removed './238/36' 2023-01-24T03:35:20.390 INFO:tasks.workunit.client.0.smithi112.stdout:removed './238/20' 2023-01-24T03:35:20.391 INFO:tasks.workunit.client.0.smithi112.stdout:removed './238/39' 2023-01-24T03:35:20.391 INFO:tasks.workunit.client.0.smithi112.stdout:removed './238/34' 2023-01-24T03:35:20.391 INFO:tasks.workunit.client.0.smithi112.stdout:removed './238/38' 2023-01-24T03:35:20.391 INFO:tasks.workunit.client.0.smithi112.stdout:removed './238/28' 2023-01-24T03:35:20.391 INFO:tasks.workunit.client.0.smithi112.stdout:removed './238/24' 2023-01-24T03:35:20.392 INFO:tasks.workunit.client.0.smithi112.stdout:removed './238/29' 2023-01-24T03:35:20.392 INFO:tasks.workunit.client.0.smithi112.stdout:removed './238/32' 2023-01-24T03:35:20.392 INFO:tasks.workunit.client.0.smithi112.stdout:removed './238/22' 2023-01-24T03:35:20.392 INFO:tasks.workunit.client.0.smithi112.stdout:removed './238/25' 2023-01-24T03:35:20.392 INFO:tasks.workunit.client.0.smithi112.stdout:removed './238/30' 2023-01-24T03:35:20.393 INFO:tasks.workunit.client.0.smithi112.stdout:removed './238/33' 2023-01-24T03:35:20.393 INFO:tasks.workunit.client.0.smithi112.stdout:removed './238/23' 2023-01-24T03:35:20.393 INFO:tasks.workunit.client.0.smithi112.stdout:removed './238/35' 2023-01-24T03:35:20.393 INFO:tasks.workunit.client.0.smithi112.stdout:removed './238/26' 2023-01-24T03:35:20.393 INFO:tasks.workunit.client.0.smithi112.stdout:removed './238/37' 2023-01-24T03:35:20.394 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './238' 2023-01-24T03:35:20.394 INFO:tasks.workunit.client.0.smithi112.stdout:removed './239/21' 2023-01-24T03:35:20.394 INFO:tasks.workunit.client.0.smithi112.stdout:removed './239/27' 2023-01-24T03:35:20.394 INFO:tasks.workunit.client.0.smithi112.stdout:removed './239/31' 2023-01-24T03:35:20.394 INFO:tasks.workunit.client.0.smithi112.stdout:removed './239/36' 2023-01-24T03:35:20.395 INFO:tasks.workunit.client.0.smithi112.stdout:removed './239/20' 2023-01-24T03:35:20.395 INFO:tasks.workunit.client.0.smithi112.stdout:removed './239/39' 2023-01-24T03:35:20.395 INFO:tasks.workunit.client.0.smithi112.stdout:removed './239/34' 2023-01-24T03:35:20.395 INFO:tasks.workunit.client.0.smithi112.stdout:removed './239/38' 2023-01-24T03:35:20.395 INFO:tasks.workunit.client.0.smithi112.stdout:removed './239/28' 2023-01-24T03:35:20.396 INFO:tasks.workunit.client.0.smithi112.stdout:removed './239/24' 2023-01-24T03:35:20.396 INFO:tasks.workunit.client.0.smithi112.stdout:removed './239/29' 2023-01-24T03:35:20.396 INFO:tasks.workunit.client.0.smithi112.stdout:removed './239/32' 2023-01-24T03:35:20.396 INFO:tasks.workunit.client.0.smithi112.stdout:removed './239/22' 2023-01-24T03:35:20.397 INFO:tasks.workunit.client.0.smithi112.stdout:removed './239/25' 2023-01-24T03:35:20.397 INFO:tasks.workunit.client.0.smithi112.stdout:removed './239/30' 2023-01-24T03:35:20.397 INFO:tasks.workunit.client.0.smithi112.stdout:removed './239/33' 2023-01-24T03:35:20.397 INFO:tasks.workunit.client.0.smithi112.stdout:removed './239/23' 2023-01-24T03:35:20.397 INFO:tasks.workunit.client.0.smithi112.stdout:removed './239/35' 2023-01-24T03:35:20.397 INFO:tasks.workunit.client.0.smithi112.stdout:removed './239/26' 2023-01-24T03:35:20.398 INFO:tasks.workunit.client.0.smithi112.stdout:removed './239/37' 2023-01-24T03:35:20.398 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './239' 2023-01-24T03:35:20.398 INFO:tasks.workunit.client.0.smithi112.stdout:removed './240/21' 2023-01-24T03:35:20.398 INFO:tasks.workunit.client.0.smithi112.stdout:removed './240/27' 2023-01-24T03:35:20.398 INFO:tasks.workunit.client.0.smithi112.stdout:removed './240/31' 2023-01-24T03:35:20.399 INFO:tasks.workunit.client.0.smithi112.stdout:removed './240/36' 2023-01-24T03:35:20.399 INFO:tasks.workunit.client.0.smithi112.stdout:removed './240/20' 2023-01-24T03:35:20.399 INFO:tasks.workunit.client.0.smithi112.stdout:removed './240/39' 2023-01-24T03:35:20.399 INFO:tasks.workunit.client.0.smithi112.stdout:removed './240/34' 2023-01-24T03:35:20.399 INFO:tasks.workunit.client.0.smithi112.stdout:removed './240/38' 2023-01-24T03:35:20.400 INFO:tasks.workunit.client.0.smithi112.stdout:removed './240/28' 2023-01-24T03:35:20.400 INFO:tasks.workunit.client.0.smithi112.stdout:removed './240/24' 2023-01-24T03:35:20.400 INFO:tasks.workunit.client.0.smithi112.stdout:removed './240/29' 2023-01-24T03:35:20.400 INFO:tasks.workunit.client.0.smithi112.stdout:removed './240/32' 2023-01-24T03:35:20.400 INFO:tasks.workunit.client.0.smithi112.stdout:removed './240/22' 2023-01-24T03:35:20.401 INFO:tasks.workunit.client.0.smithi112.stdout:removed './240/25' 2023-01-24T03:35:20.401 INFO:tasks.workunit.client.0.smithi112.stdout:removed './240/30' 2023-01-24T03:35:20.401 INFO:tasks.workunit.client.0.smithi112.stdout:removed './240/33' 2023-01-24T03:35:20.401 INFO:tasks.workunit.client.0.smithi112.stdout:removed './240/23' 2023-01-24T03:35:20.401 INFO:tasks.workunit.client.0.smithi112.stdout:removed './240/35' 2023-01-24T03:35:20.402 INFO:tasks.workunit.client.0.smithi112.stdout:removed './240/26' 2023-01-24T03:35:20.402 INFO:tasks.workunit.client.0.smithi112.stdout:removed './240/37' 2023-01-24T03:35:20.402 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './240' 2023-01-24T03:35:20.402 INFO:tasks.workunit.client.0.smithi112.stdout:removed './241/21' 2023-01-24T03:35:20.402 INFO:tasks.workunit.client.0.smithi112.stdout:removed './241/27' 2023-01-24T03:35:20.402 INFO:tasks.workunit.client.0.smithi112.stdout:removed './241/31' 2023-01-24T03:35:20.403 INFO:tasks.workunit.client.0.smithi112.stdout:removed './241/36' 2023-01-24T03:35:20.403 INFO:tasks.workunit.client.0.smithi112.stdout:removed './241/20' 2023-01-24T03:35:20.403 INFO:tasks.workunit.client.0.smithi112.stdout:removed './241/39' 2023-01-24T03:35:20.403 INFO:tasks.workunit.client.0.smithi112.stdout:removed './241/34' 2023-01-24T03:35:20.403 INFO:tasks.workunit.client.0.smithi112.stdout:removed './241/38' 2023-01-24T03:35:20.403 INFO:tasks.workunit.client.0.smithi112.stdout:removed './241/28' 2023-01-24T03:35:20.404 INFO:tasks.workunit.client.0.smithi112.stdout:removed './241/24' 2023-01-24T03:35:20.404 INFO:tasks.workunit.client.0.smithi112.stdout:removed './241/29' 2023-01-24T03:35:20.404 INFO:tasks.workunit.client.0.smithi112.stdout:removed './241/32' 2023-01-24T03:35:20.404 INFO:tasks.workunit.client.0.smithi112.stdout:removed './241/22' 2023-01-24T03:35:20.404 INFO:tasks.workunit.client.0.smithi112.stdout:removed './241/25' 2023-01-24T03:35:20.405 INFO:tasks.workunit.client.0.smithi112.stdout:removed './241/30' 2023-01-24T03:35:20.405 INFO:tasks.workunit.client.0.smithi112.stdout:removed './241/33' 2023-01-24T03:35:20.405 INFO:tasks.workunit.client.0.smithi112.stdout:removed './241/23' 2023-01-24T03:35:20.405 INFO:tasks.workunit.client.0.smithi112.stdout:removed './241/35' 2023-01-24T03:35:20.405 INFO:tasks.workunit.client.0.smithi112.stdout:removed './241/26' 2023-01-24T03:35:20.405 INFO:tasks.workunit.client.0.smithi112.stdout:removed './241/37' 2023-01-24T03:35:20.406 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './241' 2023-01-24T03:35:20.406 INFO:tasks.workunit.client.0.smithi112.stdout:removed './242/21' 2023-01-24T03:35:20.406 INFO:tasks.workunit.client.0.smithi112.stdout:removed './242/27' 2023-01-24T03:35:20.406 INFO:tasks.workunit.client.0.smithi112.stdout:removed './242/31' 2023-01-24T03:35:20.406 INFO:tasks.workunit.client.0.smithi112.stdout:removed './242/36' 2023-01-24T03:35:20.406 INFO:tasks.workunit.client.0.smithi112.stdout:removed './242/20' 2023-01-24T03:35:20.407 INFO:tasks.workunit.client.0.smithi112.stdout:removed './242/39' 2023-01-24T03:35:20.407 INFO:tasks.workunit.client.0.smithi112.stdout:removed './242/34' 2023-01-24T03:35:20.407 INFO:tasks.workunit.client.0.smithi112.stdout:removed './242/38' 2023-01-24T03:35:20.407 INFO:tasks.workunit.client.0.smithi112.stdout:removed './242/28' 2023-01-24T03:35:20.407 INFO:tasks.workunit.client.0.smithi112.stdout:removed './242/24' 2023-01-24T03:35:20.408 INFO:tasks.workunit.client.0.smithi112.stdout:removed './242/29' 2023-01-24T03:35:20.408 INFO:tasks.workunit.client.0.smithi112.stdout:removed './242/32' 2023-01-24T03:35:20.408 INFO:tasks.workunit.client.0.smithi112.stdout:removed './242/22' 2023-01-24T03:35:20.408 INFO:tasks.workunit.client.0.smithi112.stdout:removed './242/25' 2023-01-24T03:35:20.408 INFO:tasks.workunit.client.0.smithi112.stdout:removed './242/30' 2023-01-24T03:35:20.409 INFO:tasks.workunit.client.0.smithi112.stdout:removed './242/33' 2023-01-24T03:35:20.409 INFO:tasks.workunit.client.0.smithi112.stdout:removed './242/23' 2023-01-24T03:35:20.409 INFO:tasks.workunit.client.0.smithi112.stdout:removed './242/35' 2023-01-24T03:35:20.409 INFO:tasks.workunit.client.0.smithi112.stdout:removed './242/26' 2023-01-24T03:35:20.409 INFO:tasks.workunit.client.0.smithi112.stdout:removed './242/37' 2023-01-24T03:35:20.409 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './242' 2023-01-24T03:35:20.410 INFO:tasks.workunit.client.0.smithi112.stdout:removed './243/21' 2023-01-24T03:35:20.410 INFO:tasks.workunit.client.0.smithi112.stdout:removed './243/27' 2023-01-24T03:35:20.410 INFO:tasks.workunit.client.0.smithi112.stdout:removed './243/31' 2023-01-24T03:35:20.410 INFO:tasks.workunit.client.0.smithi112.stdout:removed './243/36' 2023-01-24T03:35:20.410 INFO:tasks.workunit.client.0.smithi112.stdout:removed './243/20' 2023-01-24T03:35:20.411 INFO:tasks.workunit.client.0.smithi112.stdout:removed './243/39' 2023-01-24T03:35:20.411 INFO:tasks.workunit.client.0.smithi112.stdout:removed './243/34' 2023-01-24T03:35:20.411 INFO:tasks.workunit.client.0.smithi112.stdout:removed './243/38' 2023-01-24T03:35:20.411 INFO:tasks.workunit.client.0.smithi112.stdout:removed './243/28' 2023-01-24T03:35:20.411 INFO:tasks.workunit.client.0.smithi112.stdout:removed './243/24' 2023-01-24T03:35:20.411 INFO:tasks.workunit.client.0.smithi112.stdout:removed './243/29' 2023-01-24T03:35:20.412 INFO:tasks.workunit.client.0.smithi112.stdout:removed './243/32' 2023-01-24T03:35:20.412 INFO:tasks.workunit.client.0.smithi112.stdout:removed './243/22' 2023-01-24T03:35:20.412 INFO:tasks.workunit.client.0.smithi112.stdout:removed './243/25' 2023-01-24T03:35:20.412 INFO:tasks.workunit.client.0.smithi112.stdout:removed './243/30' 2023-01-24T03:35:20.412 INFO:tasks.workunit.client.0.smithi112.stdout:removed './243/33' 2023-01-24T03:35:20.413 INFO:tasks.workunit.client.0.smithi112.stdout:removed './243/23' 2023-01-24T03:35:20.413 INFO:tasks.workunit.client.0.smithi112.stdout:removed './243/35' 2023-01-24T03:35:20.413 INFO:tasks.workunit.client.0.smithi112.stdout:removed './243/26' 2023-01-24T03:35:20.413 INFO:tasks.workunit.client.0.smithi112.stdout:removed './243/37' 2023-01-24T03:35:20.413 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './243' 2023-01-24T03:35:20.413 INFO:tasks.workunit.client.0.smithi112.stdout:removed './244/21' 2023-01-24T03:35:20.414 INFO:tasks.workunit.client.0.smithi112.stdout:removed './244/27' 2023-01-24T03:35:20.414 INFO:tasks.workunit.client.0.smithi112.stdout:removed './244/31' 2023-01-24T03:35:20.414 INFO:tasks.workunit.client.0.smithi112.stdout:removed './244/36' 2023-01-24T03:35:20.414 INFO:tasks.workunit.client.0.smithi112.stdout:removed './244/20' 2023-01-24T03:35:20.414 INFO:tasks.workunit.client.0.smithi112.stdout:removed './244/39' 2023-01-24T03:35:20.415 INFO:tasks.workunit.client.0.smithi112.stdout:removed './244/34' 2023-01-24T03:35:20.415 INFO:tasks.workunit.client.0.smithi112.stdout:removed './244/38' 2023-01-24T03:35:20.415 INFO:tasks.workunit.client.0.smithi112.stdout:removed './244/28' 2023-01-24T03:35:20.415 INFO:tasks.workunit.client.0.smithi112.stdout:removed './244/24' 2023-01-24T03:35:20.415 INFO:tasks.workunit.client.0.smithi112.stdout:removed './244/29' 2023-01-24T03:35:20.415 INFO:tasks.workunit.client.0.smithi112.stdout:removed './244/32' 2023-01-24T03:35:20.416 INFO:tasks.workunit.client.0.smithi112.stdout:removed './244/22' 2023-01-24T03:35:20.416 INFO:tasks.workunit.client.0.smithi112.stdout:removed './244/25' 2023-01-24T03:35:20.416 INFO:tasks.workunit.client.0.smithi112.stdout:removed './244/30' 2023-01-24T03:35:20.416 INFO:tasks.workunit.client.0.smithi112.stdout:removed './244/33' 2023-01-24T03:35:20.416 INFO:tasks.workunit.client.0.smithi112.stdout:removed './244/23' 2023-01-24T03:35:20.417 INFO:tasks.workunit.client.0.smithi112.stdout:removed './244/35' 2023-01-24T03:35:20.417 INFO:tasks.workunit.client.0.smithi112.stdout:removed './244/26' 2023-01-24T03:35:20.417 INFO:tasks.workunit.client.0.smithi112.stdout:removed './244/37' 2023-01-24T03:35:20.417 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './244' 2023-01-24T03:35:20.417 INFO:tasks.workunit.client.0.smithi112.stdout:removed './245/21' 2023-01-24T03:35:20.417 INFO:tasks.workunit.client.0.smithi112.stdout:removed './245/27' 2023-01-24T03:35:20.418 INFO:tasks.workunit.client.0.smithi112.stdout:removed './245/31' 2023-01-24T03:35:20.418 INFO:tasks.workunit.client.0.smithi112.stdout:removed './245/36' 2023-01-24T03:35:20.418 INFO:tasks.workunit.client.0.smithi112.stdout:removed './245/20' 2023-01-24T03:35:20.418 INFO:tasks.workunit.client.0.smithi112.stdout:removed './245/39' 2023-01-24T03:35:20.418 INFO:tasks.workunit.client.0.smithi112.stdout:removed './245/34' 2023-01-24T03:35:20.419 INFO:tasks.workunit.client.0.smithi112.stdout:removed './245/38' 2023-01-24T03:35:20.419 INFO:tasks.workunit.client.0.smithi112.stdout:removed './245/28' 2023-01-24T03:35:20.419 INFO:tasks.workunit.client.0.smithi112.stdout:removed './245/24' 2023-01-24T03:35:20.419 INFO:tasks.workunit.client.0.smithi112.stdout:removed './245/29' 2023-01-24T03:35:20.419 INFO:tasks.workunit.client.0.smithi112.stdout:removed './245/32' 2023-01-24T03:35:20.419 INFO:tasks.workunit.client.0.smithi112.stdout:removed './245/22' 2023-01-24T03:35:20.420 INFO:tasks.workunit.client.0.smithi112.stdout:removed './245/25' 2023-01-24T03:35:20.420 INFO:tasks.workunit.client.0.smithi112.stdout:removed './245/30' 2023-01-24T03:35:20.420 INFO:tasks.workunit.client.0.smithi112.stdout:removed './245/33' 2023-01-24T03:35:20.420 INFO:tasks.workunit.client.0.smithi112.stdout:removed './245/23' 2023-01-24T03:35:20.420 INFO:tasks.workunit.client.0.smithi112.stdout:removed './245/35' 2023-01-24T03:35:20.421 INFO:tasks.workunit.client.0.smithi112.stdout:removed './245/26' 2023-01-24T03:35:20.421 INFO:tasks.workunit.client.0.smithi112.stdout:removed './245/37' 2023-01-24T03:35:20.421 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './245' 2023-01-24T03:35:20.421 INFO:tasks.workunit.client.0.smithi112.stdout:removed './246/21' 2023-01-24T03:35:20.421 INFO:tasks.workunit.client.0.smithi112.stdout:removed './246/27' 2023-01-24T03:35:20.422 INFO:tasks.workunit.client.0.smithi112.stdout:removed './246/31' 2023-01-24T03:35:20.422 INFO:tasks.workunit.client.0.smithi112.stdout:removed './246/36' 2023-01-24T03:35:20.422 INFO:tasks.workunit.client.0.smithi112.stdout:removed './246/20' 2023-01-24T03:35:20.422 INFO:tasks.workunit.client.0.smithi112.stdout:removed './246/39' 2023-01-24T03:35:20.422 INFO:tasks.workunit.client.0.smithi112.stdout:removed './246/34' 2023-01-24T03:35:20.422 INFO:tasks.workunit.client.0.smithi112.stdout:removed './246/38' 2023-01-24T03:35:20.423 INFO:tasks.workunit.client.0.smithi112.stdout:removed './246/28' 2023-01-24T03:35:20.423 INFO:tasks.workunit.client.0.smithi112.stdout:removed './246/24' 2023-01-24T03:35:20.423 INFO:tasks.workunit.client.0.smithi112.stdout:removed './246/29' 2023-01-24T03:35:20.423 INFO:tasks.workunit.client.0.smithi112.stdout:removed './246/32' 2023-01-24T03:35:20.423 INFO:tasks.workunit.client.0.smithi112.stdout:removed './246/22' 2023-01-24T03:35:20.758 INFO:tasks.workunit.client.0.smithi112.stdout:removed './246/25' 2023-01-24T03:35:20.758 INFO:tasks.workunit.client.0.smithi112.stdout:removed './246/30' 2023-01-24T03:35:20.758 INFO:tasks.workunit.client.0.smithi112.stdout:removed './246/33' 2023-01-24T03:35:20.758 INFO:tasks.workunit.client.0.smithi112.stdout:removed './246/23' 2023-01-24T03:35:20.759 INFO:tasks.workunit.client.0.smithi112.stdout:removed './246/35' 2023-01-24T03:35:20.759 INFO:tasks.workunit.client.0.smithi112.stdout:removed './246/26' 2023-01-24T03:35:20.759 INFO:tasks.workunit.client.0.smithi112.stdout:removed './246/37' 2023-01-24T03:35:20.759 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './246' 2023-01-24T03:35:20.759 INFO:tasks.workunit.client.0.smithi112.stdout:removed './247/21' 2023-01-24T03:35:20.760 INFO:tasks.workunit.client.0.smithi112.stdout:removed './247/27' 2023-01-24T03:35:20.760 INFO:tasks.workunit.client.0.smithi112.stdout:removed './247/31' 2023-01-24T03:35:20.760 INFO:tasks.workunit.client.0.smithi112.stdout:removed './247/36' 2023-01-24T03:35:20.760 INFO:tasks.workunit.client.0.smithi112.stdout:removed './247/20' 2023-01-24T03:35:20.760 INFO:tasks.workunit.client.0.smithi112.stdout:removed './247/39' 2023-01-24T03:35:20.761 INFO:tasks.workunit.client.0.smithi112.stdout:removed './247/34' 2023-01-24T03:35:20.761 INFO:tasks.workunit.client.0.smithi112.stdout:removed './247/38' 2023-01-24T03:35:20.761 INFO:tasks.workunit.client.0.smithi112.stdout:removed './247/28' 2023-01-24T03:35:20.761 INFO:tasks.workunit.client.0.smithi112.stdout:removed './247/24' 2023-01-24T03:35:20.762 INFO:tasks.workunit.client.0.smithi112.stdout:removed './247/29' 2023-01-24T03:35:20.762 INFO:tasks.workunit.client.0.smithi112.stdout:removed './247/32' 2023-01-24T03:35:20.762 INFO:tasks.workunit.client.0.smithi112.stdout:removed './247/22' 2023-01-24T03:35:20.762 INFO:tasks.workunit.client.0.smithi112.stdout:removed './247/25' 2023-01-24T03:35:20.762 INFO:tasks.workunit.client.0.smithi112.stdout:removed './247/30' 2023-01-24T03:35:20.762 INFO:tasks.workunit.client.0.smithi112.stdout:removed './247/33' 2023-01-24T03:35:20.763 INFO:tasks.workunit.client.0.smithi112.stdout:removed './247/23' 2023-01-24T03:35:20.763 INFO:tasks.workunit.client.0.smithi112.stdout:removed './247/35' 2023-01-24T03:35:20.763 INFO:tasks.workunit.client.0.smithi112.stdout:removed './247/26' 2023-01-24T03:35:20.763 INFO:tasks.workunit.client.0.smithi112.stdout:removed './247/37' 2023-01-24T03:35:20.763 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './247' 2023-01-24T03:35:20.764 INFO:tasks.workunit.client.0.smithi112.stdout:removed './248/21' 2023-01-24T03:35:20.764 INFO:tasks.workunit.client.0.smithi112.stdout:removed './248/27' 2023-01-24T03:35:20.764 INFO:tasks.workunit.client.0.smithi112.stdout:removed './248/31' 2023-01-24T03:35:20.764 INFO:tasks.workunit.client.0.smithi112.stdout:removed './248/36' 2023-01-24T03:35:20.764 INFO:tasks.workunit.client.0.smithi112.stdout:removed './248/20' 2023-01-24T03:35:20.765 INFO:tasks.workunit.client.0.smithi112.stdout:removed './248/39' 2023-01-24T03:35:20.765 INFO:tasks.workunit.client.0.smithi112.stdout:removed './248/34' 2023-01-24T03:35:20.765 INFO:tasks.workunit.client.0.smithi112.stdout:removed './248/38' 2023-01-24T03:35:20.765 INFO:tasks.workunit.client.0.smithi112.stdout:removed './248/28' 2023-01-24T03:35:20.765 INFO:tasks.workunit.client.0.smithi112.stdout:removed './248/24' 2023-01-24T03:35:20.766 INFO:tasks.workunit.client.0.smithi112.stdout:removed './248/29' 2023-01-24T03:35:20.766 INFO:tasks.workunit.client.0.smithi112.stdout:removed './248/32' 2023-01-24T03:35:20.766 INFO:tasks.workunit.client.0.smithi112.stdout:removed './248/22' 2023-01-24T03:35:20.766 INFO:tasks.workunit.client.0.smithi112.stdout:removed './248/25' 2023-01-24T03:35:20.766 INFO:tasks.workunit.client.0.smithi112.stdout:removed './248/30' 2023-01-24T03:35:20.767 INFO:tasks.workunit.client.0.smithi112.stdout:removed './248/33' 2023-01-24T03:35:20.767 INFO:tasks.workunit.client.0.smithi112.stdout:removed './248/23' 2023-01-24T03:35:20.767 INFO:tasks.workunit.client.0.smithi112.stdout:removed './248/35' 2023-01-24T03:35:20.767 INFO:tasks.workunit.client.0.smithi112.stdout:removed './248/26' 2023-01-24T03:35:20.767 INFO:tasks.workunit.client.0.smithi112.stdout:removed './248/37' 2023-01-24T03:35:20.768 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './248' 2023-01-24T03:35:20.768 INFO:tasks.workunit.client.0.smithi112.stdout:removed './249/21' 2023-01-24T03:35:20.768 INFO:tasks.workunit.client.0.smithi112.stdout:removed './249/27' 2023-01-24T03:35:20.768 INFO:tasks.workunit.client.0.smithi112.stdout:removed './249/31' 2023-01-24T03:35:20.768 INFO:tasks.workunit.client.0.smithi112.stdout:removed './249/36' 2023-01-24T03:35:20.769 INFO:tasks.workunit.client.0.smithi112.stdout:removed './249/20' 2023-01-24T03:35:20.769 INFO:tasks.workunit.client.0.smithi112.stdout:removed './249/39' 2023-01-24T03:35:20.769 INFO:tasks.workunit.client.0.smithi112.stdout:removed './249/34' 2023-01-24T03:35:20.769 INFO:tasks.workunit.client.0.smithi112.stdout:removed './249/38' 2023-01-24T03:35:20.769 INFO:tasks.workunit.client.0.smithi112.stdout:removed './249/28' 2023-01-24T03:35:20.769 INFO:tasks.workunit.client.0.smithi112.stdout:removed './249/24' 2023-01-24T03:35:20.770 INFO:tasks.workunit.client.0.smithi112.stdout:removed './249/29' 2023-01-24T03:35:20.770 INFO:tasks.workunit.client.0.smithi112.stdout:removed './249/32' 2023-01-24T03:35:20.770 INFO:tasks.workunit.client.0.smithi112.stdout:removed './249/22' 2023-01-24T03:35:20.770 INFO:tasks.workunit.client.0.smithi112.stdout:removed './249/25' 2023-01-24T03:35:20.770 INFO:tasks.workunit.client.0.smithi112.stdout:removed './249/30' 2023-01-24T03:35:20.771 INFO:tasks.workunit.client.0.smithi112.stdout:removed './249/33' 2023-01-24T03:35:20.771 INFO:tasks.workunit.client.0.smithi112.stdout:removed './249/23' 2023-01-24T03:35:20.771 INFO:tasks.workunit.client.0.smithi112.stdout:removed './249/35' 2023-01-24T03:35:20.771 INFO:tasks.workunit.client.0.smithi112.stdout:removed './249/26' 2023-01-24T03:35:20.771 INFO:tasks.workunit.client.0.smithi112.stdout:removed './249/37' 2023-01-24T03:35:20.772 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './249' 2023-01-24T03:35:20.772 INFO:tasks.workunit.client.0.smithi112.stdout:removed './250/21' 2023-01-24T03:35:20.772 INFO:tasks.workunit.client.0.smithi112.stdout:removed './250/27' 2023-01-24T03:35:20.772 INFO:tasks.workunit.client.0.smithi112.stdout:removed './250/31' 2023-01-24T03:35:20.772 INFO:tasks.workunit.client.0.smithi112.stdout:removed './250/36' 2023-01-24T03:35:20.773 INFO:tasks.workunit.client.0.smithi112.stdout:removed './250/20' 2023-01-24T03:35:20.773 INFO:tasks.workunit.client.0.smithi112.stdout:removed './250/39' 2023-01-24T03:35:20.773 INFO:tasks.workunit.client.0.smithi112.stdout:removed './250/34' 2023-01-24T03:35:20.773 INFO:tasks.workunit.client.0.smithi112.stdout:removed './250/38' 2023-01-24T03:35:20.773 INFO:tasks.workunit.client.0.smithi112.stdout:removed './250/28' 2023-01-24T03:35:20.774 INFO:tasks.workunit.client.0.smithi112.stdout:removed './250/24' 2023-01-24T03:35:20.774 INFO:tasks.workunit.client.0.smithi112.stdout:removed './250/29' 2023-01-24T03:35:20.774 INFO:tasks.workunit.client.0.smithi112.stdout:removed './250/32' 2023-01-24T03:35:20.774 INFO:tasks.workunit.client.0.smithi112.stdout:removed './250/22' 2023-01-24T03:35:20.774 INFO:tasks.workunit.client.0.smithi112.stdout:removed './250/25' 2023-01-24T03:35:20.775 INFO:tasks.workunit.client.0.smithi112.stdout:removed './250/30' 2023-01-24T03:35:20.775 INFO:tasks.workunit.client.0.smithi112.stdout:removed './250/33' 2023-01-24T03:35:20.775 INFO:tasks.workunit.client.0.smithi112.stdout:removed './250/23' 2023-01-24T03:35:20.775 INFO:tasks.workunit.client.0.smithi112.stdout:removed './250/35' 2023-01-24T03:35:20.775 INFO:tasks.workunit.client.0.smithi112.stdout:removed './250/26' 2023-01-24T03:35:20.775 INFO:tasks.workunit.client.0.smithi112.stdout:removed './250/37' 2023-01-24T03:35:20.776 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './250' 2023-01-24T03:35:20.776 INFO:tasks.workunit.client.0.smithi112.stdout:removed './251/21' 2023-01-24T03:35:20.776 INFO:tasks.workunit.client.0.smithi112.stdout:removed './251/27' 2023-01-24T03:35:20.776 INFO:tasks.workunit.client.0.smithi112.stdout:removed './251/31' 2023-01-24T03:35:20.777 INFO:tasks.workunit.client.0.smithi112.stdout:removed './251/36' 2023-01-24T03:35:20.777 INFO:tasks.workunit.client.0.smithi112.stdout:removed './251/20' 2023-01-24T03:35:20.777 INFO:tasks.workunit.client.0.smithi112.stdout:removed './251/39' 2023-01-24T03:35:20.777 INFO:tasks.workunit.client.0.smithi112.stdout:removed './251/34' 2023-01-24T03:35:20.777 INFO:tasks.workunit.client.0.smithi112.stdout:removed './251/38' 2023-01-24T03:35:20.777 INFO:tasks.workunit.client.0.smithi112.stdout:removed './251/28' 2023-01-24T03:35:20.778 INFO:tasks.workunit.client.0.smithi112.stdout:removed './251/24' 2023-01-24T03:35:20.778 INFO:tasks.workunit.client.0.smithi112.stdout:removed './251/29' 2023-01-24T03:35:20.778 INFO:tasks.workunit.client.0.smithi112.stdout:removed './251/32' 2023-01-24T03:35:20.778 INFO:tasks.workunit.client.0.smithi112.stdout:removed './251/22' 2023-01-24T03:35:20.779 INFO:tasks.workunit.client.0.smithi112.stdout:removed './251/25' 2023-01-24T03:35:20.779 INFO:tasks.workunit.client.0.smithi112.stdout:removed './251/30' 2023-01-24T03:35:20.779 INFO:tasks.workunit.client.0.smithi112.stdout:removed './251/33' 2023-01-24T03:35:20.779 INFO:tasks.workunit.client.0.smithi112.stdout:removed './251/23' 2023-01-24T03:35:20.779 INFO:tasks.workunit.client.0.smithi112.stdout:removed './251/35' 2023-01-24T03:35:20.779 INFO:tasks.workunit.client.0.smithi112.stdout:removed './251/26' 2023-01-24T03:35:20.780 INFO:tasks.workunit.client.0.smithi112.stdout:removed './251/37' 2023-01-24T03:35:20.780 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './251' 2023-01-24T03:35:20.780 INFO:tasks.workunit.client.0.smithi112.stdout:removed './252/21' 2023-01-24T03:35:20.780 INFO:tasks.workunit.client.0.smithi112.stdout:removed './252/27' 2023-01-24T03:35:20.780 INFO:tasks.workunit.client.0.smithi112.stdout:removed './252/31' 2023-01-24T03:35:20.781 INFO:tasks.workunit.client.0.smithi112.stdout:removed './252/36' 2023-01-24T03:35:20.781 INFO:tasks.workunit.client.0.smithi112.stdout:removed './252/20' 2023-01-24T03:35:20.781 INFO:tasks.workunit.client.0.smithi112.stdout:removed './252/39' 2023-01-24T03:35:20.781 INFO:tasks.workunit.client.0.smithi112.stdout:removed './252/34' 2023-01-24T03:35:20.781 INFO:tasks.workunit.client.0.smithi112.stdout:removed './252/38' 2023-01-24T03:35:20.782 INFO:tasks.workunit.client.0.smithi112.stdout:removed './252/28' 2023-01-24T03:35:20.782 INFO:tasks.workunit.client.0.smithi112.stdout:removed './252/24' 2023-01-24T03:35:20.782 INFO:tasks.workunit.client.0.smithi112.stdout:removed './252/29' 2023-01-24T03:35:20.782 INFO:tasks.workunit.client.0.smithi112.stdout:removed './252/32' 2023-01-24T03:35:20.782 INFO:tasks.workunit.client.0.smithi112.stdout:removed './252/22' 2023-01-24T03:35:20.783 INFO:tasks.workunit.client.0.smithi112.stdout:removed './252/25' 2023-01-24T03:35:20.783 INFO:tasks.workunit.client.0.smithi112.stdout:removed './252/30' 2023-01-24T03:35:20.783 INFO:tasks.workunit.client.0.smithi112.stdout:removed './252/33' 2023-01-24T03:35:20.783 INFO:tasks.workunit.client.0.smithi112.stdout:removed './252/23' 2023-01-24T03:35:20.783 INFO:tasks.workunit.client.0.smithi112.stdout:removed './252/35' 2023-01-24T03:35:20.784 INFO:tasks.workunit.client.0.smithi112.stdout:removed './252/26' 2023-01-24T03:35:20.784 INFO:tasks.workunit.client.0.smithi112.stdout:removed './252/37' 2023-01-24T03:35:20.784 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './252' 2023-01-24T03:35:20.784 INFO:tasks.workunit.client.0.smithi112.stdout:removed './253/21' 2023-01-24T03:35:20.784 INFO:tasks.workunit.client.0.smithi112.stdout:removed './253/27' 2023-01-24T03:35:20.785 INFO:tasks.workunit.client.0.smithi112.stdout:removed './253/31' 2023-01-24T03:35:20.785 INFO:tasks.workunit.client.0.smithi112.stdout:removed './253/36' 2023-01-24T03:35:20.785 INFO:tasks.workunit.client.0.smithi112.stdout:removed './253/20' 2023-01-24T03:35:20.785 INFO:tasks.workunit.client.0.smithi112.stdout:removed './253/39' 2023-01-24T03:35:20.785 INFO:tasks.workunit.client.0.smithi112.stdout:removed './253/34' 2023-01-24T03:35:20.786 INFO:tasks.workunit.client.0.smithi112.stdout:removed './253/38' 2023-01-24T03:35:20.786 INFO:tasks.workunit.client.0.smithi112.stdout:removed './253/28' 2023-01-24T03:35:20.786 INFO:tasks.workunit.client.0.smithi112.stdout:removed './253/24' 2023-01-24T03:35:20.786 INFO:tasks.workunit.client.0.smithi112.stdout:removed './253/29' 2023-01-24T03:35:20.786 INFO:tasks.workunit.client.0.smithi112.stdout:removed './253/32' 2023-01-24T03:35:20.787 INFO:tasks.workunit.client.0.smithi112.stdout:removed './253/22' 2023-01-24T03:35:20.787 INFO:tasks.workunit.client.0.smithi112.stdout:removed './253/25' 2023-01-24T03:35:20.787 INFO:tasks.workunit.client.0.smithi112.stdout:removed './253/30' 2023-01-24T03:35:20.787 INFO:tasks.workunit.client.0.smithi112.stdout:removed './253/33' 2023-01-24T03:35:20.787 INFO:tasks.workunit.client.0.smithi112.stdout:removed './253/23' 2023-01-24T03:35:20.788 INFO:tasks.workunit.client.0.smithi112.stdout:removed './253/35' 2023-01-24T03:35:20.788 INFO:tasks.workunit.client.0.smithi112.stdout:removed './253/26' 2023-01-24T03:35:20.788 INFO:tasks.workunit.client.0.smithi112.stdout:removed './253/37' 2023-01-24T03:35:20.788 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './253' 2023-01-24T03:35:20.788 INFO:tasks.workunit.client.0.smithi112.stdout:removed './254/21' 2023-01-24T03:35:20.789 INFO:tasks.workunit.client.0.smithi112.stdout:removed './254/27' 2023-01-24T03:35:20.789 INFO:tasks.workunit.client.0.smithi112.stdout:removed './254/31' 2023-01-24T03:35:20.789 INFO:tasks.workunit.client.0.smithi112.stdout:removed './254/36' 2023-01-24T03:35:20.789 INFO:tasks.workunit.client.0.smithi112.stdout:removed './254/20' 2023-01-24T03:35:20.789 INFO:tasks.workunit.client.0.smithi112.stdout:removed './254/39' 2023-01-24T03:35:20.789 INFO:tasks.workunit.client.0.smithi112.stdout:removed './254/34' 2023-01-24T03:35:20.790 INFO:tasks.workunit.client.0.smithi112.stdout:removed './254/38' 2023-01-24T03:35:20.790 INFO:tasks.workunit.client.0.smithi112.stdout:removed './254/28' 2023-01-24T03:35:20.790 INFO:tasks.workunit.client.0.smithi112.stdout:removed './254/24' 2023-01-24T03:35:20.790 INFO:tasks.workunit.client.0.smithi112.stdout:removed './254/29' 2023-01-24T03:35:20.790 INFO:tasks.workunit.client.0.smithi112.stdout:removed './254/32' 2023-01-24T03:35:20.791 INFO:tasks.workunit.client.0.smithi112.stdout:removed './254/22' 2023-01-24T03:35:20.791 INFO:tasks.workunit.client.0.smithi112.stdout:removed './254/25' 2023-01-24T03:35:20.791 INFO:tasks.workunit.client.0.smithi112.stdout:removed './254/30' 2023-01-24T03:35:20.791 INFO:tasks.workunit.client.0.smithi112.stdout:removed './254/33' 2023-01-24T03:35:20.791 INFO:tasks.workunit.client.0.smithi112.stdout:removed './254/23' 2023-01-24T03:35:20.791 INFO:tasks.workunit.client.0.smithi112.stdout:removed './254/35' 2023-01-24T03:35:20.792 INFO:tasks.workunit.client.0.smithi112.stdout:removed './254/26' 2023-01-24T03:35:20.792 INFO:tasks.workunit.client.0.smithi112.stdout:removed './254/37' 2023-01-24T03:35:20.792 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './254' 2023-01-24T03:35:20.792 INFO:tasks.workunit.client.0.smithi112.stdout:removed './255/21' 2023-01-24T03:35:20.792 INFO:tasks.workunit.client.0.smithi112.stdout:removed './255/27' 2023-01-24T03:35:20.793 INFO:tasks.workunit.client.0.smithi112.stdout:removed './255/31' 2023-01-24T03:35:20.793 INFO:tasks.workunit.client.0.smithi112.stdout:removed './255/36' 2023-01-24T03:35:20.793 INFO:tasks.workunit.client.0.smithi112.stdout:removed './255/20' 2023-01-24T03:35:20.793 INFO:tasks.workunit.client.0.smithi112.stdout:removed './255/39' 2023-01-24T03:35:20.793 INFO:tasks.workunit.client.0.smithi112.stdout:removed './255/34' 2023-01-24T03:35:20.794 INFO:tasks.workunit.client.0.smithi112.stdout:removed './255/38' 2023-01-24T03:35:20.794 INFO:tasks.workunit.client.0.smithi112.stdout:removed './255/28' 2023-01-24T03:35:20.794 INFO:tasks.workunit.client.0.smithi112.stdout:removed './255/24' 2023-01-24T03:35:20.794 INFO:tasks.workunit.client.0.smithi112.stdout:removed './255/29' 2023-01-24T03:35:20.794 INFO:tasks.workunit.client.0.smithi112.stdout:removed './255/32' 2023-01-24T03:35:20.794 INFO:tasks.workunit.client.0.smithi112.stdout:removed './255/22' 2023-01-24T03:35:20.795 INFO:tasks.workunit.client.0.smithi112.stdout:removed './255/25' 2023-01-24T03:35:20.795 INFO:tasks.workunit.client.0.smithi112.stdout:removed './255/30' 2023-01-24T03:35:20.795 INFO:tasks.workunit.client.0.smithi112.stdout:removed './255/33' 2023-01-24T03:35:20.795 INFO:tasks.workunit.client.0.smithi112.stdout:removed './255/23' 2023-01-24T03:35:20.795 INFO:tasks.workunit.client.0.smithi112.stdout:removed './255/35' 2023-01-24T03:35:20.796 INFO:tasks.workunit.client.0.smithi112.stdout:removed './255/26' 2023-01-24T03:35:20.796 INFO:tasks.workunit.client.0.smithi112.stdout:removed './255/37' 2023-01-24T03:35:20.796 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './255' 2023-01-24T03:35:20.796 INFO:tasks.workunit.client.0.smithi112.stdout:removed './256/21' 2023-01-24T03:35:20.796 INFO:tasks.workunit.client.0.smithi112.stdout:removed './256/27' 2023-01-24T03:35:20.797 INFO:tasks.workunit.client.0.smithi112.stdout:removed './256/31' 2023-01-24T03:35:20.797 INFO:tasks.workunit.client.0.smithi112.stdout:removed './256/36' 2023-01-24T03:35:20.797 INFO:tasks.workunit.client.0.smithi112.stdout:removed './256/20' 2023-01-24T03:35:20.797 INFO:tasks.workunit.client.0.smithi112.stdout:removed './256/39' 2023-01-24T03:35:20.797 INFO:tasks.workunit.client.0.smithi112.stdout:removed './256/34' 2023-01-24T03:35:20.797 INFO:tasks.workunit.client.0.smithi112.stdout:removed './256/38' 2023-01-24T03:35:20.798 INFO:tasks.workunit.client.0.smithi112.stdout:removed './256/28' 2023-01-24T03:35:20.798 INFO:tasks.workunit.client.0.smithi112.stdout:removed './256/24' 2023-01-24T03:35:20.798 INFO:tasks.workunit.client.0.smithi112.stdout:removed './256/29' 2023-01-24T03:35:20.798 INFO:tasks.workunit.client.0.smithi112.stdout:removed './256/32' 2023-01-24T03:35:20.798 INFO:tasks.workunit.client.0.smithi112.stdout:removed './256/22' 2023-01-24T03:35:20.798 INFO:tasks.workunit.client.0.smithi112.stdout:removed './256/25' 2023-01-24T03:35:20.799 INFO:tasks.workunit.client.0.smithi112.stdout:removed './256/30' 2023-01-24T03:35:22.020 INFO:tasks.workunit.client.0.smithi112.stdout:removed './256/33' 2023-01-24T03:35:22.021 INFO:tasks.workunit.client.0.smithi112.stdout:removed './256/23' 2023-01-24T03:35:22.021 INFO:tasks.workunit.client.0.smithi112.stdout:removed './256/35' 2023-01-24T03:35:22.021 INFO:tasks.workunit.client.0.smithi112.stdout:removed './256/26' 2023-01-24T03:35:22.021 INFO:tasks.workunit.client.0.smithi112.stdout:removed './256/37' 2023-01-24T03:35:22.021 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './256' 2023-01-24T03:35:22.021 INFO:tasks.workunit.client.0.smithi112.stdout:removed './257/21' 2023-01-24T03:35:22.022 INFO:tasks.workunit.client.0.smithi112.stdout:removed './257/27' 2023-01-24T03:35:22.022 INFO:tasks.workunit.client.0.smithi112.stdout:removed './257/31' 2023-01-24T03:35:22.022 INFO:tasks.workunit.client.0.smithi112.stdout:removed './257/36' 2023-01-24T03:35:22.022 INFO:tasks.workunit.client.0.smithi112.stdout:removed './257/20' 2023-01-24T03:35:22.022 INFO:tasks.workunit.client.0.smithi112.stdout:removed './257/39' 2023-01-24T03:35:22.023 INFO:tasks.workunit.client.0.smithi112.stdout:removed './257/34' 2023-01-24T03:35:22.023 INFO:tasks.workunit.client.0.smithi112.stdout:removed './257/38' 2023-01-24T03:35:22.023 INFO:tasks.workunit.client.0.smithi112.stdout:removed './257/28' 2023-01-24T03:35:22.023 INFO:tasks.workunit.client.0.smithi112.stdout:removed './257/24' 2023-01-24T03:35:22.023 INFO:tasks.workunit.client.0.smithi112.stdout:removed './257/29' 2023-01-24T03:35:22.024 INFO:tasks.workunit.client.0.smithi112.stdout:removed './257/32' 2023-01-24T03:35:22.024 INFO:tasks.workunit.client.0.smithi112.stdout:removed './257/22' 2023-01-24T03:35:22.024 INFO:tasks.workunit.client.0.smithi112.stdout:removed './257/25' 2023-01-24T03:35:22.024 INFO:tasks.workunit.client.0.smithi112.stdout:removed './257/30' 2023-01-24T03:35:22.024 INFO:tasks.workunit.client.0.smithi112.stdout:removed './257/33' 2023-01-24T03:35:22.025 INFO:tasks.workunit.client.0.smithi112.stdout:removed './257/23' 2023-01-24T03:35:22.025 INFO:tasks.workunit.client.0.smithi112.stdout:removed './257/35' 2023-01-24T03:35:22.025 INFO:tasks.workunit.client.0.smithi112.stdout:removed './257/26' 2023-01-24T03:35:22.025 INFO:tasks.workunit.client.0.smithi112.stdout:removed './257/37' 2023-01-24T03:35:22.025 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './257' 2023-01-24T03:35:22.026 INFO:tasks.workunit.client.0.smithi112.stdout:removed './258/21' 2023-01-24T03:35:22.026 INFO:tasks.workunit.client.0.smithi112.stdout:removed './258/27' 2023-01-24T03:35:22.026 INFO:tasks.workunit.client.0.smithi112.stdout:removed './258/31' 2023-01-24T03:35:22.026 INFO:tasks.workunit.client.0.smithi112.stdout:removed './258/36' 2023-01-24T03:35:22.026 INFO:tasks.workunit.client.0.smithi112.stdout:removed './258/20' 2023-01-24T03:35:22.026 INFO:tasks.workunit.client.0.smithi112.stdout:removed './258/39' 2023-01-24T03:35:22.027 INFO:tasks.workunit.client.0.smithi112.stdout:removed './258/34' 2023-01-24T03:35:22.027 INFO:tasks.workunit.client.0.smithi112.stdout:removed './258/38' 2023-01-24T03:35:22.027 INFO:tasks.workunit.client.0.smithi112.stdout:removed './258/28' 2023-01-24T03:35:22.027 INFO:tasks.workunit.client.0.smithi112.stdout:removed './258/24' 2023-01-24T03:35:22.027 INFO:tasks.workunit.client.0.smithi112.stdout:removed './258/29' 2023-01-24T03:35:22.028 INFO:tasks.workunit.client.0.smithi112.stdout:removed './258/32' 2023-01-24T03:35:22.028 INFO:tasks.workunit.client.0.smithi112.stdout:removed './258/22' 2023-01-24T03:35:22.028 INFO:tasks.workunit.client.0.smithi112.stdout:removed './258/25' 2023-01-24T03:35:22.028 INFO:tasks.workunit.client.0.smithi112.stdout:removed './258/30' 2023-01-24T03:35:22.028 INFO:tasks.workunit.client.0.smithi112.stdout:removed './258/33' 2023-01-24T03:35:22.029 INFO:tasks.workunit.client.0.smithi112.stdout:removed './258/23' 2023-01-24T03:35:22.029 INFO:tasks.workunit.client.0.smithi112.stdout:removed './258/35' 2023-01-24T03:35:22.029 INFO:tasks.workunit.client.0.smithi112.stdout:removed './258/26' 2023-01-24T03:35:22.029 INFO:tasks.workunit.client.0.smithi112.stdout:removed './258/37' 2023-01-24T03:35:22.029 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './258' 2023-01-24T03:35:22.029 INFO:tasks.workunit.client.0.smithi112.stdout:removed './259/21' 2023-01-24T03:35:22.030 INFO:tasks.workunit.client.0.smithi112.stdout:removed './259/27' 2023-01-24T03:35:22.030 INFO:tasks.workunit.client.0.smithi112.stdout:removed './259/31' 2023-01-24T03:35:22.030 INFO:tasks.workunit.client.0.smithi112.stdout:removed './259/36' 2023-01-24T03:35:22.030 INFO:tasks.workunit.client.0.smithi112.stdout:removed './259/20' 2023-01-24T03:35:22.030 INFO:tasks.workunit.client.0.smithi112.stdout:removed './259/39' 2023-01-24T03:35:22.031 INFO:tasks.workunit.client.0.smithi112.stdout:removed './259/34' 2023-01-24T03:35:22.031 INFO:tasks.workunit.client.0.smithi112.stdout:removed './259/38' 2023-01-24T03:35:22.031 INFO:tasks.workunit.client.0.smithi112.stdout:removed './259/28' 2023-01-24T03:35:22.031 INFO:tasks.workunit.client.0.smithi112.stdout:removed './259/24' 2023-01-24T03:35:22.031 INFO:tasks.workunit.client.0.smithi112.stdout:removed './259/29' 2023-01-24T03:35:22.032 INFO:tasks.workunit.client.0.smithi112.stdout:removed './259/32' 2023-01-24T03:35:22.032 INFO:tasks.workunit.client.0.smithi112.stdout:removed './259/22' 2023-01-24T03:35:22.032 INFO:tasks.workunit.client.0.smithi112.stdout:removed './259/25' 2023-01-24T03:35:22.032 INFO:tasks.workunit.client.0.smithi112.stdout:removed './259/30' 2023-01-24T03:35:22.032 INFO:tasks.workunit.client.0.smithi112.stdout:removed './259/33' 2023-01-24T03:35:22.033 INFO:tasks.workunit.client.0.smithi112.stdout:removed './259/23' 2023-01-24T03:35:22.033 INFO:tasks.workunit.client.0.smithi112.stdout:removed './259/35' 2023-01-24T03:35:22.033 INFO:tasks.workunit.client.0.smithi112.stdout:removed './259/26' 2023-01-24T03:35:22.033 INFO:tasks.workunit.client.0.smithi112.stdout:removed './259/37' 2023-01-24T03:35:22.033 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './259' 2023-01-24T03:35:22.034 INFO:tasks.workunit.client.0.smithi112.stdout:removed './260/21' 2023-01-24T03:35:22.034 INFO:tasks.workunit.client.0.smithi112.stdout:removed './260/27' 2023-01-24T03:35:22.034 INFO:tasks.workunit.client.0.smithi112.stdout:removed './260/31' 2023-01-24T03:35:22.034 INFO:tasks.workunit.client.0.smithi112.stdout:removed './260/36' 2023-01-24T03:35:22.034 INFO:tasks.workunit.client.0.smithi112.stdout:removed './260/20' 2023-01-24T03:35:22.035 INFO:tasks.workunit.client.0.smithi112.stdout:removed './260/39' 2023-01-24T03:35:22.035 INFO:tasks.workunit.client.0.smithi112.stdout:removed './260/34' 2023-01-24T03:35:22.035 INFO:tasks.workunit.client.0.smithi112.stdout:removed './260/38' 2023-01-24T03:35:22.035 INFO:tasks.workunit.client.0.smithi112.stdout:removed './260/28' 2023-01-24T03:35:22.035 INFO:tasks.workunit.client.0.smithi112.stdout:removed './260/24' 2023-01-24T03:35:22.036 INFO:tasks.workunit.client.0.smithi112.stdout:removed './260/29' 2023-01-24T03:35:22.036 INFO:tasks.workunit.client.0.smithi112.stdout:removed './260/32' 2023-01-24T03:35:22.036 INFO:tasks.workunit.client.0.smithi112.stdout:removed './260/22' 2023-01-24T03:35:22.036 INFO:tasks.workunit.client.0.smithi112.stdout:removed './260/25' 2023-01-24T03:35:22.036 INFO:tasks.workunit.client.0.smithi112.stdout:removed './260/30' 2023-01-24T03:35:22.037 INFO:tasks.workunit.client.0.smithi112.stdout:removed './260/33' 2023-01-24T03:35:22.037 INFO:tasks.workunit.client.0.smithi112.stdout:removed './260/23' 2023-01-24T03:35:22.037 INFO:tasks.workunit.client.0.smithi112.stdout:removed './260/35' 2023-01-24T03:35:22.037 INFO:tasks.workunit.client.0.smithi112.stdout:removed './260/26' 2023-01-24T03:35:22.037 INFO:tasks.workunit.client.0.smithi112.stdout:removed './260/37' 2023-01-24T03:35:22.038 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './260' 2023-01-24T03:35:22.038 INFO:tasks.workunit.client.0.smithi112.stdout:removed './261/21' 2023-01-24T03:35:22.038 INFO:tasks.workunit.client.0.smithi112.stdout:removed './261/27' 2023-01-24T03:35:22.038 INFO:tasks.workunit.client.0.smithi112.stdout:removed './261/31' 2023-01-24T03:35:22.038 INFO:tasks.workunit.client.0.smithi112.stdout:removed './261/36' 2023-01-24T03:35:22.038 INFO:tasks.workunit.client.0.smithi112.stdout:removed './261/20' 2023-01-24T03:35:22.039 INFO:tasks.workunit.client.0.smithi112.stdout:removed './261/39' 2023-01-24T03:35:22.039 INFO:tasks.workunit.client.0.smithi112.stdout:removed './261/34' 2023-01-24T03:35:22.039 INFO:tasks.workunit.client.0.smithi112.stdout:removed './261/38' 2023-01-24T03:35:22.039 INFO:tasks.workunit.client.0.smithi112.stdout:removed './261/28' 2023-01-24T03:35:22.039 INFO:tasks.workunit.client.0.smithi112.stdout:removed './261/24' 2023-01-24T03:35:22.040 INFO:tasks.workunit.client.0.smithi112.stdout:removed './261/29' 2023-01-24T03:35:22.040 INFO:tasks.workunit.client.0.smithi112.stdout:removed './261/32' 2023-01-24T03:35:22.040 INFO:tasks.workunit.client.0.smithi112.stdout:removed './261/22' 2023-01-24T03:35:22.040 INFO:tasks.workunit.client.0.smithi112.stdout:removed './261/25' 2023-01-24T03:35:22.040 INFO:tasks.workunit.client.0.smithi112.stdout:removed './261/30' 2023-01-24T03:35:22.041 INFO:tasks.workunit.client.0.smithi112.stdout:removed './261/33' 2023-01-24T03:35:22.041 INFO:tasks.workunit.client.0.smithi112.stdout:removed './261/23' 2023-01-24T03:35:22.041 INFO:tasks.workunit.client.0.smithi112.stdout:removed './261/35' 2023-01-24T03:35:22.041 INFO:tasks.workunit.client.0.smithi112.stdout:removed './261/26' 2023-01-24T03:35:22.042 INFO:tasks.workunit.client.0.smithi112.stdout:removed './261/37' 2023-01-24T03:35:22.042 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './261' 2023-01-24T03:35:22.042 INFO:tasks.workunit.client.0.smithi112.stdout:removed './262/21' 2023-01-24T03:35:22.042 INFO:tasks.workunit.client.0.smithi112.stdout:removed './262/27' 2023-01-24T03:35:22.042 INFO:tasks.workunit.client.0.smithi112.stdout:removed './262/31' 2023-01-24T03:35:22.043 INFO:tasks.workunit.client.0.smithi112.stdout:removed './262/36' 2023-01-24T03:35:22.043 INFO:tasks.workunit.client.0.smithi112.stdout:removed './262/20' 2023-01-24T03:35:22.043 INFO:tasks.workunit.client.0.smithi112.stdout:removed './262/39' 2023-01-24T03:35:22.043 INFO:tasks.workunit.client.0.smithi112.stdout:removed './262/34' 2023-01-24T03:35:22.043 INFO:tasks.workunit.client.0.smithi112.stdout:removed './262/38' 2023-01-24T03:35:22.044 INFO:tasks.workunit.client.0.smithi112.stdout:removed './262/28' 2023-01-24T03:35:22.044 INFO:tasks.workunit.client.0.smithi112.stdout:removed './262/24' 2023-01-24T03:35:22.044 INFO:tasks.workunit.client.0.smithi112.stdout:removed './262/29' 2023-01-24T03:35:22.044 INFO:tasks.workunit.client.0.smithi112.stdout:removed './262/32' 2023-01-24T03:35:22.045 INFO:tasks.workunit.client.0.smithi112.stdout:removed './262/22' 2023-01-24T03:35:22.045 INFO:tasks.workunit.client.0.smithi112.stdout:removed './262/25' 2023-01-24T03:35:22.045 INFO:tasks.workunit.client.0.smithi112.stdout:removed './262/30' 2023-01-24T03:35:22.045 INFO:tasks.workunit.client.0.smithi112.stdout:removed './262/33' 2023-01-24T03:35:22.045 INFO:tasks.workunit.client.0.smithi112.stdout:removed './262/23' 2023-01-24T03:35:22.045 INFO:tasks.workunit.client.0.smithi112.stdout:removed './262/35' 2023-01-24T03:35:22.046 INFO:tasks.workunit.client.0.smithi112.stdout:removed './262/26' 2023-01-24T03:35:22.046 INFO:tasks.workunit.client.0.smithi112.stdout:removed './262/37' 2023-01-24T03:35:22.046 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './262' 2023-01-24T03:35:22.046 INFO:tasks.workunit.client.0.smithi112.stdout:removed './263/21' 2023-01-24T03:35:22.046 INFO:tasks.workunit.client.0.smithi112.stdout:removed './263/27' 2023-01-24T03:35:22.047 INFO:tasks.workunit.client.0.smithi112.stdout:removed './263/31' 2023-01-24T03:35:22.047 INFO:tasks.workunit.client.0.smithi112.stdout:removed './263/36' 2023-01-24T03:35:22.047 INFO:tasks.workunit.client.0.smithi112.stdout:removed './263/20' 2023-01-24T03:35:22.047 INFO:tasks.workunit.client.0.smithi112.stdout:removed './263/39' 2023-01-24T03:35:22.047 INFO:tasks.workunit.client.0.smithi112.stdout:removed './263/34' 2023-01-24T03:35:22.048 INFO:tasks.workunit.client.0.smithi112.stdout:removed './263/38' 2023-01-24T03:35:22.048 INFO:tasks.workunit.client.0.smithi112.stdout:removed './263/28' 2023-01-24T03:35:22.048 INFO:tasks.workunit.client.0.smithi112.stdout:removed './263/24' 2023-01-24T03:35:22.048 INFO:tasks.workunit.client.0.smithi112.stdout:removed './263/29' 2023-01-24T03:35:22.048 INFO:tasks.workunit.client.0.smithi112.stdout:removed './263/32' 2023-01-24T03:35:22.049 INFO:tasks.workunit.client.0.smithi112.stdout:removed './263/22' 2023-01-24T03:35:22.049 INFO:tasks.workunit.client.0.smithi112.stdout:removed './263/25' 2023-01-24T03:35:22.049 INFO:tasks.workunit.client.0.smithi112.stdout:removed './263/30' 2023-01-24T03:35:22.049 INFO:tasks.workunit.client.0.smithi112.stdout:removed './263/33' 2023-01-24T03:35:22.049 INFO:tasks.workunit.client.0.smithi112.stdout:removed './263/23' 2023-01-24T03:35:22.050 INFO:tasks.workunit.client.0.smithi112.stdout:removed './263/35' 2023-01-24T03:35:22.050 INFO:tasks.workunit.client.0.smithi112.stdout:removed './263/26' 2023-01-24T03:35:22.050 INFO:tasks.workunit.client.0.smithi112.stdout:removed './263/37' 2023-01-24T03:35:22.050 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './263' 2023-01-24T03:35:22.050 INFO:tasks.workunit.client.0.smithi112.stdout:removed './264/21' 2023-01-24T03:35:22.050 INFO:tasks.workunit.client.0.smithi112.stdout:removed './264/27' 2023-01-24T03:35:22.051 INFO:tasks.workunit.client.0.smithi112.stdout:removed './264/31' 2023-01-24T03:35:22.051 INFO:tasks.workunit.client.0.smithi112.stdout:removed './264/36' 2023-01-24T03:35:22.051 INFO:tasks.workunit.client.0.smithi112.stdout:removed './264/20' 2023-01-24T03:35:22.051 INFO:tasks.workunit.client.0.smithi112.stdout:removed './264/39' 2023-01-24T03:35:22.052 INFO:tasks.workunit.client.0.smithi112.stdout:removed './264/34' 2023-01-24T03:35:22.052 INFO:tasks.workunit.client.0.smithi112.stdout:removed './264/38' 2023-01-24T03:35:22.052 INFO:tasks.workunit.client.0.smithi112.stdout:removed './264/28' 2023-01-24T03:35:22.052 INFO:tasks.workunit.client.0.smithi112.stdout:removed './264/24' 2023-01-24T03:35:22.053 INFO:tasks.workunit.client.0.smithi112.stdout:removed './264/29' 2023-01-24T03:35:22.053 INFO:tasks.workunit.client.0.smithi112.stdout:removed './264/32' 2023-01-24T03:35:22.053 INFO:tasks.workunit.client.0.smithi112.stdout:removed './264/22' 2023-01-24T03:35:22.053 INFO:tasks.workunit.client.0.smithi112.stdout:removed './264/25' 2023-01-24T03:35:22.053 INFO:tasks.workunit.client.0.smithi112.stdout:removed './264/30' 2023-01-24T03:35:22.054 INFO:tasks.workunit.client.0.smithi112.stdout:removed './264/33' 2023-01-24T03:35:22.054 INFO:tasks.workunit.client.0.smithi112.stdout:removed './264/23' 2023-01-24T03:35:22.054 INFO:tasks.workunit.client.0.smithi112.stdout:removed './264/35' 2023-01-24T03:35:22.054 INFO:tasks.workunit.client.0.smithi112.stdout:removed './264/26' 2023-01-24T03:35:22.054 INFO:tasks.workunit.client.0.smithi112.stdout:removed './264/37' 2023-01-24T03:35:22.055 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './264' 2023-01-24T03:35:22.055 INFO:tasks.workunit.client.0.smithi112.stdout:removed './265/21' 2023-01-24T03:35:22.055 INFO:tasks.workunit.client.0.smithi112.stdout:removed './265/27' 2023-01-24T03:35:22.055 INFO:tasks.workunit.client.0.smithi112.stdout:removed './265/31' 2023-01-24T03:35:22.055 INFO:tasks.workunit.client.0.smithi112.stdout:removed './265/36' 2023-01-24T03:35:22.055 INFO:tasks.workunit.client.0.smithi112.stdout:removed './265/20' 2023-01-24T03:35:22.056 INFO:tasks.workunit.client.0.smithi112.stdout:removed './265/39' 2023-01-24T03:35:22.056 INFO:tasks.workunit.client.0.smithi112.stdout:removed './265/34' 2023-01-24T03:35:22.056 INFO:tasks.workunit.client.0.smithi112.stdout:removed './265/38' 2023-01-24T03:35:22.056 INFO:tasks.workunit.client.0.smithi112.stdout:removed './265/28' 2023-01-24T03:35:22.056 INFO:tasks.workunit.client.0.smithi112.stdout:removed './265/24' 2023-01-24T03:35:22.057 INFO:tasks.workunit.client.0.smithi112.stdout:removed './265/29' 2023-01-24T03:35:22.057 INFO:tasks.workunit.client.0.smithi112.stdout:removed './265/32' 2023-01-24T03:35:22.057 INFO:tasks.workunit.client.0.smithi112.stdout:removed './265/22' 2023-01-24T03:35:22.057 INFO:tasks.workunit.client.0.smithi112.stdout:removed './265/25' 2023-01-24T03:35:22.057 INFO:tasks.workunit.client.0.smithi112.stdout:removed './265/30' 2023-01-24T03:35:22.057 INFO:tasks.workunit.client.0.smithi112.stdout:removed './265/33' 2023-01-24T03:35:22.058 INFO:tasks.workunit.client.0.smithi112.stdout:removed './265/23' 2023-01-24T03:35:22.058 INFO:tasks.workunit.client.0.smithi112.stdout:removed './265/35' 2023-01-24T03:35:22.058 INFO:tasks.workunit.client.0.smithi112.stdout:removed './265/26' 2023-01-24T03:35:22.058 INFO:tasks.workunit.client.0.smithi112.stdout:removed './265/37' 2023-01-24T03:35:22.058 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './265' 2023-01-24T03:35:22.059 INFO:tasks.workunit.client.0.smithi112.stdout:removed './266/21' 2023-01-24T03:35:22.059 INFO:tasks.workunit.client.0.smithi112.stdout:removed './266/27' 2023-01-24T03:35:22.059 INFO:tasks.workunit.client.0.smithi112.stdout:removed './266/31' 2023-01-24T03:35:22.059 INFO:tasks.workunit.client.0.smithi112.stdout:removed './266/36' 2023-01-24T03:35:22.059 INFO:tasks.workunit.client.0.smithi112.stdout:removed './266/20' 2023-01-24T03:35:22.059 INFO:tasks.workunit.client.0.smithi112.stdout:removed './266/39' 2023-01-24T03:35:22.060 INFO:tasks.workunit.client.0.smithi112.stdout:removed './266/34' 2023-01-24T03:35:22.060 INFO:tasks.workunit.client.0.smithi112.stdout:removed './266/38' 2023-01-24T03:35:22.060 INFO:tasks.workunit.client.0.smithi112.stdout:removed './266/28' 2023-01-24T03:35:22.060 INFO:tasks.workunit.client.0.smithi112.stdout:removed './266/24' 2023-01-24T03:35:22.061 INFO:tasks.workunit.client.0.smithi112.stdout:removed './266/29' 2023-01-24T03:35:22.061 INFO:tasks.workunit.client.0.smithi112.stdout:removed './266/32' 2023-01-24T03:35:22.061 INFO:tasks.workunit.client.0.smithi112.stdout:removed './266/22' 2023-01-24T03:35:22.061 INFO:tasks.workunit.client.0.smithi112.stdout:removed './266/25' 2023-01-24T03:35:22.062 INFO:tasks.workunit.client.0.smithi112.stdout:removed './266/30' 2023-01-24T03:35:22.062 INFO:tasks.workunit.client.0.smithi112.stdout:removed './266/33' 2023-01-24T03:35:22.062 INFO:tasks.workunit.client.0.smithi112.stdout:removed './266/23' 2023-01-24T03:35:23.408 INFO:tasks.workunit.client.0.smithi112.stdout:removed './266/35' 2023-01-24T03:35:23.408 INFO:tasks.workunit.client.0.smithi112.stdout:removed './266/26' 2023-01-24T03:35:23.410 INFO:tasks.workunit.client.0.smithi112.stdout:removed './266/37' 2023-01-24T03:35:23.410 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './266' 2023-01-24T03:35:23.410 INFO:tasks.workunit.client.0.smithi112.stdout:removed './267/21' 2023-01-24T03:35:23.410 INFO:tasks.workunit.client.0.smithi112.stdout:removed './267/27' 2023-01-24T03:35:23.410 INFO:tasks.workunit.client.0.smithi112.stdout:removed './267/31' 2023-01-24T03:35:23.411 INFO:tasks.workunit.client.0.smithi112.stdout:removed './267/36' 2023-01-24T03:35:23.411 INFO:tasks.workunit.client.0.smithi112.stdout:removed './267/20' 2023-01-24T03:35:23.411 INFO:tasks.workunit.client.0.smithi112.stdout:removed './267/39' 2023-01-24T03:35:23.411 INFO:tasks.workunit.client.0.smithi112.stdout:removed './267/34' 2023-01-24T03:35:23.411 INFO:tasks.workunit.client.0.smithi112.stdout:removed './267/38' 2023-01-24T03:35:23.411 INFO:tasks.workunit.client.0.smithi112.stdout:removed './267/28' 2023-01-24T03:35:23.412 INFO:tasks.workunit.client.0.smithi112.stdout:removed './267/24' 2023-01-24T03:35:23.412 INFO:tasks.workunit.client.0.smithi112.stdout:removed './267/29' 2023-01-24T03:35:23.412 INFO:tasks.workunit.client.0.smithi112.stdout:removed './267/32' 2023-01-24T03:35:23.412 INFO:tasks.workunit.client.0.smithi112.stdout:removed './267/22' 2023-01-24T03:35:23.412 INFO:tasks.workunit.client.0.smithi112.stdout:removed './267/25' 2023-01-24T03:35:23.413 INFO:tasks.workunit.client.0.smithi112.stdout:removed './267/30' 2023-01-24T03:35:23.413 INFO:tasks.workunit.client.0.smithi112.stdout:removed './267/33' 2023-01-24T03:35:23.413 INFO:tasks.workunit.client.0.smithi112.stdout:removed './267/23' 2023-01-24T03:35:23.413 INFO:tasks.workunit.client.0.smithi112.stdout:removed './267/35' 2023-01-24T03:35:23.413 INFO:tasks.workunit.client.0.smithi112.stdout:removed './267/26' 2023-01-24T03:35:23.414 INFO:tasks.workunit.client.0.smithi112.stdout:removed './267/37' 2023-01-24T03:35:23.414 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './267' 2023-01-24T03:35:23.414 INFO:tasks.workunit.client.0.smithi112.stdout:removed './268/21' 2023-01-24T03:35:23.414 INFO:tasks.workunit.client.0.smithi112.stdout:removed './268/27' 2023-01-24T03:35:23.414 INFO:tasks.workunit.client.0.smithi112.stdout:removed './268/31' 2023-01-24T03:35:23.415 INFO:tasks.workunit.client.0.smithi112.stdout:removed './268/36' 2023-01-24T03:35:23.415 INFO:tasks.workunit.client.0.smithi112.stdout:removed './268/20' 2023-01-24T03:35:23.415 INFO:tasks.workunit.client.0.smithi112.stdout:removed './268/39' 2023-01-24T03:35:23.415 INFO:tasks.workunit.client.0.smithi112.stdout:removed './268/34' 2023-01-24T03:35:23.415 INFO:tasks.workunit.client.0.smithi112.stdout:removed './268/38' 2023-01-24T03:35:23.416 INFO:tasks.workunit.client.0.smithi112.stdout:removed './268/28' 2023-01-24T03:35:23.416 INFO:tasks.workunit.client.0.smithi112.stdout:removed './268/24' 2023-01-24T03:35:23.416 INFO:tasks.workunit.client.0.smithi112.stdout:removed './268/29' 2023-01-24T03:35:23.416 INFO:tasks.workunit.client.0.smithi112.stdout:removed './268/32' 2023-01-24T03:35:23.416 INFO:tasks.workunit.client.0.smithi112.stdout:removed './268/22' 2023-01-24T03:35:23.417 INFO:tasks.workunit.client.0.smithi112.stdout:removed './268/25' 2023-01-24T03:35:23.417 INFO:tasks.workunit.client.0.smithi112.stdout:removed './268/30' 2023-01-24T03:35:23.417 INFO:tasks.workunit.client.0.smithi112.stdout:removed './268/33' 2023-01-24T03:35:23.417 INFO:tasks.workunit.client.0.smithi112.stdout:removed './268/23' 2023-01-24T03:35:23.417 INFO:tasks.workunit.client.0.smithi112.stdout:removed './268/35' 2023-01-24T03:35:23.418 INFO:tasks.workunit.client.0.smithi112.stdout:removed './268/26' 2023-01-24T03:35:23.418 INFO:tasks.workunit.client.0.smithi112.stdout:removed './268/37' 2023-01-24T03:35:23.418 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './268' 2023-01-24T03:35:23.418 INFO:tasks.workunit.client.0.smithi112.stdout:removed './269/21' 2023-01-24T03:35:23.418 INFO:tasks.workunit.client.0.smithi112.stdout:removed './269/27' 2023-01-24T03:35:23.418 INFO:tasks.workunit.client.0.smithi112.stdout:removed './269/31' 2023-01-24T03:35:23.419 INFO:tasks.workunit.client.0.smithi112.stdout:removed './269/36' 2023-01-24T03:35:23.419 INFO:tasks.workunit.client.0.smithi112.stdout:removed './269/20' 2023-01-24T03:35:23.419 INFO:tasks.workunit.client.0.smithi112.stdout:removed './269/39' 2023-01-24T03:35:23.419 INFO:tasks.workunit.client.0.smithi112.stdout:removed './269/34' 2023-01-24T03:35:23.420 INFO:tasks.workunit.client.0.smithi112.stdout:removed './269/38' 2023-01-24T03:35:23.420 INFO:tasks.workunit.client.0.smithi112.stdout:removed './269/28' 2023-01-24T03:35:23.420 INFO:tasks.workunit.client.0.smithi112.stdout:removed './269/24' 2023-01-24T03:35:23.420 INFO:tasks.workunit.client.0.smithi112.stdout:removed './269/29' 2023-01-24T03:35:23.421 INFO:tasks.workunit.client.0.smithi112.stdout:removed './269/32' 2023-01-24T03:35:23.421 INFO:tasks.workunit.client.0.smithi112.stdout:removed './269/22' 2023-01-24T03:35:23.421 INFO:tasks.workunit.client.0.smithi112.stdout:removed './269/25' 2023-01-24T03:35:23.421 INFO:tasks.workunit.client.0.smithi112.stdout:removed './269/30' 2023-01-24T03:35:23.421 INFO:tasks.workunit.client.0.smithi112.stdout:removed './269/33' 2023-01-24T03:35:23.422 INFO:tasks.workunit.client.0.smithi112.stdout:removed './269/23' 2023-01-24T03:35:23.422 INFO:tasks.workunit.client.0.smithi112.stdout:removed './269/35' 2023-01-24T03:35:23.422 INFO:tasks.workunit.client.0.smithi112.stdout:removed './269/26' 2023-01-24T03:35:23.422 INFO:tasks.workunit.client.0.smithi112.stdout:removed './269/37' 2023-01-24T03:35:23.422 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './269' 2023-01-24T03:35:23.422 INFO:tasks.workunit.client.0.smithi112.stdout:removed './270/21' 2023-01-24T03:35:23.423 INFO:tasks.workunit.client.0.smithi112.stdout:removed './270/27' 2023-01-24T03:35:23.423 INFO:tasks.workunit.client.0.smithi112.stdout:removed './270/31' 2023-01-24T03:35:23.423 INFO:tasks.workunit.client.0.smithi112.stdout:removed './270/36' 2023-01-24T03:35:23.423 INFO:tasks.workunit.client.0.smithi112.stdout:removed './270/20' 2023-01-24T03:35:23.423 INFO:tasks.workunit.client.0.smithi112.stdout:removed './270/39' 2023-01-24T03:35:23.424 INFO:tasks.workunit.client.0.smithi112.stdout:removed './270/34' 2023-01-24T03:35:23.424 INFO:tasks.workunit.client.0.smithi112.stdout:removed './270/38' 2023-01-24T03:35:23.424 INFO:tasks.workunit.client.0.smithi112.stdout:removed './270/28' 2023-01-24T03:35:23.424 INFO:tasks.workunit.client.0.smithi112.stdout:removed './270/24' 2023-01-24T03:35:23.424 INFO:tasks.workunit.client.0.smithi112.stdout:removed './270/29' 2023-01-24T03:35:23.425 INFO:tasks.workunit.client.0.smithi112.stdout:removed './270/32' 2023-01-24T03:35:23.425 INFO:tasks.workunit.client.0.smithi112.stdout:removed './270/22' 2023-01-24T03:35:23.425 INFO:tasks.workunit.client.0.smithi112.stdout:removed './270/25' 2023-01-24T03:35:23.425 INFO:tasks.workunit.client.0.smithi112.stdout:removed './270/30' 2023-01-24T03:35:23.425 INFO:tasks.workunit.client.0.smithi112.stdout:removed './270/33' 2023-01-24T03:35:23.426 INFO:tasks.workunit.client.0.smithi112.stdout:removed './270/23' 2023-01-24T03:35:23.426 INFO:tasks.workunit.client.0.smithi112.stdout:removed './270/35' 2023-01-24T03:35:23.426 INFO:tasks.workunit.client.0.smithi112.stdout:removed './270/26' 2023-01-24T03:35:23.426 INFO:tasks.workunit.client.0.smithi112.stdout:removed './270/37' 2023-01-24T03:35:23.426 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './270' 2023-01-24T03:35:23.427 INFO:tasks.workunit.client.0.smithi112.stdout:removed './271/21' 2023-01-24T03:35:23.427 INFO:tasks.workunit.client.0.smithi112.stdout:removed './271/27' 2023-01-24T03:35:23.427 INFO:tasks.workunit.client.0.smithi112.stdout:removed './271/31' 2023-01-24T03:35:23.427 INFO:tasks.workunit.client.0.smithi112.stdout:removed './271/36' 2023-01-24T03:35:23.427 INFO:tasks.workunit.client.0.smithi112.stdout:removed './271/20' 2023-01-24T03:35:23.428 INFO:tasks.workunit.client.0.smithi112.stdout:removed './271/39' 2023-01-24T03:35:23.428 INFO:tasks.workunit.client.0.smithi112.stdout:removed './271/34' 2023-01-24T03:35:23.428 INFO:tasks.workunit.client.0.smithi112.stdout:removed './271/38' 2023-01-24T03:35:23.428 INFO:tasks.workunit.client.0.smithi112.stdout:removed './271/28' 2023-01-24T03:35:23.428 INFO:tasks.workunit.client.0.smithi112.stdout:removed './271/24' 2023-01-24T03:35:23.429 INFO:tasks.workunit.client.0.smithi112.stdout:removed './271/29' 2023-01-24T03:35:23.429 INFO:tasks.workunit.client.0.smithi112.stdout:removed './271/32' 2023-01-24T03:35:23.429 INFO:tasks.workunit.client.0.smithi112.stdout:removed './271/22' 2023-01-24T03:35:23.429 INFO:tasks.workunit.client.0.smithi112.stdout:removed './271/25' 2023-01-24T03:35:23.429 INFO:tasks.workunit.client.0.smithi112.stdout:removed './271/30' 2023-01-24T03:35:23.429 INFO:tasks.workunit.client.0.smithi112.stdout:removed './271/33' 2023-01-24T03:35:23.430 INFO:tasks.workunit.client.0.smithi112.stdout:removed './271/23' 2023-01-24T03:35:23.430 INFO:tasks.workunit.client.0.smithi112.stdout:removed './271/35' 2023-01-24T03:35:23.430 INFO:tasks.workunit.client.0.smithi112.stdout:removed './271/26' 2023-01-24T03:35:23.431 INFO:tasks.workunit.client.0.smithi112.stdout:removed './271/37' 2023-01-24T03:35:23.431 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './271' 2023-01-24T03:35:23.431 INFO:tasks.workunit.client.0.smithi112.stdout:removed './272/21' 2023-01-24T03:35:23.431 INFO:tasks.workunit.client.0.smithi112.stdout:removed './272/27' 2023-01-24T03:35:23.431 INFO:tasks.workunit.client.0.smithi112.stdout:removed './272/31' 2023-01-24T03:35:23.432 INFO:tasks.workunit.client.0.smithi112.stdout:removed './272/36' 2023-01-24T03:35:23.432 INFO:tasks.workunit.client.0.smithi112.stdout:removed './272/20' 2023-01-24T03:35:23.432 INFO:tasks.workunit.client.0.smithi112.stdout:removed './272/39' 2023-01-24T03:35:23.432 INFO:tasks.workunit.client.0.smithi112.stdout:removed './272/34' 2023-01-24T03:35:23.432 INFO:tasks.workunit.client.0.smithi112.stdout:removed './272/38' 2023-01-24T03:35:23.433 INFO:tasks.workunit.client.0.smithi112.stdout:removed './272/28' 2023-01-24T03:35:23.433 INFO:tasks.workunit.client.0.smithi112.stdout:removed './272/24' 2023-01-24T03:35:23.433 INFO:tasks.workunit.client.0.smithi112.stdout:removed './272/29' 2023-01-24T03:35:23.433 INFO:tasks.workunit.client.0.smithi112.stdout:removed './272/32' 2023-01-24T03:35:23.433 INFO:tasks.workunit.client.0.smithi112.stdout:removed './272/22' 2023-01-24T03:35:23.434 INFO:tasks.workunit.client.0.smithi112.stdout:removed './272/25' 2023-01-24T03:35:23.434 INFO:tasks.workunit.client.0.smithi112.stdout:removed './272/30' 2023-01-24T03:35:23.434 INFO:tasks.workunit.client.0.smithi112.stdout:removed './272/33' 2023-01-24T03:35:23.434 INFO:tasks.workunit.client.0.smithi112.stdout:removed './272/23' 2023-01-24T03:35:23.434 INFO:tasks.workunit.client.0.smithi112.stdout:removed './272/35' 2023-01-24T03:35:23.435 INFO:tasks.workunit.client.0.smithi112.stdout:removed './272/26' 2023-01-24T03:35:23.435 INFO:tasks.workunit.client.0.smithi112.stdout:removed './272/37' 2023-01-24T03:35:23.435 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './272' 2023-01-24T03:35:23.435 INFO:tasks.workunit.client.0.smithi112.stdout:removed './273/21' 2023-01-24T03:35:23.435 INFO:tasks.workunit.client.0.smithi112.stdout:removed './273/27' 2023-01-24T03:35:23.435 INFO:tasks.workunit.client.0.smithi112.stdout:removed './273/31' 2023-01-24T03:35:23.436 INFO:tasks.workunit.client.0.smithi112.stdout:removed './273/36' 2023-01-24T03:35:23.436 INFO:tasks.workunit.client.0.smithi112.stdout:removed './273/20' 2023-01-24T03:35:23.436 INFO:tasks.workunit.client.0.smithi112.stdout:removed './273/39' 2023-01-24T03:35:23.436 INFO:tasks.workunit.client.0.smithi112.stdout:removed './273/34' 2023-01-24T03:35:23.436 INFO:tasks.workunit.client.0.smithi112.stdout:removed './273/38' 2023-01-24T03:35:23.437 INFO:tasks.workunit.client.0.smithi112.stdout:removed './273/28' 2023-01-24T03:35:23.437 INFO:tasks.workunit.client.0.smithi112.stdout:removed './273/24' 2023-01-24T03:35:23.437 INFO:tasks.workunit.client.0.smithi112.stdout:removed './273/29' 2023-01-24T03:35:23.437 INFO:tasks.workunit.client.0.smithi112.stdout:removed './273/32' 2023-01-24T03:35:23.437 INFO:tasks.workunit.client.0.smithi112.stdout:removed './273/22' 2023-01-24T03:35:23.438 INFO:tasks.workunit.client.0.smithi112.stdout:removed './273/25' 2023-01-24T03:35:23.438 INFO:tasks.workunit.client.0.smithi112.stdout:removed './273/30' 2023-01-24T03:35:23.438 INFO:tasks.workunit.client.0.smithi112.stdout:removed './273/33' 2023-01-24T03:35:23.438 INFO:tasks.workunit.client.0.smithi112.stdout:removed './273/23' 2023-01-24T03:35:23.438 INFO:tasks.workunit.client.0.smithi112.stdout:removed './273/35' 2023-01-24T03:35:23.438 INFO:tasks.workunit.client.0.smithi112.stdout:removed './273/26' 2023-01-24T03:35:23.439 INFO:tasks.workunit.client.0.smithi112.stdout:removed './273/37' 2023-01-24T03:35:23.439 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './273' 2023-01-24T03:35:23.439 INFO:tasks.workunit.client.0.smithi112.stdout:removed './274/21' 2023-01-24T03:35:23.439 INFO:tasks.workunit.client.0.smithi112.stdout:removed './274/27' 2023-01-24T03:35:23.439 INFO:tasks.workunit.client.0.smithi112.stdout:removed './274/31' 2023-01-24T03:35:23.439 INFO:tasks.workunit.client.0.smithi112.stdout:removed './274/36' 2023-01-24T03:35:23.440 INFO:tasks.workunit.client.0.smithi112.stdout:removed './274/20' 2023-01-24T03:35:23.440 INFO:tasks.workunit.client.0.smithi112.stdout:removed './274/39' 2023-01-24T03:35:23.440 INFO:tasks.workunit.client.0.smithi112.stdout:removed './274/34' 2023-01-24T03:35:23.440 INFO:tasks.workunit.client.0.smithi112.stdout:removed './274/38' 2023-01-24T03:35:23.440 INFO:tasks.workunit.client.0.smithi112.stdout:removed './274/28' 2023-01-24T03:35:23.441 INFO:tasks.workunit.client.0.smithi112.stdout:removed './274/24' 2023-01-24T03:35:23.441 INFO:tasks.workunit.client.0.smithi112.stdout:removed './274/29' 2023-01-24T03:35:23.441 INFO:tasks.workunit.client.0.smithi112.stdout:removed './274/32' 2023-01-24T03:35:23.441 INFO:tasks.workunit.client.0.smithi112.stdout:removed './274/22' 2023-01-24T03:35:23.441 INFO:tasks.workunit.client.0.smithi112.stdout:removed './274/25' 2023-01-24T03:35:23.442 INFO:tasks.workunit.client.0.smithi112.stdout:removed './274/30' 2023-01-24T03:35:23.442 INFO:tasks.workunit.client.0.smithi112.stdout:removed './274/33' 2023-01-24T03:35:23.442 INFO:tasks.workunit.client.0.smithi112.stdout:removed './274/23' 2023-01-24T03:35:23.442 INFO:tasks.workunit.client.0.smithi112.stdout:removed './274/35' 2023-01-24T03:35:23.442 INFO:tasks.workunit.client.0.smithi112.stdout:removed './274/26' 2023-01-24T03:35:23.442 INFO:tasks.workunit.client.0.smithi112.stdout:removed './274/37' 2023-01-24T03:35:23.443 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './274' 2023-01-24T03:35:23.443 INFO:tasks.workunit.client.0.smithi112.stdout:removed './275/21' 2023-01-24T03:35:23.443 INFO:tasks.workunit.client.0.smithi112.stdout:removed './275/27' 2023-01-24T03:35:23.443 INFO:tasks.workunit.client.0.smithi112.stdout:removed './275/31' 2023-01-24T03:35:23.443 INFO:tasks.workunit.client.0.smithi112.stdout:removed './275/36' 2023-01-24T03:35:23.443 INFO:tasks.workunit.client.0.smithi112.stdout:removed './275/20' 2023-01-24T03:35:23.444 INFO:tasks.workunit.client.0.smithi112.stdout:removed './275/39' 2023-01-24T03:35:23.444 INFO:tasks.workunit.client.0.smithi112.stdout:removed './275/34' 2023-01-24T03:35:23.444 INFO:tasks.workunit.client.0.smithi112.stdout:removed './275/38' 2023-01-24T03:35:23.444 INFO:tasks.workunit.client.0.smithi112.stdout:removed './275/28' 2023-01-24T03:35:23.445 INFO:tasks.workunit.client.0.smithi112.stdout:removed './275/24' 2023-01-24T03:35:23.445 INFO:tasks.workunit.client.0.smithi112.stdout:removed './275/29' 2023-01-24T03:35:23.445 INFO:tasks.workunit.client.0.smithi112.stdout:removed './275/32' 2023-01-24T03:35:23.445 INFO:tasks.workunit.client.0.smithi112.stdout:removed './275/22' 2023-01-24T03:35:23.445 INFO:tasks.workunit.client.0.smithi112.stdout:removed './275/25' 2023-01-24T03:35:23.445 INFO:tasks.workunit.client.0.smithi112.stdout:removed './275/30' 2023-01-24T03:35:23.446 INFO:tasks.workunit.client.0.smithi112.stdout:removed './275/33' 2023-01-24T03:35:23.446 INFO:tasks.workunit.client.0.smithi112.stdout:removed './275/23' 2023-01-24T03:35:23.446 INFO:tasks.workunit.client.0.smithi112.stdout:removed './275/35' 2023-01-24T03:35:23.446 INFO:tasks.workunit.client.0.smithi112.stdout:removed './275/26' 2023-01-24T03:35:23.446 INFO:tasks.workunit.client.0.smithi112.stdout:removed './275/37' 2023-01-24T03:35:23.446 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './275' 2023-01-24T03:35:23.447 INFO:tasks.workunit.client.0.smithi112.stdout:removed './276/21' 2023-01-24T03:35:23.447 INFO:tasks.workunit.client.0.smithi112.stdout:removed './276/27' 2023-01-24T03:35:23.447 INFO:tasks.workunit.client.0.smithi112.stdout:removed './276/31' 2023-01-24T03:35:23.447 INFO:tasks.workunit.client.0.smithi112.stdout:removed './276/36' 2023-01-24T03:35:23.447 INFO:tasks.workunit.client.0.smithi112.stdout:removed './276/20' 2023-01-24T03:35:23.448 INFO:tasks.workunit.client.0.smithi112.stdout:removed './276/39' 2023-01-24T03:35:23.448 INFO:tasks.workunit.client.0.smithi112.stdout:removed './276/34' 2023-01-24T03:35:23.448 INFO:tasks.workunit.client.0.smithi112.stdout:removed './276/38' 2023-01-24T03:35:23.448 INFO:tasks.workunit.client.0.smithi112.stdout:removed './276/28' 2023-01-24T03:35:23.448 INFO:tasks.workunit.client.0.smithi112.stdout:removed './276/24' 2023-01-24T03:35:23.449 INFO:tasks.workunit.client.0.smithi112.stdout:removed './276/29' 2023-01-24T03:35:23.449 INFO:tasks.workunit.client.0.smithi112.stdout:removed './276/32' 2023-01-24T03:35:23.449 INFO:tasks.workunit.client.0.smithi112.stdout:removed './276/22' 2023-01-24T03:35:23.449 INFO:tasks.workunit.client.0.smithi112.stdout:removed './276/25' 2023-01-24T03:35:23.449 INFO:tasks.workunit.client.0.smithi112.stdout:removed './276/30' 2023-01-24T03:35:23.450 INFO:tasks.workunit.client.0.smithi112.stdout:removed './276/33' 2023-01-24T03:35:23.450 INFO:tasks.workunit.client.0.smithi112.stdout:removed './276/23' 2023-01-24T03:35:23.450 INFO:tasks.workunit.client.0.smithi112.stdout:removed './276/35' 2023-01-24T03:35:23.450 INFO:tasks.workunit.client.0.smithi112.stdout:removed './276/26' 2023-01-24T03:35:25.261 INFO:tasks.workunit.client.0.smithi112.stdout:removed './276/37' 2023-01-24T03:35:25.261 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './276' 2023-01-24T03:35:25.261 INFO:tasks.workunit.client.0.smithi112.stdout:removed './277/21' 2023-01-24T03:35:25.261 INFO:tasks.workunit.client.0.smithi112.stdout:removed './277/27' 2023-01-24T03:35:25.262 INFO:tasks.workunit.client.0.smithi112.stdout:removed './277/31' 2023-01-24T03:35:25.262 INFO:tasks.workunit.client.0.smithi112.stdout:removed './277/36' 2023-01-24T03:35:25.262 INFO:tasks.workunit.client.0.smithi112.stdout:removed './277/20' 2023-01-24T03:35:25.262 INFO:tasks.workunit.client.0.smithi112.stdout:removed './277/39' 2023-01-24T03:35:25.262 INFO:tasks.workunit.client.0.smithi112.stdout:removed './277/34' 2023-01-24T03:35:25.263 INFO:tasks.workunit.client.0.smithi112.stdout:removed './277/38' 2023-01-24T03:35:25.263 INFO:tasks.workunit.client.0.smithi112.stdout:removed './277/28' 2023-01-24T03:35:25.263 INFO:tasks.workunit.client.0.smithi112.stdout:removed './277/24' 2023-01-24T03:35:25.263 INFO:tasks.workunit.client.0.smithi112.stdout:removed './277/29' 2023-01-24T03:35:25.264 INFO:tasks.workunit.client.0.smithi112.stdout:removed './277/32' 2023-01-24T03:35:25.264 INFO:tasks.workunit.client.0.smithi112.stdout:removed './277/22' 2023-01-24T03:35:25.264 INFO:tasks.workunit.client.0.smithi112.stdout:removed './277/25' 2023-01-24T03:35:25.264 INFO:tasks.workunit.client.0.smithi112.stdout:removed './277/30' 2023-01-24T03:35:25.264 INFO:tasks.workunit.client.0.smithi112.stdout:removed './277/33' 2023-01-24T03:35:25.265 INFO:tasks.workunit.client.0.smithi112.stdout:removed './277/23' 2023-01-24T03:35:25.265 INFO:tasks.workunit.client.0.smithi112.stdout:removed './277/35' 2023-01-24T03:35:25.265 INFO:tasks.workunit.client.0.smithi112.stdout:removed './277/26' 2023-01-24T03:35:25.265 INFO:tasks.workunit.client.0.smithi112.stdout:removed './277/37' 2023-01-24T03:35:25.265 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './277' 2023-01-24T03:35:25.266 INFO:tasks.workunit.client.0.smithi112.stdout:removed './278/21' 2023-01-24T03:35:25.266 INFO:tasks.workunit.client.0.smithi112.stdout:removed './278/27' 2023-01-24T03:35:25.266 INFO:tasks.workunit.client.0.smithi112.stdout:removed './278/31' 2023-01-24T03:35:25.266 INFO:tasks.workunit.client.0.smithi112.stdout:removed './278/36' 2023-01-24T03:35:25.266 INFO:tasks.workunit.client.0.smithi112.stdout:removed './278/20' 2023-01-24T03:35:25.267 INFO:tasks.workunit.client.0.smithi112.stdout:removed './278/39' 2023-01-24T03:35:25.267 INFO:tasks.workunit.client.0.smithi112.stdout:removed './278/34' 2023-01-24T03:35:25.267 INFO:tasks.workunit.client.0.smithi112.stdout:removed './278/38' 2023-01-24T03:35:25.267 INFO:tasks.workunit.client.0.smithi112.stdout:removed './278/28' 2023-01-24T03:35:25.267 INFO:tasks.workunit.client.0.smithi112.stdout:removed './278/24' 2023-01-24T03:35:25.268 INFO:tasks.workunit.client.0.smithi112.stdout:removed './278/29' 2023-01-24T03:35:25.268 INFO:tasks.workunit.client.0.smithi112.stdout:removed './278/32' 2023-01-24T03:35:25.268 INFO:tasks.workunit.client.0.smithi112.stdout:removed './278/22' 2023-01-24T03:35:25.268 INFO:tasks.workunit.client.0.smithi112.stdout:removed './278/25' 2023-01-24T03:35:25.268 INFO:tasks.workunit.client.0.smithi112.stdout:removed './278/30' 2023-01-24T03:35:25.269 INFO:tasks.workunit.client.0.smithi112.stdout:removed './278/33' 2023-01-24T03:35:25.269 INFO:tasks.workunit.client.0.smithi112.stdout:removed './278/23' 2023-01-24T03:35:25.269 INFO:tasks.workunit.client.0.smithi112.stdout:removed './278/35' 2023-01-24T03:35:25.269 INFO:tasks.workunit.client.0.smithi112.stdout:removed './278/26' 2023-01-24T03:35:25.269 INFO:tasks.workunit.client.0.smithi112.stdout:removed './278/37' 2023-01-24T03:35:25.269 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './278' 2023-01-24T03:35:25.270 INFO:tasks.workunit.client.0.smithi112.stdout:removed './279/21' 2023-01-24T03:35:25.270 INFO:tasks.workunit.client.0.smithi112.stdout:removed './279/27' 2023-01-24T03:35:25.270 INFO:tasks.workunit.client.0.smithi112.stdout:removed './279/31' 2023-01-24T03:35:25.270 INFO:tasks.workunit.client.0.smithi112.stdout:removed './279/36' 2023-01-24T03:35:25.271 INFO:tasks.workunit.client.0.smithi112.stdout:removed './279/20' 2023-01-24T03:35:25.271 INFO:tasks.workunit.client.0.smithi112.stdout:removed './279/39' 2023-01-24T03:35:25.271 INFO:tasks.workunit.client.0.smithi112.stdout:removed './279/34' 2023-01-24T03:35:25.271 INFO:tasks.workunit.client.0.smithi112.stdout:removed './279/38' 2023-01-24T03:35:25.271 INFO:tasks.workunit.client.0.smithi112.stdout:removed './279/28' 2023-01-24T03:35:25.271 INFO:tasks.workunit.client.0.smithi112.stdout:removed './279/24' 2023-01-24T03:35:25.272 INFO:tasks.workunit.client.0.smithi112.stdout:removed './279/29' 2023-01-24T03:35:25.272 INFO:tasks.workunit.client.0.smithi112.stdout:removed './279/32' 2023-01-24T03:35:25.272 INFO:tasks.workunit.client.0.smithi112.stdout:removed './279/22' 2023-01-24T03:35:25.272 INFO:tasks.workunit.client.0.smithi112.stdout:removed './279/25' 2023-01-24T03:35:25.272 INFO:tasks.workunit.client.0.smithi112.stdout:removed './279/30' 2023-01-24T03:35:25.273 INFO:tasks.workunit.client.0.smithi112.stdout:removed './279/33' 2023-01-24T03:35:25.273 INFO:tasks.workunit.client.0.smithi112.stdout:removed './279/23' 2023-01-24T03:35:25.273 INFO:tasks.workunit.client.0.smithi112.stdout:removed './279/35' 2023-01-24T03:35:25.273 INFO:tasks.workunit.client.0.smithi112.stdout:removed './279/26' 2023-01-24T03:35:25.273 INFO:tasks.workunit.client.0.smithi112.stdout:removed './279/37' 2023-01-24T03:35:25.274 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './279' 2023-01-24T03:35:25.274 INFO:tasks.workunit.client.0.smithi112.stdout:removed './280/21' 2023-01-24T03:35:25.274 INFO:tasks.workunit.client.0.smithi112.stdout:removed './280/27' 2023-01-24T03:35:25.274 INFO:tasks.workunit.client.0.smithi112.stdout:removed './280/31' 2023-01-24T03:35:25.274 INFO:tasks.workunit.client.0.smithi112.stdout:removed './280/36' 2023-01-24T03:35:25.275 INFO:tasks.workunit.client.0.smithi112.stdout:removed './280/20' 2023-01-24T03:35:25.275 INFO:tasks.workunit.client.0.smithi112.stdout:removed './280/39' 2023-01-24T03:35:25.275 INFO:tasks.workunit.client.0.smithi112.stdout:removed './280/34' 2023-01-24T03:35:25.275 INFO:tasks.workunit.client.0.smithi112.stdout:removed './280/38' 2023-01-24T03:35:25.275 INFO:tasks.workunit.client.0.smithi112.stdout:removed './280/28' 2023-01-24T03:35:25.276 INFO:tasks.workunit.client.0.smithi112.stdout:removed './280/24' 2023-01-24T03:35:25.276 INFO:tasks.workunit.client.0.smithi112.stdout:removed './280/29' 2023-01-24T03:35:25.276 INFO:tasks.workunit.client.0.smithi112.stdout:removed './280/32' 2023-01-24T03:35:25.276 INFO:tasks.workunit.client.0.smithi112.stdout:removed './280/22' 2023-01-24T03:35:25.276 INFO:tasks.workunit.client.0.smithi112.stdout:removed './280/25' 2023-01-24T03:35:25.277 INFO:tasks.workunit.client.0.smithi112.stdout:removed './280/30' 2023-01-24T03:35:25.277 INFO:tasks.workunit.client.0.smithi112.stdout:removed './280/33' 2023-01-24T03:35:25.277 INFO:tasks.workunit.client.0.smithi112.stdout:removed './280/23' 2023-01-24T03:35:25.277 INFO:tasks.workunit.client.0.smithi112.stdout:removed './280/35' 2023-01-24T03:35:25.277 INFO:tasks.workunit.client.0.smithi112.stdout:removed './280/26' 2023-01-24T03:35:25.277 INFO:tasks.workunit.client.0.smithi112.stdout:removed './280/37' 2023-01-24T03:35:25.278 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './280' 2023-01-24T03:35:25.278 INFO:tasks.workunit.client.0.smithi112.stdout:removed './281/21' 2023-01-24T03:35:25.278 INFO:tasks.workunit.client.0.smithi112.stdout:removed './281/27' 2023-01-24T03:35:25.278 INFO:tasks.workunit.client.0.smithi112.stdout:removed './281/31' 2023-01-24T03:35:25.278 INFO:tasks.workunit.client.0.smithi112.stdout:removed './281/36' 2023-01-24T03:35:25.279 INFO:tasks.workunit.client.0.smithi112.stdout:removed './281/20' 2023-01-24T03:35:25.279 INFO:tasks.workunit.client.0.smithi112.stdout:removed './281/39' 2023-01-24T03:35:25.279 INFO:tasks.workunit.client.0.smithi112.stdout:removed './281/34' 2023-01-24T03:35:25.279 INFO:tasks.workunit.client.0.smithi112.stdout:removed './281/38' 2023-01-24T03:35:25.279 INFO:tasks.workunit.client.0.smithi112.stdout:removed './281/28' 2023-01-24T03:35:25.279 INFO:tasks.workunit.client.0.smithi112.stdout:removed './281/24' 2023-01-24T03:35:25.280 INFO:tasks.workunit.client.0.smithi112.stdout:removed './281/29' 2023-01-24T03:35:25.280 INFO:tasks.workunit.client.0.smithi112.stdout:removed './281/32' 2023-01-24T03:35:25.280 INFO:tasks.workunit.client.0.smithi112.stdout:removed './281/22' 2023-01-24T03:35:25.280 INFO:tasks.workunit.client.0.smithi112.stdout:removed './281/25' 2023-01-24T03:35:25.280 INFO:tasks.workunit.client.0.smithi112.stdout:removed './281/30' 2023-01-24T03:35:25.281 INFO:tasks.workunit.client.0.smithi112.stdout:removed './281/33' 2023-01-24T03:35:25.281 INFO:tasks.workunit.client.0.smithi112.stdout:removed './281/23' 2023-01-24T03:35:25.281 INFO:tasks.workunit.client.0.smithi112.stdout:removed './281/35' 2023-01-24T03:35:25.281 INFO:tasks.workunit.client.0.smithi112.stdout:removed './281/26' 2023-01-24T03:35:25.281 INFO:tasks.workunit.client.0.smithi112.stdout:removed './281/37' 2023-01-24T03:35:25.282 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './281' 2023-01-24T03:35:25.282 INFO:tasks.workunit.client.0.smithi112.stdout:removed './282/21' 2023-01-24T03:35:25.282 INFO:tasks.workunit.client.0.smithi112.stdout:removed './282/27' 2023-01-24T03:35:25.282 INFO:tasks.workunit.client.0.smithi112.stdout:removed './282/31' 2023-01-24T03:35:25.282 INFO:tasks.workunit.client.0.smithi112.stdout:removed './282/36' 2023-01-24T03:35:25.283 INFO:tasks.workunit.client.0.smithi112.stdout:removed './282/20' 2023-01-24T03:35:25.283 INFO:tasks.workunit.client.0.smithi112.stdout:removed './282/39' 2023-01-24T03:35:25.283 INFO:tasks.workunit.client.0.smithi112.stdout:removed './282/34' 2023-01-24T03:35:25.283 INFO:tasks.workunit.client.0.smithi112.stdout:removed './282/38' 2023-01-24T03:35:25.283 INFO:tasks.workunit.client.0.smithi112.stdout:removed './282/28' 2023-01-24T03:35:25.284 INFO:tasks.workunit.client.0.smithi112.stdout:removed './282/24' 2023-01-24T03:35:25.284 INFO:tasks.workunit.client.0.smithi112.stdout:removed './282/29' 2023-01-24T03:35:25.284 INFO:tasks.workunit.client.0.smithi112.stdout:removed './282/32' 2023-01-24T03:35:25.284 INFO:tasks.workunit.client.0.smithi112.stdout:removed './282/22' 2023-01-24T03:35:25.284 INFO:tasks.workunit.client.0.smithi112.stdout:removed './282/25' 2023-01-24T03:35:25.285 INFO:tasks.workunit.client.0.smithi112.stdout:removed './282/30' 2023-01-24T03:35:25.285 INFO:tasks.workunit.client.0.smithi112.stdout:removed './282/33' 2023-01-24T03:35:25.285 INFO:tasks.workunit.client.0.smithi112.stdout:removed './282/23' 2023-01-24T03:35:25.285 INFO:tasks.workunit.client.0.smithi112.stdout:removed './282/35' 2023-01-24T03:35:25.285 INFO:tasks.workunit.client.0.smithi112.stdout:removed './282/26' 2023-01-24T03:35:25.286 INFO:tasks.workunit.client.0.smithi112.stdout:removed './282/37' 2023-01-24T03:35:25.286 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './282' 2023-01-24T03:35:25.286 INFO:tasks.workunit.client.0.smithi112.stdout:removed './283/21' 2023-01-24T03:35:25.286 INFO:tasks.workunit.client.0.smithi112.stdout:removed './283/27' 2023-01-24T03:35:25.286 INFO:tasks.workunit.client.0.smithi112.stdout:removed './283/31' 2023-01-24T03:35:25.286 INFO:tasks.workunit.client.0.smithi112.stdout:removed './283/36' 2023-01-24T03:35:25.287 INFO:tasks.workunit.client.0.smithi112.stdout:removed './283/20' 2023-01-24T03:35:25.287 INFO:tasks.workunit.client.0.smithi112.stdout:removed './283/39' 2023-01-24T03:35:25.287 INFO:tasks.workunit.client.0.smithi112.stdout:removed './283/34' 2023-01-24T03:35:25.287 INFO:tasks.workunit.client.0.smithi112.stdout:removed './283/38' 2023-01-24T03:35:25.287 INFO:tasks.workunit.client.0.smithi112.stdout:removed './283/28' 2023-01-24T03:35:25.288 INFO:tasks.workunit.client.0.smithi112.stdout:removed './283/24' 2023-01-24T03:35:25.288 INFO:tasks.workunit.client.0.smithi112.stdout:removed './283/29' 2023-01-24T03:35:25.288 INFO:tasks.workunit.client.0.smithi112.stdout:removed './283/32' 2023-01-24T03:35:25.288 INFO:tasks.workunit.client.0.smithi112.stdout:removed './283/22' 2023-01-24T03:35:25.288 INFO:tasks.workunit.client.0.smithi112.stdout:removed './283/25' 2023-01-24T03:35:25.289 INFO:tasks.workunit.client.0.smithi112.stdout:removed './283/30' 2023-01-24T03:35:25.289 INFO:tasks.workunit.client.0.smithi112.stdout:removed './283/33' 2023-01-24T03:35:25.289 INFO:tasks.workunit.client.0.smithi112.stdout:removed './283/23' 2023-01-24T03:35:25.289 INFO:tasks.workunit.client.0.smithi112.stdout:removed './283/35' 2023-01-24T03:35:25.289 INFO:tasks.workunit.client.0.smithi112.stdout:removed './283/26' 2023-01-24T03:35:25.290 INFO:tasks.workunit.client.0.smithi112.stdout:removed './283/37' 2023-01-24T03:35:25.290 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './283' 2023-01-24T03:35:25.290 INFO:tasks.workunit.client.0.smithi112.stdout:removed './284/21' 2023-01-24T03:35:25.290 INFO:tasks.workunit.client.0.smithi112.stdout:removed './284/27' 2023-01-24T03:35:25.290 INFO:tasks.workunit.client.0.smithi112.stdout:removed './284/31' 2023-01-24T03:35:25.291 INFO:tasks.workunit.client.0.smithi112.stdout:removed './284/36' 2023-01-24T03:35:25.291 INFO:tasks.workunit.client.0.smithi112.stdout:removed './284/20' 2023-01-24T03:35:25.291 INFO:tasks.workunit.client.0.smithi112.stdout:removed './284/39' 2023-01-24T03:35:25.291 INFO:tasks.workunit.client.0.smithi112.stdout:removed './284/34' 2023-01-24T03:35:25.291 INFO:tasks.workunit.client.0.smithi112.stdout:removed './284/38' 2023-01-24T03:35:25.291 INFO:tasks.workunit.client.0.smithi112.stdout:removed './284/28' 2023-01-24T03:35:25.292 INFO:tasks.workunit.client.0.smithi112.stdout:removed './284/24' 2023-01-24T03:35:25.292 INFO:tasks.workunit.client.0.smithi112.stdout:removed './284/29' 2023-01-24T03:35:25.292 INFO:tasks.workunit.client.0.smithi112.stdout:removed './284/32' 2023-01-24T03:35:25.292 INFO:tasks.workunit.client.0.smithi112.stdout:removed './284/22' 2023-01-24T03:35:25.293 INFO:tasks.workunit.client.0.smithi112.stdout:removed './284/25' 2023-01-24T03:35:25.293 INFO:tasks.workunit.client.0.smithi112.stdout:removed './284/30' 2023-01-24T03:35:25.293 INFO:tasks.workunit.client.0.smithi112.stdout:removed './284/33' 2023-01-24T03:35:25.293 INFO:tasks.workunit.client.0.smithi112.stdout:removed './284/23' 2023-01-24T03:35:25.293 INFO:tasks.workunit.client.0.smithi112.stdout:removed './284/35' 2023-01-24T03:35:25.294 INFO:tasks.workunit.client.0.smithi112.stdout:removed './284/26' 2023-01-24T03:35:25.294 INFO:tasks.workunit.client.0.smithi112.stdout:removed './284/37' 2023-01-24T03:35:25.294 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './284' 2023-01-24T03:35:25.294 INFO:tasks.workunit.client.0.smithi112.stdout:removed './285/21' 2023-01-24T03:35:25.294 INFO:tasks.workunit.client.0.smithi112.stdout:removed './285/27' 2023-01-24T03:35:25.295 INFO:tasks.workunit.client.0.smithi112.stdout:removed './285/31' 2023-01-24T03:35:25.295 INFO:tasks.workunit.client.0.smithi112.stdout:removed './285/36' 2023-01-24T03:35:25.295 INFO:tasks.workunit.client.0.smithi112.stdout:removed './285/20' 2023-01-24T03:35:25.295 INFO:tasks.workunit.client.0.smithi112.stdout:removed './285/39' 2023-01-24T03:35:25.295 INFO:tasks.workunit.client.0.smithi112.stdout:removed './285/34' 2023-01-24T03:35:25.296 INFO:tasks.workunit.client.0.smithi112.stdout:removed './285/38' 2023-01-24T03:35:25.296 INFO:tasks.workunit.client.0.smithi112.stdout:removed './285/28' 2023-01-24T03:35:25.296 INFO:tasks.workunit.client.0.smithi112.stdout:removed './285/24' 2023-01-24T03:35:25.296 INFO:tasks.workunit.client.0.smithi112.stdout:removed './285/29' 2023-01-24T03:35:25.297 INFO:tasks.workunit.client.0.smithi112.stdout:removed './285/32' 2023-01-24T03:35:25.297 INFO:tasks.workunit.client.0.smithi112.stdout:removed './285/22' 2023-01-24T03:35:25.297 INFO:tasks.workunit.client.0.smithi112.stdout:removed './285/25' 2023-01-24T03:35:25.297 INFO:tasks.workunit.client.0.smithi112.stdout:removed './285/30' 2023-01-24T03:35:25.297 INFO:tasks.workunit.client.0.smithi112.stdout:removed './285/33' 2023-01-24T03:35:25.298 INFO:tasks.workunit.client.0.smithi112.stdout:removed './285/23' 2023-01-24T03:35:25.298 INFO:tasks.workunit.client.0.smithi112.stdout:removed './285/35' 2023-01-24T03:35:25.298 INFO:tasks.workunit.client.0.smithi112.stdout:removed './285/26' 2023-01-24T03:35:25.298 INFO:tasks.workunit.client.0.smithi112.stdout:removed './285/37' 2023-01-24T03:35:25.298 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './285' 2023-01-24T03:35:25.299 INFO:tasks.workunit.client.0.smithi112.stdout:removed './286/21' 2023-01-24T03:35:25.299 INFO:tasks.workunit.client.0.smithi112.stdout:removed './286/27' 2023-01-24T03:35:25.299 INFO:tasks.workunit.client.0.smithi112.stdout:removed './286/31' 2023-01-24T03:35:25.299 INFO:tasks.workunit.client.0.smithi112.stdout:removed './286/36' 2023-01-24T03:35:25.299 INFO:tasks.workunit.client.0.smithi112.stdout:removed './286/20' 2023-01-24T03:35:25.300 INFO:tasks.workunit.client.0.smithi112.stdout:removed './286/39' 2023-01-24T03:35:25.300 INFO:tasks.workunit.client.0.smithi112.stdout:removed './286/34' 2023-01-24T03:35:25.300 INFO:tasks.workunit.client.0.smithi112.stdout:removed './286/38' 2023-01-24T03:35:25.300 INFO:tasks.workunit.client.0.smithi112.stdout:removed './286/28' 2023-01-24T03:35:25.300 INFO:tasks.workunit.client.0.smithi112.stdout:removed './286/24' 2023-01-24T03:35:25.301 INFO:tasks.workunit.client.0.smithi112.stdout:removed './286/29' 2023-01-24T03:35:25.301 INFO:tasks.workunit.client.0.smithi112.stdout:removed './286/32' 2023-01-24T03:35:25.301 INFO:tasks.workunit.client.0.smithi112.stdout:removed './286/22' 2023-01-24T03:35:25.301 INFO:tasks.workunit.client.0.smithi112.stdout:removed './286/25' 2023-01-24T03:35:25.301 INFO:tasks.workunit.client.0.smithi112.stdout:removed './286/30' 2023-01-24T03:35:25.301 INFO:tasks.workunit.client.0.smithi112.stdout:removed './286/33' 2023-01-24T03:35:25.302 INFO:tasks.workunit.client.0.smithi112.stdout:removed './286/23' 2023-01-24T03:35:25.302 INFO:tasks.workunit.client.0.smithi112.stdout:removed './286/35' 2023-01-24T03:35:25.302 INFO:tasks.workunit.client.0.smithi112.stdout:removed './286/26' 2023-01-24T03:35:25.302 INFO:tasks.workunit.client.0.smithi112.stdout:removed './286/37' 2023-01-24T03:35:25.302 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './286' 2023-01-24T03:35:25.934 INFO:tasks.workunit.client.0.smithi112.stdout:removed './287/21' 2023-01-24T03:35:25.934 INFO:tasks.workunit.client.0.smithi112.stdout:removed './287/27' 2023-01-24T03:35:25.934 INFO:tasks.workunit.client.0.smithi112.stdout:removed './287/31' 2023-01-24T03:35:25.934 INFO:tasks.workunit.client.0.smithi112.stdout:removed './287/36' 2023-01-24T03:35:25.935 INFO:tasks.workunit.client.0.smithi112.stdout:removed './287/20' 2023-01-24T03:35:25.935 INFO:tasks.workunit.client.0.smithi112.stdout:removed './287/39' 2023-01-24T03:35:25.935 INFO:tasks.workunit.client.0.smithi112.stdout:removed './287/34' 2023-01-24T03:35:25.935 INFO:tasks.workunit.client.0.smithi112.stdout:removed './287/38' 2023-01-24T03:35:25.935 INFO:tasks.workunit.client.0.smithi112.stdout:removed './287/28' 2023-01-24T03:35:25.935 INFO:tasks.workunit.client.0.smithi112.stdout:removed './287/24' 2023-01-24T03:35:25.936 INFO:tasks.workunit.client.0.smithi112.stdout:removed './287/29' 2023-01-24T03:35:25.936 INFO:tasks.workunit.client.0.smithi112.stdout:removed './287/32' 2023-01-24T03:35:25.936 INFO:tasks.workunit.client.0.smithi112.stdout:removed './287/22' 2023-01-24T03:35:25.936 INFO:tasks.workunit.client.0.smithi112.stdout:removed './287/25' 2023-01-24T03:35:25.936 INFO:tasks.workunit.client.0.smithi112.stdout:removed './287/30' 2023-01-24T03:35:25.937 INFO:tasks.workunit.client.0.smithi112.stdout:removed './287/33' 2023-01-24T03:35:25.937 INFO:tasks.workunit.client.0.smithi112.stdout:removed './287/23' 2023-01-24T03:35:25.937 INFO:tasks.workunit.client.0.smithi112.stdout:removed './287/35' 2023-01-24T03:35:25.937 INFO:tasks.workunit.client.0.smithi112.stdout:removed './287/26' 2023-01-24T03:35:25.938 INFO:tasks.workunit.client.0.smithi112.stdout:removed './287/37' 2023-01-24T03:35:25.938 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './287' 2023-01-24T03:35:25.938 INFO:tasks.workunit.client.0.smithi112.stdout:removed './288/21' 2023-01-24T03:35:25.938 INFO:tasks.workunit.client.0.smithi112.stdout:removed './288/27' 2023-01-24T03:35:25.938 INFO:tasks.workunit.client.0.smithi112.stdout:removed './288/31' 2023-01-24T03:35:25.939 INFO:tasks.workunit.client.0.smithi112.stdout:removed './288/36' 2023-01-24T03:35:25.939 INFO:tasks.workunit.client.0.smithi112.stdout:removed './288/20' 2023-01-24T03:35:25.939 INFO:tasks.workunit.client.0.smithi112.stdout:removed './288/39' 2023-01-24T03:35:25.939 INFO:tasks.workunit.client.0.smithi112.stdout:removed './288/34' 2023-01-24T03:35:25.939 INFO:tasks.workunit.client.0.smithi112.stdout:removed './288/38' 2023-01-24T03:35:25.939 INFO:tasks.workunit.client.0.smithi112.stdout:removed './288/28' 2023-01-24T03:35:25.940 INFO:tasks.workunit.client.0.smithi112.stdout:removed './288/24' 2023-01-24T03:35:25.940 INFO:tasks.workunit.client.0.smithi112.stdout:removed './288/29' 2023-01-24T03:35:25.940 INFO:tasks.workunit.client.0.smithi112.stdout:removed './288/32' 2023-01-24T03:35:25.940 INFO:tasks.workunit.client.0.smithi112.stdout:removed './288/22' 2023-01-24T03:35:25.940 INFO:tasks.workunit.client.0.smithi112.stdout:removed './288/25' 2023-01-24T03:35:25.941 INFO:tasks.workunit.client.0.smithi112.stdout:removed './288/30' 2023-01-24T03:35:25.941 INFO:tasks.workunit.client.0.smithi112.stdout:removed './288/33' 2023-01-24T03:35:25.941 INFO:tasks.workunit.client.0.smithi112.stdout:removed './288/23' 2023-01-24T03:35:25.941 INFO:tasks.workunit.client.0.smithi112.stdout:removed './288/35' 2023-01-24T03:35:25.941 INFO:tasks.workunit.client.0.smithi112.stdout:removed './288/26' 2023-01-24T03:35:25.941 INFO:tasks.workunit.client.0.smithi112.stdout:removed './288/37' 2023-01-24T03:35:25.942 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './288' 2023-01-24T03:35:25.942 INFO:tasks.workunit.client.0.smithi112.stdout:removed './289/21' 2023-01-24T03:35:25.942 INFO:tasks.workunit.client.0.smithi112.stdout:removed './289/27' 2023-01-24T03:35:25.942 INFO:tasks.workunit.client.0.smithi112.stdout:removed './289/31' 2023-01-24T03:35:25.942 INFO:tasks.workunit.client.0.smithi112.stdout:removed './289/36' 2023-01-24T03:35:25.943 INFO:tasks.workunit.client.0.smithi112.stdout:removed './289/20' 2023-01-24T03:35:25.943 INFO:tasks.workunit.client.0.smithi112.stdout:removed './289/39' 2023-01-24T03:35:25.943 INFO:tasks.workunit.client.0.smithi112.stdout:removed './289/34' 2023-01-24T03:35:25.943 INFO:tasks.workunit.client.0.smithi112.stdout:removed './289/38' 2023-01-24T03:35:25.943 INFO:tasks.workunit.client.0.smithi112.stdout:removed './289/28' 2023-01-24T03:35:25.943 INFO:tasks.workunit.client.0.smithi112.stdout:removed './289/24' 2023-01-24T03:35:25.944 INFO:tasks.workunit.client.0.smithi112.stdout:removed './289/29' 2023-01-24T03:35:25.944 INFO:tasks.workunit.client.0.smithi112.stdout:removed './289/32' 2023-01-24T03:35:25.944 INFO:tasks.workunit.client.0.smithi112.stdout:removed './289/22' 2023-01-24T03:35:25.944 INFO:tasks.workunit.client.0.smithi112.stdout:removed './289/25' 2023-01-24T03:35:25.944 INFO:tasks.workunit.client.0.smithi112.stdout:removed './289/30' 2023-01-24T03:35:25.945 INFO:tasks.workunit.client.0.smithi112.stdout:removed './289/33' 2023-01-24T03:35:25.945 INFO:tasks.workunit.client.0.smithi112.stdout:removed './289/23' 2023-01-24T03:35:25.945 INFO:tasks.workunit.client.0.smithi112.stdout:removed './289/35' 2023-01-24T03:35:25.945 INFO:tasks.workunit.client.0.smithi112.stdout:removed './289/26' 2023-01-24T03:35:25.945 INFO:tasks.workunit.client.0.smithi112.stdout:removed './289/37' 2023-01-24T03:35:25.946 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './289' 2023-01-24T03:35:25.946 INFO:tasks.workunit.client.0.smithi112.stdout:removed './290/21' 2023-01-24T03:35:25.946 INFO:tasks.workunit.client.0.smithi112.stdout:removed './290/27' 2023-01-24T03:35:25.946 INFO:tasks.workunit.client.0.smithi112.stdout:removed './290/31' 2023-01-24T03:35:25.946 INFO:tasks.workunit.client.0.smithi112.stdout:removed './290/36' 2023-01-24T03:35:25.946 INFO:tasks.workunit.client.0.smithi112.stdout:removed './290/20' 2023-01-24T03:35:25.947 INFO:tasks.workunit.client.0.smithi112.stdout:removed './290/39' 2023-01-24T03:35:25.947 INFO:tasks.workunit.client.0.smithi112.stdout:removed './290/34' 2023-01-24T03:35:25.947 INFO:tasks.workunit.client.0.smithi112.stdout:removed './290/38' 2023-01-24T03:35:25.947 INFO:tasks.workunit.client.0.smithi112.stdout:removed './290/28' 2023-01-24T03:35:25.947 INFO:tasks.workunit.client.0.smithi112.stdout:removed './290/24' 2023-01-24T03:35:25.948 INFO:tasks.workunit.client.0.smithi112.stdout:removed './290/29' 2023-01-24T03:35:25.948 INFO:tasks.workunit.client.0.smithi112.stdout:removed './290/32' 2023-01-24T03:35:25.948 INFO:tasks.workunit.client.0.smithi112.stdout:removed './290/22' 2023-01-24T03:35:25.948 INFO:tasks.workunit.client.0.smithi112.stdout:removed './290/25' 2023-01-24T03:35:25.949 INFO:tasks.workunit.client.0.smithi112.stdout:removed './290/30' 2023-01-24T03:35:25.949 INFO:tasks.workunit.client.0.smithi112.stdout:removed './290/33' 2023-01-24T03:35:25.949 INFO:tasks.workunit.client.0.smithi112.stdout:removed './290/23' 2023-01-24T03:35:25.949 INFO:tasks.workunit.client.0.smithi112.stdout:removed './290/35' 2023-01-24T03:35:25.949 INFO:tasks.workunit.client.0.smithi112.stdout:removed './290/26' 2023-01-24T03:35:25.949 INFO:tasks.workunit.client.0.smithi112.stdout:removed './290/37' 2023-01-24T03:35:25.950 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './290' 2023-01-24T03:35:25.950 INFO:tasks.workunit.client.0.smithi112.stdout:removed './291/21' 2023-01-24T03:35:25.950 INFO:tasks.workunit.client.0.smithi112.stdout:removed './291/27' 2023-01-24T03:35:25.950 INFO:tasks.workunit.client.0.smithi112.stdout:removed './291/31' 2023-01-24T03:35:25.951 INFO:tasks.workunit.client.0.smithi112.stdout:removed './291/36' 2023-01-24T03:35:25.951 INFO:tasks.workunit.client.0.smithi112.stdout:removed './291/20' 2023-01-24T03:35:25.951 INFO:tasks.workunit.client.0.smithi112.stdout:removed './291/39' 2023-01-24T03:35:25.951 INFO:tasks.workunit.client.0.smithi112.stdout:removed './291/34' 2023-01-24T03:35:25.952 INFO:tasks.workunit.client.0.smithi112.stdout:removed './291/38' 2023-01-24T03:35:25.952 INFO:tasks.workunit.client.0.smithi112.stdout:removed './291/28' 2023-01-24T03:35:25.952 INFO:tasks.workunit.client.0.smithi112.stdout:removed './291/24' 2023-01-24T03:35:25.953 INFO:tasks.workunit.client.0.smithi112.stdout:removed './291/29' 2023-01-24T03:35:25.953 INFO:tasks.workunit.client.0.smithi112.stdout:removed './291/32' 2023-01-24T03:35:25.954 INFO:tasks.workunit.client.0.smithi112.stdout:removed './291/22' 2023-01-24T03:35:25.954 INFO:tasks.workunit.client.0.smithi112.stdout:removed './291/25' 2023-01-24T03:35:25.955 INFO:tasks.workunit.client.0.smithi112.stdout:removed './291/30' 2023-01-24T03:35:25.955 INFO:tasks.workunit.client.0.smithi112.stdout:removed './291/33' 2023-01-24T03:35:25.955 INFO:tasks.workunit.client.0.smithi112.stdout:removed './291/23' 2023-01-24T03:35:25.956 INFO:tasks.workunit.client.0.smithi112.stdout:removed './291/35' 2023-01-24T03:35:25.956 INFO:tasks.workunit.client.0.smithi112.stdout:removed './291/26' 2023-01-24T03:35:25.956 INFO:tasks.workunit.client.0.smithi112.stdout:removed './291/37' 2023-01-24T03:35:25.957 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './291' 2023-01-24T03:35:25.957 INFO:tasks.workunit.client.0.smithi112.stdout:removed './292/21' 2023-01-24T03:35:25.957 INFO:tasks.workunit.client.0.smithi112.stdout:removed './292/27' 2023-01-24T03:35:25.958 INFO:tasks.workunit.client.0.smithi112.stdout:removed './292/31' 2023-01-24T03:35:25.958 INFO:tasks.workunit.client.0.smithi112.stdout:removed './292/36' 2023-01-24T03:35:25.958 INFO:tasks.workunit.client.0.smithi112.stdout:removed './292/20' 2023-01-24T03:35:25.959 INFO:tasks.workunit.client.0.smithi112.stdout:removed './292/39' 2023-01-24T03:35:25.959 INFO:tasks.workunit.client.0.smithi112.stdout:removed './292/34' 2023-01-24T03:35:25.959 INFO:tasks.workunit.client.0.smithi112.stdout:removed './292/38' 2023-01-24T03:35:25.959 INFO:tasks.workunit.client.0.smithi112.stdout:removed './292/28' 2023-01-24T03:35:25.960 INFO:tasks.workunit.client.0.smithi112.stdout:removed './292/24' 2023-01-24T03:35:25.960 INFO:tasks.workunit.client.0.smithi112.stdout:removed './292/29' 2023-01-24T03:35:25.960 INFO:tasks.workunit.client.0.smithi112.stdout:removed './292/32' 2023-01-24T03:35:25.960 INFO:tasks.workunit.client.0.smithi112.stdout:removed './292/22' 2023-01-24T03:35:25.961 INFO:tasks.workunit.client.0.smithi112.stdout:removed './292/25' 2023-01-24T03:35:25.961 INFO:tasks.workunit.client.0.smithi112.stdout:removed './292/30' 2023-01-24T03:35:25.961 INFO:tasks.workunit.client.0.smithi112.stdout:removed './292/33' 2023-01-24T03:35:25.961 INFO:tasks.workunit.client.0.smithi112.stdout:removed './292/23' 2023-01-24T03:35:25.961 INFO:tasks.workunit.client.0.smithi112.stdout:removed './292/35' 2023-01-24T03:35:25.962 INFO:tasks.workunit.client.0.smithi112.stdout:removed './292/26' 2023-01-24T03:35:25.962 INFO:tasks.workunit.client.0.smithi112.stdout:removed './292/37' 2023-01-24T03:35:25.962 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './292' 2023-01-24T03:35:25.962 INFO:tasks.workunit.client.0.smithi112.stdout:removed './293/21' 2023-01-24T03:35:25.962 INFO:tasks.workunit.client.0.smithi112.stdout:removed './293/27' 2023-01-24T03:35:25.963 INFO:tasks.workunit.client.0.smithi112.stdout:removed './293/31' 2023-01-24T03:35:25.963 INFO:tasks.workunit.client.0.smithi112.stdout:removed './293/36' 2023-01-24T03:35:25.963 INFO:tasks.workunit.client.0.smithi112.stdout:removed './293/20' 2023-01-24T03:35:25.963 INFO:tasks.workunit.client.0.smithi112.stdout:removed './293/39' 2023-01-24T03:35:25.963 INFO:tasks.workunit.client.0.smithi112.stdout:removed './293/34' 2023-01-24T03:35:25.963 INFO:tasks.workunit.client.0.smithi112.stdout:removed './293/38' 2023-01-24T03:35:25.964 INFO:tasks.workunit.client.0.smithi112.stdout:removed './293/28' 2023-01-24T03:35:25.964 INFO:tasks.workunit.client.0.smithi112.stdout:removed './293/24' 2023-01-24T03:35:25.964 INFO:tasks.workunit.client.0.smithi112.stdout:removed './293/29' 2023-01-24T03:35:25.964 INFO:tasks.workunit.client.0.smithi112.stdout:removed './293/32' 2023-01-24T03:35:25.964 INFO:tasks.workunit.client.0.smithi112.stdout:removed './293/22' 2023-01-24T03:35:25.965 INFO:tasks.workunit.client.0.smithi112.stdout:removed './293/25' 2023-01-24T03:35:25.965 INFO:tasks.workunit.client.0.smithi112.stdout:removed './293/30' 2023-01-24T03:35:25.965 INFO:tasks.workunit.client.0.smithi112.stdout:removed './293/33' 2023-01-24T03:35:25.965 INFO:tasks.workunit.client.0.smithi112.stdout:removed './293/23' 2023-01-24T03:35:25.965 INFO:tasks.workunit.client.0.smithi112.stdout:removed './293/35' 2023-01-24T03:35:25.965 INFO:tasks.workunit.client.0.smithi112.stdout:removed './293/26' 2023-01-24T03:35:25.966 INFO:tasks.workunit.client.0.smithi112.stdout:removed './293/37' 2023-01-24T03:35:25.966 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './293' 2023-01-24T03:35:25.966 INFO:tasks.workunit.client.0.smithi112.stdout:removed './294/21' 2023-01-24T03:35:25.966 INFO:tasks.workunit.client.0.smithi112.stdout:removed './294/27' 2023-01-24T03:35:25.966 INFO:tasks.workunit.client.0.smithi112.stdout:removed './294/31' 2023-01-24T03:35:25.967 INFO:tasks.workunit.client.0.smithi112.stdout:removed './294/36' 2023-01-24T03:35:25.967 INFO:tasks.workunit.client.0.smithi112.stdout:removed './294/20' 2023-01-24T03:35:25.967 INFO:tasks.workunit.client.0.smithi112.stdout:removed './294/39' 2023-01-24T03:35:25.967 INFO:tasks.workunit.client.0.smithi112.stdout:removed './294/34' 2023-01-24T03:35:25.967 INFO:tasks.workunit.client.0.smithi112.stdout:removed './294/38' 2023-01-24T03:35:25.967 INFO:tasks.workunit.client.0.smithi112.stdout:removed './294/28' 2023-01-24T03:35:25.968 INFO:tasks.workunit.client.0.smithi112.stdout:removed './294/24' 2023-01-24T03:35:25.968 INFO:tasks.workunit.client.0.smithi112.stdout:removed './294/29' 2023-01-24T03:35:25.968 INFO:tasks.workunit.client.0.smithi112.stdout:removed './294/32' 2023-01-24T03:35:25.968 INFO:tasks.workunit.client.0.smithi112.stdout:removed './294/22' 2023-01-24T03:35:25.968 INFO:tasks.workunit.client.0.smithi112.stdout:removed './294/25' 2023-01-24T03:35:25.969 INFO:tasks.workunit.client.0.smithi112.stdout:removed './294/30' 2023-01-24T03:35:25.969 INFO:tasks.workunit.client.0.smithi112.stdout:removed './294/33' 2023-01-24T03:35:25.969 INFO:tasks.workunit.client.0.smithi112.stdout:removed './294/23' 2023-01-24T03:35:25.969 INFO:tasks.workunit.client.0.smithi112.stdout:removed './294/35' 2023-01-24T03:35:25.969 INFO:tasks.workunit.client.0.smithi112.stdout:removed './294/26' 2023-01-24T03:35:25.969 INFO:tasks.workunit.client.0.smithi112.stdout:removed './294/37' 2023-01-24T03:35:25.970 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './294' 2023-01-24T03:35:25.970 INFO:tasks.workunit.client.0.smithi112.stdout:removed './295/21' 2023-01-24T03:35:25.970 INFO:tasks.workunit.client.0.smithi112.stdout:removed './295/27' 2023-01-24T03:35:25.970 INFO:tasks.workunit.client.0.smithi112.stdout:removed './295/31' 2023-01-24T03:35:25.970 INFO:tasks.workunit.client.0.smithi112.stdout:removed './295/36' 2023-01-24T03:35:25.970 INFO:tasks.workunit.client.0.smithi112.stdout:removed './295/20' 2023-01-24T03:35:25.971 INFO:tasks.workunit.client.0.smithi112.stdout:removed './295/39' 2023-01-24T03:35:25.971 INFO:tasks.workunit.client.0.smithi112.stdout:removed './295/34' 2023-01-24T03:35:25.971 INFO:tasks.workunit.client.0.smithi112.stdout:removed './295/38' 2023-01-24T03:35:25.971 INFO:tasks.workunit.client.0.smithi112.stdout:removed './295/28' 2023-01-24T03:35:25.971 INFO:tasks.workunit.client.0.smithi112.stdout:removed './295/24' 2023-01-24T03:35:25.972 INFO:tasks.workunit.client.0.smithi112.stdout:removed './295/29' 2023-01-24T03:35:25.972 INFO:tasks.workunit.client.0.smithi112.stdout:removed './295/32' 2023-01-24T03:35:25.972 INFO:tasks.workunit.client.0.smithi112.stdout:removed './295/22' 2023-01-24T03:35:25.972 INFO:tasks.workunit.client.0.smithi112.stdout:removed './295/25' 2023-01-24T03:35:25.973 INFO:tasks.workunit.client.0.smithi112.stdout:removed './295/30' 2023-01-24T03:35:25.973 INFO:tasks.workunit.client.0.smithi112.stdout:removed './295/33' 2023-01-24T03:35:25.973 INFO:tasks.workunit.client.0.smithi112.stdout:removed './295/23' 2023-01-24T03:35:25.973 INFO:tasks.workunit.client.0.smithi112.stdout:removed './295/35' 2023-01-24T03:35:25.974 INFO:tasks.workunit.client.0.smithi112.stdout:removed './295/26' 2023-01-24T03:35:25.974 INFO:tasks.workunit.client.0.smithi112.stdout:removed './295/37' 2023-01-24T03:35:25.974 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './295' 2023-01-24T03:35:25.975 INFO:tasks.workunit.client.0.smithi112.stdout:removed './296/21' 2023-01-24T03:35:25.975 INFO:tasks.workunit.client.0.smithi112.stdout:removed './296/27' 2023-01-24T03:35:25.975 INFO:tasks.workunit.client.0.smithi112.stdout:removed './296/31' 2023-01-24T03:35:25.975 INFO:tasks.workunit.client.0.smithi112.stdout:removed './296/36' 2023-01-24T03:35:25.975 INFO:tasks.workunit.client.0.smithi112.stdout:removed './296/20' 2023-01-24T03:35:25.975 INFO:tasks.workunit.client.0.smithi112.stdout:removed './296/39' 2023-01-24T03:35:25.976 INFO:tasks.workunit.client.0.smithi112.stdout:removed './296/34' 2023-01-24T03:35:25.976 INFO:tasks.workunit.client.0.smithi112.stdout:removed './296/38' 2023-01-24T03:35:25.976 INFO:tasks.workunit.client.0.smithi112.stdout:removed './296/28' 2023-01-24T03:35:25.976 INFO:tasks.workunit.client.0.smithi112.stdout:removed './296/24' 2023-01-24T03:35:25.976 INFO:tasks.workunit.client.0.smithi112.stdout:removed './296/29' 2023-01-24T03:35:25.977 INFO:tasks.workunit.client.0.smithi112.stdout:removed './296/32' 2023-01-24T03:35:25.977 INFO:tasks.workunit.client.0.smithi112.stdout:removed './296/22' 2023-01-24T03:35:25.977 INFO:tasks.workunit.client.0.smithi112.stdout:removed './296/25' 2023-01-24T03:35:25.977 INFO:tasks.workunit.client.0.smithi112.stdout:removed './296/30' 2023-01-24T03:35:25.977 INFO:tasks.workunit.client.0.smithi112.stdout:removed './296/33' 2023-01-24T03:35:25.978 INFO:tasks.workunit.client.0.smithi112.stdout:removed './296/23' 2023-01-24T03:35:25.978 INFO:tasks.workunit.client.0.smithi112.stdout:removed './296/35' 2023-01-24T03:35:25.978 INFO:tasks.workunit.client.0.smithi112.stdout:removed './296/26' 2023-01-24T03:35:25.978 INFO:tasks.workunit.client.0.smithi112.stdout:removed './296/37' 2023-01-24T03:35:25.978 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './296' 2023-01-24T03:35:25.978 INFO:tasks.workunit.client.0.smithi112.stdout:removed './297/21' 2023-01-24T03:35:25.979 INFO:tasks.workunit.client.0.smithi112.stdout:removed './297/27' 2023-01-24T03:35:27.351 INFO:tasks.workunit.client.0.smithi112.stdout:removed './297/31' 2023-01-24T03:35:27.352 INFO:tasks.workunit.client.0.smithi112.stdout:removed './297/36' 2023-01-24T03:35:27.352 INFO:tasks.workunit.client.0.smithi112.stdout:removed './297/20' 2023-01-24T03:35:27.352 INFO:tasks.workunit.client.0.smithi112.stdout:removed './297/39' 2023-01-24T03:35:27.352 INFO:tasks.workunit.client.0.smithi112.stdout:removed './297/34' 2023-01-24T03:35:27.353 INFO:tasks.workunit.client.0.smithi112.stdout:removed './297/38' 2023-01-24T03:35:27.353 INFO:tasks.workunit.client.0.smithi112.stdout:removed './297/28' 2023-01-24T03:35:27.353 INFO:tasks.workunit.client.0.smithi112.stdout:removed './297/24' 2023-01-24T03:35:27.353 INFO:tasks.workunit.client.0.smithi112.stdout:removed './297/29' 2023-01-24T03:35:27.353 INFO:tasks.workunit.client.0.smithi112.stdout:removed './297/32' 2023-01-24T03:35:27.354 INFO:tasks.workunit.client.0.smithi112.stdout:removed './297/22' 2023-01-24T03:35:27.354 INFO:tasks.workunit.client.0.smithi112.stdout:removed './297/25' 2023-01-24T03:35:27.354 INFO:tasks.workunit.client.0.smithi112.stdout:removed './297/30' 2023-01-24T03:35:27.354 INFO:tasks.workunit.client.0.smithi112.stdout:removed './297/33' 2023-01-24T03:35:27.355 INFO:tasks.workunit.client.0.smithi112.stdout:removed './297/23' 2023-01-24T03:35:27.355 INFO:tasks.workunit.client.0.smithi112.stdout:removed './297/35' 2023-01-24T03:35:27.355 INFO:tasks.workunit.client.0.smithi112.stdout:removed './297/26' 2023-01-24T03:35:27.355 INFO:tasks.workunit.client.0.smithi112.stdout:removed './297/37' 2023-01-24T03:35:27.355 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './297' 2023-01-24T03:35:27.355 INFO:tasks.workunit.client.0.smithi112.stdout:removed './298/21' 2023-01-24T03:35:27.356 INFO:tasks.workunit.client.0.smithi112.stdout:removed './298/27' 2023-01-24T03:35:27.356 INFO:tasks.workunit.client.0.smithi112.stdout:removed './298/31' 2023-01-24T03:35:27.356 INFO:tasks.workunit.client.0.smithi112.stdout:removed './298/36' 2023-01-24T03:35:27.356 INFO:tasks.workunit.client.0.smithi112.stdout:removed './298/20' 2023-01-24T03:35:27.357 INFO:tasks.workunit.client.0.smithi112.stdout:removed './298/39' 2023-01-24T03:35:27.357 INFO:tasks.workunit.client.0.smithi112.stdout:removed './298/34' 2023-01-24T03:35:27.357 INFO:tasks.workunit.client.0.smithi112.stdout:removed './298/38' 2023-01-24T03:35:27.357 INFO:tasks.workunit.client.0.smithi112.stdout:removed './298/28' 2023-01-24T03:35:27.357 INFO:tasks.workunit.client.0.smithi112.stdout:removed './298/24' 2023-01-24T03:35:27.358 INFO:tasks.workunit.client.0.smithi112.stdout:removed './298/29' 2023-01-24T03:35:27.358 INFO:tasks.workunit.client.0.smithi112.stdout:removed './298/32' 2023-01-24T03:35:27.358 INFO:tasks.workunit.client.0.smithi112.stdout:removed './298/22' 2023-01-24T03:35:27.358 INFO:tasks.workunit.client.0.smithi112.stdout:removed './298/25' 2023-01-24T03:35:27.358 INFO:tasks.workunit.client.0.smithi112.stdout:removed './298/30' 2023-01-24T03:35:27.358 INFO:tasks.workunit.client.0.smithi112.stdout:removed './298/33' 2023-01-24T03:35:27.359 INFO:tasks.workunit.client.0.smithi112.stdout:removed './298/23' 2023-01-24T03:35:27.359 INFO:tasks.workunit.client.0.smithi112.stdout:removed './298/35' 2023-01-24T03:35:27.359 INFO:tasks.workunit.client.0.smithi112.stdout:removed './298/26' 2023-01-24T03:35:27.359 INFO:tasks.workunit.client.0.smithi112.stdout:removed './298/37' 2023-01-24T03:35:27.359 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './298' 2023-01-24T03:35:27.360 INFO:tasks.workunit.client.0.smithi112.stdout:removed './299/21' 2023-01-24T03:35:27.360 INFO:tasks.workunit.client.0.smithi112.stdout:removed './299/27' 2023-01-24T03:35:27.360 INFO:tasks.workunit.client.0.smithi112.stdout:removed './299/31' 2023-01-24T03:35:27.360 INFO:tasks.workunit.client.0.smithi112.stdout:removed './299/36' 2023-01-24T03:35:27.360 INFO:tasks.workunit.client.0.smithi112.stdout:removed './299/20' 2023-01-24T03:35:27.361 INFO:tasks.workunit.client.0.smithi112.stdout:removed './299/39' 2023-01-24T03:35:27.361 INFO:tasks.workunit.client.0.smithi112.stdout:removed './299/34' 2023-01-24T03:35:27.361 INFO:tasks.workunit.client.0.smithi112.stdout:removed './299/38' 2023-01-24T03:35:27.361 INFO:tasks.workunit.client.0.smithi112.stdout:removed './299/28' 2023-01-24T03:35:27.361 INFO:tasks.workunit.client.0.smithi112.stdout:removed './299/24' 2023-01-24T03:35:27.362 INFO:tasks.workunit.client.0.smithi112.stdout:removed './299/29' 2023-01-24T03:35:27.362 INFO:tasks.workunit.client.0.smithi112.stdout:removed './299/32' 2023-01-24T03:35:27.362 INFO:tasks.workunit.client.0.smithi112.stdout:removed './299/22' 2023-01-24T03:35:27.362 INFO:tasks.workunit.client.0.smithi112.stdout:removed './299/25' 2023-01-24T03:35:27.362 INFO:tasks.workunit.client.0.smithi112.stdout:removed './299/30' 2023-01-24T03:35:27.362 INFO:tasks.workunit.client.0.smithi112.stdout:removed './299/33' 2023-01-24T03:35:27.363 INFO:tasks.workunit.client.0.smithi112.stdout:removed './299/23' 2023-01-24T03:35:27.363 INFO:tasks.workunit.client.0.smithi112.stdout:removed './299/35' 2023-01-24T03:35:27.363 INFO:tasks.workunit.client.0.smithi112.stdout:removed './299/26' 2023-01-24T03:35:27.363 INFO:tasks.workunit.client.0.smithi112.stdout:removed './299/37' 2023-01-24T03:35:27.364 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './299' 2023-01-24T03:35:27.364 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/166/19' 2023-01-24T03:35:27.364 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/166/15' 2023-01-24T03:35:27.364 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/166/20' 2023-01-24T03:35:27.365 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/166/12' 2023-01-24T03:35:27.365 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/166/10' 2023-01-24T03:35:27.365 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/166/11' 2023-01-24T03:35:27.365 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/166/16' 2023-01-24T03:35:27.365 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/166/14' 2023-01-24T03:35:27.366 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/166/18' 2023-01-24T03:35:27.366 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/166/13' 2023-01-24T03:35:27.366 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/166/17' 2023-01-24T03:35:27.366 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/166' 2023-01-24T03:35:27.366 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/147/19' 2023-01-24T03:35:27.367 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/147/15' 2023-01-24T03:35:27.367 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/147/20' 2023-01-24T03:35:27.367 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/147/12' 2023-01-24T03:35:27.367 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/147/10' 2023-01-24T03:35:27.367 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/147/11' 2023-01-24T03:35:27.367 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/147/16' 2023-01-24T03:35:27.368 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/147/14' 2023-01-24T03:35:27.368 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/147/18' 2023-01-24T03:35:27.368 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/147/13' 2023-01-24T03:35:27.368 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/147/17' 2023-01-24T03:35:27.368 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/147' 2023-01-24T03:35:27.369 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/151/19' 2023-01-24T03:35:27.369 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/151/15' 2023-01-24T03:35:27.369 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/151/20' 2023-01-24T03:35:27.369 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/151/12' 2023-01-24T03:35:27.369 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/151/10' 2023-01-24T03:35:27.370 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/151/11' 2023-01-24T03:35:27.370 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/151/16' 2023-01-24T03:35:27.370 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/151/14' 2023-01-24T03:35:27.370 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/151/18' 2023-01-24T03:35:27.370 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/151/13' 2023-01-24T03:35:27.371 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/151/17' 2023-01-24T03:35:27.371 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/151' 2023-01-24T03:35:27.371 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/170/19' 2023-01-24T03:35:27.371 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/170/15' 2023-01-24T03:35:27.371 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/170/20' 2023-01-24T03:35:27.372 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/170/12' 2023-01-24T03:35:27.372 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/170/10' 2023-01-24T03:35:27.372 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/170/11' 2023-01-24T03:35:27.372 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/170/16' 2023-01-24T03:35:27.372 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/170/14' 2023-01-24T03:35:27.373 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/170/18' 2023-01-24T03:35:27.373 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/170/13' 2023-01-24T03:35:27.373 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/170/17' 2023-01-24T03:35:27.373 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/170' 2023-01-24T03:35:27.373 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/167/19' 2023-01-24T03:35:27.374 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/167/15' 2023-01-24T03:35:27.374 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/167/20' 2023-01-24T03:35:27.374 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/167/12' 2023-01-24T03:35:27.374 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/167/10' 2023-01-24T03:35:27.374 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/167/11' 2023-01-24T03:35:27.375 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/167/16' 2023-01-24T03:35:27.375 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/167/14' 2023-01-24T03:35:27.375 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/167/18' 2023-01-24T03:35:27.375 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/167/13' 2023-01-24T03:35:27.375 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/167/17' 2023-01-24T03:35:27.376 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/167' 2023-01-24T03:35:27.376 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/140/19' 2023-01-24T03:35:27.376 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/140/15' 2023-01-24T03:35:27.376 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/140/20' 2023-01-24T03:35:27.376 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/140/12' 2023-01-24T03:35:27.377 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/140/10' 2023-01-24T03:35:27.377 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/140/11' 2023-01-24T03:35:27.377 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/140/16' 2023-01-24T03:35:27.377 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/140/14' 2023-01-24T03:35:27.378 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/140/18' 2023-01-24T03:35:27.378 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/140/13' 2023-01-24T03:35:27.378 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/140/17' 2023-01-24T03:35:27.378 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/140' 2023-01-24T03:35:27.378 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/142/19' 2023-01-24T03:35:27.379 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/142/15' 2023-01-24T03:35:27.379 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/142/20' 2023-01-24T03:35:27.379 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/142/12' 2023-01-24T03:35:27.379 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/142/10' 2023-01-24T03:35:28.420 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/142/11' 2023-01-24T03:35:28.420 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/142/16' 2023-01-24T03:35:28.420 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/142/14' 2023-01-24T03:35:28.421 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/142/18' 2023-01-24T03:35:28.421 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/142/13' 2023-01-24T03:35:28.421 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/142/17' 2023-01-24T03:35:28.421 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/142' 2023-01-24T03:35:28.421 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/150/19' 2023-01-24T03:35:28.422 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/150/15' 2023-01-24T03:35:28.422 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/150/20' 2023-01-24T03:35:28.422 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/150/12' 2023-01-24T03:35:28.422 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/150/10' 2023-01-24T03:35:28.422 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/150/11' 2023-01-24T03:35:28.423 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/150/16' 2023-01-24T03:35:28.423 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/150/14' 2023-01-24T03:35:28.423 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/150/18' 2023-01-24T03:35:28.423 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/150/13' 2023-01-24T03:35:28.423 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/150/17' 2023-01-24T03:35:28.424 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/150' 2023-01-24T03:35:28.424 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/105/19' 2023-01-24T03:35:28.424 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/105/15' 2023-01-24T03:35:28.424 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/105/20' 2023-01-24T03:35:28.425 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/105/12' 2023-01-24T03:35:28.425 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/105/10' 2023-01-24T03:35:28.425 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/105/11' 2023-01-24T03:35:28.425 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/105/16' 2023-01-24T03:35:28.425 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/105/14' 2023-01-24T03:35:28.426 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/105/18' 2023-01-24T03:35:28.426 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/105/13' 2023-01-24T03:35:28.426 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/105/17' 2023-01-24T03:35:28.426 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/105' 2023-01-24T03:35:28.426 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/149/19' 2023-01-24T03:35:28.427 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/149/15' 2023-01-24T03:35:28.427 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/149/20' 2023-01-24T03:35:28.427 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/149/12' 2023-01-24T03:35:28.427 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/149/10' 2023-01-24T03:35:28.427 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/149/11' 2023-01-24T03:35:28.427 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/149/16' 2023-01-24T03:35:28.428 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/149/14' 2023-01-24T03:35:28.428 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/149/18' 2023-01-24T03:35:28.428 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/149/13' 2023-01-24T03:35:28.428 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/149/17' 2023-01-24T03:35:28.428 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/149' 2023-01-24T03:35:28.429 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/109/19' 2023-01-24T03:35:28.429 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/109/15' 2023-01-24T03:35:28.429 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/109/20' 2023-01-24T03:35:28.429 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/109/12' 2023-01-24T03:35:28.429 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/109/10' 2023-01-24T03:35:28.430 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/109/11' 2023-01-24T03:35:28.430 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/109/16' 2023-01-24T03:35:28.430 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/109/14' 2023-01-24T03:35:28.430 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/109/18' 2023-01-24T03:35:28.430 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/109/13' 2023-01-24T03:35:28.430 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/109/17' 2023-01-24T03:35:28.431 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/109' 2023-01-24T03:35:28.431 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/158/19' 2023-01-24T03:35:28.431 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/158/15' 2023-01-24T03:35:28.431 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/158/20' 2023-01-24T03:35:28.431 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/158/12' 2023-01-24T03:35:28.432 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/158/10' 2023-01-24T03:35:28.432 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/158/11' 2023-01-24T03:35:28.432 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/158/16' 2023-01-24T03:35:28.432 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/158/14' 2023-01-24T03:35:28.432 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/158/18' 2023-01-24T03:35:28.433 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/158/13' 2023-01-24T03:35:28.433 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/158/17' 2023-01-24T03:35:28.433 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/158' 2023-01-24T03:35:28.433 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/106/19' 2023-01-24T03:35:28.433 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/106/15' 2023-01-24T03:35:28.433 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/106/20' 2023-01-24T03:35:28.434 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/106/12' 2023-01-24T03:35:28.434 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/106/10' 2023-01-24T03:35:28.434 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/106/11' 2023-01-24T03:35:28.434 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/106/16' 2023-01-24T03:35:28.434 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/106/14' 2023-01-24T03:35:28.435 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/106/18' 2023-01-24T03:35:28.435 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/106/13' 2023-01-24T03:35:28.435 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/106/17' 2023-01-24T03:35:28.435 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/106' 2023-01-24T03:35:28.435 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/189/19' 2023-01-24T03:35:28.435 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/189/15' 2023-01-24T03:35:28.436 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/189/20' 2023-01-24T03:35:28.436 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/189/12' 2023-01-24T03:35:28.436 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/189/10' 2023-01-24T03:35:28.436 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/189/11' 2023-01-24T03:35:28.436 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/189/16' 2023-01-24T03:35:28.437 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/189/14' 2023-01-24T03:35:28.437 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/189/18' 2023-01-24T03:35:28.437 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/189/13' 2023-01-24T03:35:28.437 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/189/17' 2023-01-24T03:35:28.437 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/189' 2023-01-24T03:35:28.438 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/133/19' 2023-01-24T03:35:28.438 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/133/15' 2023-01-24T03:35:28.438 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/133/20' 2023-01-24T03:35:28.438 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/133/12' 2023-01-24T03:35:28.438 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/133/10' 2023-01-24T03:35:28.439 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/133/11' 2023-01-24T03:35:28.439 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/133/16' 2023-01-24T03:35:28.439 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/133/14' 2023-01-24T03:35:28.439 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/133/18' 2023-01-24T03:35:28.439 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/133/13' 2023-01-24T03:35:28.440 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/133/17' 2023-01-24T03:35:28.440 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/133' 2023-01-24T03:35:28.440 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/173/19' 2023-01-24T03:35:28.440 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/173/15' 2023-01-24T03:35:28.440 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/173/20' 2023-01-24T03:35:28.441 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/173/12' 2023-01-24T03:35:28.441 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/173/10' 2023-01-24T03:35:28.441 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/173/11' 2023-01-24T03:35:29.463 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/173/16' 2023-01-24T03:35:29.463 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/173/14' 2023-01-24T03:35:29.463 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/173/18' 2023-01-24T03:35:29.464 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/173/13' 2023-01-24T03:35:29.464 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/173/17' 2023-01-24T03:35:29.464 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/173' 2023-01-24T03:35:29.465 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/174/19' 2023-01-24T03:35:29.465 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/174/15' 2023-01-24T03:35:29.465 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/174/20' 2023-01-24T03:35:29.465 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/174/12' 2023-01-24T03:35:29.466 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/174/10' 2023-01-24T03:35:29.466 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/174/11' 2023-01-24T03:35:29.466 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/174/16' 2023-01-24T03:35:29.467 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/174/14' 2023-01-24T03:35:29.467 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/174/18' 2023-01-24T03:35:29.467 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/174/13' 2023-01-24T03:35:29.467 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/174/17' 2023-01-24T03:35:29.468 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/174' 2023-01-24T03:35:29.468 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/155/19' 2023-01-24T03:35:29.468 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/155/15' 2023-01-24T03:35:29.469 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/155/20' 2023-01-24T03:35:29.469 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/155/12' 2023-01-24T03:35:29.469 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/155/10' 2023-01-24T03:35:29.469 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/155/11' 2023-01-24T03:35:29.470 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/155/16' 2023-01-24T03:35:29.470 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/155/14' 2023-01-24T03:35:29.470 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/155/18' 2023-01-24T03:35:29.470 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/155/13' 2023-01-24T03:35:29.471 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/155/17' 2023-01-24T03:35:29.471 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/155' 2023-01-24T03:35:29.471 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/114/19' 2023-01-24T03:35:29.472 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/114/15' 2023-01-24T03:35:29.472 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/114/20' 2023-01-24T03:35:29.472 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/114/12' 2023-01-24T03:35:29.472 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/114/10' 2023-01-24T03:35:29.473 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/114/11' 2023-01-24T03:35:29.473 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/114/16' 2023-01-24T03:35:29.473 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/114/14' 2023-01-24T03:35:29.474 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/114/18' 2023-01-24T03:35:29.474 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/114/13' 2023-01-24T03:35:29.474 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/114/17' 2023-01-24T03:35:29.474 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/114' 2023-01-24T03:35:29.475 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/159/19' 2023-01-24T03:35:29.475 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/159/15' 2023-01-24T03:35:29.475 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/159/20' 2023-01-24T03:35:29.475 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/159/12' 2023-01-24T03:35:29.476 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/159/10' 2023-01-24T03:35:29.476 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/159/11' 2023-01-24T03:35:29.476 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/159/16' 2023-01-24T03:35:29.476 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/159/14' 2023-01-24T03:35:29.477 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/159/18' 2023-01-24T03:35:29.477 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/159/13' 2023-01-24T03:35:29.477 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/159/17' 2023-01-24T03:35:29.477 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/159' 2023-01-24T03:35:29.478 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/187/19' 2023-01-24T03:35:29.478 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/187/15' 2023-01-24T03:35:29.478 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/187/20' 2023-01-24T03:35:29.479 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/187/12' 2023-01-24T03:35:29.479 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/187/10' 2023-01-24T03:35:29.479 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/187/11' 2023-01-24T03:35:29.479 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/187/16' 2023-01-24T03:35:29.480 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/187/14' 2023-01-24T03:35:29.480 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/187/18' 2023-01-24T03:35:29.480 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/187/13' 2023-01-24T03:35:29.480 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/187/17' 2023-01-24T03:35:29.481 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/187' 2023-01-24T03:35:29.481 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/185/19' 2023-01-24T03:35:29.481 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/185/15' 2023-01-24T03:35:29.482 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/185/20' 2023-01-24T03:35:29.482 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/185/12' 2023-01-24T03:35:29.482 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/185/10' 2023-01-24T03:35:29.482 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/185/11' 2023-01-24T03:35:29.483 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/185/16' 2023-01-24T03:35:29.483 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/185/14' 2023-01-24T03:35:29.483 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/185/18' 2023-01-24T03:35:29.483 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/185/13' 2023-01-24T03:35:29.484 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/185/17' 2023-01-24T03:35:29.484 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/185' 2023-01-24T03:35:29.484 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/120/19' 2023-01-24T03:35:29.485 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/120/15' 2023-01-24T03:35:29.485 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/120/20' 2023-01-24T03:35:29.485 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/120/12' 2023-01-24T03:35:29.486 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/120/10' 2023-01-24T03:35:29.486 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/120/11' 2023-01-24T03:35:29.486 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/120/16' 2023-01-24T03:35:29.486 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/120/14' 2023-01-24T03:35:29.486 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/120/18' 2023-01-24T03:35:29.487 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/120/13' 2023-01-24T03:35:29.487 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/120/17' 2023-01-24T03:35:29.487 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/120' 2023-01-24T03:35:29.487 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/110/19' 2023-01-24T03:35:29.487 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/110/15' 2023-01-24T03:35:29.488 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/110/20' 2023-01-24T03:35:29.488 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/110/12' 2023-01-24T03:35:29.488 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/110/10' 2023-01-24T03:35:29.488 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/110/11' 2023-01-24T03:35:29.488 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/110/16' 2023-01-24T03:35:29.489 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/110/14' 2023-01-24T03:35:29.489 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/110/18' 2023-01-24T03:35:29.489 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/110/13' 2023-01-24T03:35:29.489 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/110/17' 2023-01-24T03:35:29.489 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/110' 2023-01-24T03:35:29.490 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/137/19' 2023-01-24T03:35:29.490 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/137/15' 2023-01-24T03:35:29.490 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/137/20' 2023-01-24T03:35:29.490 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/137/12' 2023-01-24T03:35:29.490 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/137/10' 2023-01-24T03:35:29.491 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/137/11' 2023-01-24T03:35:29.491 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/137/16' 2023-01-24T03:35:29.501 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:35:29.504 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:35:31.451 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/137/14' 2023-01-24T03:35:31.452 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/137/18' 2023-01-24T03:35:31.452 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/137/13' 2023-01-24T03:35:31.452 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/137/17' 2023-01-24T03:35:31.452 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/137' 2023-01-24T03:35:31.453 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/198/19' 2023-01-24T03:35:31.453 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/198/15' 2023-01-24T03:35:31.453 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/198/20' 2023-01-24T03:35:31.453 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/198/12' 2023-01-24T03:35:31.454 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/198/10' 2023-01-24T03:35:31.454 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/198/11' 2023-01-24T03:35:31.454 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/198/16' 2023-01-24T03:35:31.454 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/198/14' 2023-01-24T03:35:31.454 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/198/18' 2023-01-24T03:35:31.455 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/198/13' 2023-01-24T03:35:31.455 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/198/17' 2023-01-24T03:35:31.455 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/198' 2023-01-24T03:35:31.455 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/136/19' 2023-01-24T03:35:31.455 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/136/15' 2023-01-24T03:35:31.456 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/136/20' 2023-01-24T03:35:31.456 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/136/12' 2023-01-24T03:35:31.456 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/136/10' 2023-01-24T03:35:31.456 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/136/11' 2023-01-24T03:35:31.456 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/136/16' 2023-01-24T03:35:31.456 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/136/14' 2023-01-24T03:35:31.457 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/136/18' 2023-01-24T03:35:31.457 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/136/13' 2023-01-24T03:35:31.457 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/136/17' 2023-01-24T03:35:31.457 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/136' 2023-01-24T03:35:31.457 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/125/19' 2023-01-24T03:35:31.458 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/125/15' 2023-01-24T03:35:31.458 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/125/20' 2023-01-24T03:35:31.458 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/125/12' 2023-01-24T03:35:31.458 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/125/10' 2023-01-24T03:35:31.458 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/125/11' 2023-01-24T03:35:31.458 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/125/16' 2023-01-24T03:35:31.459 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/125/14' 2023-01-24T03:35:31.459 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/125/18' 2023-01-24T03:35:31.459 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/125/13' 2023-01-24T03:35:31.459 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/125/17' 2023-01-24T03:35:31.459 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/125' 2023-01-24T03:35:31.460 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/194/19' 2023-01-24T03:35:31.460 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/194/15' 2023-01-24T03:35:31.460 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/194/20' 2023-01-24T03:35:31.460 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/194/12' 2023-01-24T03:35:31.460 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/194/10' 2023-01-24T03:35:31.461 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/194/11' 2023-01-24T03:35:31.461 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/194/16' 2023-01-24T03:35:31.461 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/194/14' 2023-01-24T03:35:31.461 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/194/18' 2023-01-24T03:35:31.461 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/194/13' 2023-01-24T03:35:31.462 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/194/17' 2023-01-24T03:35:31.462 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/194' 2023-01-24T03:35:31.462 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/119/19' 2023-01-24T03:35:31.462 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/119/15' 2023-01-24T03:35:31.462 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/119/20' 2023-01-24T03:35:31.463 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/119/12' 2023-01-24T03:35:31.463 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/119/10' 2023-01-24T03:35:31.463 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/119/11' 2023-01-24T03:35:31.463 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/119/16' 2023-01-24T03:35:31.464 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/119/14' 2023-01-24T03:35:31.464 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/119/18' 2023-01-24T03:35:31.464 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/119/13' 2023-01-24T03:35:31.464 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/119/17' 2023-01-24T03:35:31.464 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/119' 2023-01-24T03:35:31.465 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/157/19' 2023-01-24T03:35:31.465 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/157/15' 2023-01-24T03:35:31.465 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/157/20' 2023-01-24T03:35:31.465 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/157/12' 2023-01-24T03:35:31.465 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/157/10' 2023-01-24T03:35:31.466 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/157/11' 2023-01-24T03:35:31.466 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/157/16' 2023-01-24T03:35:31.466 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/157/14' 2023-01-24T03:35:31.466 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/157/18' 2023-01-24T03:35:31.466 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/157/13' 2023-01-24T03:35:31.467 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/157/17' 2023-01-24T03:35:31.467 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/157' 2023-01-24T03:35:31.467 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/172/19' 2023-01-24T03:35:31.467 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/172/15' 2023-01-24T03:35:31.467 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/172/20' 2023-01-24T03:35:31.467 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/172/12' 2023-01-24T03:35:31.468 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/172/10' 2023-01-24T03:35:31.468 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/172/11' 2023-01-24T03:35:31.468 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/172/16' 2023-01-24T03:35:31.468 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/172/14' 2023-01-24T03:35:31.468 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/172/18' 2023-01-24T03:35:31.469 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/172/13' 2023-01-24T03:35:31.469 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/172/17' 2023-01-24T03:35:31.469 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/172' 2023-01-24T03:35:31.469 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/186/19' 2023-01-24T03:35:31.469 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/186/15' 2023-01-24T03:35:31.470 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/186/20' 2023-01-24T03:35:31.470 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/186/12' 2023-01-24T03:35:31.470 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/186/10' 2023-01-24T03:35:31.470 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/186/11' 2023-01-24T03:35:31.470 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/186/16' 2023-01-24T03:35:31.471 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/186/14' 2023-01-24T03:35:31.471 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/186/18' 2023-01-24T03:35:31.471 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/186/13' 2023-01-24T03:35:31.471 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/186/17' 2023-01-24T03:35:31.471 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/186' 2023-01-24T03:35:31.472 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/190/19' 2023-01-24T03:35:31.472 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/190/15' 2023-01-24T03:35:31.472 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/190/20' 2023-01-24T03:35:31.472 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/190/12' 2023-01-24T03:35:31.472 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/190/10' 2023-01-24T03:35:31.472 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/190/11' 2023-01-24T03:35:31.473 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/190/16' 2023-01-24T03:35:31.473 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/190/14' 2023-01-24T03:35:31.700 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/190/18' 2023-01-24T03:35:31.701 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/190/13' 2023-01-24T03:35:31.701 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/190/17' 2023-01-24T03:35:31.701 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/190' 2023-01-24T03:35:31.701 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/104/19' 2023-01-24T03:35:31.701 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/104/15' 2023-01-24T03:35:31.702 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/104/20' 2023-01-24T03:35:31.702 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/104/12' 2023-01-24T03:35:31.702 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/104/10' 2023-01-24T03:35:31.702 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/104/11' 2023-01-24T03:35:31.702 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/104/16' 2023-01-24T03:35:31.703 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/104/14' 2023-01-24T03:35:31.703 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/104/18' 2023-01-24T03:35:31.703 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/104/13' 2023-01-24T03:35:31.703 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/104/17' 2023-01-24T03:35:31.704 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/104' 2023-01-24T03:35:31.704 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/164/19' 2023-01-24T03:35:31.704 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/164/15' 2023-01-24T03:35:31.704 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/164/20' 2023-01-24T03:35:31.705 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/164/12' 2023-01-24T03:35:31.705 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/164/10' 2023-01-24T03:35:31.705 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/164/11' 2023-01-24T03:35:31.705 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/164/16' 2023-01-24T03:35:31.705 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/164/14' 2023-01-24T03:35:31.705 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/164/18' 2023-01-24T03:35:31.706 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/164/13' 2023-01-24T03:35:31.706 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/164/17' 2023-01-24T03:35:31.706 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/164' 2023-01-24T03:35:31.706 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/112/19' 2023-01-24T03:35:31.706 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/112/15' 2023-01-24T03:35:31.707 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/112/20' 2023-01-24T03:35:31.707 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/112/12' 2023-01-24T03:35:31.707 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/112/10' 2023-01-24T03:35:31.707 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/112/11' 2023-01-24T03:35:31.707 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/112/16' 2023-01-24T03:35:31.708 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/112/14' 2023-01-24T03:35:31.708 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/112/18' 2023-01-24T03:35:31.708 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/112/13' 2023-01-24T03:35:31.708 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/112/17' 2023-01-24T03:35:31.708 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/112' 2023-01-24T03:35:31.709 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/100/19' 2023-01-24T03:35:31.709 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/100/15' 2023-01-24T03:35:31.709 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/100/20' 2023-01-24T03:35:31.709 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/100/12' 2023-01-24T03:35:31.709 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/100/10' 2023-01-24T03:35:31.709 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/100/11' 2023-01-24T03:35:31.710 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/100/16' 2023-01-24T03:35:31.710 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/100/14' 2023-01-24T03:35:31.710 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/100/18' 2023-01-24T03:35:31.710 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/100/13' 2023-01-24T03:35:31.710 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/100/17' 2023-01-24T03:35:31.711 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/100' 2023-01-24T03:35:31.711 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/134/19' 2023-01-24T03:35:31.711 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/134/15' 2023-01-24T03:35:31.711 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/134/20' 2023-01-24T03:35:31.711 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/134/12' 2023-01-24T03:35:31.712 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/134/10' 2023-01-24T03:35:31.712 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/134/11' 2023-01-24T03:35:31.712 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/134/16' 2023-01-24T03:35:31.712 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/134/14' 2023-01-24T03:35:31.713 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/134/18' 2023-01-24T03:35:31.713 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/134/13' 2023-01-24T03:35:31.713 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/134/17' 2023-01-24T03:35:31.713 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/134' 2023-01-24T03:35:31.713 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/115/19' 2023-01-24T03:35:31.714 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/115/15' 2023-01-24T03:35:31.714 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/115/20' 2023-01-24T03:35:31.714 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/115/12' 2023-01-24T03:35:31.714 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/115/10' 2023-01-24T03:35:31.714 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/115/11' 2023-01-24T03:35:31.714 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/115/16' 2023-01-24T03:35:31.715 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/115/14' 2023-01-24T03:35:31.715 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/115/18' 2023-01-24T03:35:31.715 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/115/13' 2023-01-24T03:35:31.715 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/115/17' 2023-01-24T03:35:31.715 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/115' 2023-01-24T03:35:31.715 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/116/19' 2023-01-24T03:35:31.716 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/116/15' 2023-01-24T03:35:31.716 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/116/20' 2023-01-24T03:35:31.716 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/116/12' 2023-01-24T03:35:31.716 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/116/10' 2023-01-24T03:35:31.716 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/116/11' 2023-01-24T03:35:31.717 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/116/16' 2023-01-24T03:35:31.717 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/116/14' 2023-01-24T03:35:31.717 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/116/18' 2023-01-24T03:35:31.717 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/116/13' 2023-01-24T03:35:31.717 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/116/17' 2023-01-24T03:35:31.718 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/116' 2023-01-24T03:35:31.718 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/195/19' 2023-01-24T03:35:31.718 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/195/15' 2023-01-24T03:35:31.718 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/195/20' 2023-01-24T03:35:31.718 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/195/12' 2023-01-24T03:35:31.718 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/195/10' 2023-01-24T03:35:31.719 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/195/11' 2023-01-24T03:35:31.719 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/195/16' 2023-01-24T03:35:31.719 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/195/14' 2023-01-24T03:35:31.719 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/195/18' 2023-01-24T03:35:31.719 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/195/13' 2023-01-24T03:35:31.719 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/195/17' 2023-01-24T03:35:31.720 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/195' 2023-01-24T03:35:31.720 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/138/19' 2023-01-24T03:35:31.720 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/138/15' 2023-01-24T03:35:31.720 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/138/20' 2023-01-24T03:35:31.720 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/138/12' 2023-01-24T03:35:31.721 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/138/10' 2023-01-24T03:35:31.721 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/138/11' 2023-01-24T03:35:31.721 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/138/16' 2023-01-24T03:35:31.721 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/138/14' 2023-01-24T03:35:31.722 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/138/18' 2023-01-24T03:35:33.430 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/138/13' 2023-01-24T03:35:33.430 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/138/17' 2023-01-24T03:35:33.430 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/138' 2023-01-24T03:35:33.431 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/135/19' 2023-01-24T03:35:33.431 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/135/15' 2023-01-24T03:35:33.431 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/135/20' 2023-01-24T03:35:33.431 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/135/12' 2023-01-24T03:35:33.431 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/135/10' 2023-01-24T03:35:33.432 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/135/11' 2023-01-24T03:35:33.432 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/135/16' 2023-01-24T03:35:33.432 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/135/14' 2023-01-24T03:35:33.432 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/135/18' 2023-01-24T03:35:33.432 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/135/13' 2023-01-24T03:35:33.433 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/135/17' 2023-01-24T03:35:33.433 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/135' 2023-01-24T03:35:33.433 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/117/19' 2023-01-24T03:35:33.433 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/117/15' 2023-01-24T03:35:33.433 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/117/20' 2023-01-24T03:35:33.434 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/117/12' 2023-01-24T03:35:33.434 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/117/10' 2023-01-24T03:35:33.434 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/117/11' 2023-01-24T03:35:33.434 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/117/16' 2023-01-24T03:35:33.434 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/117/14' 2023-01-24T03:35:33.435 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/117/18' 2023-01-24T03:35:33.435 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/117/13' 2023-01-24T03:35:33.435 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/117/17' 2023-01-24T03:35:33.435 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/117' 2023-01-24T03:35:33.435 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/143/19' 2023-01-24T03:35:33.436 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/143/15' 2023-01-24T03:35:33.436 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/143/20' 2023-01-24T03:35:33.436 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/143/12' 2023-01-24T03:35:33.436 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/143/10' 2023-01-24T03:35:33.436 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/143/11' 2023-01-24T03:35:33.437 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/143/16' 2023-01-24T03:35:33.437 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/143/14' 2023-01-24T03:35:33.437 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/143/18' 2023-01-24T03:35:33.437 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/143/13' 2023-01-24T03:35:33.438 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/143/17' 2023-01-24T03:35:33.438 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/143' 2023-01-24T03:35:33.438 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/180/19' 2023-01-24T03:35:33.438 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/180/15' 2023-01-24T03:35:33.438 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/180/20' 2023-01-24T03:35:33.439 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/180/12' 2023-01-24T03:35:33.439 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/180/10' 2023-01-24T03:35:33.439 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/180/11' 2023-01-24T03:35:33.439 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/180/16' 2023-01-24T03:35:33.440 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/180/14' 2023-01-24T03:35:33.440 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/180/18' 2023-01-24T03:35:33.440 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/180/13' 2023-01-24T03:35:33.440 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/180/17' 2023-01-24T03:35:33.440 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/180' 2023-01-24T03:35:33.441 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/126/19' 2023-01-24T03:35:33.441 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/126/15' 2023-01-24T03:35:33.441 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/126/20' 2023-01-24T03:35:33.441 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/126/12' 2023-01-24T03:35:33.441 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/126/10' 2023-01-24T03:35:33.442 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/126/11' 2023-01-24T03:35:33.442 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/126/16' 2023-01-24T03:35:33.442 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/126/14' 2023-01-24T03:35:33.442 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/126/18' 2023-01-24T03:35:33.442 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/126/13' 2023-01-24T03:35:33.443 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/126/17' 2023-01-24T03:35:33.443 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/126' 2023-01-24T03:35:33.443 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/161/19' 2023-01-24T03:35:33.443 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/161/15' 2023-01-24T03:35:33.443 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/161/20' 2023-01-24T03:35:33.444 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/161/12' 2023-01-24T03:35:33.444 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/161/10' 2023-01-24T03:35:33.444 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/161/11' 2023-01-24T03:35:33.444 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/161/16' 2023-01-24T03:35:33.444 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/161/14' 2023-01-24T03:35:33.444 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/161/18' 2023-01-24T03:35:33.445 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/161/13' 2023-01-24T03:35:33.445 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/161/17' 2023-01-24T03:35:33.445 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/161' 2023-01-24T03:35:33.445 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/176/19' 2023-01-24T03:35:33.445 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/176/15' 2023-01-24T03:35:33.446 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/176/20' 2023-01-24T03:35:33.446 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/176/12' 2023-01-24T03:35:33.446 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/176/10' 2023-01-24T03:35:33.446 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/176/11' 2023-01-24T03:35:33.446 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/176/16' 2023-01-24T03:35:33.447 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/176/14' 2023-01-24T03:35:33.447 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/176/18' 2023-01-24T03:35:33.447 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/176/13' 2023-01-24T03:35:33.447 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/176/17' 2023-01-24T03:35:33.447 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/176' 2023-01-24T03:35:33.447 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/111/19' 2023-01-24T03:35:33.448 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/111/15' 2023-01-24T03:35:33.448 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/111/20' 2023-01-24T03:35:33.448 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/111/12' 2023-01-24T03:35:33.449 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/111/10' 2023-01-24T03:35:33.449 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/111/11' 2023-01-24T03:35:33.449 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/111/16' 2023-01-24T03:35:33.449 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/111/14' 2023-01-24T03:35:33.449 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/111/18' 2023-01-24T03:35:33.450 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/111/13' 2023-01-24T03:35:33.450 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/111/17' 2023-01-24T03:35:33.450 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/111' 2023-01-24T03:35:33.450 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/191/19' 2023-01-24T03:35:33.450 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/191/15' 2023-01-24T03:35:33.451 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/191/20' 2023-01-24T03:35:33.451 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/191/12' 2023-01-24T03:35:33.451 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/191/10' 2023-01-24T03:35:33.451 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/191/11' 2023-01-24T03:35:33.451 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/191/16' 2023-01-24T03:35:33.452 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/191/14' 2023-01-24T03:35:33.452 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/191/18' 2023-01-24T03:35:33.452 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/191/13' 2023-01-24T03:35:34.427 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/191/17' 2023-01-24T03:35:34.428 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/191' 2023-01-24T03:35:34.428 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/184/19' 2023-01-24T03:35:34.428 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/184/15' 2023-01-24T03:35:34.428 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/184/20' 2023-01-24T03:35:34.429 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/184/12' 2023-01-24T03:35:34.429 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/184/10' 2023-01-24T03:35:34.429 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/184/11' 2023-01-24T03:35:34.429 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/184/16' 2023-01-24T03:35:34.429 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/184/14' 2023-01-24T03:35:34.429 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/184/18' 2023-01-24T03:35:34.430 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/184/13' 2023-01-24T03:35:34.430 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/184/17' 2023-01-24T03:35:34.430 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/184' 2023-01-24T03:35:34.430 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/192/19' 2023-01-24T03:35:34.430 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/192/15' 2023-01-24T03:35:34.431 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/192/20' 2023-01-24T03:35:34.431 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/192/12' 2023-01-24T03:35:34.431 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/192/10' 2023-01-24T03:35:34.431 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/192/11' 2023-01-24T03:35:34.431 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/192/16' 2023-01-24T03:35:34.432 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/192/14' 2023-01-24T03:35:34.432 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/192/18' 2023-01-24T03:35:34.432 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/192/13' 2023-01-24T03:35:34.432 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/192/17' 2023-01-24T03:35:34.432 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/192' 2023-01-24T03:35:34.433 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/103/19' 2023-01-24T03:35:34.433 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/103/15' 2023-01-24T03:35:34.433 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/103/20' 2023-01-24T03:35:34.433 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/103/12' 2023-01-24T03:35:34.433 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/103/10' 2023-01-24T03:35:34.434 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/103/11' 2023-01-24T03:35:34.434 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/103/16' 2023-01-24T03:35:34.434 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/103/14' 2023-01-24T03:35:34.434 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/103/18' 2023-01-24T03:35:34.434 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/103/13' 2023-01-24T03:35:34.434 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/103/17' 2023-01-24T03:35:34.435 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/103' 2023-01-24T03:35:34.435 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/128/19' 2023-01-24T03:35:34.435 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/128/15' 2023-01-24T03:35:34.435 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/128/20' 2023-01-24T03:35:34.436 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/128/12' 2023-01-24T03:35:34.436 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/128/10' 2023-01-24T03:35:34.436 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/128/11' 2023-01-24T03:35:34.436 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/128/16' 2023-01-24T03:35:34.436 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/128/14' 2023-01-24T03:35:34.437 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/128/18' 2023-01-24T03:35:34.437 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/128/13' 2023-01-24T03:35:34.437 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/128/17' 2023-01-24T03:35:34.437 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/128' 2023-01-24T03:35:34.437 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/144/19' 2023-01-24T03:35:34.438 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/144/15' 2023-01-24T03:35:34.438 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/144/20' 2023-01-24T03:35:34.438 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/144/12' 2023-01-24T03:35:34.438 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/144/10' 2023-01-24T03:35:34.438 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/144/11' 2023-01-24T03:35:34.438 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/144/16' 2023-01-24T03:35:34.439 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/144/14' 2023-01-24T03:35:34.439 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/144/18' 2023-01-24T03:35:34.439 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/144/13' 2023-01-24T03:35:34.439 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/144/17' 2023-01-24T03:35:34.439 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/144' 2023-01-24T03:35:34.440 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/146/19' 2023-01-24T03:35:34.440 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/146/15' 2023-01-24T03:35:34.440 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/146/20' 2023-01-24T03:35:34.440 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/146/12' 2023-01-24T03:35:34.440 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/146/10' 2023-01-24T03:35:34.441 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/146/11' 2023-01-24T03:35:34.441 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/146/16' 2023-01-24T03:35:34.441 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/146/14' 2023-01-24T03:35:34.441 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/146/18' 2023-01-24T03:35:34.441 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/146/13' 2023-01-24T03:35:34.442 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/146/17' 2023-01-24T03:35:34.442 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/146' 2023-01-24T03:35:34.442 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/141/19' 2023-01-24T03:35:34.442 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/141/15' 2023-01-24T03:35:34.442 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/141/20' 2023-01-24T03:35:34.442 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/141/12' 2023-01-24T03:35:34.443 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/141/10' 2023-01-24T03:35:34.443 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/141/11' 2023-01-24T03:35:34.443 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/141/16' 2023-01-24T03:35:34.443 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/141/14' 2023-01-24T03:35:34.443 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/141/18' 2023-01-24T03:35:34.444 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/141/13' 2023-01-24T03:35:34.444 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/141/17' 2023-01-24T03:35:34.444 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/141' 2023-01-24T03:35:34.444 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/182/19' 2023-01-24T03:35:34.444 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/182/15' 2023-01-24T03:35:34.445 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/182/20' 2023-01-24T03:35:34.445 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/182/12' 2023-01-24T03:35:34.445 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/182/10' 2023-01-24T03:35:34.445 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/182/11' 2023-01-24T03:35:34.446 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/182/16' 2023-01-24T03:35:34.446 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/182/14' 2023-01-24T03:35:34.446 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/182/18' 2023-01-24T03:35:34.446 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/182/13' 2023-01-24T03:35:34.446 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/182/17' 2023-01-24T03:35:34.447 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/182' 2023-01-24T03:35:34.447 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/107/19' 2023-01-24T03:35:34.447 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/107/15' 2023-01-24T03:35:34.447 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/107/20' 2023-01-24T03:35:34.447 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/107/12' 2023-01-24T03:35:34.448 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/107/10' 2023-01-24T03:35:34.448 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/107/11' 2023-01-24T03:35:34.448 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/107/16' 2023-01-24T03:35:34.448 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/107/14' 2023-01-24T03:35:34.448 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/107/18' 2023-01-24T03:35:34.449 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/107/13' 2023-01-24T03:35:34.449 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/107/17' 2023-01-24T03:35:35.070 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/107' 2023-01-24T03:35:35.071 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/178/19' 2023-01-24T03:35:35.071 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/178/15' 2023-01-24T03:35:35.071 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/178/20' 2023-01-24T03:35:35.071 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/178/12' 2023-01-24T03:35:35.072 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/178/10' 2023-01-24T03:35:35.072 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/178/11' 2023-01-24T03:35:35.072 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/178/16' 2023-01-24T03:35:35.072 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/178/14' 2023-01-24T03:35:35.072 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/178/18' 2023-01-24T03:35:35.073 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/178/13' 2023-01-24T03:35:35.073 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/178/17' 2023-01-24T03:35:35.073 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/178' 2023-01-24T03:35:35.073 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/156/19' 2023-01-24T03:35:35.073 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/156/15' 2023-01-24T03:35:35.073 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/156/20' 2023-01-24T03:35:35.074 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/156/12' 2023-01-24T03:35:35.074 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/156/10' 2023-01-24T03:35:35.074 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/156/11' 2023-01-24T03:35:35.074 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/156/16' 2023-01-24T03:35:35.074 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/156/14' 2023-01-24T03:35:35.075 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/156/18' 2023-01-24T03:35:35.075 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/156/13' 2023-01-24T03:35:35.075 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/156/17' 2023-01-24T03:35:35.075 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/156' 2023-01-24T03:35:35.075 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/131/19' 2023-01-24T03:35:35.076 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/131/15' 2023-01-24T03:35:35.076 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/131/20' 2023-01-24T03:35:35.076 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/131/12' 2023-01-24T03:35:35.076 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/131/10' 2023-01-24T03:35:35.077 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/131/11' 2023-01-24T03:35:35.077 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/131/16' 2023-01-24T03:35:35.077 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/131/14' 2023-01-24T03:35:35.077 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/131/18' 2023-01-24T03:35:35.077 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/131/13' 2023-01-24T03:35:35.078 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/131/17' 2023-01-24T03:35:35.078 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/131' 2023-01-24T03:35:35.078 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/123/19' 2023-01-24T03:35:35.078 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/123/15' 2023-01-24T03:35:35.078 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/123/20' 2023-01-24T03:35:35.078 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/123/12' 2023-01-24T03:35:35.079 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/123/10' 2023-01-24T03:35:35.079 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/123/11' 2023-01-24T03:35:35.079 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/123/16' 2023-01-24T03:35:35.079 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/123/14' 2023-01-24T03:35:35.079 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/123/18' 2023-01-24T03:35:35.079 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/123/13' 2023-01-24T03:35:35.080 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/123/17' 2023-01-24T03:35:35.080 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/123' 2023-01-24T03:35:35.080 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/160/19' 2023-01-24T03:35:35.080 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/160/15' 2023-01-24T03:35:35.081 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/160/20' 2023-01-24T03:35:35.081 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/160/12' 2023-01-24T03:35:35.081 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/160/10' 2023-01-24T03:35:35.081 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/160/11' 2023-01-24T03:35:35.081 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/160/16' 2023-01-24T03:35:35.082 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/160/14' 2023-01-24T03:35:35.082 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/160/18' 2023-01-24T03:35:35.082 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/160/13' 2023-01-24T03:35:35.082 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/160/17' 2023-01-24T03:35:35.082 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/160' 2023-01-24T03:35:35.083 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/165/19' 2023-01-24T03:35:35.083 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/165/15' 2023-01-24T03:35:35.083 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/165/20' 2023-01-24T03:35:35.083 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/165/12' 2023-01-24T03:35:35.083 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/165/10' 2023-01-24T03:35:35.084 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/165/11' 2023-01-24T03:35:35.084 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/165/16' 2023-01-24T03:35:35.084 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/165/14' 2023-01-24T03:35:35.084 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/165/18' 2023-01-24T03:35:35.084 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/165/13' 2023-01-24T03:35:35.085 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/165/17' 2023-01-24T03:35:35.085 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/165' 2023-01-24T03:35:35.085 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/101/19' 2023-01-24T03:35:35.085 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/101/15' 2023-01-24T03:35:35.085 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/101/20' 2023-01-24T03:35:35.086 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/101/12' 2023-01-24T03:35:35.086 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/101/10' 2023-01-24T03:35:35.086 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/101/11' 2023-01-24T03:35:35.086 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/101/16' 2023-01-24T03:35:35.086 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/101/14' 2023-01-24T03:35:35.086 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/101/18' 2023-01-24T03:35:35.087 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/101/13' 2023-01-24T03:35:35.087 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/101/17' 2023-01-24T03:35:35.087 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/101' 2023-01-24T03:35:35.087 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/148/19' 2023-01-24T03:35:35.087 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/148/15' 2023-01-24T03:35:35.088 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/148/20' 2023-01-24T03:35:35.088 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/148/12' 2023-01-24T03:35:35.088 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/148/10' 2023-01-24T03:35:35.088 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/148/11' 2023-01-24T03:35:35.088 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/148/16' 2023-01-24T03:35:35.089 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/148/14' 2023-01-24T03:35:35.089 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/148/18' 2023-01-24T03:35:35.089 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/148/13' 2023-01-24T03:35:35.089 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/148/17' 2023-01-24T03:35:35.089 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/148' 2023-01-24T03:35:35.089 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/168/19' 2023-01-24T03:35:35.090 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/168/15' 2023-01-24T03:35:35.090 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/168/20' 2023-01-24T03:35:35.090 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/168/12' 2023-01-24T03:35:35.090 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/168/10' 2023-01-24T03:35:35.090 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/168/11' 2023-01-24T03:35:35.090 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/168/16' 2023-01-24T03:35:35.091 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/168/14' 2023-01-24T03:35:35.091 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/168/18' 2023-01-24T03:35:35.091 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/168/13' 2023-01-24T03:35:35.091 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/168/17' 2023-01-24T03:35:35.091 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/168' 2023-01-24T03:35:35.372 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/179/19' 2023-01-24T03:35:35.372 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/179/15' 2023-01-24T03:35:35.372 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/179/20' 2023-01-24T03:35:35.373 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/179/12' 2023-01-24T03:35:35.373 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/179/10' 2023-01-24T03:35:35.373 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/179/11' 2023-01-24T03:35:35.373 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/179/16' 2023-01-24T03:35:35.373 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/179/14' 2023-01-24T03:35:35.374 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/179/18' 2023-01-24T03:35:35.374 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/179/13' 2023-01-24T03:35:35.374 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/179/17' 2023-01-24T03:35:35.374 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/179' 2023-01-24T03:35:35.375 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/132/19' 2023-01-24T03:35:35.375 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/132/15' 2023-01-24T03:35:35.375 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/132/20' 2023-01-24T03:35:35.375 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/132/12' 2023-01-24T03:35:35.376 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/132/10' 2023-01-24T03:35:35.376 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/132/11' 2023-01-24T03:35:35.376 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/132/16' 2023-01-24T03:35:35.376 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/132/14' 2023-01-24T03:35:35.376 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/132/18' 2023-01-24T03:35:35.377 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/132/13' 2023-01-24T03:35:35.377 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/132/17' 2023-01-24T03:35:35.377 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/132' 2023-01-24T03:35:35.377 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/199/19' 2023-01-24T03:35:35.377 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/199/15' 2023-01-24T03:35:35.377 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/199/20' 2023-01-24T03:35:35.378 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/199/12' 2023-01-24T03:35:35.378 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/199/10' 2023-01-24T03:35:35.378 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/199/11' 2023-01-24T03:35:35.378 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/199/16' 2023-01-24T03:35:35.378 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/199/14' 2023-01-24T03:35:35.379 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/199/18' 2023-01-24T03:35:35.379 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/199/13' 2023-01-24T03:35:35.379 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/199/17' 2023-01-24T03:35:35.379 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/199' 2023-01-24T03:35:35.379 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/163/19' 2023-01-24T03:35:35.380 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/163/15' 2023-01-24T03:35:35.380 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/163/20' 2023-01-24T03:35:35.380 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/163/12' 2023-01-24T03:35:35.380 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/163/10' 2023-01-24T03:35:35.380 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/163/11' 2023-01-24T03:35:35.381 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/163/16' 2023-01-24T03:35:35.381 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/163/14' 2023-01-24T03:35:35.381 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/163/18' 2023-01-24T03:35:35.381 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/163/13' 2023-01-24T03:35:35.381 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/163/17' 2023-01-24T03:35:35.381 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/163' 2023-01-24T03:35:35.382 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/153/19' 2023-01-24T03:35:35.382 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/153/15' 2023-01-24T03:35:35.382 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/153/20' 2023-01-24T03:35:35.382 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/153/12' 2023-01-24T03:35:35.382 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/153/10' 2023-01-24T03:35:35.383 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/153/11' 2023-01-24T03:35:35.383 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/153/16' 2023-01-24T03:35:35.383 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/153/14' 2023-01-24T03:35:35.384 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/153/18' 2023-01-24T03:35:35.384 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/153/13' 2023-01-24T03:35:35.384 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/153/17' 2023-01-24T03:35:35.384 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/153' 2023-01-24T03:35:35.384 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/122/19' 2023-01-24T03:35:35.385 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/122/15' 2023-01-24T03:35:35.385 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/122/20' 2023-01-24T03:35:35.385 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/122/12' 2023-01-24T03:35:35.385 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/122/10' 2023-01-24T03:35:35.385 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/122/11' 2023-01-24T03:35:35.386 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/122/16' 2023-01-24T03:35:35.386 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/122/14' 2023-01-24T03:35:35.386 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/122/18' 2023-01-24T03:35:35.386 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/122/13' 2023-01-24T03:35:35.386 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/122/17' 2023-01-24T03:35:35.387 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/122' 2023-01-24T03:35:35.387 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/162/19' 2023-01-24T03:35:35.387 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/162/15' 2023-01-24T03:35:35.387 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/162/20' 2023-01-24T03:35:35.387 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/162/12' 2023-01-24T03:35:35.388 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/162/10' 2023-01-24T03:35:35.388 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/162/11' 2023-01-24T03:35:35.388 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/162/16' 2023-01-24T03:35:35.388 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/162/14' 2023-01-24T03:35:35.388 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/162/18' 2023-01-24T03:35:35.388 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/162/13' 2023-01-24T03:35:35.389 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/162/17' 2023-01-24T03:35:35.389 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/162' 2023-01-24T03:35:35.389 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/197/19' 2023-01-24T03:35:35.389 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/197/15' 2023-01-24T03:35:35.389 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/197/20' 2023-01-24T03:35:35.390 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/197/12' 2023-01-24T03:35:35.390 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/197/10' 2023-01-24T03:35:35.390 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/197/11' 2023-01-24T03:35:35.390 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/197/16' 2023-01-24T03:35:35.390 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/197/14' 2023-01-24T03:35:35.391 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/197/18' 2023-01-24T03:35:35.391 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/197/13' 2023-01-24T03:35:35.391 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/197/17' 2023-01-24T03:35:35.391 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/197' 2023-01-24T03:35:35.391 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/113/19' 2023-01-24T03:35:35.392 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/113/15' 2023-01-24T03:35:35.392 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/113/20' 2023-01-24T03:35:35.392 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/113/12' 2023-01-24T03:35:35.392 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/113/10' 2023-01-24T03:35:35.392 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/113/11' 2023-01-24T03:35:35.393 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/113/16' 2023-01-24T03:35:35.393 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/113/14' 2023-01-24T03:35:35.393 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/113/18' 2023-01-24T03:35:35.393 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/113/13' 2023-01-24T03:35:35.393 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/113/17' 2023-01-24T03:35:35.394 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/113' 2023-01-24T03:35:35.394 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/152/19' 2023-01-24T03:35:36.884 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/152/15' 2023-01-24T03:35:36.884 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/152/20' 2023-01-24T03:35:36.885 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/152/12' 2023-01-24T03:35:36.885 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/152/10' 2023-01-24T03:35:36.885 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/152/11' 2023-01-24T03:35:36.885 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/152/16' 2023-01-24T03:35:36.885 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/152/14' 2023-01-24T03:35:36.886 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/152/18' 2023-01-24T03:35:36.886 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/152/13' 2023-01-24T03:35:36.886 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/152/17' 2023-01-24T03:35:36.886 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/152' 2023-01-24T03:35:36.886 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/121/19' 2023-01-24T03:35:36.886 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/121/15' 2023-01-24T03:35:36.887 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/121/20' 2023-01-24T03:35:36.887 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/121/12' 2023-01-24T03:35:36.887 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/121/10' 2023-01-24T03:35:36.887 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/121/11' 2023-01-24T03:35:36.887 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/121/16' 2023-01-24T03:35:36.888 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/121/14' 2023-01-24T03:35:36.888 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/121/18' 2023-01-24T03:35:36.888 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/121/13' 2023-01-24T03:35:36.888 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/121/17' 2023-01-24T03:35:36.888 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/121' 2023-01-24T03:35:36.888 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/118/19' 2023-01-24T03:35:36.889 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/118/15' 2023-01-24T03:35:36.889 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/118/20' 2023-01-24T03:35:36.889 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/118/12' 2023-01-24T03:35:36.889 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/118/10' 2023-01-24T03:35:36.889 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/118/11' 2023-01-24T03:35:36.890 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/118/16' 2023-01-24T03:35:36.890 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/118/14' 2023-01-24T03:35:36.890 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/118/18' 2023-01-24T03:35:36.890 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/118/13' 2023-01-24T03:35:36.890 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/118/17' 2023-01-24T03:35:36.891 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/118' 2023-01-24T03:35:36.891 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/196/19' 2023-01-24T03:35:36.891 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/196/15' 2023-01-24T03:35:36.891 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/196/20' 2023-01-24T03:35:36.891 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/196/12' 2023-01-24T03:35:36.892 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/196/10' 2023-01-24T03:35:36.892 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/196/11' 2023-01-24T03:35:36.892 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/196/16' 2023-01-24T03:35:36.892 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/196/14' 2023-01-24T03:35:36.892 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/196/18' 2023-01-24T03:35:36.893 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/196/13' 2023-01-24T03:35:36.893 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/196/17' 2023-01-24T03:35:36.893 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/196' 2023-01-24T03:35:36.893 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/124/19' 2023-01-24T03:35:36.893 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/124/15' 2023-01-24T03:35:36.894 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/124/20' 2023-01-24T03:35:36.894 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/124/12' 2023-01-24T03:35:36.894 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/124/10' 2023-01-24T03:35:36.894 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/124/11' 2023-01-24T03:35:36.894 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/124/16' 2023-01-24T03:35:36.895 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/124/14' 2023-01-24T03:35:36.895 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/124/18' 2023-01-24T03:35:36.895 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/124/13' 2023-01-24T03:35:36.895 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/124/17' 2023-01-24T03:35:36.895 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/124' 2023-01-24T03:35:36.896 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/188/19' 2023-01-24T03:35:36.896 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/188/15' 2023-01-24T03:35:36.896 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/188/20' 2023-01-24T03:35:36.896 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/188/12' 2023-01-24T03:35:36.896 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/188/10' 2023-01-24T03:35:36.897 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/188/11' 2023-01-24T03:35:36.897 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/188/16' 2023-01-24T03:35:36.897 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/188/14' 2023-01-24T03:35:36.897 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/188/18' 2023-01-24T03:35:36.897 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/188/13' 2023-01-24T03:35:36.898 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/188/17' 2023-01-24T03:35:36.898 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/188' 2023-01-24T03:35:36.898 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/145/19' 2023-01-24T03:35:36.898 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/145/15' 2023-01-24T03:35:36.898 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/145/20' 2023-01-24T03:35:36.899 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/145/12' 2023-01-24T03:35:36.899 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/145/10' 2023-01-24T03:35:36.899 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/145/11' 2023-01-24T03:35:36.899 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/145/16' 2023-01-24T03:35:36.899 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/145/14' 2023-01-24T03:35:36.899 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/145/18' 2023-01-24T03:35:36.900 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/145/13' 2023-01-24T03:35:36.900 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/145/17' 2023-01-24T03:35:36.900 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/145' 2023-01-24T03:35:36.900 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/130/19' 2023-01-24T03:35:36.900 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/130/15' 2023-01-24T03:35:36.901 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/130/20' 2023-01-24T03:35:36.901 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/130/12' 2023-01-24T03:35:36.901 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/130/10' 2023-01-24T03:35:36.901 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/130/11' 2023-01-24T03:35:36.901 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/130/16' 2023-01-24T03:35:36.902 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/130/14' 2023-01-24T03:35:36.902 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/130/18' 2023-01-24T03:35:36.902 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/130/13' 2023-01-24T03:35:36.902 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/130/17' 2023-01-24T03:35:36.902 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/130' 2023-01-24T03:35:36.902 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/193/19' 2023-01-24T03:35:36.903 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/193/15' 2023-01-24T03:35:36.903 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/193/20' 2023-01-24T03:35:36.903 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/193/12' 2023-01-24T03:35:36.903 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/193/10' 2023-01-24T03:35:36.903 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/193/11' 2023-01-24T03:35:36.904 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/193/16' 2023-01-24T03:35:36.904 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/193/14' 2023-01-24T03:35:36.904 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/193/18' 2023-01-24T03:35:36.904 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/193/13' 2023-01-24T03:35:36.905 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/193/17' 2023-01-24T03:35:36.905 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/193' 2023-01-24T03:35:36.905 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/175/19' 2023-01-24T03:35:36.905 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/175/15' 2023-01-24T03:35:37.393 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/175/20' 2023-01-24T03:35:37.393 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/175/12' 2023-01-24T03:35:37.393 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/175/10' 2023-01-24T03:35:37.394 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/175/11' 2023-01-24T03:35:37.394 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/175/16' 2023-01-24T03:35:37.394 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/175/14' 2023-01-24T03:35:37.394 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/175/18' 2023-01-24T03:35:37.394 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/175/13' 2023-01-24T03:35:37.395 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/175/17' 2023-01-24T03:35:37.395 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/175' 2023-01-24T03:35:37.395 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/181/19' 2023-01-24T03:35:37.395 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/181/15' 2023-01-24T03:35:37.395 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/181/20' 2023-01-24T03:35:37.395 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/181/12' 2023-01-24T03:35:37.396 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/181/10' 2023-01-24T03:35:37.396 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/181/11' 2023-01-24T03:35:37.396 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/181/16' 2023-01-24T03:35:37.396 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/181/14' 2023-01-24T03:35:37.396 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/181/18' 2023-01-24T03:35:37.397 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/181/13' 2023-01-24T03:35:37.397 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/181/17' 2023-01-24T03:35:37.397 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/181' 2023-01-24T03:35:37.397 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/127/19' 2023-01-24T03:35:37.397 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/127/15' 2023-01-24T03:35:37.398 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/127/20' 2023-01-24T03:35:37.398 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/127/12' 2023-01-24T03:35:37.398 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/127/10' 2023-01-24T03:35:37.398 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/127/11' 2023-01-24T03:35:37.398 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/127/16' 2023-01-24T03:35:37.399 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/127/14' 2023-01-24T03:35:37.399 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/127/18' 2023-01-24T03:35:37.399 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/127/13' 2023-01-24T03:35:37.399 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/127/17' 2023-01-24T03:35:37.399 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/127' 2023-01-24T03:35:37.399 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/108/19' 2023-01-24T03:35:37.400 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/108/15' 2023-01-24T03:35:37.400 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/108/20' 2023-01-24T03:35:37.400 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/108/12' 2023-01-24T03:35:37.401 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/108/10' 2023-01-24T03:35:37.401 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/108/11' 2023-01-24T03:35:37.401 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/108/16' 2023-01-24T03:35:37.401 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/108/14' 2023-01-24T03:35:37.401 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/108/18' 2023-01-24T03:35:37.401 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/108/13' 2023-01-24T03:35:37.402 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/108/17' 2023-01-24T03:35:37.402 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/108' 2023-01-24T03:35:37.402 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/183/19' 2023-01-24T03:35:37.402 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/183/15' 2023-01-24T03:35:37.402 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/183/20' 2023-01-24T03:35:37.403 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/183/12' 2023-01-24T03:35:37.403 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/183/10' 2023-01-24T03:35:37.403 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/183/11' 2023-01-24T03:35:37.404 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/183/16' 2023-01-24T03:35:37.404 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/183/14' 2023-01-24T03:35:37.404 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/183/18' 2023-01-24T03:35:37.404 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/183/13' 2023-01-24T03:35:37.404 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/183/17' 2023-01-24T03:35:37.404 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/183' 2023-01-24T03:35:37.405 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/177/19' 2023-01-24T03:35:37.405 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/177/15' 2023-01-24T03:35:37.405 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/177/20' 2023-01-24T03:35:37.405 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/177/12' 2023-01-24T03:35:37.405 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/177/10' 2023-01-24T03:35:37.406 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/177/11' 2023-01-24T03:35:37.406 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/177/16' 2023-01-24T03:35:37.406 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/177/14' 2023-01-24T03:35:37.406 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/177/18' 2023-01-24T03:35:37.406 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/177/13' 2023-01-24T03:35:37.407 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/177/17' 2023-01-24T03:35:37.407 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/177' 2023-01-24T03:35:37.407 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/169/19' 2023-01-24T03:35:37.407 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/169/15' 2023-01-24T03:35:37.407 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/169/20' 2023-01-24T03:35:37.408 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/169/12' 2023-01-24T03:35:37.408 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/169/10' 2023-01-24T03:35:37.408 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/169/11' 2023-01-24T03:35:37.408 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/169/16' 2023-01-24T03:35:37.408 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/169/14' 2023-01-24T03:35:37.409 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/169/18' 2023-01-24T03:35:37.409 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/169/13' 2023-01-24T03:35:37.409 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/169/17' 2023-01-24T03:35:37.409 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/169' 2023-01-24T03:35:37.409 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/154/19' 2023-01-24T03:35:37.410 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/154/15' 2023-01-24T03:35:37.410 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/154/20' 2023-01-24T03:35:37.410 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/154/12' 2023-01-24T03:35:37.410 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/154/10' 2023-01-24T03:35:37.410 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/154/11' 2023-01-24T03:35:37.410 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/154/16' 2023-01-24T03:35:37.411 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/154/14' 2023-01-24T03:35:37.411 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/154/18' 2023-01-24T03:35:37.411 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/154/13' 2023-01-24T03:35:37.411 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/154/17' 2023-01-24T03:35:37.412 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/154' 2023-01-24T03:35:37.412 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/171/19' 2023-01-24T03:35:37.412 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/171/15' 2023-01-24T03:35:37.412 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/171/20' 2023-01-24T03:35:37.412 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/171/12' 2023-01-24T03:35:37.413 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/171/10' 2023-01-24T03:35:37.413 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/171/11' 2023-01-24T03:35:37.413 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/171/16' 2023-01-24T03:35:37.413 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/171/14' 2023-01-24T03:35:37.414 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/171/18' 2023-01-24T03:35:37.414 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/171/13' 2023-01-24T03:35:37.414 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/171/17' 2023-01-24T03:35:37.414 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/171' 2023-01-24T03:35:37.414 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/139/19' 2023-01-24T03:35:37.415 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/139/15' 2023-01-24T03:35:37.415 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/139/20' 2023-01-24T03:35:37.485 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/139/12' 2023-01-24T03:35:37.486 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/139/10' 2023-01-24T03:35:37.486 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/139/11' 2023-01-24T03:35:37.486 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/139/16' 2023-01-24T03:35:37.486 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/139/14' 2023-01-24T03:35:37.486 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/139/18' 2023-01-24T03:35:37.487 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/139/13' 2023-01-24T03:35:37.487 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/139/17' 2023-01-24T03:35:37.487 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/139' 2023-01-24T03:35:37.487 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/102/19' 2023-01-24T03:35:37.487 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/102/15' 2023-01-24T03:35:37.487 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/102/20' 2023-01-24T03:35:37.488 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/102/12' 2023-01-24T03:35:37.488 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/102/10' 2023-01-24T03:35:37.488 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/102/11' 2023-01-24T03:35:37.488 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/102/16' 2023-01-24T03:35:37.488 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/102/14' 2023-01-24T03:35:37.489 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/102/18' 2023-01-24T03:35:37.489 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/102/13' 2023-01-24T03:35:37.489 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/102/17' 2023-01-24T03:35:37.489 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/102' 2023-01-24T03:35:37.489 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/129/19' 2023-01-24T03:35:37.490 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/129/15' 2023-01-24T03:35:37.490 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/129/20' 2023-01-24T03:35:37.490 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/129/12' 2023-01-24T03:35:37.490 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/129/10' 2023-01-24T03:35:37.490 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/129/11' 2023-01-24T03:35:37.491 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/129/16' 2023-01-24T03:35:37.491 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/129/14' 2023-01-24T03:35:37.491 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/129/18' 2023-01-24T03:35:37.491 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/129/13' 2023-01-24T03:35:37.491 INFO:tasks.workunit.client.0.smithi112.stdout:removed './copyofsnap1/test1/129/17' 2023-01-24T03:35:37.491 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1/129' 2023-01-24T03:35:37.492 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1/test1' 2023-01-24T03:35:37.492 INFO:tasks.workunit.client.0.smithi112.stdout:removed directory './copyofsnap1' 2023-01-24T03:35:37.492 INFO:tasks.workunit.client.0.smithi112.stdout:OK 2023-01-24T03:35:37.493 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-01-24T03:35:37.493 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-01-24T03:35:37.528 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-authwb.sh... 2023-01-24T03:35:37.529 DEBUG:teuthology.orchestra.run.smithi112:workunit test fs/snaps/snaptest-authwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8387d427d91a301da19bd82e090b96e1348bfadb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-authwb.sh 2023-01-24T03:35:37.591 INFO:tasks.workunit.client.0.smithi112.stderr:+ set -e 2023-01-24T03:35:37.592 INFO:tasks.workunit.client.0.smithi112.stderr:+ touch foo 2023-01-24T03:35:37.593 INFO:tasks.workunit.client.0.smithi112.stderr:+ chmod +x foo 2023-01-24T03:35:37.594 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir .snap/s 2023-01-24T03:35:37.620 INFO:tasks.workunit.client.0.smithi112.stderr:+ find .snap/s/foo -executable 2023-01-24T03:35:37.621 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep foo 2023-01-24T03:35:37.622 INFO:tasks.workunit.client.0.smithi112.stdout:.snap/s/foo 2023-01-24T03:35:37.623 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir .snap/s 2023-01-24T03:35:37.628 INFO:tasks.workunit.client.0.smithi112.stderr:+ rm foo 2023-01-24T03:35:37.630 INFO:tasks.workunit.client.0.smithi112.stdout:OK 2023-01-24T03:35:37.631 INFO:tasks.workunit.client.0.smithi112.stderr:+ echo OK 2023-01-24T03:35:37.631 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-01-24T03:35:37.631 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-01-24T03:35:37.704 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-capwb.sh... 2023-01-24T03:35:37.704 DEBUG:teuthology.orchestra.run.smithi112:workunit test fs/snaps/snaptest-capwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8387d427d91a301da19bd82e090b96e1348bfadb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-capwb.sh 2023-01-24T03:35:37.764 INFO:tasks.workunit.client.0.smithi112.stderr:+ set -e 2023-01-24T03:35:37.765 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir foo 2023-01-24T03:35:37.766 INFO:tasks.workunit.client.0.smithi112.stderr:+ echo x 2023-01-24T03:35:37.768 INFO:tasks.workunit.client.0.smithi112.stderr:+ sync 2023-01-24T03:35:38.991 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir foo/.snap/ss 2023-01-24T03:35:39.014 INFO:tasks.workunit.client.0.smithi112.stderr:+ ln foo/x foo/xx 2023-01-24T03:35:39.017 INFO:tasks.workunit.client.0.smithi112.stderr:+ cat foo/.snap/ss/x 2023-01-24T03:35:39.023 INFO:tasks.workunit.client.0.smithi112.stdout:x 2023-01-24T03:35:39.023 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir foo/.snap/ss 2023-01-24T03:35:39.027 INFO:tasks.workunit.client.0.smithi112.stderr:+ echo a 2023-01-24T03:35:39.028 INFO:tasks.workunit.client.0.smithi112.stderr:+ echo b 2023-01-24T03:35:39.029 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir foo/.snap/s 2023-01-24T03:35:39.052 INFO:tasks.workunit.client.0.smithi112.stderr:++ cat foo/.snap/s/a 2023-01-24T03:35:39.064 INFO:tasks.workunit.client.0.smithi112.stderr:+ r=a 2023-01-24T03:35:39.064 INFO:tasks.workunit.client.0.smithi112.stderr:+ '[' -z a ']' 2023-01-24T03:35:39.064 INFO:tasks.workunit.client.0.smithi112.stderr:+ ln foo/b foo/b2 2023-01-24T03:35:39.065 INFO:tasks.workunit.client.0.smithi112.stderr:+ cat foo/.snap/s/b 2023-01-24T03:35:39.333 INFO:tasks.workunit.client.0.smithi112.stdout:b 2023-01-24T03:35:39.334 INFO:tasks.workunit.client.0.smithi112.stderr:+ echo 'this used to hang:' 2023-01-24T03:35:39.334 INFO:tasks.workunit.client.0.smithi112.stdout:this used to hang: 2023-01-24T03:35:39.335 INFO:tasks.workunit.client.0.smithi112.stderr:+ echo more 2023-01-24T03:35:39.335 INFO:tasks.workunit.client.0.smithi112.stdout:oh, it didn't hang! good job. 2023-01-24T03:35:39.336 INFO:tasks.workunit.client.0.smithi112.stderr:+ echo 'oh, it didn'\''t hang! good job.' 2023-01-24T03:35:39.336 INFO:tasks.workunit.client.0.smithi112.stderr:+ cat foo/b 2023-01-24T03:35:39.336 INFO:tasks.workunit.client.0.smithi112.stdout:b 2023-01-24T03:35:39.337 INFO:tasks.workunit.client.0.smithi112.stdout:more 2023-01-24T03:35:39.337 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir foo/.snap/s 2023-01-24T03:35:39.340 INFO:tasks.workunit.client.0.smithi112.stderr:+ rm -r foo 2023-01-24T03:35:39.353 INFO:tasks.workunit.client.0.smithi112.stdout:OK 2023-01-24T03:35:39.354 INFO:tasks.workunit.client.0.smithi112.stderr:+ echo OK 2023-01-24T03:35:39.355 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-01-24T03:35:39.355 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-01-24T03:35:39.426 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-dir-rename.sh... 2023-01-24T03:35:39.427 DEBUG:teuthology.orchestra.run.smithi112:workunit test fs/snaps/snaptest-dir-rename.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8387d427d91a301da19bd82e090b96e1348bfadb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-dir-rename.sh 2023-01-24T03:35:39.487 INFO:tasks.workunit.client.0.smithi112.stderr:+ set -e 2023-01-24T03:35:39.487 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a 2023-01-24T03:35:39.489 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir .snap/bar 2023-01-24T03:35:39.516 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/foo 2023-01-24T03:35:39.537 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a/.snap/foo 2023-01-24T03:35:39.542 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a 2023-01-24T03:35:39.544 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/bar/a 2023-01-24T03:35:39.547 INFO:tasks.workunit.client.0.smithi112.stdout: File: .snap/bar/a 2023-01-24T03:35:39.547 INFO:tasks.workunit.client.0.smithi112.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2023-01-24T03:35:39.547 INFO:tasks.workunit.client.0.smithi112.stdout:Device: 100001h/1048577d Inode: 1099511633797 Links: 2 2023-01-24T03:35:39.548 INFO:tasks.workunit.client.0.smithi112.stdout:Access: (0775/drwxrwxr-x) Uid: ( 1000/ ubuntu) Gid: ( 1275/ ubuntu) 2023-01-24T03:35:39.548 INFO:tasks.workunit.client.0.smithi112.stdout:Context: unconfined_u:object_r:cephfs_t:s0 2023-01-24T03:35:39.548 INFO:tasks.workunit.client.0.smithi112.stdout:Access: 2023-01-24 03:35:39.485850143 +0000 2023-01-24T03:35:39.548 INFO:tasks.workunit.client.0.smithi112.stdout:Modify: 2023-01-24 03:35:39.485850143 +0000 2023-01-24T03:35:39.548 INFO:tasks.workunit.client.0.smithi112.stdout:Change: 2023-01-24 03:35:39.485850143 +0000 2023-01-24T03:35:39.549 INFO:tasks.workunit.client.0.smithi112.stdout: Birth: - 2023-01-24T03:35:39.549 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir .snap/bar 2023-01-24T03:35:39.555 INFO:tasks.workunit.client.0.smithi112.stderr:+ echo OK 2023-01-24T03:35:39.556 INFO:tasks.workunit.client.0.smithi112.stdout:OK 2023-01-24T03:35:39.557 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-01-24T03:35:39.557 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-01-24T03:35:39.623 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-double-null.sh... 2023-01-24T03:35:39.623 DEBUG:teuthology.orchestra.run.smithi112:workunit test fs/snaps/snaptest-double-null.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8387d427d91a301da19bd82e090b96e1348bfadb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-double-null.sh 2023-01-24T03:35:39.683 INFO:tasks.workunit.client.0.smithi112.stderr:+ set -e 2023-01-24T03:35:39.683 INFO:tasks.workunit.client.0.smithi112.stderr:++ seq 1 20 2023-01-24T03:35:39.683 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 20` 2023-01-24T03:35:39.684 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a 2023-01-24T03:35:39.685 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/one 2023-01-24T03:35:39.685 INFO:tasks.workunit.client.0.smithi112.stderr:+ cat 2023-01-24T03:35:40.501 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/two 2023-01-24T03:35:40.523 INFO:tasks.workunit.client.0.smithi112.stderr:+ chmod 777 a/foo 2023-01-24T03:35:40.524 INFO:tasks.workunit.client.0.smithi112.stderr:+ sync 2023-01-24T03:35:40.610 INFO:tasks.workunit.client.0.smithi112.stderr:+ ps 2023-01-24T03:35:40.625 INFO:tasks.workunit.client.0.smithi112.stdout: PID TTY TIME CMD 2023-01-24T03:35:40.625 INFO:tasks.workunit.client.0.smithi112.stdout: 6336 ? 00:00:00 systemd 2023-01-24T03:35:40.625 INFO:tasks.workunit.client.0.smithi112.stdout: 6339 ? 00:00:00 (sd-pam) 2023-01-24T03:35:40.626 INFO:tasks.workunit.client.0.smithi112.stdout: 72022 ? 00:00:00 sshd 2023-01-24T03:35:40.626 INFO:tasks.workunit.client.0.smithi112.stdout: 98853 ? 00:00:00 timeout 2023-01-24T03:35:40.626 INFO:tasks.workunit.client.0.smithi112.stdout: 98870 ? 00:00:00 snaptest-double 2023-01-24T03:35:40.626 INFO:tasks.workunit.client.0.smithi112.stdout: 98878 ? 00:00:00 ps 2023-01-24T03:35:40.627 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a/.snap/one a/.snap/two 2023-01-24T03:35:41.332 INFO:tasks.workunit.client.0.smithi112.stderr:+ rm a/foo 2023-01-24T03:35:41.335 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a 2023-01-24T03:35:41.342 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 20` 2023-01-24T03:35:41.342 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a 2023-01-24T03:35:41.345 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/one 2023-01-24T03:35:41.345 INFO:tasks.workunit.client.0.smithi112.stderr:+ cat 2023-01-24T03:35:41.370 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/two 2023-01-24T03:35:41.392 INFO:tasks.workunit.client.0.smithi112.stderr:+ chmod 777 a/foo 2023-01-24T03:35:41.393 INFO:tasks.workunit.client.0.smithi112.stderr:+ sync 2023-01-24T03:35:41.467 INFO:tasks.workunit.client.0.smithi112.stderr:+ ps 2023-01-24T03:35:41.475 INFO:tasks.workunit.client.0.smithi112.stdout: PID TTY TIME CMD 2023-01-24T03:35:41.475 INFO:tasks.workunit.client.0.smithi112.stdout: 6336 ? 00:00:00 systemd 2023-01-24T03:35:41.475 INFO:tasks.workunit.client.0.smithi112.stdout: 6339 ? 00:00:00 (sd-pam) 2023-01-24T03:35:41.475 INFO:tasks.workunit.client.0.smithi112.stdout: 72022 ? 00:00:00 sshd 2023-01-24T03:35:41.476 INFO:tasks.workunit.client.0.smithi112.stdout: 98853 ? 00:00:00 timeout 2023-01-24T03:35:41.476 INFO:tasks.workunit.client.0.smithi112.stdout: 98870 ? 00:00:00 snaptest-double 2023-01-24T03:35:41.476 INFO:tasks.workunit.client.0.smithi112.stdout: 98888 ? 00:00:00 ps 2023-01-24T03:35:41.477 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a/.snap/one a/.snap/two 2023-01-24T03:35:41.487 INFO:tasks.workunit.client.0.smithi112.stderr:+ rm a/foo 2023-01-24T03:35:41.490 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a 2023-01-24T03:35:41.497 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 20` 2023-01-24T03:35:41.498 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a 2023-01-24T03:35:41.500 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/one 2023-01-24T03:35:41.500 INFO:tasks.workunit.client.0.smithi112.stderr:+ cat 2023-01-24T03:35:41.524 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/two 2023-01-24T03:35:41.539 INFO:tasks.workunit.client.0.smithi112.stderr:+ chmod 777 a/foo 2023-01-24T03:35:41.540 INFO:tasks.workunit.client.0.smithi112.stderr:+ sync 2023-01-24T03:35:41.584 INFO:tasks.workunit.client.0.smithi112.stderr:+ ps 2023-01-24T03:35:41.591 INFO:tasks.workunit.client.0.smithi112.stdout: PID TTY TIME CMD 2023-01-24T03:35:41.591 INFO:tasks.workunit.client.0.smithi112.stdout: 6336 ? 00:00:00 systemd 2023-01-24T03:35:41.591 INFO:tasks.workunit.client.0.smithi112.stdout: 6339 ? 00:00:00 (sd-pam) 2023-01-24T03:35:41.591 INFO:tasks.workunit.client.0.smithi112.stdout: 72022 ? 00:00:00 sshd 2023-01-24T03:35:41.592 INFO:tasks.workunit.client.0.smithi112.stdout: 98853 ? 00:00:00 timeout 2023-01-24T03:35:41.592 INFO:tasks.workunit.client.0.smithi112.stdout: 98870 ? 00:00:00 snaptest-double 2023-01-24T03:35:41.592 INFO:tasks.workunit.client.0.smithi112.stdout: 98898 ? 00:00:00 ps 2023-01-24T03:35:41.593 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a/.snap/one a/.snap/two 2023-01-24T03:35:41.603 INFO:tasks.workunit.client.0.smithi112.stderr:+ rm a/foo 2023-01-24T03:35:41.605 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a 2023-01-24T03:35:41.610 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 20` 2023-01-24T03:35:41.611 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a 2023-01-24T03:35:41.613 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/one 2023-01-24T03:35:41.613 INFO:tasks.workunit.client.0.smithi112.stderr:+ cat 2023-01-24T03:35:41.631 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/two 2023-01-24T03:35:41.645 INFO:tasks.workunit.client.0.smithi112.stderr:+ chmod 777 a/foo 2023-01-24T03:35:41.646 INFO:tasks.workunit.client.0.smithi112.stderr:+ sync 2023-01-24T03:35:41.718 INFO:tasks.workunit.client.0.smithi112.stderr:+ ps 2023-01-24T03:35:41.725 INFO:tasks.workunit.client.0.smithi112.stdout: PID TTY TIME CMD 2023-01-24T03:35:41.725 INFO:tasks.workunit.client.0.smithi112.stdout: 6336 ? 00:00:00 systemd 2023-01-24T03:35:41.725 INFO:tasks.workunit.client.0.smithi112.stdout: 6339 ? 00:00:00 (sd-pam) 2023-01-24T03:35:41.726 INFO:tasks.workunit.client.0.smithi112.stdout: 72022 ? 00:00:00 sshd 2023-01-24T03:35:41.726 INFO:tasks.workunit.client.0.smithi112.stdout: 98853 ? 00:00:00 timeout 2023-01-24T03:35:41.726 INFO:tasks.workunit.client.0.smithi112.stdout: 98870 ? 00:00:00 snaptest-double 2023-01-24T03:35:41.726 INFO:tasks.workunit.client.0.smithi112.stdout: 98908 ? 00:00:00 ps 2023-01-24T03:35:41.727 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a/.snap/one a/.snap/two 2023-01-24T03:35:41.736 INFO:tasks.workunit.client.0.smithi112.stderr:+ rm a/foo 2023-01-24T03:35:41.738 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a 2023-01-24T03:35:41.743 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 20` 2023-01-24T03:35:41.744 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a 2023-01-24T03:35:41.746 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/one 2023-01-24T03:35:41.746 INFO:tasks.workunit.client.0.smithi112.stderr:+ cat 2023-01-24T03:35:41.764 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/two 2023-01-24T03:35:41.779 INFO:tasks.workunit.client.0.smithi112.stderr:+ chmod 777 a/foo 2023-01-24T03:35:41.780 INFO:tasks.workunit.client.0.smithi112.stderr:+ sync 2023-01-24T03:35:41.825 INFO:tasks.workunit.client.0.smithi112.stderr:+ ps 2023-01-24T03:35:41.832 INFO:tasks.workunit.client.0.smithi112.stdout: PID TTY TIME CMD 2023-01-24T03:35:41.832 INFO:tasks.workunit.client.0.smithi112.stdout: 6336 ? 00:00:00 systemd 2023-01-24T03:35:41.833 INFO:tasks.workunit.client.0.smithi112.stdout: 6339 ? 00:00:00 (sd-pam) 2023-01-24T03:35:41.833 INFO:tasks.workunit.client.0.smithi112.stdout: 72022 ? 00:00:00 sshd 2023-01-24T03:35:41.833 INFO:tasks.workunit.client.0.smithi112.stdout: 98853 ? 00:00:00 timeout 2023-01-24T03:35:41.833 INFO:tasks.workunit.client.0.smithi112.stdout: 98870 ? 00:00:00 snaptest-double 2023-01-24T03:35:41.834 INFO:tasks.workunit.client.0.smithi112.stdout: 98918 ? 00:00:00 ps 2023-01-24T03:35:41.834 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a/.snap/one a/.snap/two 2023-01-24T03:35:41.845 INFO:tasks.workunit.client.0.smithi112.stderr:+ rm a/foo 2023-01-24T03:35:41.847 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a 2023-01-24T03:35:41.853 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 20` 2023-01-24T03:35:41.854 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a 2023-01-24T03:35:41.855 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/one 2023-01-24T03:35:41.855 INFO:tasks.workunit.client.0.smithi112.stderr:+ cat 2023-01-24T03:35:41.873 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/two 2023-01-24T03:35:41.887 INFO:tasks.workunit.client.0.smithi112.stderr:+ chmod 777 a/foo 2023-01-24T03:35:41.887 INFO:tasks.workunit.client.0.smithi112.stderr:+ sync 2023-01-24T03:35:41.934 INFO:tasks.workunit.client.0.smithi112.stderr:+ ps 2023-01-24T03:35:41.942 INFO:tasks.workunit.client.0.smithi112.stdout: PID TTY TIME CMD 2023-01-24T03:35:41.942 INFO:tasks.workunit.client.0.smithi112.stdout: 6336 ? 00:00:00 systemd 2023-01-24T03:35:41.942 INFO:tasks.workunit.client.0.smithi112.stdout: 6339 ? 00:00:00 (sd-pam) 2023-01-24T03:35:41.942 INFO:tasks.workunit.client.0.smithi112.stdout: 72022 ? 00:00:00 sshd 2023-01-24T03:35:41.943 INFO:tasks.workunit.client.0.smithi112.stdout: 98853 ? 00:00:00 timeout 2023-01-24T03:35:41.943 INFO:tasks.workunit.client.0.smithi112.stdout: 98870 ? 00:00:00 snaptest-double 2023-01-24T03:35:41.943 INFO:tasks.workunit.client.0.smithi112.stdout: 98928 ? 00:00:00 ps 2023-01-24T03:35:41.943 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a/.snap/one a/.snap/two 2023-01-24T03:35:42.215 INFO:tasks.workunit.client.0.smithi112.stderr:+ rm a/foo 2023-01-24T03:35:42.221 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a 2023-01-24T03:35:42.434 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 20` 2023-01-24T03:35:42.434 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a 2023-01-24T03:35:42.436 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/one 2023-01-24T03:35:42.437 INFO:tasks.workunit.client.0.smithi112.stderr:+ cat 2023-01-24T03:35:42.455 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/two 2023-01-24T03:35:42.679 INFO:tasks.workunit.client.0.smithi112.stderr:+ chmod 777 a/foo 2023-01-24T03:35:42.680 INFO:tasks.workunit.client.0.smithi112.stderr:+ sync 2023-01-24T03:35:42.795 INFO:tasks.workunit.client.0.smithi112.stderr:+ ps 2023-01-24T03:35:42.806 INFO:tasks.workunit.client.0.smithi112.stdout: PID TTY TIME CMD 2023-01-24T03:35:42.806 INFO:tasks.workunit.client.0.smithi112.stdout: 6336 ? 00:00:00 systemd 2023-01-24T03:35:42.806 INFO:tasks.workunit.client.0.smithi112.stdout: 6339 ? 00:00:00 (sd-pam) 2023-01-24T03:35:42.806 INFO:tasks.workunit.client.0.smithi112.stdout: 72022 ? 00:00:00 sshd 2023-01-24T03:35:42.807 INFO:tasks.workunit.client.0.smithi112.stdout: 98853 ? 00:00:00 timeout 2023-01-24T03:35:42.807 INFO:tasks.workunit.client.0.smithi112.stdout: 98870 ? 00:00:00 snaptest-double 2023-01-24T03:35:42.807 INFO:tasks.workunit.client.0.smithi112.stdout: 98938 ? 00:00:00 ps 2023-01-24T03:35:42.808 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a/.snap/one a/.snap/two 2023-01-24T03:35:42.849 INFO:tasks.workunit.client.0.smithi112.stderr:+ rm a/foo 2023-01-24T03:35:42.857 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a 2023-01-24T03:35:42.871 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 20` 2023-01-24T03:35:42.871 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a 2023-01-24T03:35:42.875 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/one 2023-01-24T03:35:42.875 INFO:tasks.workunit.client.0.smithi112.stderr:+ cat 2023-01-24T03:35:42.923 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/two 2023-01-24T03:35:42.965 INFO:tasks.workunit.client.0.smithi112.stderr:+ chmod 777 a/foo 2023-01-24T03:35:42.965 INFO:tasks.workunit.client.0.smithi112.stderr:+ sync 2023-01-24T03:35:43.461 INFO:tasks.workunit.client.0.smithi112.stderr:+ ps 2023-01-24T03:35:43.469 INFO:tasks.workunit.client.0.smithi112.stdout: PID TTY TIME CMD 2023-01-24T03:35:43.469 INFO:tasks.workunit.client.0.smithi112.stdout: 6336 ? 00:00:00 systemd 2023-01-24T03:35:43.469 INFO:tasks.workunit.client.0.smithi112.stdout: 6339 ? 00:00:00 (sd-pam) 2023-01-24T03:35:43.469 INFO:tasks.workunit.client.0.smithi112.stdout: 72022 ? 00:00:00 sshd 2023-01-24T03:35:43.470 INFO:tasks.workunit.client.0.smithi112.stdout: 98853 ? 00:00:00 timeout 2023-01-24T03:35:43.470 INFO:tasks.workunit.client.0.smithi112.stdout: 98870 ? 00:00:00 snaptest-double 2023-01-24T03:35:43.470 INFO:tasks.workunit.client.0.smithi112.stdout: 98948 ? 00:00:00 ps 2023-01-24T03:35:43.471 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a/.snap/one a/.snap/two 2023-01-24T03:35:43.480 INFO:tasks.workunit.client.0.smithi112.stderr:+ rm a/foo 2023-01-24T03:35:43.482 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a 2023-01-24T03:35:43.487 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 20` 2023-01-24T03:35:43.488 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a 2023-01-24T03:35:43.490 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/one 2023-01-24T03:35:43.490 INFO:tasks.workunit.client.0.smithi112.stderr:+ cat 2023-01-24T03:35:43.514 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/two 2023-01-24T03:35:43.535 INFO:tasks.workunit.client.0.smithi112.stderr:+ chmod 777 a/foo 2023-01-24T03:35:43.536 INFO:tasks.workunit.client.0.smithi112.stderr:+ sync 2023-01-24T03:35:43.569 INFO:tasks.workunit.client.0.smithi112.stderr:+ ps 2023-01-24T03:35:43.576 INFO:tasks.workunit.client.0.smithi112.stdout: PID TTY TIME CMD 2023-01-24T03:35:43.577 INFO:tasks.workunit.client.0.smithi112.stdout: 6336 ? 00:00:00 systemd 2023-01-24T03:35:43.577 INFO:tasks.workunit.client.0.smithi112.stdout: 6339 ? 00:00:00 (sd-pam) 2023-01-24T03:35:43.577 INFO:tasks.workunit.client.0.smithi112.stdout: 72022 ? 00:00:00 sshd 2023-01-24T03:35:43.577 INFO:tasks.workunit.client.0.smithi112.stdout: 98853 ? 00:00:00 timeout 2023-01-24T03:35:43.577 INFO:tasks.workunit.client.0.smithi112.stdout: 98870 ? 00:00:00 snaptest-double 2023-01-24T03:35:43.578 INFO:tasks.workunit.client.0.smithi112.stdout: 98958 ? 00:00:00 ps 2023-01-24T03:35:43.578 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a/.snap/one a/.snap/two 2023-01-24T03:35:43.586 INFO:tasks.workunit.client.0.smithi112.stderr:+ rm a/foo 2023-01-24T03:35:43.588 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a 2023-01-24T03:35:43.594 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 20` 2023-01-24T03:35:43.594 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a 2023-01-24T03:35:43.596 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/one 2023-01-24T03:35:43.596 INFO:tasks.workunit.client.0.smithi112.stderr:+ cat 2023-01-24T03:35:43.621 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/two 2023-01-24T03:35:43.641 INFO:tasks.workunit.client.0.smithi112.stderr:+ chmod 777 a/foo 2023-01-24T03:35:43.642 INFO:tasks.workunit.client.0.smithi112.stderr:+ sync 2023-01-24T03:35:43.676 INFO:tasks.workunit.client.0.smithi112.stderr:+ ps 2023-01-24T03:35:43.684 INFO:tasks.workunit.client.0.smithi112.stdout: PID TTY TIME CMD 2023-01-24T03:35:43.684 INFO:tasks.workunit.client.0.smithi112.stdout: 6336 ? 00:00:00 systemd 2023-01-24T03:35:43.685 INFO:tasks.workunit.client.0.smithi112.stdout: 6339 ? 00:00:00 (sd-pam) 2023-01-24T03:35:43.685 INFO:tasks.workunit.client.0.smithi112.stdout: 72022 ? 00:00:00 sshd 2023-01-24T03:35:43.685 INFO:tasks.workunit.client.0.smithi112.stdout: 98853 ? 00:00:00 timeout 2023-01-24T03:35:43.685 INFO:tasks.workunit.client.0.smithi112.stdout: 98870 ? 00:00:00 snaptest-double 2023-01-24T03:35:43.685 INFO:tasks.workunit.client.0.smithi112.stdout: 98968 ? 00:00:00 ps 2023-01-24T03:35:43.686 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a/.snap/one a/.snap/two 2023-01-24T03:35:43.719 INFO:tasks.workunit.client.0.smithi112.stderr:+ rm a/foo 2023-01-24T03:35:43.721 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a 2023-01-24T03:35:43.726 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 20` 2023-01-24T03:35:43.726 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a 2023-01-24T03:35:43.728 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/one 2023-01-24T03:35:43.728 INFO:tasks.workunit.client.0.smithi112.stderr:+ cat 2023-01-24T03:35:43.752 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/two 2023-01-24T03:35:43.773 INFO:tasks.workunit.client.0.smithi112.stderr:+ chmod 777 a/foo 2023-01-24T03:35:43.774 INFO:tasks.workunit.client.0.smithi112.stderr:+ sync 2023-01-24T03:35:43.854 INFO:tasks.workunit.client.0.smithi112.stderr:+ ps 2023-01-24T03:35:43.861 INFO:tasks.workunit.client.0.smithi112.stdout: PID TTY TIME CMD 2023-01-24T03:35:43.862 INFO:tasks.workunit.client.0.smithi112.stdout: 6336 ? 00:00:00 systemd 2023-01-24T03:35:43.862 INFO:tasks.workunit.client.0.smithi112.stdout: 6339 ? 00:00:00 (sd-pam) 2023-01-24T03:35:43.862 INFO:tasks.workunit.client.0.smithi112.stdout: 72022 ? 00:00:00 sshd 2023-01-24T03:35:43.862 INFO:tasks.workunit.client.0.smithi112.stdout: 98853 ? 00:00:00 timeout 2023-01-24T03:35:43.863 INFO:tasks.workunit.client.0.smithi112.stdout: 98870 ? 00:00:00 snaptest-double 2023-01-24T03:35:43.863 INFO:tasks.workunit.client.0.smithi112.stdout: 98978 ? 00:00:00 ps 2023-01-24T03:35:43.863 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a/.snap/one a/.snap/two 2023-01-24T03:35:43.872 INFO:tasks.workunit.client.0.smithi112.stderr:+ rm a/foo 2023-01-24T03:35:43.874 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a 2023-01-24T03:35:43.885 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 20` 2023-01-24T03:35:43.885 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a 2023-01-24T03:35:43.887 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/one 2023-01-24T03:35:43.887 INFO:tasks.workunit.client.0.smithi112.stderr:+ cat 2023-01-24T03:35:43.911 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/two 2023-01-24T03:35:44.185 INFO:tasks.workunit.client.0.smithi112.stderr:+ chmod 777 a/foo 2023-01-24T03:35:44.186 INFO:tasks.workunit.client.0.smithi112.stderr:+ sync 2023-01-24T03:35:44.296 INFO:tasks.workunit.client.0.smithi112.stderr:+ ps 2023-01-24T03:35:44.304 INFO:tasks.workunit.client.0.smithi112.stdout: PID TTY TIME CMD 2023-01-24T03:35:44.304 INFO:tasks.workunit.client.0.smithi112.stdout: 6336 ? 00:00:00 systemd 2023-01-24T03:35:44.304 INFO:tasks.workunit.client.0.smithi112.stdout: 6339 ? 00:00:00 (sd-pam) 2023-01-24T03:35:44.304 INFO:tasks.workunit.client.0.smithi112.stdout: 72022 ? 00:00:00 sshd 2023-01-24T03:35:44.305 INFO:tasks.workunit.client.0.smithi112.stdout: 98853 ? 00:00:00 timeout 2023-01-24T03:35:44.305 INFO:tasks.workunit.client.0.smithi112.stdout: 98870 ? 00:00:00 snaptest-double 2023-01-24T03:35:44.305 INFO:tasks.workunit.client.0.smithi112.stdout: 98988 ? 00:00:00 ps 2023-01-24T03:35:44.305 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a/.snap/one a/.snap/two 2023-01-24T03:35:44.316 INFO:tasks.workunit.client.0.smithi112.stderr:+ rm a/foo 2023-01-24T03:35:44.318 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a 2023-01-24T03:35:44.324 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 20` 2023-01-24T03:35:44.325 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a 2023-01-24T03:35:44.327 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/one 2023-01-24T03:35:44.327 INFO:tasks.workunit.client.0.smithi112.stderr:+ cat 2023-01-24T03:35:44.345 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/two 2023-01-24T03:35:44.361 INFO:tasks.workunit.client.0.smithi112.stderr:+ chmod 777 a/foo 2023-01-24T03:35:44.361 INFO:tasks.workunit.client.0.smithi112.stderr:+ sync 2023-01-24T03:35:44.428 INFO:tasks.workunit.client.0.smithi112.stderr:+ ps 2023-01-24T03:35:44.435 INFO:tasks.workunit.client.0.smithi112.stdout: PID TTY TIME CMD 2023-01-24T03:35:44.435 INFO:tasks.workunit.client.0.smithi112.stdout: 6336 ? 00:00:00 systemd 2023-01-24T03:35:44.436 INFO:tasks.workunit.client.0.smithi112.stdout: 6339 ? 00:00:00 (sd-pam) 2023-01-24T03:35:44.436 INFO:tasks.workunit.client.0.smithi112.stdout: 72022 ? 00:00:00 sshd 2023-01-24T03:35:44.436 INFO:tasks.workunit.client.0.smithi112.stdout: 98853 ? 00:00:00 timeout 2023-01-24T03:35:44.436 INFO:tasks.workunit.client.0.smithi112.stdout: 98870 ? 00:00:00 snaptest-double 2023-01-24T03:35:44.437 INFO:tasks.workunit.client.0.smithi112.stdout: 98998 ? 00:00:00 ps 2023-01-24T03:35:44.437 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a/.snap/one a/.snap/two 2023-01-24T03:35:44.448 INFO:tasks.workunit.client.0.smithi112.stderr:+ rm a/foo 2023-01-24T03:35:44.450 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a 2023-01-24T03:35:44.457 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 20` 2023-01-24T03:35:44.457 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a 2023-01-24T03:35:44.459 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/one 2023-01-24T03:35:44.459 INFO:tasks.workunit.client.0.smithi112.stderr:+ cat 2023-01-24T03:35:44.478 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/two 2023-01-24T03:35:44.494 INFO:tasks.workunit.client.0.smithi112.stderr:+ chmod 777 a/foo 2023-01-24T03:35:44.494 INFO:tasks.workunit.client.0.smithi112.stderr:+ sync 2023-01-24T03:35:44.549 INFO:tasks.workunit.client.0.smithi112.stderr:+ ps 2023-01-24T03:35:44.556 INFO:tasks.workunit.client.0.smithi112.stdout: PID TTY TIME CMD 2023-01-24T03:35:44.556 INFO:tasks.workunit.client.0.smithi112.stdout: 6336 ? 00:00:00 systemd 2023-01-24T03:35:44.556 INFO:tasks.workunit.client.0.smithi112.stdout: 6339 ? 00:00:00 (sd-pam) 2023-01-24T03:35:44.557 INFO:tasks.workunit.client.0.smithi112.stdout: 72022 ? 00:00:00 sshd 2023-01-24T03:35:44.557 INFO:tasks.workunit.client.0.smithi112.stdout: 98853 ? 00:00:00 timeout 2023-01-24T03:35:44.557 INFO:tasks.workunit.client.0.smithi112.stdout: 98870 ? 00:00:00 snaptest-double 2023-01-24T03:35:44.557 INFO:tasks.workunit.client.0.smithi112.stdout: 99008 ? 00:00:00 ps 2023-01-24T03:35:44.558 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a/.snap/one a/.snap/two 2023-01-24T03:35:44.569 INFO:tasks.workunit.client.0.smithi112.stderr:+ rm a/foo 2023-01-24T03:35:44.587 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a 2023-01-24T03:35:44.594 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 20` 2023-01-24T03:35:44.594 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a 2023-01-24T03:35:44.596 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/one 2023-01-24T03:35:44.596 INFO:tasks.workunit.client.0.smithi112.stderr:+ cat 2023-01-24T03:35:44.616 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/two 2023-01-24T03:35:44.631 INFO:tasks.workunit.client.0.smithi112.stderr:+ chmod 777 a/foo 2023-01-24T03:35:44.632 INFO:tasks.workunit.client.0.smithi112.stderr:+ sync 2023-01-24T03:35:44.694 INFO:tasks.workunit.client.0.smithi112.stderr:+ ps 2023-01-24T03:35:44.702 INFO:tasks.workunit.client.0.smithi112.stdout: PID TTY TIME CMD 2023-01-24T03:35:44.702 INFO:tasks.workunit.client.0.smithi112.stdout: 6336 ? 00:00:00 systemd 2023-01-24T03:35:44.702 INFO:tasks.workunit.client.0.smithi112.stdout: 6339 ? 00:00:00 (sd-pam) 2023-01-24T03:35:44.703 INFO:tasks.workunit.client.0.smithi112.stdout: 72022 ? 00:00:00 sshd 2023-01-24T03:35:44.703 INFO:tasks.workunit.client.0.smithi112.stdout: 98853 ? 00:00:00 timeout 2023-01-24T03:35:44.703 INFO:tasks.workunit.client.0.smithi112.stdout: 98870 ? 00:00:00 snaptest-double 2023-01-24T03:35:44.703 INFO:tasks.workunit.client.0.smithi112.stdout: 99018 ? 00:00:00 ps 2023-01-24T03:35:44.704 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a/.snap/one a/.snap/two 2023-01-24T03:35:44.738 INFO:tasks.workunit.client.0.smithi112.stderr:+ rm a/foo 2023-01-24T03:35:44.742 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a 2023-01-24T03:35:44.752 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 20` 2023-01-24T03:35:44.752 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a 2023-01-24T03:35:44.754 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/one 2023-01-24T03:35:44.754 INFO:tasks.workunit.client.0.smithi112.stderr:+ cat 2023-01-24T03:35:44.779 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/two 2023-01-24T03:35:44.801 INFO:tasks.workunit.client.0.smithi112.stderr:+ chmod 777 a/foo 2023-01-24T03:35:44.802 INFO:tasks.workunit.client.0.smithi112.stderr:+ sync 2023-01-24T03:35:44.879 INFO:tasks.workunit.client.0.smithi112.stderr:+ ps 2023-01-24T03:35:44.886 INFO:tasks.workunit.client.0.smithi112.stdout: PID TTY TIME CMD 2023-01-24T03:35:44.886 INFO:tasks.workunit.client.0.smithi112.stdout: 6336 ? 00:00:00 systemd 2023-01-24T03:35:44.887 INFO:tasks.workunit.client.0.smithi112.stdout: 6339 ? 00:00:00 (sd-pam) 2023-01-24T03:35:44.887 INFO:tasks.workunit.client.0.smithi112.stdout: 72022 ? 00:00:00 sshd 2023-01-24T03:35:44.887 INFO:tasks.workunit.client.0.smithi112.stdout: 98853 ? 00:00:00 timeout 2023-01-24T03:35:44.887 INFO:tasks.workunit.client.0.smithi112.stdout: 98870 ? 00:00:00 snaptest-double 2023-01-24T03:35:44.887 INFO:tasks.workunit.client.0.smithi112.stdout: 99028 ? 00:00:00 ps 2023-01-24T03:35:44.888 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a/.snap/one a/.snap/two 2023-01-24T03:35:44.898 INFO:tasks.workunit.client.0.smithi112.stderr:+ rm a/foo 2023-01-24T03:35:44.900 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a 2023-01-24T03:35:44.906 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 20` 2023-01-24T03:35:44.906 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a 2023-01-24T03:35:44.908 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/one 2023-01-24T03:35:44.908 INFO:tasks.workunit.client.0.smithi112.stderr:+ cat 2023-01-24T03:35:44.926 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/two 2023-01-24T03:35:44.941 INFO:tasks.workunit.client.0.smithi112.stderr:+ chmod 777 a/foo 2023-01-24T03:35:44.942 INFO:tasks.workunit.client.0.smithi112.stderr:+ sync 2023-01-24T03:35:45.056 INFO:tasks.workunit.client.0.smithi112.stderr:+ ps 2023-01-24T03:35:45.063 INFO:tasks.workunit.client.0.smithi112.stdout: PID TTY TIME CMD 2023-01-24T03:35:45.064 INFO:tasks.workunit.client.0.smithi112.stdout: 6336 ? 00:00:00 systemd 2023-01-24T03:35:45.064 INFO:tasks.workunit.client.0.smithi112.stdout: 6339 ? 00:00:00 (sd-pam) 2023-01-24T03:35:45.064 INFO:tasks.workunit.client.0.smithi112.stdout: 72022 ? 00:00:00 sshd 2023-01-24T03:35:45.064 INFO:tasks.workunit.client.0.smithi112.stdout: 98853 ? 00:00:00 timeout 2023-01-24T03:35:45.065 INFO:tasks.workunit.client.0.smithi112.stdout: 98870 ? 00:00:00 snaptest-double 2023-01-24T03:35:45.065 INFO:tasks.workunit.client.0.smithi112.stdout: 99038 ? 00:00:00 ps 2023-01-24T03:35:45.065 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a/.snap/one a/.snap/two 2023-01-24T03:35:45.074 INFO:tasks.workunit.client.0.smithi112.stderr:+ rm a/foo 2023-01-24T03:35:45.076 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a 2023-01-24T03:35:45.081 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 20` 2023-01-24T03:35:45.081 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a 2023-01-24T03:35:45.083 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/one 2023-01-24T03:35:45.084 INFO:tasks.workunit.client.0.smithi112.stderr:+ cat 2023-01-24T03:35:45.101 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/two 2023-01-24T03:35:45.115 INFO:tasks.workunit.client.0.smithi112.stderr:+ chmod 777 a/foo 2023-01-24T03:35:45.115 INFO:tasks.workunit.client.0.smithi112.stderr:+ sync 2023-01-24T03:35:45.196 INFO:tasks.workunit.client.0.smithi112.stderr:+ ps 2023-01-24T03:35:45.205 INFO:tasks.workunit.client.0.smithi112.stdout: PID TTY TIME CMD 2023-01-24T03:35:45.205 INFO:tasks.workunit.client.0.smithi112.stdout: 6336 ? 00:00:00 systemd 2023-01-24T03:35:45.205 INFO:tasks.workunit.client.0.smithi112.stdout: 6339 ? 00:00:00 (sd-pam) 2023-01-24T03:35:45.205 INFO:tasks.workunit.client.0.smithi112.stdout: 72022 ? 00:00:00 sshd 2023-01-24T03:35:45.206 INFO:tasks.workunit.client.0.smithi112.stdout: 98853 ? 00:00:00 timeout 2023-01-24T03:35:45.206 INFO:tasks.workunit.client.0.smithi112.stdout: 98870 ? 00:00:00 snaptest-double 2023-01-24T03:35:45.206 INFO:tasks.workunit.client.0.smithi112.stdout: 99048 ? 00:00:00 ps 2023-01-24T03:35:45.208 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a/.snap/one a/.snap/two 2023-01-24T03:35:45.234 INFO:tasks.workunit.client.0.smithi112.stderr:+ rm a/foo 2023-01-24T03:35:45.236 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a 2023-01-24T03:35:45.241 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 20` 2023-01-24T03:35:45.241 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a 2023-01-24T03:35:45.243 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/one 2023-01-24T03:35:45.243 INFO:tasks.workunit.client.0.smithi112.stderr:+ cat 2023-01-24T03:35:45.262 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/two 2023-01-24T03:35:45.276 INFO:tasks.workunit.client.0.smithi112.stderr:+ chmod 777 a/foo 2023-01-24T03:35:45.277 INFO:tasks.workunit.client.0.smithi112.stderr:+ sync 2023-01-24T03:35:46.025 INFO:tasks.workunit.client.0.smithi112.stderr:+ ps 2023-01-24T03:35:46.032 INFO:tasks.workunit.client.0.smithi112.stdout: PID TTY TIME CMD 2023-01-24T03:35:46.033 INFO:tasks.workunit.client.0.smithi112.stdout: 6336 ? 00:00:00 systemd 2023-01-24T03:35:46.033 INFO:tasks.workunit.client.0.smithi112.stdout: 6339 ? 00:00:00 (sd-pam) 2023-01-24T03:35:46.033 INFO:tasks.workunit.client.0.smithi112.stdout: 72022 ? 00:00:00 sshd 2023-01-24T03:35:46.034 INFO:tasks.workunit.client.0.smithi112.stdout: 98853 ? 00:00:00 timeout 2023-01-24T03:35:46.034 INFO:tasks.workunit.client.0.smithi112.stdout: 98870 ? 00:00:00 snaptest-double 2023-01-24T03:35:46.034 INFO:tasks.workunit.client.0.smithi112.stdout: 99058 ? 00:00:00 ps 2023-01-24T03:35:46.034 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a/.snap/one a/.snap/two 2023-01-24T03:35:46.045 INFO:tasks.workunit.client.0.smithi112.stderr:+ rm a/foo 2023-01-24T03:35:46.047 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a 2023-01-24T03:35:46.053 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 20` 2023-01-24T03:35:46.053 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a 2023-01-24T03:35:46.055 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/one 2023-01-24T03:35:46.055 INFO:tasks.workunit.client.0.smithi112.stderr:+ cat 2023-01-24T03:35:46.073 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir a/.snap/two 2023-01-24T03:35:46.087 INFO:tasks.workunit.client.0.smithi112.stderr:+ chmod 777 a/foo 2023-01-24T03:35:46.088 INFO:tasks.workunit.client.0.smithi112.stderr:+ sync 2023-01-24T03:35:46.472 INFO:tasks.workunit.client.0.smithi112.stderr:+ ps 2023-01-24T03:35:46.480 INFO:tasks.workunit.client.0.smithi112.stdout: PID TTY TIME CMD 2023-01-24T03:35:46.480 INFO:tasks.workunit.client.0.smithi112.stdout: 6336 ? 00:00:00 systemd 2023-01-24T03:35:46.480 INFO:tasks.workunit.client.0.smithi112.stdout: 6339 ? 00:00:00 (sd-pam) 2023-01-24T03:35:46.481 INFO:tasks.workunit.client.0.smithi112.stdout: 72022 ? 00:00:00 sshd 2023-01-24T03:35:46.481 INFO:tasks.workunit.client.0.smithi112.stdout: 98853 ? 00:00:00 timeout 2023-01-24T03:35:46.481 INFO:tasks.workunit.client.0.smithi112.stdout: 98870 ? 00:00:00 snaptest-double 2023-01-24T03:35:46.481 INFO:tasks.workunit.client.0.smithi112.stdout: 99068 ? 00:00:00 ps 2023-01-24T03:35:46.482 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a/.snap/one a/.snap/two 2023-01-24T03:35:46.493 INFO:tasks.workunit.client.0.smithi112.stderr:+ rm a/foo 2023-01-24T03:35:46.496 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir a 2023-01-24T03:35:46.502 INFO:tasks.workunit.client.0.smithi112.stdout:OK 2023-01-24T03:35:46.503 INFO:tasks.workunit.client.0.smithi112.stderr:+ echo OK 2023-01-24T03:35:46.504 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-01-24T03:35:46.504 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-01-24T03:35:46.579 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-estale.sh... 2023-01-24T03:35:46.579 DEBUG:teuthology.orchestra.run.smithi112:workunit test fs/snaps/snaptest-estale.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8387d427d91a301da19bd82e090b96e1348bfadb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-estale.sh 2023-01-24T03:35:46.640 INFO:tasks.workunit.client.0.smithi112.stderr:+ mkdir .snap/foo 2023-01-24T03:35:46.658 INFO:tasks.workunit.client.0.smithi112.stderr:+ echo 'We want ENOENT, not ESTALE, here.' 2023-01-24T03:35:46.658 INFO:tasks.workunit.client.0.smithi112.stdout:We want ENOENT, not ESTALE, here. 2023-01-24T03:35:46.659 INFO:tasks.workunit.client.0.smithi112.stderr:++ seq 1 100 2023-01-24T03:35:46.659 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.659 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/1 2023-01-24T03:35:46.660 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.661 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/1': No such file or directory 2023-01-24T03:35:46.662 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.662 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/2 2023-01-24T03:35:46.662 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.663 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/2': No such file or directory 2023-01-24T03:35:46.663 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.663 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/3 2023-01-24T03:35:46.664 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.664 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/3': No such file or directory 2023-01-24T03:35:46.665 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.665 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/4 2023-01-24T03:35:46.665 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.666 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/4': No such file or directory 2023-01-24T03:35:46.666 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.667 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/5 2023-01-24T03:35:46.667 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.667 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/5': No such file or directory 2023-01-24T03:35:46.668 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.668 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/6 2023-01-24T03:35:46.668 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.669 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/6': No such file or directory 2023-01-24T03:35:46.669 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.670 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/7 2023-01-24T03:35:46.670 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.670 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/7': No such file or directory 2023-01-24T03:35:46.671 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.671 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/8 2023-01-24T03:35:46.671 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.672 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/8': No such file or directory 2023-01-24T03:35:46.672 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.673 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/9 2023-01-24T03:35:46.673 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.673 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/9': No such file or directory 2023-01-24T03:35:46.674 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.674 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/10 2023-01-24T03:35:46.674 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.675 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/10': No such file or directory 2023-01-24T03:35:46.675 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.675 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/11 2023-01-24T03:35:46.676 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.676 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/11': No such file or directory 2023-01-24T03:35:46.677 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.677 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/12 2023-01-24T03:35:46.677 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.677 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/12': No such file or directory 2023-01-24T03:35:46.678 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.678 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/13 2023-01-24T03:35:46.678 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.679 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/13': No such file or directory 2023-01-24T03:35:46.679 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.680 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/14 2023-01-24T03:35:46.680 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.680 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/14': No such file or directory 2023-01-24T03:35:46.681 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.681 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/15 2023-01-24T03:35:46.681 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.682 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/15': No such file or directory 2023-01-24T03:35:46.682 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.682 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/16 2023-01-24T03:35:46.683 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.683 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/16': No such file or directory 2023-01-24T03:35:46.684 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.684 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/17 2023-01-24T03:35:46.684 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.685 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/17': No such file or directory 2023-01-24T03:35:46.685 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.685 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/18 2023-01-24T03:35:46.686 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.686 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/18': No such file or directory 2023-01-24T03:35:46.686 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.687 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/19 2023-01-24T03:35:46.687 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.687 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/19': No such file or directory 2023-01-24T03:35:46.688 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.688 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/20 2023-01-24T03:35:46.688 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.689 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/20': No such file or directory 2023-01-24T03:35:46.689 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.689 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/21 2023-01-24T03:35:46.690 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.691 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/21': No such file or directory 2023-01-24T03:35:46.691 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.691 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/22 2023-01-24T03:35:46.692 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.693 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/22': No such file or directory 2023-01-24T03:35:46.693 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.694 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/23 2023-01-24T03:35:46.694 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.694 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/23': No such file or directory 2023-01-24T03:35:46.695 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.695 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/24 2023-01-24T03:35:46.695 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.697 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/24': No such file or directory 2023-01-24T03:35:46.697 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.697 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/25 2023-01-24T03:35:46.698 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.699 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/25': No such file or directory 2023-01-24T03:35:46.699 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.699 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/26 2023-01-24T03:35:46.700 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.701 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/26': No such file or directory 2023-01-24T03:35:46.701 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.701 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/27 2023-01-24T03:35:46.702 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.703 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/27': No such file or directory 2023-01-24T03:35:46.703 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.704 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/28 2023-01-24T03:35:46.704 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.706 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/28': No such file or directory 2023-01-24T03:35:46.706 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.707 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/29 2023-01-24T03:35:46.707 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.707 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/29': No such file or directory 2023-01-24T03:35:46.708 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.708 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/30 2023-01-24T03:35:46.708 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.709 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/30': No such file or directory 2023-01-24T03:35:46.710 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.710 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/31 2023-01-24T03:35:46.710 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.710 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/31': No such file or directory 2023-01-24T03:35:46.711 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.711 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/32 2023-01-24T03:35:46.712 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.712 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/32': No such file or directory 2023-01-24T03:35:46.713 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.713 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/33 2023-01-24T03:35:46.713 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.713 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/33': No such file or directory 2023-01-24T03:35:46.714 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.714 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/34 2023-01-24T03:35:46.714 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.715 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/34': No such file or directory 2023-01-24T03:35:46.716 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.716 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/35 2023-01-24T03:35:46.716 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.717 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/35': No such file or directory 2023-01-24T03:35:46.717 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.718 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/36 2023-01-24T03:35:46.718 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.718 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/36': No such file or directory 2023-01-24T03:35:46.719 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.719 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/37 2023-01-24T03:35:46.719 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.721 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/37': No such file or directory 2023-01-24T03:35:46.721 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.721 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/38 2023-01-24T03:35:46.722 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.722 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/38': No such file or directory 2023-01-24T03:35:46.723 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.723 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/39 2023-01-24T03:35:46.723 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.724 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/39': No such file or directory 2023-01-24T03:35:46.724 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.725 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/40 2023-01-24T03:35:46.725 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.727 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/40': No such file or directory 2023-01-24T03:35:46.727 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.727 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/41 2023-01-24T03:35:46.728 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.730 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/41': No such file or directory 2023-01-24T03:35:46.730 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.730 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/42 2023-01-24T03:35:46.731 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.731 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/42': No such file or directory 2023-01-24T03:35:46.732 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.732 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/43 2023-01-24T03:35:46.732 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.733 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/43': No such file or directory 2023-01-24T03:35:46.733 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.733 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/44 2023-01-24T03:35:46.734 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.734 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/44': No such file or directory 2023-01-24T03:35:46.735 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.735 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/45 2023-01-24T03:35:46.735 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.735 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/45': No such file or directory 2023-01-24T03:35:46.736 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.736 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/46 2023-01-24T03:35:46.737 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.737 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/46': No such file or directory 2023-01-24T03:35:46.738 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.738 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/47 2023-01-24T03:35:46.738 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.738 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/47': No such file or directory 2023-01-24T03:35:46.739 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.739 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/48 2023-01-24T03:35:46.739 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.740 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/48': No such file or directory 2023-01-24T03:35:46.740 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.741 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/49 2023-01-24T03:35:46.741 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.741 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/49': No such file or directory 2023-01-24T03:35:46.742 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.742 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/50 2023-01-24T03:35:46.742 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.742 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/50': No such file or directory 2023-01-24T03:35:46.743 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.743 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/51 2023-01-24T03:35:46.743 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.744 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/51': No such file or directory 2023-01-24T03:35:46.744 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.745 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/52 2023-01-24T03:35:46.745 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.745 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/52': No such file or directory 2023-01-24T03:35:46.746 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.746 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/53 2023-01-24T03:35:46.746 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.747 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/53': No such file or directory 2023-01-24T03:35:46.747 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.747 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/54 2023-01-24T03:35:46.748 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.748 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/54': No such file or directory 2023-01-24T03:35:46.749 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.749 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/55 2023-01-24T03:35:46.749 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.749 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/55': No such file or directory 2023-01-24T03:35:46.750 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.750 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/56 2023-01-24T03:35:46.750 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.751 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/56': No such file or directory 2023-01-24T03:35:46.751 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.752 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/57 2023-01-24T03:35:46.752 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.752 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/57': No such file or directory 2023-01-24T03:35:46.753 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.753 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/58 2023-01-24T03:35:46.753 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.754 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/58': No such file or directory 2023-01-24T03:35:46.754 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.754 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/59 2023-01-24T03:35:46.755 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.755 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/59': No such file or directory 2023-01-24T03:35:46.756 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.756 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/60 2023-01-24T03:35:46.756 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.756 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/60': No such file or directory 2023-01-24T03:35:46.757 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.757 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/61 2023-01-24T03:35:46.758 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.758 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/61': No such file or directory 2023-01-24T03:35:46.759 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.759 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/62 2023-01-24T03:35:46.759 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.759 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/62': No such file or directory 2023-01-24T03:35:46.760 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.760 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/63 2023-01-24T03:35:46.760 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.761 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/63': No such file or directory 2023-01-24T03:35:46.761 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.762 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/64 2023-01-24T03:35:46.762 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.762 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/64': No such file or directory 2023-01-24T03:35:46.763 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.763 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/65 2023-01-24T03:35:46.763 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.764 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/65': No such file or directory 2023-01-24T03:35:46.765 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.765 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/66 2023-01-24T03:35:46.765 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.765 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/66': No such file or directory 2023-01-24T03:35:46.766 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.766 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/67 2023-01-24T03:35:46.766 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.767 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/67': No such file or directory 2023-01-24T03:35:46.767 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.768 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/68 2023-01-24T03:35:46.768 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.768 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/68': No such file or directory 2023-01-24T03:35:46.769 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.769 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/69 2023-01-24T03:35:46.769 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.770 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/69': No such file or directory 2023-01-24T03:35:46.770 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.770 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/70 2023-01-24T03:35:46.771 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.771 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/70': No such file or directory 2023-01-24T03:35:46.772 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.772 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/71 2023-01-24T03:35:46.772 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.772 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/71': No such file or directory 2023-01-24T03:35:46.773 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.773 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/72 2023-01-24T03:35:46.773 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.774 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/72': No such file or directory 2023-01-24T03:35:46.774 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.775 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/73 2023-01-24T03:35:46.775 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.775 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/73': No such file or directory 2023-01-24T03:35:46.776 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.776 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/74 2023-01-24T03:35:46.776 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.777 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/74': No such file or directory 2023-01-24T03:35:46.777 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.777 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/75 2023-01-24T03:35:46.778 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.778 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/75': No such file or directory 2023-01-24T03:35:46.779 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.779 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/76 2023-01-24T03:35:46.779 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.779 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/76': No such file or directory 2023-01-24T03:35:46.780 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.780 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/77 2023-01-24T03:35:46.781 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.781 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/77': No such file or directory 2023-01-24T03:35:46.782 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.782 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/78 2023-01-24T03:35:46.782 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.782 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/78': No such file or directory 2023-01-24T03:35:46.783 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.783 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/79 2023-01-24T03:35:46.784 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.784 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/79': No such file or directory 2023-01-24T03:35:46.785 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.785 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/80 2023-01-24T03:35:46.785 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.785 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/80': No such file or directory 2023-01-24T03:35:46.786 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.786 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/81 2023-01-24T03:35:46.786 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.787 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/81': No such file or directory 2023-01-24T03:35:46.788 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.788 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/82 2023-01-24T03:35:46.788 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.788 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/82': No such file or directory 2023-01-24T03:35:46.789 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.789 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/83 2023-01-24T03:35:46.790 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.790 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/83': No such file or directory 2023-01-24T03:35:46.791 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.791 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/84 2023-01-24T03:35:46.791 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.791 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/84': No such file or directory 2023-01-24T03:35:46.792 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.792 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/85 2023-01-24T03:35:46.792 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.793 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/85': No such file or directory 2023-01-24T03:35:46.793 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.794 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/86 2023-01-24T03:35:46.794 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.794 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/86': No such file or directory 2023-01-24T03:35:46.795 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.795 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/87 2023-01-24T03:35:46.795 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.796 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/87': No such file or directory 2023-01-24T03:35:46.797 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.797 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/88 2023-01-24T03:35:46.797 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.797 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/88': No such file or directory 2023-01-24T03:35:46.798 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.798 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/89 2023-01-24T03:35:46.798 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.799 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/89': No such file or directory 2023-01-24T03:35:46.799 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.800 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/90 2023-01-24T03:35:46.800 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.800 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/90': No such file or directory 2023-01-24T03:35:46.801 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.801 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/91 2023-01-24T03:35:46.801 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.802 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/91': No such file or directory 2023-01-24T03:35:46.802 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.802 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/92 2023-01-24T03:35:46.802 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.803 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/92': No such file or directory 2023-01-24T03:35:46.803 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.804 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/93 2023-01-24T03:35:46.804 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.804 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/93': No such file or directory 2023-01-24T03:35:46.805 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.805 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/94 2023-01-24T03:35:46.805 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.806 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/94': No such file or directory 2023-01-24T03:35:46.806 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.806 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/95 2023-01-24T03:35:46.807 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.807 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/95': No such file or directory 2023-01-24T03:35:46.807 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.808 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/96 2023-01-24T03:35:46.808 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.808 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/96': No such file or directory 2023-01-24T03:35:46.809 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.809 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/97 2023-01-24T03:35:46.809 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.810 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/97': No such file or directory 2023-01-24T03:35:46.810 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.810 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/98 2023-01-24T03:35:46.811 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.811 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/98': No such file or directory 2023-01-24T03:35:46.811 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.812 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/99 2023-01-24T03:35:46.812 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.812 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/99': No such file or directory 2023-01-24T03:35:46.813 INFO:tasks.workunit.client.0.smithi112.stderr:+ for f in `seq 1 100` 2023-01-24T03:35:46.813 INFO:tasks.workunit.client.0.smithi112.stderr:+ stat .snap/foo/100 2023-01-24T03:35:46.813 INFO:tasks.workunit.client.0.smithi112.stderr:+ grep 'No such file' 2023-01-24T03:35:46.814 INFO:tasks.workunit.client.0.smithi112.stdout:stat: cannot stat '.snap/foo/100': No such file or directory 2023-01-24T03:35:46.814 INFO:tasks.workunit.client.0.smithi112.stderr:+ rmdir .snap/foo 2023-01-24T03:35:46.837 INFO:tasks.workunit.client.0.smithi112.stdout:OK 2023-01-24T03:35:46.838 INFO:tasks.workunit.client.0.smithi112.stderr:+ echo OK 2023-01-24T03:35:46.838 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-01-24T03:35:46.839 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-01-24T03:35:46.906 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-git-ceph.sh... 2023-01-24T03:35:46.906 DEBUG:teuthology.orchestra.run.smithi112:workunit test fs/snaps/snaptest-git-ceph.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8387d427d91a301da19bd82e090b96e1348bfadb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh 2023-01-24T03:35:46.978 INFO:tasks.workunit.client.0.smithi112.stderr:+ set -e 2023-01-24T03:35:46.978 INFO:tasks.workunit.client.0.smithi112.stderr:+ retried=false 2023-01-24T03:35:46.978 INFO:tasks.workunit.client.0.smithi112.stderr:+ trap -- retry EXIT 2023-01-24T03:35:46.978 INFO:tasks.workunit.client.0.smithi112.stderr:+ rm -rf ceph 2023-01-24T03:35:46.979 INFO:tasks.workunit.client.0.smithi112.stderr:+ timeout 1800 git clone https://git.ceph.com/ceph.git 2023-01-24T03:35:46.981 INFO:tasks.workunit.client.0.smithi112.stderr:Cloning into 'ceph'... 2023-01-24T03:35:59.565 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:35:59.569 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:36:06.197 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:36:06.312 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.310+0000 7fab57560700 1 Processor -- start 2023-01-24T03:36:06.312 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.310+0000 7fab57560700 1 -- start start 2023-01-24T03:36:06.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.311+0000 7fab57560700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fab5014c290 0x7fab5006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:06.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.311+0000 7fab57560700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab500c18f0 con 0x7fab500c2490 2023-01-24T03:36:06.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.311+0000 7fab57560700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab500c1030 con 0x7fab5014c660 2023-01-24T03:36:06.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.311+0000 7fab57560700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab5006bad0 con 0x7fab5014c290 2023-01-24T03:36:06.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.311+0000 7fab55afd700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fab500c2490 0x7fab50155710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:34844/0 (socket says 172.21.15.112:34844) 2023-01-24T03:36:06.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.311+0000 7fab55afd700 1 -- 172.21.15.112:0/2883015041 learned_addr learned my addr 172.21.15.112:0/2883015041 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:36:06.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.311+0000 7fab552fc700 1 --2- 172.21.15.112:0/2883015041 >> v2:172.21.15.142:3300/0 conn(0x7fab5014c290 0x7fab5006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:06.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.311+0000 7fab3ffff700 1 -- 172.21.15.112:0/2883015041 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1740944360 0 0) 0x7fab500c18f0 con 0x7fab500c2490 2023-01-24T03:36:06.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.311+0000 7fab3ffff700 1 -- 172.21.15.112:0/2883015041 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab380033f0 con 0x7fab500c2490 2023-01-24T03:36:06.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.312+0000 7fab3ffff700 1 -- 172.21.15.112:0/2883015041 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 993223754 0 0) 0x7fab380033f0 con 0x7fab500c2490 2023-01-24T03:36:06.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.312+0000 7fab552fc700 1 -- 172.21.15.112:0/2883015041 >> v1:172.21.15.142:6789/0 conn(0x7fab5014c660 legacy=0x7fab500c1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:06.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.312+0000 7fab552fc700 1 -- 172.21.15.112:0/2883015041 >> v1:172.21.15.112:6789/0 conn(0x7fab500c2490 legacy=0x7fab50155710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:06.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.312+0000 7fab552fc700 1 -- 172.21.15.112:0/2883015041 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab5006bdb0 con 0x7fab5014c290 2023-01-24T03:36:06.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.312+0000 7fab552fc700 1 --2- 172.21.15.112:0/2883015041 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab5014c290 0x7fab5006ec80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fab40009760 tx=0x7fab40009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19c147a15986b541 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:06.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.312+0000 7fab3ffff700 1 -- 172.21.15.112:0/2883015041 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab480017f0 con 0x7fab5014c290 2023-01-24T03:36:06.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.312+0000 7fab3ffff700 1 -- 172.21.15.112:0/2883015041 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab40011440 con 0x7fab5014c290 2023-01-24T03:36:06.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.312+0000 7fab3ffff700 1 -- 172.21.15.112:0/2883015041 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab400117e0 con 0x7fab5014c290 2023-01-24T03:36:06.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.312+0000 7fab57560700 1 -- 172.21.15.112:0/2883015041 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab5014c290 msgr2=0x7fab5006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:06.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.312+0000 7fab57560700 1 --2- 172.21.15.112:0/2883015041 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab5014c290 0x7fab5006ec80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fab40009760 tx=0x7fab40009060 comp rx=0 tx=0).stop 2023-01-24T03:36:06.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.312+0000 7fab57560700 1 -- 172.21.15.112:0/2883015041 shutdown_connections 2023-01-24T03:36:06.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.312+0000 7fab57560700 1 --2- 172.21.15.112:0/2883015041 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab5014c290 0x7fab5006ec80 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:06.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.312+0000 7fab57560700 1 -- 172.21.15.112:0/2883015041 >> 172.21.15.112:0/2883015041 conn(0x7fab50067f20 msgr2=0x7fab50068320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:06.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.313+0000 7fab57560700 1 -- 172.21.15.112:0/2883015041 shutdown_connections 2023-01-24T03:36:06.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.313+0000 7fab57560700 1 -- 172.21.15.112:0/2883015041 wait complete. 2023-01-24T03:36:06.324 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.314+0000 7fab57560700 1 Processor -- start 2023-01-24T03:36:06.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.314+0000 7fab57560700 1 -- start start 2023-01-24T03:36:06.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.314+0000 7fab57560700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab500c2490 0x7fab500c50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:06.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.314+0000 7fab57560700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab5014c290 0x7fab50166220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:06.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.314+0000 7fab57560700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab5014c660 0x7fab50157c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:06.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.314+0000 7fab57560700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab50140e30 con 0x7fab5014c290 2023-01-24T03:36:06.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.314+0000 7fab57560700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab500bcab0 con 0x7fab500c2490 2023-01-24T03:36:06.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.314+0000 7fab57560700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab500c4f20 con 0x7fab5014c660 2023-01-24T03:36:06.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.315+0000 7fab54afb700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab5014c290 0x7fab50166220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:06.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.315+0000 7fab54afb700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab5014c290 0x7fab50166220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:53862/0 (socket says 172.21.15.112:53862) 2023-01-24T03:36:06.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.315+0000 7fab54afb700 1 -- 172.21.15.112:0/1318237352 learned_addr learned my addr 172.21.15.112:0/1318237352 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:36:06.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.315+0000 7fab552fc700 1 --2- 172.21.15.112:0/1318237352 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab500c2490 0x7fab500c50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:06.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.315+0000 7fab55afd700 1 --2- 172.21.15.112:0/1318237352 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab5014c660 0x7fab50157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:06.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.315+0000 7fab54afb700 1 -- 172.21.15.112:0/1318237352 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab5014c660 msgr2=0x7fab50157c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:06.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.315+0000 7fab54afb700 1 --2- 172.21.15.112:0/1318237352 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab5014c660 0x7fab50157c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:06.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.315+0000 7fab54afb700 1 -- 172.21.15.112:0/1318237352 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab500c2490 msgr2=0x7fab500c50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:06.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.315+0000 7fab54afb700 1 --2- 172.21.15.112:0/1318237352 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab500c2490 0x7fab500c50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:06.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.315+0000 7fab54afb700 1 -- 172.21.15.112:0/1318237352 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab400092a0 con 0x7fab5014c290 2023-01-24T03:36:06.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.315+0000 7fab54afb700 1 --2- 172.21.15.112:0/1318237352 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab5014c290 0x7fab50166220 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fab4c00c540 tx=0x7fab4c007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:06.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.315+0000 7fab3d7fa700 1 -- 172.21.15.112:0/1318237352 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab4c00ce50 con 0x7fab5014c290 2023-01-24T03:36:06.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.315+0000 7fab3d7fa700 1 -- 172.21.15.112:0/1318237352 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab4c011590 con 0x7fab5014c290 2023-01-24T03:36:06.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.315+0000 7fab57560700 1 -- 172.21.15.112:0/1318237352 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab501623d0 con 0x7fab5014c290 2023-01-24T03:36:06.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.315+0000 7fab3d7fa700 1 -- 172.21.15.112:0/1318237352 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab4c011950 con 0x7fab5014c290 2023-01-24T03:36:06.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.315+0000 7fab57560700 1 -- 172.21.15.112:0/1318237352 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab50162950 con 0x7fab5014c290 2023-01-24T03:36:06.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.317+0000 7fab57560700 1 -- 172.21.15.112:0/1318237352 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab5004bb80 con 0x7fab5014c290 2023-01-24T03:36:06.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.317+0000 7fab3d7fa700 1 -- 172.21.15.112:0/1318237352 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fab4c019030 con 0x7fab5014c290 2023-01-24T03:36:06.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.317+0000 7fab3d7fa700 1 --2- 172.21.15.112:0/1318237352 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fab300611c0 0x7fab30063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:06.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.317+0000 7fab3d7fa700 1 -- 172.21.15.112:0/1318237352 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 7181+0+0 (secure 0 0 0) 0x7fab4c07cb50 con 0x7fab5014c290 2023-01-24T03:36:06.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.317+0000 7fab552fc700 1 --2- 172.21.15.112:0/1318237352 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fab300611c0 0x7fab30063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:06.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.317+0000 7fab552fc700 1 --2- 172.21.15.112:0/1318237352 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fab300611c0 0x7fab30063670 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fab40003080 tx=0x7fab40014000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:06.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.322+0000 7fab3d7fa700 1 -- 172.21.15.112:0/1318237352 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fab4c04c820 con 0x7fab5014c290 2023-01-24T03:36:06.573 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.571+0000 7fab57560700 1 -- 172.21.15.112:0/1318237352 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab500c12a0 con 0x7fab5014c290 2023-01-24T03:36:06.573 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.572+0000 7fab3d7fa700 1 -- 172.21.15.112:0/1318237352 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+6115 (secure 0 0 0) 0x7fab4c017090 con 0x7fab5014c290 2023-01-24T03:36:06.574 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:36:06.574 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4907,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/824478388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":824478388},{"type":"v1","addr":"172.21.15.142:6835","nonce":824478388}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:34:46.842740+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":22,"addr":"172.21.15.142:6834/612058861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":612058861},{"type":"v1","addr":"172.21.15.142:6834","nonce":612058861}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:36:06.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.573+0000 7fab57560700 1 -- 172.21.15.112:0/1318237352 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fab300611c0 msgr2=0x7fab30063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:06.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.573+0000 7fab57560700 1 --2- 172.21.15.112:0/1318237352 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fab300611c0 0x7fab30063670 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fab40003080 tx=0x7fab40014000 comp rx=0 tx=0).stop 2023-01-24T03:36:06.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.573+0000 7fab57560700 1 -- 172.21.15.112:0/1318237352 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab5014c290 msgr2=0x7fab50166220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:06.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.573+0000 7fab57560700 1 --2- 172.21.15.112:0/1318237352 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab5014c290 0x7fab50166220 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fab4c00c540 tx=0x7fab4c007e00 comp rx=0 tx=0).stop 2023-01-24T03:36:06.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.573+0000 7fab57560700 1 -- 172.21.15.112:0/1318237352 shutdown_connections 2023-01-24T03:36:06.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.574+0000 7fab57560700 1 --2- 172.21.15.112:0/1318237352 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fab300611c0 0x7fab30063670 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:06.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.574+0000 7fab57560700 1 --2- 172.21.15.112:0/1318237352 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fab5014c660 0x7fab50157c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:06.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.574+0000 7fab57560700 1 --2- 172.21.15.112:0/1318237352 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab500c2490 0x7fab500c50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:06.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.574+0000 7fab57560700 1 --2- 172.21.15.112:0/1318237352 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab5014c290 0x7fab50166220 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:06.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.574+0000 7fab57560700 1 -- 172.21.15.112:0/1318237352 >> 172.21.15.112:0/1318237352 conn(0x7fab50067f20 msgr2=0x7fab500682f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:06.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.574+0000 7fab57560700 1 -- 172.21.15.112:0/1318237352 shutdown_connections 2023-01-24T03:36:06.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.574+0000 7fab57560700 1 -- 172.21.15.112:0/1318237352 wait complete. 2023-01-24T03:36:06.578 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 26 2023-01-24T03:36:06.587 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2023-01-24T03:36:06.587 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-01-24T03:36:06.588 INFO:teuthology.orchestra.run:waiting for 300 2023-01-24T03:36:06.600 INFO:tasks.ceph.mds.b:Stopped 2023-01-24T03:36:06.601 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:36:06.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.722+0000 7ff893c92700 1 Processor -- start 2023-01-24T03:36:06.725 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.722+0000 7ff893c92700 1 -- start start 2023-01-24T03:36:06.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.723+0000 7ff893c92700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7ff88c06cf50 0x7ff88c155db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:06.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.723+0000 7ff893c92700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7ff88c06d320 0x7ff88c0c5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:06.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.723+0000 7ff893c92700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff88c0c54d0 con 0x7ff88c067f20 2023-01-24T03:36:06.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.723+0000 7ff893c92700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff88c0c49c0 con 0x7ff88c06cf50 2023-01-24T03:36:06.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.723+0000 7ff893c92700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff88c1562f0 con 0x7ff88c06d320 2023-01-24T03:36:06.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.723+0000 7ff891a2e700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7ff88c06cf50 0x7ff88c155db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:06.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.723+0000 7ff891a2e700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7ff88c06cf50 0x7ff88c155db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:53866/0 (socket says 172.21.15.112:53866) 2023-01-24T03:36:06.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.723+0000 7ff891a2e700 1 -- 172.21.15.112:0/1228534611 learned_addr learned my addr 172.21.15.112:0/1228534611 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:36:06.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.723+0000 7ff89122d700 1 --2- 172.21.15.112:0/1228534611 >> v2:172.21.15.142:3300/0 conn(0x7ff88c06d320 0x7ff88c0c5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:06.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.724+0000 7ff891a2e700 1 -- 172.21.15.112:0/1228534611 >> v2:172.21.15.142:3300/0 conn(0x7ff88c06d320 msgr2=0x7ff88c0c5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:36:06.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.724+0000 7ff891a2e700 1 --2- 172.21.15.112:0/1228534611 >> v2:172.21.15.142:3300/0 conn(0x7ff88c06d320 0x7ff88c0c5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:06.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.724+0000 7ff891a2e700 1 -- 172.21.15.112:0/1228534611 >> v1:172.21.15.142:6789/0 conn(0x7ff88c067f20 legacy=0x7ff88c14efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:36:06.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.724+0000 7ff891a2e700 1 -- 172.21.15.112:0/1228534611 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff88c068920 con 0x7ff88c06cf50 2023-01-24T03:36:06.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.724+0000 7ff891a2e700 1 --2- 172.21.15.112:0/1228534611 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff88c06cf50 0x7ff88c155db0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ff88c0c43b0 tx=0x7ff8880090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4c3ae6e02d6101e server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:06.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.724+0000 7ff890a2c700 1 -- 172.21.15.112:0/1228534611 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff888011630 con 0x7ff88c06cf50 2023-01-24T03:36:06.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.724+0000 7ff890a2c700 1 -- 172.21.15.112:0/1228534611 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8880117d0 con 0x7ff88c06cf50 2023-01-24T03:36:06.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.724+0000 7ff890a2c700 1 -- 172.21.15.112:0/1228534611 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff888011b90 con 0x7ff88c06cf50 2023-01-24T03:36:06.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.724+0000 7ff893c92700 1 -- 172.21.15.112:0/1228534611 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff88c06cf50 msgr2=0x7ff88c155db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:06.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.724+0000 7ff893c92700 1 --2- 172.21.15.112:0/1228534611 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff88c06cf50 0x7ff88c155db0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ff88c0c43b0 tx=0x7ff8880090f0 comp rx=0 tx=0).stop 2023-01-24T03:36:06.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.724+0000 7ff893c92700 1 -- 172.21.15.112:0/1228534611 shutdown_connections 2023-01-24T03:36:06.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.724+0000 7ff893c92700 1 --2- 172.21.15.112:0/1228534611 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff88c06cf50 0x7ff88c155db0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:06.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.724+0000 7ff893c92700 1 --2- 172.21.15.112:0/1228534611 >> v2:172.21.15.142:3300/0 conn(0x7ff88c06d320 0x7ff88c0c5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:06.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.724+0000 7ff893c92700 1 -- 172.21.15.112:0/1228534611 >> 172.21.15.112:0/1228534611 conn(0x7ff88c0c9820 msgr2=0x7ff88c0c9c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:06.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.725+0000 7ff893c92700 1 -- 172.21.15.112:0/1228534611 shutdown_connections 2023-01-24T03:36:06.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.725+0000 7ff893c92700 1 -- 172.21.15.112:0/1228534611 wait complete. 2023-01-24T03:36:06.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.725+0000 7ff893c92700 1 Processor -- start 2023-01-24T03:36:06.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.726+0000 7ff893c92700 1 -- start start 2023-01-24T03:36:06.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.726+0000 7ff893c92700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff88c067f20 0x7ff88c15afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:06.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.726+0000 7ff893c92700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff88c06cf50 0x7ff88c0c8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:06.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.726+0000 7ff893c92700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff88c06d320 0x7ff88c168970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:06.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.726+0000 7ff893c92700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c0c1f80 con 0x7ff88c067f20 2023-01-24T03:36:06.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.726+0000 7ff893c92700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c154ba0 con 0x7ff88c06d320 2023-01-24T03:36:06.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.726+0000 7ff893c92700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c0c3160 con 0x7ff88c06cf50 2023-01-24T03:36:06.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.726+0000 7ff891a2e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff88c067f20 0x7ff88c15afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:06.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.726+0000 7ff891a2e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff88c067f20 0x7ff88c15afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:53872/0 (socket says 172.21.15.112:53872) 2023-01-24T03:36:06.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.726+0000 7ff891a2e700 1 -- 172.21.15.112:0/1740024069 learned_addr learned my addr 172.21.15.112:0/1740024069 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:36:06.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.726+0000 7ff891a2e700 1 -- 172.21.15.112:0/1740024069 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff88c06cf50 msgr2=0x7ff88c0c8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:06.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.726+0000 7ff89222f700 1 --2- 172.21.15.112:0/1740024069 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff88c06d320 0x7ff88c168970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:06.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.726+0000 7ff89122d700 1 --2- 172.21.15.112:0/1740024069 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff88c06cf50 0x7ff88c0c8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:06.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.726+0000 7ff891a2e700 1 --2- 172.21.15.112:0/1740024069 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff88c06cf50 0x7ff88c0c8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:06.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.727+0000 7ff891a2e700 1 -- 172.21.15.112:0/1740024069 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff88c06d320 msgr2=0x7ff88c168970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:06.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.727+0000 7ff891a2e700 1 --2- 172.21.15.112:0/1740024069 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff88c06d320 0x7ff88c168970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:06.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.727+0000 7ff891a2e700 1 -- 172.21.15.112:0/1740024069 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff888009370 con 0x7ff88c067f20 2023-01-24T03:36:06.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.727+0000 7ff89222f700 1 --2- 172.21.15.112:0/1740024069 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff88c06d320 0x7ff88c168970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-01-24T03:36:06.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.727+0000 7ff891a2e700 1 --2- 172.21.15.112:0/1740024069 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff88c067f20 0x7ff88c15afd0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7ff88c06e780 tx=0x7ff888016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:06.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.727+0000 7ff879ffb700 1 -- 172.21.15.112:0/1740024069 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff888011b50 con 0x7ff88c067f20 2023-01-24T03:36:06.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.727+0000 7ff879ffb700 1 -- 172.21.15.112:0/1740024069 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff888020070 con 0x7ff88c067f20 2023-01-24T03:36:06.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.727+0000 7ff893c92700 1 -- 172.21.15.112:0/1740024069 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff88c154df0 con 0x7ff88c067f20 2023-01-24T03:36:06.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.727+0000 7ff879ffb700 1 -- 172.21.15.112:0/1740024069 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8880276f0 con 0x7ff88c067f20 2023-01-24T03:36:06.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.727+0000 7ff893c92700 1 -- 172.21.15.112:0/1740024069 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff88c0c2b10 con 0x7ff88c067f20 2023-01-24T03:36:06.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.728+0000 7ff893c92700 1 -- 172.21.15.112:0/1740024069 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff88c0c34d0 con 0x7ff88c067f20 2023-01-24T03:36:06.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.728+0000 7ff879ffb700 1 -- 172.21.15.112:0/1740024069 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7ff888017020 con 0x7ff88c067f20 2023-01-24T03:36:06.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.729+0000 7ff879ffb700 1 --2- 172.21.15.112:0/1740024069 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff86c0611c0 0x7ff86c063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:06.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.729+0000 7ff879ffb700 1 -- 172.21.15.112:0/1740024069 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 7181+0+0 (secure 0 0 0) 0x7ff888022070 con 0x7ff88c067f20 2023-01-24T03:36:06.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.729+0000 7ff89122d700 1 --2- 172.21.15.112:0/1740024069 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff86c0611c0 0x7ff86c063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:06.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.729+0000 7ff89122d700 1 --2- 172.21.15.112:0/1740024069 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff86c0611c0 0x7ff86c063670 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff87c006fd0 tx=0x7ff87c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:06.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.733+0000 7ff879ffb700 1 -- 172.21.15.112:0/1740024069 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7ff88804f500 con 0x7ff88c067f20 2023-01-24T03:36:06.974 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.973+0000 7ff893c92700 1 -- 172.21.15.112:0/1740024069 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff88c04bb80 con 0x7ff88c067f20 2023-01-24T03:36:06.975 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.973+0000 7ff879ffb700 1 -- 172.21.15.112:0/1740024069 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+6115 (secure 0 0 0) 0x7ff888052b20 con 0x7ff88c067f20 2023-01-24T03:36:06.976 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:36:06.976 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4907,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/824478388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":824478388},{"type":"v1","addr":"172.21.15.142:6835","nonce":824478388}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:34:46.842740+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":22,"addr":"172.21.15.142:6834/612058861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":612058861},{"type":"v1","addr":"172.21.15.142:6834","nonce":612058861}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:36:06.977 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.975+0000 7ff893c92700 1 -- 172.21.15.112:0/1740024069 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff86c0611c0 msgr2=0x7ff86c063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:06.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.975+0000 7ff893c92700 1 --2- 172.21.15.112:0/1740024069 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff86c0611c0 0x7ff86c063670 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff87c006fd0 tx=0x7ff87c008040 comp rx=0 tx=0).stop 2023-01-24T03:36:06.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.975+0000 7ff893c92700 1 -- 172.21.15.112:0/1740024069 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff88c067f20 msgr2=0x7ff88c15afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:06.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.975+0000 7ff893c92700 1 --2- 172.21.15.112:0/1740024069 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff88c067f20 0x7ff88c15afd0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7ff88c06e780 tx=0x7ff888016000 comp rx=0 tx=0).stop 2023-01-24T03:36:06.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.975+0000 7ff893c92700 1 -- 172.21.15.112:0/1740024069 shutdown_connections 2023-01-24T03:36:06.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.975+0000 7ff893c92700 1 --2- 172.21.15.112:0/1740024069 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff86c0611c0 0x7ff86c063670 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:06.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.975+0000 7ff893c92700 1 --2- 172.21.15.112:0/1740024069 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff88c067f20 0x7ff88c15afd0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:06.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.975+0000 7ff893c92700 1 --2- 172.21.15.112:0/1740024069 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff88c06d320 0x7ff88c168970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:06.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.975+0000 7ff893c92700 1 --2- 172.21.15.112:0/1740024069 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff88c06cf50 0x7ff88c0c8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:06.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.975+0000 7ff893c92700 1 -- 172.21.15.112:0/1740024069 >> 172.21.15.112:0/1740024069 conn(0x7ff88c0c9820 msgr2=0x7ff88c155810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:06.980 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.975+0000 7ff893c92700 1 -- 172.21.15.112:0/1740024069 shutdown_connections 2023-01-24T03:36:06.980 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:06.975+0000 7ff893c92700 1 -- 172.21.15.112:0/1740024069 wait complete. 2023-01-24T03:36:06.980 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 26 2023-01-24T03:36:06.989 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-01-24T03:36:08.990 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:36:09.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.109+0000 7f5f4823c700 1 Processor -- start 2023-01-24T03:36:09.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.109+0000 7f5f4823c700 1 -- start start 2023-01-24T03:36:09.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.110+0000 7f5f4823c700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f5f400c7e40 0x7f5f400b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:09.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.110+0000 7f5f4823c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f5f40155da0 0x7f5f400ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:09.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.110+0000 7f5f4823c700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f40156240 con 0x7f5f400aef20 2023-01-24T03:36:09.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.110+0000 7f5f4823c700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f400af320 con 0x7f5f400c7e40 2023-01-24T03:36:09.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.110+0000 7f5f4823c700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f400b1170 con 0x7f5f40155da0 2023-01-24T03:36:09.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.110+0000 7f5f45fd8700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f5f400c7e40 0x7f5f400b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:09.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.110+0000 7f5f45fd8700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f5f400c7e40 0x7f5f400b0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:53890/0 (socket says 172.21.15.112:53890) 2023-01-24T03:36:09.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.110+0000 7f5f45fd8700 1 -- 172.21.15.112:0/3457916561 learned_addr learned my addr 172.21.15.112:0/3457916561 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:36:09.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.110+0000 7f5f457d7700 1 --2- 172.21.15.112:0/3457916561 >> v2:172.21.15.142:3300/0 conn(0x7f5f40155da0 0x7f5f400ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:09.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.110+0000 7f5f45fd8700 1 -- 172.21.15.112:0/3457916561 >> v2:172.21.15.142:3300/0 conn(0x7f5f40155da0 msgr2=0x7f5f400ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:36:09.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.110+0000 7f5f45fd8700 1 --2- 172.21.15.112:0/3457916561 >> v2:172.21.15.142:3300/0 conn(0x7f5f40155da0 0x7f5f400ae9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:09.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.110+0000 7f5f45fd8700 1 -- 172.21.15.112:0/3457916561 >> v1:172.21.15.112:6789/0 conn(0x7f5f400aef20 legacy=0x7f5f400ca3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:09.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.110+0000 7f5f45fd8700 1 -- 172.21.15.112:0/3457916561 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f400b1450 con 0x7f5f400c7e40 2023-01-24T03:36:09.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.110+0000 7f5f45fd8700 1 --2- 172.21.15.112:0/3457916561 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5f400c7e40 0x7f5f400b0aa0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f5f40152ea0 tx=0x7f5f3c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f88205ee5ea2937c server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:09.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.111+0000 7f5f44fd6700 1 -- 172.21.15.112:0/3457916561 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f3c011630 con 0x7f5f400c7e40 2023-01-24T03:36:09.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.111+0000 7f5f44fd6700 1 -- 172.21.15.112:0/3457916561 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f3c011790 con 0x7f5f400c7e40 2023-01-24T03:36:09.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.111+0000 7f5f44fd6700 1 -- 172.21.15.112:0/3457916561 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f3c011b50 con 0x7f5f400c7e40 2023-01-24T03:36:09.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.111+0000 7f5f4823c700 1 -- 172.21.15.112:0/3457916561 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5f400c7e40 msgr2=0x7f5f400b0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:09.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.111+0000 7f5f4823c700 1 --2- 172.21.15.112:0/3457916561 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5f400c7e40 0x7f5f400b0aa0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f5f40152ea0 tx=0x7f5f3c0090f0 comp rx=0 tx=0).stop 2023-01-24T03:36:09.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.111+0000 7f5f4823c700 1 -- 172.21.15.112:0/3457916561 shutdown_connections 2023-01-24T03:36:09.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.111+0000 7f5f4823c700 1 --2- 172.21.15.112:0/3457916561 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5f400c7e40 0x7f5f400b0aa0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:09.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.111+0000 7f5f4823c700 1 --2- 172.21.15.112:0/3457916561 >> v2:172.21.15.142:3300/0 conn(0x7f5f40155da0 0x7f5f400ae9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:09.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.111+0000 7f5f4823c700 1 -- 172.21.15.112:0/3457916561 >> 172.21.15.112:0/3457916561 conn(0x7f5f4005e2c0 msgr2=0x7f5f400cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:09.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.111+0000 7f5f4823c700 1 -- 172.21.15.112:0/3457916561 shutdown_connections 2023-01-24T03:36:09.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.111+0000 7f5f4823c700 1 -- 172.21.15.112:0/3457916561 wait complete. 2023-01-24T03:36:09.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.112+0000 7f5f4823c700 1 Processor -- start 2023-01-24T03:36:09.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.112+0000 7f5f4823c700 1 -- start start 2023-01-24T03:36:09.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.113+0000 7f5f4823c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5f400aef20 0x7f5f400cb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:09.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.113+0000 7f5f4823c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5f400c7e40 0x7f5f400c34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:09.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.113+0000 7f5f4823c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5f40155da0 0x7f5f400c39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:09.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.113+0000 7f5f4823c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f40151150 con 0x7f5f400aef20 2023-01-24T03:36:09.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.113+0000 7f5f4823c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f40157170 con 0x7f5f400c7e40 2023-01-24T03:36:09.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.113+0000 7f5f4823c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f40169690 con 0x7f5f40155da0 2023-01-24T03:36:09.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.113+0000 7f5f45fd8700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5f400aef20 0x7f5f400cb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:09.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.113+0000 7f5f45fd8700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5f400aef20 0x7f5f400cb4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:53896/0 (socket says 172.21.15.112:53896) 2023-01-24T03:36:09.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.113+0000 7f5f45fd8700 1 -- 172.21.15.112:0/3037079127 learned_addr learned my addr 172.21.15.112:0/3037079127 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:36:09.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.113+0000 7f5f457d7700 1 --2- 172.21.15.112:0/3037079127 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5f400c7e40 0x7f5f400c34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:09.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.113+0000 7f5f467d9700 1 --2- 172.21.15.112:0/3037079127 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5f40155da0 0x7f5f400c39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:09.124 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.113+0000 7f5f45fd8700 1 -- 172.21.15.112:0/3037079127 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5f40155da0 msgr2=0x7f5f400c39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:09.124 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.113+0000 7f5f45fd8700 1 --2- 172.21.15.112:0/3037079127 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5f40155da0 0x7f5f400c39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:09.124 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.113+0000 7f5f45fd8700 1 -- 172.21.15.112:0/3037079127 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5f400c7e40 msgr2=0x7f5f400c34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:09.124 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.113+0000 7f5f45fd8700 1 --2- 172.21.15.112:0/3037079127 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5f400c7e40 0x7f5f400c34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:09.124 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.113+0000 7f5f45fd8700 1 -- 172.21.15.112:0/3037079127 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f3c009370 con 0x7f5f400aef20 2023-01-24T03:36:09.125 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.113+0000 7f5f457d7700 1 --2- 172.21.15.112:0/3037079127 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5f400c7e40 0x7f5f400c34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:36:09.125 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.114+0000 7f5f45fd8700 1 --2- 172.21.15.112:0/3037079127 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5f400aef20 0x7f5f400cb4a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c002000 tx=0x7f5f3c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:09.125 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.114+0000 7f5f2dffb700 1 -- 172.21.15.112:0/3037079127 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f3c01bcd0 con 0x7f5f400aef20 2023-01-24T03:36:09.125 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.114+0000 7f5f2dffb700 1 -- 172.21.15.112:0/3037079127 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f3c020070 con 0x7f5f400aef20 2023-01-24T03:36:09.125 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.114+0000 7f5f4823c700 1 -- 172.21.15.112:0/3037079127 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f400c4040 con 0x7f5f400aef20 2023-01-24T03:36:09.126 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.114+0000 7f5f2dffb700 1 -- 172.21.15.112:0/3037079127 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f3c0276d0 con 0x7f5f400aef20 2023-01-24T03:36:09.126 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.114+0000 7f5f4823c700 1 -- 172.21.15.112:0/3037079127 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f40168d60 con 0x7f5f400aef20 2023-01-24T03:36:09.126 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.115+0000 7f5f4823c700 1 -- 172.21.15.112:0/3037079127 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f4004bb80 con 0x7f5f400aef20 2023-01-24T03:36:09.126 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.115+0000 7f5f2dffb700 1 -- 172.21.15.112:0/3037079127 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f5f3c017020 con 0x7f5f400aef20 2023-01-24T03:36:09.126 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.116+0000 7f5f2dffb700 1 --2- 172.21.15.112:0/3037079127 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5f200611f0 0x7f5f200636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:09.127 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.116+0000 7f5f2dffb700 1 -- 172.21.15.112:0/3037079127 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 7181+0+0 (secure 0 0 0) 0x7f5f3c022070 con 0x7f5f400aef20 2023-01-24T03:36:09.127 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.116+0000 7f5f457d7700 1 --2- 172.21.15.112:0/3037079127 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5f200611f0 0x7f5f200636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:09.127 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.116+0000 7f5f457d7700 1 --2- 172.21.15.112:0/3037079127 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5f200611f0 0x7f5f200636a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5f40068cc0 tx=0x7f5f30008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:09.128 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.120+0000 7f5f2dffb700 1 -- 172.21.15.112:0/3037079127 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f5f3c04ecf0 con 0x7f5f400aef20 2023-01-24T03:36:09.356 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.354+0000 7f5f4823c700 1 -- 172.21.15.112:0/3037079127 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f4006efe0 con 0x7f5f400aef20 2023-01-24T03:36:09.356 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.354+0000 7f5f2dffb700 1 -- 172.21.15.112:0/3037079127 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+6115 (secure 0 0 0) 0x7f5f3c052310 con 0x7f5f400aef20 2023-01-24T03:36:09.357 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:36:09.357 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4907,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/824478388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":824478388},{"type":"v1","addr":"172.21.15.142:6835","nonce":824478388}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:34:46.842740+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":22,"addr":"172.21.15.142:6834/612058861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":612058861},{"type":"v1","addr":"172.21.15.142:6834","nonce":612058861}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:36:09.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.356+0000 7f5f4823c700 1 -- 172.21.15.112:0/3037079127 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5f200611f0 msgr2=0x7f5f200636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:09.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.356+0000 7f5f4823c700 1 --2- 172.21.15.112:0/3037079127 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5f200611f0 0x7f5f200636a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5f40068cc0 tx=0x7f5f30008040 comp rx=0 tx=0).stop 2023-01-24T03:36:09.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.356+0000 7f5f4823c700 1 -- 172.21.15.112:0/3037079127 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5f400aef20 msgr2=0x7f5f400cb4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:09.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.356+0000 7f5f4823c700 1 --2- 172.21.15.112:0/3037079127 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5f400aef20 0x7f5f400cb4a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c002000 tx=0x7f5f3c016000 comp rx=0 tx=0).stop 2023-01-24T03:36:09.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.357+0000 7f5f4823c700 1 -- 172.21.15.112:0/3037079127 shutdown_connections 2023-01-24T03:36:09.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.357+0000 7f5f4823c700 1 --2- 172.21.15.112:0/3037079127 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5f200611f0 0x7f5f200636a0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:09.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.357+0000 7f5f4823c700 1 --2- 172.21.15.112:0/3037079127 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5f400aef20 0x7f5f400cb4a0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:09.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.357+0000 7f5f4823c700 1 --2- 172.21.15.112:0/3037079127 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5f40155da0 0x7f5f400c39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:09.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.357+0000 7f5f4823c700 1 --2- 172.21.15.112:0/3037079127 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5f400c7e40 0x7f5f400c34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:09.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.357+0000 7f5f4823c700 1 -- 172.21.15.112:0/3037079127 >> 172.21.15.112:0/3037079127 conn(0x7f5f4005e2c0 msgr2=0x7f5f401551d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:09.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.357+0000 7f5f4823c700 1 -- 172.21.15.112:0/3037079127 shutdown_connections 2023-01-24T03:36:09.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:09.357+0000 7f5f4823c700 1 -- 172.21.15.112:0/3037079127 wait complete. 2023-01-24T03:36:09.362 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 26 2023-01-24T03:36:09.370 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-01-24T03:36:11.371 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:36:11.496 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.493+0000 7f9a32a56700 1 Processor -- start 2023-01-24T03:36:11.497 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.493+0000 7f9a32a56700 1 -- start start 2023-01-24T03:36:11.499 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.494+0000 7f9a32a56700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9a2c0c7e40 0x7f9a2c0b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:11.499 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.494+0000 7f9a32a56700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f9a2c155da0 0x7f9a2c0ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:11.499 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.494+0000 7f9a32a56700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a2c156210 con 0x7f9a2c0aef20 2023-01-24T03:36:11.499 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.494+0000 7f9a32a56700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a2c0b1170 con 0x7f9a2c155da0 2023-01-24T03:36:11.499 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.494+0000 7f9a32a56700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2c0b1450 con 0x7f9a2c0c7e40 2023-01-24T03:36:11.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.494+0000 7f9a2b7fe700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f9a2c155da0 0x7f9a2c0ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:11.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.494+0000 7f9a2b7fe700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f9a2c155da0 0x7f9a2c0ae9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:53908/0 (socket says 172.21.15.112:53908) 2023-01-24T03:36:11.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.494+0000 7f9a2b7fe700 1 -- 172.21.15.112:0/2483339909 learned_addr learned my addr 172.21.15.112:0/2483339909 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:36:11.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.494+0000 7f9a30ff3700 1 --1- 172.21.15.112:0/2483339909 >> v1:172.21.15.112:6789/0 conn(0x7f9a2c0aef20 0x7f9a2c0ca3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:34896/0 (socket says 172.21.15.112:34896) 2023-01-24T03:36:11.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.494+0000 7f9a2bfff700 1 --2- 172.21.15.112:0/2483339909 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9a2c0c7e40 0x7f9a2c0b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:11.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.494+0000 7f9a2b7fe700 1 -- 172.21.15.112:0/2483339909 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9a2c0c7e40 msgr2=0x7f9a2c0b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:36:11.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.494+0000 7f9a2b7fe700 1 --2- 172.21.15.112:0/2483339909 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9a2c0c7e40 0x7f9a2c0b0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:11.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.494+0000 7f9a2b7fe700 1 -- 172.21.15.112:0/2483339909 >> v1:172.21.15.112:6789/0 conn(0x7f9a2c0aef20 legacy=0x7f9a2c0ca3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:11.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.494+0000 7f9a2b7fe700 1 -- 172.21.15.112:0/2483339909 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a2c0b1730 con 0x7f9a2c155da0 2023-01-24T03:36:11.502 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.495+0000 7f9a2b7fe700 1 --2- 172.21.15.112:0/2483339909 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9a2c155da0 0x7f9a2c0ae9e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f9a20006fd0 tx=0x7f9a200090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d378268d9cba5c server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:11.502 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.495+0000 7f9a2affd700 1 -- 172.21.15.112:0/2483339909 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a20011630 con 0x7f9a2c155da0 2023-01-24T03:36:11.502 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.495+0000 7f9a2affd700 1 -- 172.21.15.112:0/2483339909 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a200117d0 con 0x7f9a2c155da0 2023-01-24T03:36:11.502 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.495+0000 7f9a2affd700 1 -- 172.21.15.112:0/2483339909 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a20011b90 con 0x7f9a2c155da0 2023-01-24T03:36:11.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.495+0000 7f9a32a56700 1 -- 172.21.15.112:0/2483339909 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9a2c155da0 msgr2=0x7f9a2c0ae9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:11.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.495+0000 7f9a32a56700 1 --2- 172.21.15.112:0/2483339909 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9a2c155da0 0x7f9a2c0ae9e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f9a20006fd0 tx=0x7f9a200090f0 comp rx=0 tx=0).stop 2023-01-24T03:36:11.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.495+0000 7f9a32a56700 1 -- 172.21.15.112:0/2483339909 shutdown_connections 2023-01-24T03:36:11.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.495+0000 7f9a32a56700 1 --2- 172.21.15.112:0/2483339909 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9a2c0c7e40 0x7f9a2c0b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:11.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.495+0000 7f9a32a56700 1 --2- 172.21.15.112:0/2483339909 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9a2c155da0 0x7f9a2c0ae9e0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:11.504 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.495+0000 7f9a32a56700 1 -- 172.21.15.112:0/2483339909 >> 172.21.15.112:0/2483339909 conn(0x7f9a2c05e2c0 msgr2=0x7f9a2c0cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:11.504 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.495+0000 7f9a32a56700 1 -- 172.21.15.112:0/2483339909 shutdown_connections 2023-01-24T03:36:11.504 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.495+0000 7f9a32a56700 1 -- 172.21.15.112:0/2483339909 wait complete. 2023-01-24T03:36:11.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.496+0000 7f9a32a56700 1 Processor -- start 2023-01-24T03:36:11.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.497+0000 7f9a32a56700 1 -- start start 2023-01-24T03:36:11.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.497+0000 7f9a32a56700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9a2c0aef20 0x7f9a2c14c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:11.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.497+0000 7f9a32a56700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9a2c0c7e40 0x7f9a2c168a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:11.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.497+0000 7f9a32a56700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9a2c155da0 0x7f9a2c168fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:11.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.497+0000 7f9a32a56700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2c154560 con 0x7f9a2c155da0 2023-01-24T03:36:11.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.497+0000 7f9a32a56700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2c0c3880 con 0x7f9a2c0aef20 2023-01-24T03:36:11.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.497+0000 7f9a32a56700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2c0c2c70 con 0x7f9a2c0c7e40 2023-01-24T03:36:11.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.497+0000 7f9a30ff3700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9a2c155da0 0x7f9a2c168fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:11.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.497+0000 7f9a30ff3700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9a2c155da0 0x7f9a2c168fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:53914/0 (socket says 172.21.15.112:53914) 2023-01-24T03:36:11.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.497+0000 7f9a30ff3700 1 -- 172.21.15.112:0/3229120217 learned_addr learned my addr 172.21.15.112:0/3229120217 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:36:11.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.498+0000 7f9a2b7fe700 1 --2- 172.21.15.112:0/3229120217 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9a2c0c7e40 0x7f9a2c168a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:11.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.498+0000 7f9a2bfff700 1 --2- 172.21.15.112:0/3229120217 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9a2c0aef20 0x7f9a2c14c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:11.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.498+0000 7f9a30ff3700 1 -- 172.21.15.112:0/3229120217 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9a2c0c7e40 msgr2=0x7f9a2c168a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:11.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.498+0000 7f9a30ff3700 1 --2- 172.21.15.112:0/3229120217 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9a2c0c7e40 0x7f9a2c168a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:11.510 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.498+0000 7f9a30ff3700 1 -- 172.21.15.112:0/3229120217 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9a2c0aef20 msgr2=0x7f9a2c14c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:11.510 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.498+0000 7f9a30ff3700 1 --2- 172.21.15.112:0/3229120217 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9a2c0aef20 0x7f9a2c14c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:11.510 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.498+0000 7f9a30ff3700 1 -- 172.21.15.112:0/3229120217 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a20009370 con 0x7f9a2c155da0 2023-01-24T03:36:11.510 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.498+0000 7f9a2b7fe700 1 --2- 172.21.15.112:0/3229120217 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9a2c0c7e40 0x7f9a2c168a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:36:11.510 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.498+0000 7f9a2bfff700 1 --2- 172.21.15.112:0/3229120217 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9a2c0aef20 0x7f9a2c14c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:36:11.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.498+0000 7f9a30ff3700 1 --2- 172.21.15.112:0/3229120217 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9a2c155da0 0x7f9a2c168fd0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f9a2400c790 tx=0x7f9a240015e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:11.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.498+0000 7f9a13fff700 1 -- 172.21.15.112:0/3229120217 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a24005c70 con 0x7f9a2c155da0 2023-01-24T03:36:11.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.498+0000 7f9a13fff700 1 -- 172.21.15.112:0/3229120217 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a24005dd0 con 0x7f9a2c155da0 2023-01-24T03:36:11.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.498+0000 7f9a13fff700 1 -- 172.21.15.112:0/3229120217 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a24007320 con 0x7f9a2c155da0 2023-01-24T03:36:11.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.498+0000 7f9a32a56700 1 -- 172.21.15.112:0/3229120217 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a2c0c2f50 con 0x7f9a2c155da0 2023-01-24T03:36:11.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.498+0000 7f9a32a56700 1 -- 172.21.15.112:0/3229120217 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a2c0c3510 con 0x7f9a2c155da0 2023-01-24T03:36:11.512 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.499+0000 7f9a32a56700 1 -- 172.21.15.112:0/3229120217 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a2c1687b0 con 0x7f9a2c155da0 2023-01-24T03:36:11.512 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.499+0000 7f9a13fff700 1 -- 172.21.15.112:0/3229120217 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f9a24010070 con 0x7f9a2c155da0 2023-01-24T03:36:11.512 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.500+0000 7f9a13fff700 1 --2- 172.21.15.112:0/3229120217 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9a0c0611c0 0x7f9a0c063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:11.512 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.500+0000 7f9a13fff700 1 -- 172.21.15.112:0/3229120217 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 7181+0+0 (secure 0 0 0) 0x7f9a2407d440 con 0x7f9a2c155da0 2023-01-24T03:36:11.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.500+0000 7f9a2bfff700 1 --2- 172.21.15.112:0/3229120217 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9a0c0611c0 0x7f9a0c063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:11.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.500+0000 7f9a2bfff700 1 --2- 172.21.15.112:0/3229120217 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9a0c0611c0 0x7f9a0c063670 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c009c00 tx=0x7f9a1c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:11.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.504+0000 7f9a13fff700 1 -- 172.21.15.112:0/3229120217 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f9a2c1687b0 con 0x7f9a2c155da0 2023-01-24T03:36:11.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.726+0000 7f9a32a56700 1 -- 172.21.15.112:0/3229120217 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a2c1687b0 con 0x7f9a2c155da0 2023-01-24T03:36:11.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.726+0000 7f9a13fff700 1 -- 172.21.15.112:0/3229120217 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+6115 (secure 0 0 0) 0x7f9a2c1687b0 con 0x7f9a2c155da0 2023-01-24T03:36:11.730 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:36:11.731 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4907,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/824478388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":824478388},{"type":"v1","addr":"172.21.15.142:6835","nonce":824478388}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:34:46.842740+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":22,"addr":"172.21.15.142:6834/612058861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":612058861},{"type":"v1","addr":"172.21.15.142:6834","nonce":612058861}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:36:11.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.728+0000 7f9a32a56700 1 -- 172.21.15.112:0/3229120217 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9a0c0611c0 msgr2=0x7f9a0c063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:11.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.728+0000 7f9a32a56700 1 --2- 172.21.15.112:0/3229120217 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9a0c0611c0 0x7f9a0c063670 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c009c00 tx=0x7f9a1c008040 comp rx=0 tx=0).stop 2023-01-24T03:36:11.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.728+0000 7f9a32a56700 1 -- 172.21.15.112:0/3229120217 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9a2c155da0 msgr2=0x7f9a2c168fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:11.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.728+0000 7f9a32a56700 1 --2- 172.21.15.112:0/3229120217 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9a2c155da0 0x7f9a2c168fd0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f9a2400c790 tx=0x7f9a240015e0 comp rx=0 tx=0).stop 2023-01-24T03:36:11.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.728+0000 7f9a32a56700 1 -- 172.21.15.112:0/3229120217 shutdown_connections 2023-01-24T03:36:11.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.728+0000 7f9a32a56700 1 --2- 172.21.15.112:0/3229120217 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9a0c0611c0 0x7f9a0c063670 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:11.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.728+0000 7f9a32a56700 1 --2- 172.21.15.112:0/3229120217 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9a2c155da0 0x7f9a2c168fd0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:11.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.728+0000 7f9a32a56700 1 --2- 172.21.15.112:0/3229120217 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9a2c0aef20 0x7f9a2c14c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:11.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.728+0000 7f9a32a56700 1 --2- 172.21.15.112:0/3229120217 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9a2c0c7e40 0x7f9a2c168a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:11.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.728+0000 7f9a32a56700 1 -- 172.21.15.112:0/3229120217 >> 172.21.15.112:0/3229120217 conn(0x7f9a2c05e2c0 msgr2=0x7f9a2c159040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:11.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.728+0000 7f9a32a56700 1 -- 172.21.15.112:0/3229120217 shutdown_connections 2023-01-24T03:36:11.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:11.728+0000 7f9a32a56700 1 -- 172.21.15.112:0/3229120217 wait complete. 2023-01-24T03:36:11.735 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 26 2023-01-24T03:36:11.743 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-01-24T03:36:13.744 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:36:13.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.870+0000 7f8a51cf8700 1 Processor -- start 2023-01-24T03:36:13.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.870+0000 7f8a51cf8700 1 -- start start 2023-01-24T03:36:13.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.871+0000 7f8a51cf8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a4c0aeb90 0x7f8a4c0cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:13.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.871+0000 7f8a51cf8700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f8a4c0aef60 0x7f8a4c0b14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:13.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.871+0000 7f8a51cf8700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a4c1560e0 con 0x7f8a4c068400 2023-01-24T03:36:13.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.871+0000 7f8a51cf8700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a4c155530 con 0x7f8a4c0aef60 2023-01-24T03:36:13.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.871+0000 7f8a51cf8700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4c069ac0 con 0x7f8a4c0aeb90 2023-01-24T03:36:13.877 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.871+0000 7f8a43fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f8a4c0aef60 0x7f8a4c0b14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:13.877 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.871+0000 7f8a4b7fe700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a4c0aeb90 0x7f8a4c0cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:13.877 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.871+0000 7f8a4bfff700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7f8a4c068400 0x7f8a4c14f370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.112:48702/0 (socket says 172.21.15.112:48702) 2023-01-24T03:36:13.877 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.871+0000 7f8a4bfff700 1 -- 172.21.15.112:0/531880696 learned_addr learned my addr 172.21.15.112:0/531880696 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:36:13.877 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.871+0000 7f8a43fff700 1 -- 172.21.15.112:0/531880696 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a4c0aeb90 msgr2=0x7f8a4c0cc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:36:13.878 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.871+0000 7f8a43fff700 1 --2- 172.21.15.112:0/531880696 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a4c0aeb90 0x7f8a4c0cc3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:13.878 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.871+0000 7f8a43fff700 1 -- 172.21.15.112:0/531880696 >> v1:172.21.15.142:6789/0 conn(0x7f8a4c068400 legacy=0x7f8a4c14f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:13.878 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.871+0000 7f8a43fff700 1 -- 172.21.15.112:0/531880696 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a4c069d80 con 0x7f8a4c0aef60 2023-01-24T03:36:13.878 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.871+0000 7f8a4affd700 1 -- 172.21.15.112:0/531880696 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2537240179 0 0) 0x7f8a4c1560e0 con 0x7f8a4c068400 2023-01-24T03:36:13.878 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.871+0000 7f8a4b7fe700 1 --2- 172.21.15.112:0/531880696 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a4c0aeb90 0x7f8a4c0cc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:36:13.879 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.871+0000 7f8a43fff700 1 --2- 172.21.15.112:0/531880696 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a4c0aef60 0x7f8a4c0b14c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8a34009760 tx=0x7f8a34009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dbe395dd057feec server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:13.879 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.872+0000 7f8a4affd700 1 -- 172.21.15.112:0/531880696 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a34011630 con 0x7f8a4c0aef60 2023-01-24T03:36:13.879 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.872+0000 7f8a4affd700 1 -- 172.21.15.112:0/531880696 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a34011790 con 0x7f8a4c0aef60 2023-01-24T03:36:13.879 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.872+0000 7f8a4affd700 1 -- 172.21.15.112:0/531880696 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a34011b50 con 0x7f8a4c0aef60 2023-01-24T03:36:13.879 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.872+0000 7f8a51cf8700 1 -- 172.21.15.112:0/531880696 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a4c0aef60 msgr2=0x7f8a4c0b14c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:13.880 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.872+0000 7f8a51cf8700 1 --2- 172.21.15.112:0/531880696 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a4c0aef60 0x7f8a4c0b14c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8a34009760 tx=0x7f8a34009060 comp rx=0 tx=0).stop 2023-01-24T03:36:13.880 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.872+0000 7f8a51cf8700 1 -- 172.21.15.112:0/531880696 shutdown_connections 2023-01-24T03:36:13.880 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.872+0000 7f8a51cf8700 1 --2- 172.21.15.112:0/531880696 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a4c0aeb90 0x7f8a4c0cc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:13.880 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.872+0000 7f8a51cf8700 1 --2- 172.21.15.112:0/531880696 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a4c0aef60 0x7f8a4c0b14c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:13.881 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.872+0000 7f8a51cf8700 1 -- 172.21.15.112:0/531880696 >> 172.21.15.112:0/531880696 conn(0x7f8a4c05e300 msgr2=0x7f8a4c0cba20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:13.881 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.872+0000 7f8a51cf8700 1 -- 172.21.15.112:0/531880696 shutdown_connections 2023-01-24T03:36:13.881 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.872+0000 7f8a51cf8700 1 -- 172.21.15.112:0/531880696 wait complete. 2023-01-24T03:36:13.884 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.873+0000 7f8a51cf8700 1 Processor -- start 2023-01-24T03:36:13.884 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.873+0000 7f8a51cf8700 1 -- start start 2023-01-24T03:36:13.884 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.874+0000 7f8a51cf8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a4c068400 0x7f8a4c154260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:13.885 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.874+0000 7f8a51cf8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a4c0aeb90 0x7f8a4c1658f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:13.885 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.874+0000 7f8a51cf8700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a4c0aef60 0x7f8a4c14d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:13.885 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.874+0000 7f8a51cf8700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4c154910 con 0x7f8a4c0aef60 2023-01-24T03:36:13.885 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.874+0000 7f8a51cf8700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4c14e940 con 0x7f8a4c068400 2023-01-24T03:36:13.885 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.874+0000 7f8a51cf8700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4c165e30 con 0x7f8a4c0aeb90 2023-01-24T03:36:13.886 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.874+0000 7f8a4bfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a4c0aef60 0x7f8a4c14d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:13.886 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.874+0000 7f8a4bfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a4c0aef60 0x7f8a4c14d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:53928/0 (socket says 172.21.15.112:53928) 2023-01-24T03:36:13.886 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.874+0000 7f8a4bfff700 1 -- 172.21.15.112:0/3552829767 learned_addr learned my addr 172.21.15.112:0/3552829767 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:36:13.886 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.874+0000 7f8a4b7fe700 1 --2- 172.21.15.112:0/3552829767 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a4c068400 0x7f8a4c154260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:13.887 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.874+0000 7f8a43fff700 1 --2- 172.21.15.112:0/3552829767 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a4c0aeb90 0x7f8a4c1658f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:13.887 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.874+0000 7f8a4bfff700 1 -- 172.21.15.112:0/3552829767 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a4c0aeb90 msgr2=0x7f8a4c1658f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:13.887 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.874+0000 7f8a4bfff700 1 --2- 172.21.15.112:0/3552829767 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a4c0aeb90 0x7f8a4c1658f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:13.887 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.874+0000 7f8a4bfff700 1 -- 172.21.15.112:0/3552829767 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a4c068400 msgr2=0x7f8a4c154260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:13.887 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.874+0000 7f8a4bfff700 1 --2- 172.21.15.112:0/3552829767 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a4c068400 0x7f8a4c154260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:13.888 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.874+0000 7f8a4bfff700 1 -- 172.21.15.112:0/3552829767 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a3c0092a0 con 0x7f8a4c0aef60 2023-01-24T03:36:13.888 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.875+0000 7f8a4bfff700 1 --2- 172.21.15.112:0/3552829767 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a4c0aef60 0x7f8a4c14d9a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8a4400c810 tx=0x7f8a440014d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:13.888 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.875+0000 7f8a437fe700 1 -- 172.21.15.112:0/3552829767 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a44005d30 con 0x7f8a4c0aef60 2023-01-24T03:36:13.888 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.875+0000 7f8a437fe700 1 -- 172.21.15.112:0/3552829767 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a44005e90 con 0x7f8a4c0aef60 2023-01-24T03:36:13.888 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.875+0000 7f8a51cf8700 1 -- 172.21.15.112:0/3552829767 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a340092a0 con 0x7f8a4c0aef60 2023-01-24T03:36:13.889 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.875+0000 7f8a437fe700 1 -- 172.21.15.112:0/3552829767 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a44007300 con 0x7f8a4c0aef60 2023-01-24T03:36:13.889 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.875+0000 7f8a51cf8700 1 -- 172.21.15.112:0/3552829767 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a4c14a550 con 0x7f8a4c0aef60 2023-01-24T03:36:13.889 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.876+0000 7f8a51cf8700 1 -- 172.21.15.112:0/3552829767 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a4c04bb80 con 0x7f8a4c0aef60 2023-01-24T03:36:13.889 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.876+0000 7f8a437fe700 1 -- 172.21.15.112:0/3552829767 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f8a44019030 con 0x7f8a4c0aef60 2023-01-24T03:36:13.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.876+0000 7f8a437fe700 1 --2- 172.21.15.112:0/3552829767 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f8a300611f0 0x7f8a300636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:13.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.877+0000 7f8a437fe700 1 -- 172.21.15.112:0/3552829767 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 7181+0+0 (secure 0 0 0) 0x7f8a4407ce90 con 0x7f8a4c0aef60 2023-01-24T03:36:13.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.877+0000 7f8a4b7fe700 1 --2- 172.21.15.112:0/3552829767 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f8a300611f0 0x7f8a300636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:13.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.877+0000 7f8a4b7fe700 1 --2- 172.21.15.112:0/3552829767 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f8a300611f0 0x7f8a300636a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c009d20 tx=0x7f8a3c009060 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:13.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:13.881+0000 7f8a437fe700 1 -- 172.21.15.112:0/3552829767 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f8a4404ca70 con 0x7f8a4c0aef60 2023-01-24T03:36:14.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:14.101+0000 7f8a51cf8700 1 -- 172.21.15.112:0/3552829767 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a4c0af470 con 0x7f8a4c0aef60 2023-01-24T03:36:14.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:14.102+0000 7f8a437fe700 1 -- 172.21.15.112:0/3552829767 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+6115 (secure 0 0 0) 0x7f8a44017090 con 0x7f8a4c0aef60 2023-01-24T03:36:14.105 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:36:14.106 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4907,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/824478388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":824478388},{"type":"v1","addr":"172.21.15.142:6835","nonce":824478388}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:34:46.842740+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":22,"addr":"172.21.15.142:6834/612058861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":612058861},{"type":"v1","addr":"172.21.15.142:6834","nonce":612058861}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:36:14.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:14.104+0000 7f8a51cf8700 1 -- 172.21.15.112:0/3552829767 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f8a300611f0 msgr2=0x7f8a300636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:14.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:14.104+0000 7f8a51cf8700 1 --2- 172.21.15.112:0/3552829767 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f8a300611f0 0x7f8a300636a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c009d20 tx=0x7f8a3c009060 comp rx=0 tx=0).stop 2023-01-24T03:36:14.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:14.104+0000 7f8a51cf8700 1 -- 172.21.15.112:0/3552829767 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a4c0aef60 msgr2=0x7f8a4c14d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:14.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:14.104+0000 7f8a51cf8700 1 --2- 172.21.15.112:0/3552829767 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a4c0aef60 0x7f8a4c14d9a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8a4400c810 tx=0x7f8a440014d0 comp rx=0 tx=0).stop 2023-01-24T03:36:14.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:14.104+0000 7f8a51cf8700 1 -- 172.21.15.112:0/3552829767 shutdown_connections 2023-01-24T03:36:14.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:14.104+0000 7f8a51cf8700 1 --2- 172.21.15.112:0/3552829767 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f8a300611f0 0x7f8a300636a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:14.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:14.104+0000 7f8a51cf8700 1 --2- 172.21.15.112:0/3552829767 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8a4c0aef60 0x7f8a4c14d9a0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:14.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:14.104+0000 7f8a51cf8700 1 --2- 172.21.15.112:0/3552829767 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8a4c068400 0x7f8a4c154260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:14.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:14.104+0000 7f8a51cf8700 1 --2- 172.21.15.112:0/3552829767 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8a4c0aeb90 0x7f8a4c1658f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:14.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:14.104+0000 7f8a51cf8700 1 -- 172.21.15.112:0/3552829767 >> 172.21.15.112:0/3552829767 conn(0x7f8a4c05e300 msgr2=0x7f8a4c15aaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:14.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:14.104+0000 7f8a51cf8700 1 -- 172.21.15.112:0/3552829767 shutdown_connections 2023-01-24T03:36:14.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:14.104+0000 7f8a51cf8700 1 -- 172.21.15.112:0/3552829767 wait complete. 2023-01-24T03:36:14.110 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 26 2023-01-24T03:36:14.118 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-01-24T03:36:16.119 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:36:16.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.242+0000 7f3b2d169700 1 Processor -- start 2023-01-24T03:36:16.246 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.242+0000 7f3b2d169700 1 -- start start 2023-01-24T03:36:16.248 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.243+0000 7f3b2d169700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f3b280b0aa0 0x7f3b28155da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:16.248 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.243+0000 7f3b2d169700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b280ca3e0 0x7f3b280ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:16.248 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.243+0000 7f3b2d169700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b280b0f40 con 0x7f3b280c7e40 2023-01-24T03:36:16.249 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.243+0000 7f3b2d169700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b280b1170 con 0x7f3b280b0aa0 2023-01-24T03:36:16.249 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.243+0000 7f3b2d169700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b280b1450 con 0x7f3b280ca3e0 2023-01-24T03:36:16.249 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.243+0000 7f3b2659c700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f3b280b0aa0 0x7f3b28155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:16.249 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.243+0000 7f3b2659c700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f3b280b0aa0 0x7f3b28155da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:53944/0 (socket says 172.21.15.112:53944) 2023-01-24T03:36:16.249 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.243+0000 7f3b2659c700 1 -- 172.21.15.112:0/763744591 learned_addr learned my addr 172.21.15.112:0/763744591 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:36:16.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.243+0000 7f3b26d9d700 1 --1- 172.21.15.112:0/763744591 >> v1:172.21.15.112:6789/0 conn(0x7f3b280c7e40 0x7f3b280ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:34930/0 (socket says 172.21.15.112:34930) 2023-01-24T03:36:16.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.243+0000 7f3b2759e700 1 --2- 172.21.15.112:0/763744591 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b280ca3e0 0x7f3b280ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:16.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.244+0000 7f3b2659c700 1 -- 172.21.15.112:0/763744591 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b280ca3e0 msgr2=0x7f3b280ca7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:36:16.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.244+0000 7f3b2659c700 1 --2- 172.21.15.112:0/763744591 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b280ca3e0 0x7f3b280ca7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:16.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.244+0000 7f3b2659c700 1 -- 172.21.15.112:0/763744591 >> v1:172.21.15.112:6789/0 conn(0x7f3b280c7e40 legacy=0x7f3b280ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:16.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.244+0000 7f3b2659c700 1 -- 172.21.15.112:0/763744591 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b280b1730 con 0x7f3b280b0aa0 2023-01-24T03:36:16.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.244+0000 7f3b2659c700 1 --2- 172.21.15.112:0/763744591 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b280b0aa0 0x7f3b28155da0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f3b10006fd0 tx=0x7f3b100090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9a43a16f335ff62 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:16.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.244+0000 7f3b25d9b700 1 -- 172.21.15.112:0/763744591 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b10011630 con 0x7f3b280b0aa0 2023-01-24T03:36:16.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.244+0000 7f3b25d9b700 1 -- 172.21.15.112:0/763744591 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b100117d0 con 0x7f3b280b0aa0 2023-01-24T03:36:16.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.244+0000 7f3b25d9b700 1 -- 172.21.15.112:0/763744591 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b10011b90 con 0x7f3b280b0aa0 2023-01-24T03:36:16.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.244+0000 7f3b2d169700 1 -- 172.21.15.112:0/763744591 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b280b0aa0 msgr2=0x7f3b28155da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:16.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.244+0000 7f3b2d169700 1 --2- 172.21.15.112:0/763744591 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b280b0aa0 0x7f3b28155da0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f3b10006fd0 tx=0x7f3b100090f0 comp rx=0 tx=0).stop 2023-01-24T03:36:16.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.244+0000 7f3b2d169700 1 -- 172.21.15.112:0/763744591 shutdown_connections 2023-01-24T03:36:16.252 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.244+0000 7f3b2d169700 1 --2- 172.21.15.112:0/763744591 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b280ca3e0 0x7f3b280ca7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:16.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.244+0000 7f3b2d169700 1 --2- 172.21.15.112:0/763744591 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b280b0aa0 0x7f3b28155da0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:16.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.244+0000 7f3b2d169700 1 -- 172.21.15.112:0/763744591 >> 172.21.15.112:0/763744591 conn(0x7f3b2805e2c0 msgr2=0x7f3b280cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:16.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.244+0000 7f3b2d169700 1 -- 172.21.15.112:0/763744591 shutdown_connections 2023-01-24T03:36:16.253 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.244+0000 7f3b2d169700 1 -- 172.21.15.112:0/763744591 wait complete. 2023-01-24T03:36:16.256 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.246+0000 7f3b2d169700 1 Processor -- start 2023-01-24T03:36:16.256 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.246+0000 7f3b2d169700 1 -- start start 2023-01-24T03:36:16.256 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.246+0000 7f3b2d169700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b280b0aa0 0x7f3b280c1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:16.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.246+0000 7f3b2d169700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b280c7e40 0x7f3b2814ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:16.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.246+0000 7f3b2d169700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b280ca3e0 0x7f3b2814f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:16.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.246+0000 7f3b2d169700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b28166210 con 0x7f3b280ca3e0 2023-01-24T03:36:16.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.246+0000 7f3b2d169700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b280c3c50 con 0x7f3b280c7e40 2023-01-24T03:36:16.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.246+0000 7f3b2d169700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b28153b10 con 0x7f3b280b0aa0 2023-01-24T03:36:16.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.246+0000 7f3b2759e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b280ca3e0 0x7f3b2814f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:16.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.247+0000 7f3b2759e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b280ca3e0 0x7f3b2814f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:53952/0 (socket says 172.21.15.112:53952) 2023-01-24T03:36:16.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.247+0000 7f3b2759e700 1 -- 172.21.15.112:0/1005090262 learned_addr learned my addr 172.21.15.112:0/1005090262 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:36:16.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.247+0000 7f3b2659c700 1 --2- 172.21.15.112:0/1005090262 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b280c7e40 0x7f3b2814ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:16.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.247+0000 7f3b26d9d700 1 --2- 172.21.15.112:0/1005090262 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b280b0aa0 0x7f3b280c1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:16.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.247+0000 7f3b2759e700 1 -- 172.21.15.112:0/1005090262 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b280b0aa0 msgr2=0x7f3b280c1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:16.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.247+0000 7f3b2759e700 1 --2- 172.21.15.112:0/1005090262 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b280b0aa0 0x7f3b280c1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:16.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.247+0000 7f3b2759e700 1 -- 172.21.15.112:0/1005090262 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b280c7e40 msgr2=0x7f3b2814ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:16.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.247+0000 7f3b2759e700 1 --2- 172.21.15.112:0/1005090262 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b280c7e40 0x7f3b2814ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:16.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.247+0000 7f3b2759e700 1 -- 172.21.15.112:0/1005090262 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b10009370 con 0x7f3b280ca3e0 2023-01-24T03:36:16.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.247+0000 7f3b26d9d700 1 --2- 172.21.15.112:0/1005090262 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b280b0aa0 0x7f3b280c1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:36:16.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.247+0000 7f3b2759e700 1 --2- 172.21.15.112:0/1005090262 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b280ca3e0 0x7f3b2814f2a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f3b2000e900 tx=0x7f3b20009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:16.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.247+0000 7f3b1effd700 1 -- 172.21.15.112:0/1005090262 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b20013680 con 0x7f3b280ca3e0 2023-01-24T03:36:16.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.247+0000 7f3b1effd700 1 -- 172.21.15.112:0/1005090262 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b200137e0 con 0x7f3b280ca3e0 2023-01-24T03:36:16.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.247+0000 7f3b1effd700 1 -- 172.21.15.112:0/1005090262 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b20013ba0 con 0x7f3b280ca3e0 2023-01-24T03:36:16.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.247+0000 7f3b2d169700 1 -- 172.21.15.112:0/1005090262 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b28153df0 con 0x7f3b280ca3e0 2023-01-24T03:36:16.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.247+0000 7f3b2d169700 1 -- 172.21.15.112:0/1005090262 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b281543b0 con 0x7f3b280ca3e0 2023-01-24T03:36:16.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.248+0000 7f3b2d169700 1 -- 172.21.15.112:0/1005090262 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b280c3de0 con 0x7f3b280ca3e0 2023-01-24T03:36:16.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.249+0000 7f3b1effd700 1 -- 172.21.15.112:0/1005090262 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f3b20019030 con 0x7f3b280ca3e0 2023-01-24T03:36:16.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.249+0000 7f3b1effd700 1 --2- 172.21.15.112:0/1005090262 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3b0c0611f0 0x7f3b0c0636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:16.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.249+0000 7f3b1effd700 1 -- 172.21.15.112:0/1005090262 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 7181+0+0 (secure 0 0 0) 0x7f3b2007e120 con 0x7f3b280ca3e0 2023-01-24T03:36:16.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.249+0000 7f3b26d9d700 1 --2- 172.21.15.112:0/1005090262 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3b0c0611f0 0x7f3b0c0636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:16.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.250+0000 7f3b26d9d700 1 --2- 172.21.15.112:0/1005090262 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3b0c0611f0 0x7f3b0c0636a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3b18007500 tx=0x7f3b18005ad0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:16.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.253+0000 7f3b1effd700 1 -- 172.21.15.112:0/1005090262 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f3b2004de70 con 0x7f3b280ca3e0 2023-01-24T03:36:16.477 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.475+0000 7f3b2d169700 1 -- 172.21.15.112:0/1005090262 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b2804bb80 con 0x7f3b280ca3e0 2023-01-24T03:36:16.478 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.475+0000 7f3b1effd700 1 -- 172.21.15.112:0/1005090262 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+6115 (secure 0 0 0) 0x7f3b20051490 con 0x7f3b280ca3e0 2023-01-24T03:36:16.478 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:36:16.479 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4907,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/824478388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":824478388},{"type":"v1","addr":"172.21.15.142:6835","nonce":824478388}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:34:46.842740+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":22,"addr":"172.21.15.142:6834/612058861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":612058861},{"type":"v1","addr":"172.21.15.142:6834","nonce":612058861}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:36:16.481 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.477+0000 7f3b2d169700 1 -- 172.21.15.112:0/1005090262 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3b0c0611f0 msgr2=0x7f3b0c0636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:16.481 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.477+0000 7f3b2d169700 1 --2- 172.21.15.112:0/1005090262 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3b0c0611f0 0x7f3b0c0636a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3b18007500 tx=0x7f3b18005ad0 comp rx=0 tx=0).stop 2023-01-24T03:36:16.481 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.477+0000 7f3b2d169700 1 -- 172.21.15.112:0/1005090262 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b280ca3e0 msgr2=0x7f3b2814f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:16.481 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.477+0000 7f3b2d169700 1 --2- 172.21.15.112:0/1005090262 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b280ca3e0 0x7f3b2814f2a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f3b2000e900 tx=0x7f3b20009060 comp rx=0 tx=0).stop 2023-01-24T03:36:16.481 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.477+0000 7f3b2d169700 1 -- 172.21.15.112:0/1005090262 shutdown_connections 2023-01-24T03:36:16.482 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.477+0000 7f3b2d169700 1 --2- 172.21.15.112:0/1005090262 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3b0c0611f0 0x7f3b0c0636a0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:16.482 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.477+0000 7f3b2d169700 1 --2- 172.21.15.112:0/1005090262 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b280ca3e0 0x7f3b2814f2a0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:16.482 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.477+0000 7f3b2d169700 1 --2- 172.21.15.112:0/1005090262 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b280b0aa0 0x7f3b280c1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:16.482 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.477+0000 7f3b2d169700 1 --2- 172.21.15.112:0/1005090262 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b280c7e40 0x7f3b2814ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:16.482 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.477+0000 7f3b2d169700 1 -- 172.21.15.112:0/1005090262 >> 172.21.15.112:0/1005090262 conn(0x7f3b2805e2c0 msgr2=0x7f3b280c8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:16.482 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.478+0000 7f3b2d169700 1 -- 172.21.15.112:0/1005090262 shutdown_connections 2023-01-24T03:36:16.483 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:16.478+0000 7f3b2d169700 1 -- 172.21.15.112:0/1005090262 wait complete. 2023-01-24T03:36:16.483 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 26 2023-01-24T03:36:16.492 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-01-24T03:36:18.493 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:36:18.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.614+0000 7fbd8b313700 1 Processor -- start 2023-01-24T03:36:18.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.615+0000 7fbd8b313700 1 -- start start 2023-01-24T03:36:18.620 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.615+0000 7fbd8b313700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fbd8414f850 0x7fbd8406c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:18.621 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.615+0000 7fbd8b313700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fbd840c4c80 0x7fbd840b8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:18.621 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.615+0000 7fbd8b313700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd8406c7a0 con 0x7fbd840c4830 2023-01-24T03:36:18.621 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.615+0000 7fbd8b313700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd840c3ca0 con 0x7fbd840c4c80 2023-01-24T03:36:18.621 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.615+0000 7fbd8b313700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd8406ccd0 con 0x7fbd8414f850 2023-01-24T03:36:18.621 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.615+0000 7fbd898b0700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fbd840c4c80 0x7fbd840b8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:18.622 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.616+0000 7fbd898b0700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fbd840c4c80 0x7fbd840b8310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:53958/0 (socket says 172.21.15.112:53958) 2023-01-24T03:36:18.622 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.616+0000 7fbd898b0700 1 -- 172.21.15.112:0/4015084772 learned_addr learned my addr 172.21.15.112:0/4015084772 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:36:18.622 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.616+0000 7fbd888ae700 1 --1- 172.21.15.112:0/4015084772 >> v1:172.21.15.112:6789/0 conn(0x7fbd840c4830 0x7fbd84155590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:34946/0 (socket says 172.21.15.112:34946) 2023-01-24T03:36:18.622 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.616+0000 7fbd890af700 1 --2- 172.21.15.112:0/4015084772 >> v2:172.21.15.142:3300/0 conn(0x7fbd8414f850 0x7fbd8406c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:18.623 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.616+0000 7fbd898b0700 1 -- 172.21.15.112:0/4015084772 >> v2:172.21.15.142:3300/0 conn(0x7fbd8414f850 msgr2=0x7fbd8406c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:36:18.623 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.616+0000 7fbd898b0700 1 --2- 172.21.15.112:0/4015084772 >> v2:172.21.15.142:3300/0 conn(0x7fbd8414f850 0x7fbd8406c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:18.623 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.616+0000 7fbd898b0700 1 -- 172.21.15.112:0/4015084772 >> v1:172.21.15.112:6789/0 conn(0x7fbd840c4830 legacy=0x7fbd84155590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:18.623 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.616+0000 7fbd898b0700 1 -- 172.21.15.112:0/4015084772 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd8406c9f0 con 0x7fbd840c4c80 2023-01-24T03:36:18.623 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.616+0000 7fbd898b0700 1 --2- 172.21.15.112:0/4015084772 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd840c4c80 0x7fbd840b8310 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fbd84067130 tx=0x7fbd7c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75759988a190fbaf server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:18.624 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.616+0000 7fbd83fff700 1 -- 172.21.15.112:0/4015084772 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd7c011630 con 0x7fbd840c4c80 2023-01-24T03:36:18.624 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.616+0000 7fbd83fff700 1 -- 172.21.15.112:0/4015084772 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd7c0117d0 con 0x7fbd840c4c80 2023-01-24T03:36:18.624 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.616+0000 7fbd83fff700 1 -- 172.21.15.112:0/4015084772 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd7c011b90 con 0x7fbd840c4c80 2023-01-24T03:36:18.624 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.617+0000 7fbd8b313700 1 -- 172.21.15.112:0/4015084772 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd840c4c80 msgr2=0x7fbd840b8310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:18.624 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.617+0000 7fbd8b313700 1 --2- 172.21.15.112:0/4015084772 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd840c4c80 0x7fbd840b8310 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fbd84067130 tx=0x7fbd7c0090f0 comp rx=0 tx=0).stop 2023-01-24T03:36:18.625 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.617+0000 7fbd8b313700 1 -- 172.21.15.112:0/4015084772 shutdown_connections 2023-01-24T03:36:18.625 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.617+0000 7fbd8b313700 1 --2- 172.21.15.112:0/4015084772 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd840c4c80 0x7fbd840b8310 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:18.625 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.617+0000 7fbd8b313700 1 --2- 172.21.15.112:0/4015084772 >> v2:172.21.15.142:3300/0 conn(0x7fbd8414f850 0x7fbd8406c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:18.625 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.617+0000 7fbd8b313700 1 -- 172.21.15.112:0/4015084772 >> 172.21.15.112:0/4015084772 conn(0x7fbd84067f20 msgr2=0x7fbd84068320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:18.625 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.617+0000 7fbd8b313700 1 -- 172.21.15.112:0/4015084772 shutdown_connections 2023-01-24T03:36:18.626 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.617+0000 7fbd8b313700 1 -- 172.21.15.112:0/4015084772 wait complete. 2023-01-24T03:36:18.626 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.618+0000 7fbd8b313700 1 Processor -- start 2023-01-24T03:36:18.626 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.618+0000 7fbd8b313700 1 -- start start 2023-01-24T03:36:18.628 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.619+0000 7fbd8b313700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd840c4830 0x7fbd840c0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:18.629 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.619+0000 7fbd8b313700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd840c4c80 0x7fbd84155450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:18.629 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.619+0000 7fbd8b313700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd8414f850 0x7fbd84155990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:18.629 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.619+0000 7fbd8b313700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd840c1300 con 0x7fbd840c4830 2023-01-24T03:36:18.629 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.619+0000 7fbd8b313700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd84158fe0 con 0x7fbd840c4c80 2023-01-24T03:36:18.630 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.619+0000 7fbd8b313700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd8414e730 con 0x7fbd8414f850 2023-01-24T03:36:18.630 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.619+0000 7fbd890af700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd840c4830 0x7fbd840c0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:18.630 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.619+0000 7fbd890af700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd840c4830 0x7fbd840c0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:53962/0 (socket says 172.21.15.112:53962) 2023-01-24T03:36:18.630 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.619+0000 7fbd890af700 1 -- 172.21.15.112:0/3217347708 learned_addr learned my addr 172.21.15.112:0/3217347708 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:36:18.631 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.619+0000 7fbd888ae700 1 --2- 172.21.15.112:0/3217347708 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd840c4c80 0x7fbd84155450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:18.631 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.619+0000 7fbd898b0700 1 --2- 172.21.15.112:0/3217347708 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd8414f850 0x7fbd84155990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:18.631 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.619+0000 7fbd890af700 1 -- 172.21.15.112:0/3217347708 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd8414f850 msgr2=0x7fbd84155990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:18.631 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.619+0000 7fbd890af700 1 --2- 172.21.15.112:0/3217347708 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd8414f850 0x7fbd84155990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:18.631 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.619+0000 7fbd890af700 1 -- 172.21.15.112:0/3217347708 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd840c4c80 msgr2=0x7fbd84155450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:18.632 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.619+0000 7fbd890af700 1 --2- 172.21.15.112:0/3217347708 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd840c4c80 0x7fbd84155450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:18.632 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.619+0000 7fbd890af700 1 -- 172.21.15.112:0/3217347708 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd7c009370 con 0x7fbd840c4830 2023-01-24T03:36:18.632 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.619+0000 7fbd890af700 1 --2- 172.21.15.112:0/3217347708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd840c4830 0x7fbd840c0dc0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fbd74009fa0 tx=0x7fbd74009880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:18.632 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.619+0000 7fbd817fa700 1 -- 172.21.15.112:0/3217347708 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd7400ec70 con 0x7fbd840c4830 2023-01-24T03:36:18.632 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.619+0000 7fbd817fa700 1 -- 172.21.15.112:0/3217347708 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd7400edd0 con 0x7fbd840c4830 2023-01-24T03:36:18.633 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.619+0000 7fbd8b313700 1 -- 172.21.15.112:0/3217347708 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd84152a00 con 0x7fbd840c4830 2023-01-24T03:36:18.633 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.619+0000 7fbd817fa700 1 -- 172.21.15.112:0/3217347708 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd74014670 con 0x7fbd840c4830 2023-01-24T03:36:18.633 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.619+0000 7fbd8b313700 1 -- 172.21.15.112:0/3217347708 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd840c05c0 con 0x7fbd840c4830 2023-01-24T03:36:18.633 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.620+0000 7fbd8b313700 1 -- 172.21.15.112:0/3217347708 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd8404bb80 con 0x7fbd840c4830 2023-01-24T03:36:18.633 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.621+0000 7fbd817fa700 1 -- 172.21.15.112:0/3217347708 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fbd74012050 con 0x7fbd840c4830 2023-01-24T03:36:18.634 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.621+0000 7fbd817fa700 1 --2- 172.21.15.112:0/3217347708 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fbd640611c0 0x7fbd64063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:18.634 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.621+0000 7fbd817fa700 1 -- 172.21.15.112:0/3217347708 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 7181+0+0 (secure 0 0 0) 0x7fbd7407dee0 con 0x7fbd840c4830 2023-01-24T03:36:18.634 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.621+0000 7fbd888ae700 1 --2- 172.21.15.112:0/3217347708 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fbd640611c0 0x7fbd64063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:18.634 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.621+0000 7fbd888ae700 1 --2- 172.21.15.112:0/3217347708 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fbd640611c0 0x7fbd64063670 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbd8414ea90 tx=0x7fbd78007d30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:18.634 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.625+0000 7fbd817fa700 1 -- 172.21.15.112:0/3217347708 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fbd7404db50 con 0x7fbd840c4830 2023-01-24T03:36:18.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.846+0000 7fbd8b313700 1 -- 172.21.15.112:0/3217347708 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd840c3de0 con 0x7fbd840c4830 2023-01-24T03:36:18.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.847+0000 7fbd817fa700 1 -- 172.21.15.112:0/3217347708 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+6115 (secure 0 0 0) 0x7fbd74051170 con 0x7fbd840c4830 2023-01-24T03:36:18.850 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:36:18.851 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4907,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/824478388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":824478388},{"type":"v1","addr":"172.21.15.142:6835","nonce":824478388}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:34:46.842740+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":22,"addr":"172.21.15.142:6834/612058861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":612058861},{"type":"v1","addr":"172.21.15.142:6834","nonce":612058861}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:36:18.852 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.849+0000 7fbd8b313700 1 -- 172.21.15.112:0/3217347708 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fbd640611c0 msgr2=0x7fbd64063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:18.852 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.849+0000 7fbd8b313700 1 --2- 172.21.15.112:0/3217347708 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fbd640611c0 0x7fbd64063670 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbd8414ea90 tx=0x7fbd78007d30 comp rx=0 tx=0).stop 2023-01-24T03:36:18.852 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.849+0000 7fbd8b313700 1 -- 172.21.15.112:0/3217347708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd840c4830 msgr2=0x7fbd840c0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:18.853 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.849+0000 7fbd8b313700 1 --2- 172.21.15.112:0/3217347708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd840c4830 0x7fbd840c0dc0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fbd74009fa0 tx=0x7fbd74009880 comp rx=0 tx=0).stop 2023-01-24T03:36:18.853 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.849+0000 7fbd8b313700 1 -- 172.21.15.112:0/3217347708 shutdown_connections 2023-01-24T03:36:18.853 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.849+0000 7fbd8b313700 1 --2- 172.21.15.112:0/3217347708 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fbd640611c0 0x7fbd64063670 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:18.853 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.849+0000 7fbd8b313700 1 --2- 172.21.15.112:0/3217347708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd840c4830 0x7fbd840c0dc0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:18.853 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.849+0000 7fbd8b313700 1 --2- 172.21.15.112:0/3217347708 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd8414f850 0x7fbd84155990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:18.853 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.849+0000 7fbd8b313700 1 --2- 172.21.15.112:0/3217347708 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd840c4c80 0x7fbd84155450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:18.854 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.849+0000 7fbd8b313700 1 -- 172.21.15.112:0/3217347708 >> 172.21.15.112:0/3217347708 conn(0x7fbd84067f20 msgr2=0x7fbd840b8310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:18.854 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.849+0000 7fbd8b313700 1 -- 172.21.15.112:0/3217347708 shutdown_connections 2023-01-24T03:36:18.854 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:18.849+0000 7fbd8b313700 1 -- 172.21.15.112:0/3217347708 wait complete. 2023-01-24T03:36:18.855 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 26 2023-01-24T03:36:18.863 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-01-24T03:36:20.865 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:36:20.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.985+0000 7f1fd1e50700 1 Processor -- start 2023-01-24T03:36:20.998 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.985+0000 7f1fd1e50700 1 -- start start 2023-01-24T03:36:20.998 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.986+0000 7f1fd1e50700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1fcc068b10 0x7f1fcc0cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:20.998 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.986+0000 7f1fd1e50700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fcc1560e0 con 0x7f1fcc0aef60 2023-01-24T03:36:20.998 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.986+0000 7f1fd1e50700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fcc155530 con 0x7f1fcc0aeb90 2023-01-24T03:36:20.999 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.986+0000 7f1fd1e50700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fcc069ac0 con 0x7f1fcc068b10 2023-01-24T03:36:20.999 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.986+0000 7f1fcbfff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f1fcc0aef60 0x7f1fcc14f370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:34974/0 (socket says 172.21.15.112:34974) 2023-01-24T03:36:20.999 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.986+0000 7f1fcbfff700 1 -- 172.21.15.112:0/268691946 learned_addr learned my addr 172.21.15.112:0/268691946 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:36:20.999 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.987+0000 7f1fcaffd700 1 --2- 172.21.15.112:0/268691946 >> v2:172.21.15.142:3300/0 conn(0x7f1fcc068b10 0x7f1fcc0cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:20.999 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.987+0000 7f1fca7fc700 1 -- 172.21.15.112:0/268691946 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2264109173 0 0) 0x7f1fcc1560e0 con 0x7f1fcc0aef60 2023-01-24T03:36:21.000 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.987+0000 7f1fca7fc700 1 -- 172.21.15.112:0/268691946 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fb80033f0 con 0x7f1fcc0aef60 2023-01-24T03:36:21.000 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.987+0000 7f1fca7fc700 1 -- 172.21.15.112:0/268691946 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4025247880 0 0) 0x7f1fb80033f0 con 0x7f1fcc0aef60 2023-01-24T03:36:21.000 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.987+0000 7f1fca7fc700 1 -- 172.21.15.112:0/268691946 >> v2:172.21.15.142:3300/0 conn(0x7f1fcc068b10 msgr2=0x7f1fcc0cc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:36:21.000 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.987+0000 7f1fca7fc700 1 --2- 172.21.15.112:0/268691946 >> v2:172.21.15.142:3300/0 conn(0x7f1fcc068b10 0x7f1fcc0cc3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:21.000 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.987+0000 7f1fca7fc700 1 -- 172.21.15.112:0/268691946 >> v1:172.21.15.142:6789/0 conn(0x7f1fcc0aeb90 legacy=0x7f1fcc068400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:21.001 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.987+0000 7f1fca7fc700 1 -- 172.21.15.112:0/268691946 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fcc069d80 con 0x7f1fcc0aef60 2023-01-24T03:36:21.001 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.987+0000 7f1fca7fc700 1 -- 172.21.15.112:0/268691946 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7f1fc4001470 con 0x7f1fcc0aef60 2023-01-24T03:36:21.001 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.987+0000 7f1fca7fc700 1 -- 172.21.15.112:0/268691946 >> v1:172.21.15.112:6789/0 conn(0x7f1fcc0aef60 legacy=0x7f1fcc14f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:21.001 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.987+0000 7f1fca7fc700 1 --2- 172.21.15.112:0/268691946 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fb80038e0 0x7f1fb8003cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:21.001 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.987+0000 7f1fca7fc700 1 --2- 172.21.15.112:0/268691946 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fb8004690 0x7f1fb8004b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:21.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.987+0000 7f1fca7fc700 1 --2- 172.21.15.112:0/268691946 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1fb8005160 0x7f1fb800b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:21.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.987+0000 7f1fca7fc700 1 -- 172.21.15.112:0/268691946 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc069ac0 con 0x7f1fb8005160 2023-01-24T03:36:21.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.987+0000 7f1fca7fc700 1 -- 172.21.15.112:0/268691946 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb800c120 con 0x7f1fb8004690 2023-01-24T03:36:21.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.987+0000 7f1fca7fc700 1 -- 172.21.15.112:0/268691946 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb800c2e0 con 0x7f1fb80038e0 2023-01-24T03:36:21.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.987+0000 7f1fcbfff700 1 --2- 172.21.15.112:0/268691946 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1fb8005160 0x7f1fb800b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:21.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.987+0000 7f1fcaffd700 1 --2- 172.21.15.112:0/268691946 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fb8004690 0x7f1fb8004b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:21.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.987+0000 7f1fcb7fe700 1 --2- 172.21.15.112:0/268691946 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fb80038e0 0x7f1fb8003cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:21.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.987+0000 7f1fcbfff700 1 -- 172.21.15.112:0/268691946 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fb80038e0 msgr2=0x7f1fb8003cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:36:21.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.987+0000 7f1fcbfff700 1 --2- 172.21.15.112:0/268691946 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fb80038e0 0x7f1fb8003cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:21.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.987+0000 7f1fcbfff700 1 -- 172.21.15.112:0/268691946 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fb8004690 msgr2=0x7f1fb8004b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:36:21.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.987+0000 7f1fcbfff700 1 --2- 172.21.15.112:0/268691946 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fb8004690 0x7f1fb8004b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:21.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.987+0000 7f1fcbfff700 1 -- 172.21.15.112:0/268691946 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb80033f0 con 0x7f1fb8005160 2023-01-24T03:36:21.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.987+0000 7f1fcbfff700 1 -- 172.21.15.112:0/268691946 reap_dead start 2023-01-24T03:36:21.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.987+0000 7f1fcbfff700 1 --2- 172.21.15.112:0/268691946 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1fb8005160 0x7f1fb800b8e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f1fc400c190 tx=0x7f1fc4001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f309dabc4bdef8c server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:21.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.988+0000 7f1fca7fc700 1 -- 172.21.15.112:0/268691946 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc400cac0 con 0x7f1fb8005160 2023-01-24T03:36:21.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.988+0000 7f1fca7fc700 1 -- 172.21.15.112:0/268691946 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fc400cc20 con 0x7f1fb8005160 2023-01-24T03:36:21.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.988+0000 7f1fca7fc700 1 -- 172.21.15.112:0/268691946 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc40070d0 con 0x7f1fb8005160 2023-01-24T03:36:21.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.988+0000 7f1fd1e50700 1 -- 172.21.15.112:0/268691946 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1fb8005160 msgr2=0x7f1fb800b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:21.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.988+0000 7f1fd1e50700 1 --2- 172.21.15.112:0/268691946 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1fb8005160 0x7f1fb800b8e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f1fc400c190 tx=0x7f1fc4001e50 comp rx=0 tx=0).stop 2023-01-24T03:36:21.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.988+0000 7f1fd1e50700 1 -- 172.21.15.112:0/268691946 shutdown_connections 2023-01-24T03:36:21.006 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.988+0000 7f1fd1e50700 1 --2- 172.21.15.112:0/268691946 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1fb8005160 0x7f1fb800b8e0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:21.006 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.988+0000 7f1fd1e50700 1 -- 172.21.15.112:0/268691946 >> 172.21.15.112:0/268691946 conn(0x7f1fcc05e300 msgr2=0x7f1fcc0cba20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:21.006 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.988+0000 7f1fd1e50700 1 -- 172.21.15.112:0/268691946 shutdown_connections 2023-01-24T03:36:21.006 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.988+0000 7f1fd1e50700 1 -- 172.21.15.112:0/268691946 wait complete. 2023-01-24T03:36:21.006 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.989+0000 7f1fd1e50700 1 Processor -- start 2023-01-24T03:36:21.007 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.989+0000 7f1fd1e50700 1 -- start start 2023-01-24T03:36:21.007 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.990+0000 7f1fd1e50700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fb8005160 0x7f1fcc0bf830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:21.007 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.990+0000 7f1fd1e50700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc0bff10 0x7f1fcc149010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:21.007 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.990+0000 7f1fd1e50700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1fcc0c1470 0x7f1fcc0c0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:21.007 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.990+0000 7f1fd1e50700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc14f860 con 0x7f1fcc0c1470 2023-01-24T03:36:21.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.990+0000 7f1fd1e50700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc150db0 con 0x7f1fb8005160 2023-01-24T03:36:21.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.990+0000 7f1fd1e50700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc1503d0 con 0x7f1fcc0bff10 2023-01-24T03:36:21.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.990+0000 7f1fcbfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1fcc0c1470 0x7f1fcc0c0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:21.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.990+0000 7f1fcbfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1fcc0c1470 0x7f1fcc0c0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:53998/0 (socket says 172.21.15.112:53998) 2023-01-24T03:36:21.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.990+0000 7f1fcbfff700 1 -- 172.21.15.112:0/1873652106 learned_addr learned my addr 172.21.15.112:0/1873652106 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:36:21.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.990+0000 7f1fcb7fe700 1 --2- 172.21.15.112:0/1873652106 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fb8005160 0x7f1fcc0bf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:21.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.990+0000 7f1fcaffd700 1 --2- 172.21.15.112:0/1873652106 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc0bff10 0x7f1fcc149010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:21.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.991+0000 7f1fcbfff700 1 -- 172.21.15.112:0/1873652106 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc0bff10 msgr2=0x7f1fcc149010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:21.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.991+0000 7f1fcbfff700 1 --2- 172.21.15.112:0/1873652106 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc0bff10 0x7f1fcc149010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:21.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.991+0000 7f1fcbfff700 1 -- 172.21.15.112:0/1873652106 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fb8005160 msgr2=0x7f1fcc0bf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:21.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.991+0000 7f1fcbfff700 1 --2- 172.21.15.112:0/1873652106 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fb8005160 0x7f1fcc0bf830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:21.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.991+0000 7f1fcbfff700 1 -- 172.21.15.112:0/1873652106 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fcc169070 con 0x7f1fcc0c1470 2023-01-24T03:36:21.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.991+0000 7f1fcbfff700 1 --2- 172.21.15.112:0/1873652106 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1fcc0c1470 0x7f1fcc0c0a80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc067210 tx=0x7f1fc4013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:21.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.991+0000 7f1fb37fe700 1 -- 172.21.15.112:0/1873652106 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc400ce60 con 0x7f1fcc0c1470 2023-01-24T03:36:21.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.991+0000 7f1fb37fe700 1 -- 172.21.15.112:0/1873652106 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fc40046b0 con 0x7f1fcc0c1470 2023-01-24T03:36:21.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.991+0000 7f1fd1e50700 1 -- 172.21.15.112:0/1873652106 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fcc169350 con 0x7f1fcc0c1470 2023-01-24T03:36:21.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.991+0000 7f1fb37fe700 1 -- 172.21.15.112:0/1873652106 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc4004a30 con 0x7f1fcc0c1470 2023-01-24T03:36:21.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.991+0000 7f1fd1e50700 1 -- 172.21.15.112:0/1873652106 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fcc1500f0 con 0x7f1fcc0c1470 2023-01-24T03:36:21.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.992+0000 7f1fd1e50700 1 -- 172.21.15.112:0/1873652106 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fcc04bb80 con 0x7f1fcc0c1470 2023-01-24T03:36:21.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.992+0000 7f1fb37fe700 1 -- 172.21.15.112:0/1873652106 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f1fc4010070 con 0x7f1fcc0c1470 2023-01-24T03:36:21.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.992+0000 7f1fb37fe700 1 --2- 172.21.15.112:0/1873652106 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f1fac0611f0 0x7f1fac0636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:21.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.993+0000 7f1fcb7fe700 1 --2- 172.21.15.112:0/1873652106 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f1fac0611f0 0x7f1fac0636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:21.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.993+0000 7f1fb37fe700 1 -- 172.21.15.112:0/1873652106 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 7181+0+0 (secure 0 0 0) 0x7f1fc407e1d0 con 0x7f1fcc0c1470 2023-01-24T03:36:21.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.993+0000 7f1fcb7fe700 1 --2- 172.21.15.112:0/1873652106 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f1fac0611f0 0x7f1fac0636a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc000f80 tx=0x7f1fbc007f30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:21.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:20.998+0000 7f1fb37fe700 1 -- 172.21.15.112:0/1873652106 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f1fc404dea0 con 0x7f1fcc0c1470 2023-01-24T03:36:21.223 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:21.220+0000 7f1fd1e50700 1 -- 172.21.15.112:0/1873652106 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1fcc0af470 con 0x7f1fcc0c1470 2023-01-24T03:36:21.223 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:21.220+0000 7f1fb37fe700 1 -- 172.21.15.112:0/1873652106 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+6115 (secure 0 0 0) 0x7f1fc40514c0 con 0x7f1fcc0c1470 2023-01-24T03:36:21.224 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:36:21.224 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4907,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/824478388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":824478388},{"type":"v1","addr":"172.21.15.142:6835","nonce":824478388}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:34:46.842740+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":22,"addr":"172.21.15.142:6834/612058861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":612058861},{"type":"v1","addr":"172.21.15.142:6834","nonce":612058861}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:36:21.226 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:21.222+0000 7f1fd1e50700 1 -- 172.21.15.112:0/1873652106 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f1fac0611f0 msgr2=0x7f1fac0636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:21.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:21.222+0000 7f1fd1e50700 1 --2- 172.21.15.112:0/1873652106 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f1fac0611f0 0x7f1fac0636a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc000f80 tx=0x7f1fbc007f30 comp rx=0 tx=0).stop 2023-01-24T03:36:21.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:21.222+0000 7f1fd1e50700 1 -- 172.21.15.112:0/1873652106 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1fcc0c1470 msgr2=0x7f1fcc0c0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:21.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:21.222+0000 7f1fd1e50700 1 --2- 172.21.15.112:0/1873652106 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1fcc0c1470 0x7f1fcc0c0a80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc067210 tx=0x7f1fc4013000 comp rx=0 tx=0).stop 2023-01-24T03:36:21.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:21.223+0000 7f1fd1e50700 1 -- 172.21.15.112:0/1873652106 shutdown_connections 2023-01-24T03:36:21.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:21.223+0000 7f1fd1e50700 1 --2- 172.21.15.112:0/1873652106 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f1fac0611f0 0x7f1fac0636a0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:21.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:21.223+0000 7f1fd1e50700 1 --2- 172.21.15.112:0/1873652106 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1fcc0c1470 0x7f1fcc0c0a80 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:21.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:21.223+0000 7f1fd1e50700 1 --2- 172.21.15.112:0/1873652106 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fb8005160 0x7f1fcc0bf830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:21.229 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:21.223+0000 7f1fd1e50700 1 --2- 172.21.15.112:0/1873652106 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1fcc0bff10 0x7f1fcc149010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:21.229 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:21.223+0000 7f1fd1e50700 1 -- 172.21.15.112:0/1873652106 >> 172.21.15.112:0/1873652106 conn(0x7f1fcc05e300 msgr2=0x7f1fcc154280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:21.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:21.223+0000 7f1fd1e50700 1 -- 172.21.15.112:0/1873652106 shutdown_connections 2023-01-24T03:36:21.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:21.223+0000 7f1fd1e50700 1 -- 172.21.15.112:0/1873652106 wait complete. 2023-01-24T03:36:21.230 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 26 2023-01-24T03:36:21.237 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-01-24T03:36:23.238 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:36:23.354 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.351+0000 7f64bcee9700 1 Processor -- start 2023-01-24T03:36:23.355 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.351+0000 7f64bcee9700 1 -- start start 2023-01-24T03:36:23.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.352+0000 7f64bcee9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f64b80c4fb0 0x7f64b815e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:23.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.352+0000 7f64bcee9700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b8153e30 con 0x7f64b80c0bb0 2023-01-24T03:36:23.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.352+0000 7f64bcee9700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b8152fd0 con 0x7f64b806c4e0 2023-01-24T03:36:23.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.352+0000 7f64bcee9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64b81532b0 con 0x7f64b80c4fb0 2023-01-24T03:36:23.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.352+0000 7f64b77fe700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f64b80c0bb0 0x7f64b80c5670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35006/0 (socket says 172.21.15.112:35006) 2023-01-24T03:36:23.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.352+0000 7f64b77fe700 1 -- 172.21.15.112:0/3725243726 learned_addr learned my addr 172.21.15.112:0/3725243726 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:36:23.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.352+0000 7f64b7fff700 1 --2- 172.21.15.112:0/3725243726 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f64b80c4fb0 0x7f64b815e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:23.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.352+0000 7f64b67fc700 1 -- 172.21.15.112:0/3725243726 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3155810162 0 0) 0x7f64b8153e30 con 0x7f64b80c0bb0 2023-01-24T03:36:23.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.352+0000 7f64b67fc700 1 -- 172.21.15.112:0/3725243726 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64a40033f0 con 0x7f64b80c0bb0 2023-01-24T03:36:23.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.352+0000 7f64b67fc700 1 -- 172.21.15.112:0/3725243726 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 97377130 0 0) 0x7f64b8152fd0 con 0x7f64b806c4e0 2023-01-24T03:36:23.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.352+0000 7f64b67fc700 1 -- 172.21.15.112:0/3725243726 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64b8153e30 con 0x7f64b806c4e0 2023-01-24T03:36:23.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.352+0000 7f64b67fc700 1 -- 172.21.15.112:0/3725243726 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3280784041 0 0) 0x7f64a40033f0 con 0x7f64b80c0bb0 2023-01-24T03:36:23.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.352+0000 7f64b67fc700 1 -- 172.21.15.112:0/3725243726 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f64b80c4fb0 msgr2=0x7f64b815e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:36:23.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.352+0000 7f64b67fc700 1 --2- 172.21.15.112:0/3725243726 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f64b80c4fb0 0x7f64b815e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:23.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.352+0000 7f64b67fc700 1 -- 172.21.15.112:0/3725243726 >> v1:172.21.15.142:6789/0 conn(0x7f64b806c4e0 legacy=0x7f64b80c48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:23.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.352+0000 7f64b67fc700 1 -- 172.21.15.112:0/3725243726 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b8153590 con 0x7f64b80c0bb0 2023-01-24T03:36:23.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.353+0000 7f64b67fc700 1 -- 172.21.15.112:0/3725243726 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7f64a80017f0 con 0x7f64b80c0bb0 2023-01-24T03:36:23.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.353+0000 7f64b67fc700 1 -- 172.21.15.112:0/3725243726 >> v1:172.21.15.112:6789/0 conn(0x7f64b80c0bb0 legacy=0x7f64b80c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:23.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.353+0000 7f64b67fc700 1 --2- 172.21.15.112:0/3725243726 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64a4003aa0 0x7f64a4003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:23.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.353+0000 7f64b67fc700 1 --2- 172.21.15.112:0/3725243726 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f64a4004690 0x7f64a4004b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:23.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.353+0000 7f64b67fc700 1 --2- 172.21.15.112:0/3725243726 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f64b80c4fb0 0x7f64a40050a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:23.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.353+0000 7f64b67fc700 1 -- 172.21.15.112:0/3725243726 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b81532b0 con 0x7f64a4004690 2023-01-24T03:36:23.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.353+0000 7f64b67fc700 1 -- 172.21.15.112:0/3725243726 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a400ba70 con 0x7f64a4003aa0 2023-01-24T03:36:23.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.353+0000 7f64b67fc700 1 -- 172.21.15.112:0/3725243726 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64a400bc30 con 0x7f64b80c4fb0 2023-01-24T03:36:23.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.353+0000 7f64b6ffd700 1 --2- 172.21.15.112:0/3725243726 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f64a4004690 0x7f64a4004b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:23.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.353+0000 7f64b77fe700 1 --2- 172.21.15.112:0/3725243726 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64a4003aa0 0x7f64a4003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:23.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.353+0000 7f64b7fff700 1 --2- 172.21.15.112:0/3725243726 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f64b80c4fb0 0x7f64a40050a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:23.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.353+0000 7f64b6ffd700 1 -- 172.21.15.112:0/3725243726 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f64b80c4fb0 msgr2=0x7f64a40050a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:36:23.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.353+0000 7f64b6ffd700 1 --2- 172.21.15.112:0/3725243726 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f64b80c4fb0 0x7f64a40050a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:23.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.353+0000 7f64b6ffd700 1 -- 172.21.15.112:0/3725243726 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64a4003aa0 msgr2=0x7f64a4003e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:36:23.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.353+0000 7f64b6ffd700 1 --2- 172.21.15.112:0/3725243726 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64a4003aa0 0x7f64a4003e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:23.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.353+0000 7f64b6ffd700 1 -- 172.21.15.112:0/3725243726 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a40033f0 con 0x7f64a4004690 2023-01-24T03:36:23.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.353+0000 7f64b6ffd700 1 --2- 172.21.15.112:0/3725243726 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f64a4004690 0x7f64a4004b00 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f64ac001860 tx=0x7f64ac007bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36872f8d7069e2a2 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:23.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.353+0000 7f64b67fc700 1 -- 172.21.15.112:0/3725243726 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ac012420 con 0x7f64a4004690 2023-01-24T03:36:23.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.353+0000 7f64b67fc700 1 -- 172.21.15.112:0/3725243726 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64ac012580 con 0x7f64a4004690 2023-01-24T03:36:23.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.353+0000 7f64b67fc700 1 -- 172.21.15.112:0/3725243726 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ac012940 con 0x7f64a4004690 2023-01-24T03:36:23.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.354+0000 7f64bcee9700 1 -- 172.21.15.112:0/3725243726 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f64a4004690 msgr2=0x7f64a4004b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:23.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.354+0000 7f64bcee9700 1 --2- 172.21.15.112:0/3725243726 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f64a4004690 0x7f64a4004b00 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f64ac001860 tx=0x7f64ac007bb0 comp rx=0 tx=0).stop 2023-01-24T03:36:23.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.354+0000 7f64b7fff700 1 -- 172.21.15.112:0/3725243726 reap_dead start 2023-01-24T03:36:23.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.354+0000 7f64bcee9700 1 -- 172.21.15.112:0/3725243726 shutdown_connections 2023-01-24T03:36:23.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.354+0000 7f64bcee9700 1 -- 172.21.15.112:0/3725243726 >> 172.21.15.112:0/3725243726 conn(0x7f64b80ca150 msgr2=0x7f64b80c6120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:23.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.354+0000 7f64bcee9700 1 -- 172.21.15.112:0/3725243726 shutdown_connections 2023-01-24T03:36:23.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.354+0000 7f64bcee9700 1 -- 172.21.15.112:0/3725243726 wait complete. 2023-01-24T03:36:23.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.355+0000 7f64bcee9700 1 Processor -- start 2023-01-24T03:36:23.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.355+0000 7f64bcee9700 1 -- start start 2023-01-24T03:36:23.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.356+0000 7f64bcee9700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64a4004690 0x7f64b81549f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:23.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.356+0000 7f64bcee9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f64b81502f0 0x7f64b8150d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:23.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.356+0000 7f64bcee9700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f64b81639c0 0x7f64b8163e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:23.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.356+0000 7f64bcee9700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b8163530 con 0x7f64b81502f0 2023-01-24T03:36:23.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.356+0000 7f64bcee9700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b8166960 con 0x7f64a4004690 2023-01-24T03:36:23.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.356+0000 7f64bcee9700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64b8166c00 con 0x7f64b81639c0 2023-01-24T03:36:23.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.356+0000 7f64b6ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f64b81502f0 0x7f64b8150d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:23.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.356+0000 7f64b77fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64a4004690 0x7f64b81549f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:23.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.356+0000 7f64b6ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f64b81502f0 0x7f64b8150d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54032/0 (socket says 172.21.15.112:54032) 2023-01-24T03:36:23.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.356+0000 7f64b6ffd700 1 -- 172.21.15.112:0/3128812636 learned_addr learned my addr 172.21.15.112:0/3128812636 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:36:23.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.356+0000 7f64b7fff700 1 --2- 172.21.15.112:0/3128812636 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f64b81639c0 0x7f64b8163e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:23.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.356+0000 7f64b77fe700 1 -- 172.21.15.112:0/3128812636 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f64b81639c0 msgr2=0x7f64b8163e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:23.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.356+0000 7f64b77fe700 1 --2- 172.21.15.112:0/3128812636 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f64b81639c0 0x7f64b8163e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:23.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.356+0000 7f64b77fe700 1 -- 172.21.15.112:0/3128812636 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f64b81502f0 msgr2=0x7f64b8150d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:23.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.356+0000 7f64b77fe700 1 --2- 172.21.15.112:0/3128812636 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f64b81502f0 0x7f64b8150d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:23.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.356+0000 7f64b77fe700 1 -- 172.21.15.112:0/3128812636 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b8166ee0 con 0x7f64a4004690 2023-01-24T03:36:23.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.356+0000 7f64b6ffd700 1 --2- 172.21.15.112:0/3128812636 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f64b81502f0 0x7f64b8150d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:36:23.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.356+0000 7f64b77fe700 1 --2- 172.21.15.112:0/3128812636 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64a4004690 0x7f64b81549f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f64a8007410 tx=0x7f64a8007f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:23.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.356+0000 7f649f7fe700 1 -- 172.21.15.112:0/3128812636 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a8018030 con 0x7f64a4004690 2023-01-24T03:36:23.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.356+0000 7f649f7fe700 1 -- 172.21.15.112:0/3128812636 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64a800f5f0 con 0x7f64a4004690 2023-01-24T03:36:23.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.356+0000 7f64bcee9700 1 -- 172.21.15.112:0/3128812636 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64b81671c0 con 0x7f64a4004690 2023-01-24T03:36:23.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.356+0000 7f649f7fe700 1 -- 172.21.15.112:0/3128812636 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a800f9d0 con 0x7f64a4004690 2023-01-24T03:36:23.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.356+0000 7f64bcee9700 1 -- 172.21.15.112:0/3128812636 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64b814d050 con 0x7f64a4004690 2023-01-24T03:36:23.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.357+0000 7f64bcee9700 1 -- 172.21.15.112:0/3128812636 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64b805ea20 con 0x7f64a4004690 2023-01-24T03:36:23.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.358+0000 7f649f7fe700 1 -- 172.21.15.112:0/3128812636 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f64a801e470 con 0x7f64a4004690 2023-01-24T03:36:23.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.358+0000 7f649f7fe700 1 --2- 172.21.15.112:0/3128812636 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f64980611f0 0x7f64980636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:23.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.358+0000 7f649f7fe700 1 -- 172.21.15.112:0/3128812636 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 7181+0+0 (secure 0 0 0) 0x7f64a8019070 con 0x7f64a4004690 2023-01-24T03:36:23.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.358+0000 7f64b6ffd700 1 --2- 172.21.15.112:0/3128812636 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f64980611f0 0x7f64980636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:23.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.358+0000 7f64b6ffd700 1 --2- 172.21.15.112:0/3128812636 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f64980611f0 0x7f64980636a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f64b814ce80 tx=0x7f64ac015000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:23.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.362+0000 7f649f7fe700 1 -- 172.21.15.112:0/3128812636 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f64a804d8a0 con 0x7f64a4004690 2023-01-24T03:36:23.586 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.583+0000 7f64bcee9700 1 -- 172.21.15.112:0/3128812636 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64b8153870 con 0x7f64a4004690 2023-01-24T03:36:23.586 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.584+0000 7f649f7fe700 1 -- 172.21.15.112:0/3128812636 <== mon.1 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+6115 (secure 0 0 0) 0x7f64a8050ec0 con 0x7f64a4004690 2023-01-24T03:36:23.587 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:36:23.587 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4907,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/824478388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":824478388},{"type":"v1","addr":"172.21.15.142:6835","nonce":824478388}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:34:46.842740+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5041":{"gid":5041,"name":"b","rank":0,"incarnation":22,"state":"up:active","state_seq":22,"addr":"172.21.15.142:6834/612058861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":612058861},{"type":"v1","addr":"172.21.15.142:6834","nonce":612058861}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:36:23.589 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.586+0000 7f64bcee9700 1 -- 172.21.15.112:0/3128812636 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f64980611f0 msgr2=0x7f64980636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:23.589 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.586+0000 7f64bcee9700 1 --2- 172.21.15.112:0/3128812636 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f64980611f0 0x7f64980636a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f64b814ce80 tx=0x7f64ac015000 comp rx=0 tx=0).stop 2023-01-24T03:36:23.589 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.586+0000 7f64bcee9700 1 -- 172.21.15.112:0/3128812636 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64a4004690 msgr2=0x7f64b81549f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:23.590 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.586+0000 7f64bcee9700 1 --2- 172.21.15.112:0/3128812636 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64a4004690 0x7f64b81549f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f64a8007410 tx=0x7f64a8007f20 comp rx=0 tx=0).stop 2023-01-24T03:36:23.590 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.586+0000 7f64bcee9700 1 -- 172.21.15.112:0/3128812636 shutdown_connections 2023-01-24T03:36:23.590 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.586+0000 7f64bcee9700 1 --2- 172.21.15.112:0/3128812636 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f64980611f0 0x7f64980636a0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:23.590 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.586+0000 7f64bcee9700 1 --2- 172.21.15.112:0/3128812636 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f64b81639c0 0x7f64b8163e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:23.590 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.586+0000 7f64bcee9700 1 --2- 172.21.15.112:0/3128812636 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f64a4004690 0x7f64b81549f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:23.591 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.586+0000 7f64bcee9700 1 --2- 172.21.15.112:0/3128812636 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f64b81502f0 0x7f64b8150d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:23.591 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.586+0000 7f64bcee9700 1 -- 172.21.15.112:0/3128812636 >> 172.21.15.112:0/3128812636 conn(0x7f64b80ca150 msgr2=0x7f64b80c6850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:23.591 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.586+0000 7f64bcee9700 1 -- 172.21.15.112:0/3128812636 shutdown_connections 2023-01-24T03:36:23.591 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:23.586+0000 7f64bcee9700 1 -- 172.21.15.112:0/3128812636 wait complete. 2023-01-24T03:36:23.591 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 26 2023-01-24T03:36:23.600 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-01-24T03:36:25.601 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:36:25.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.724+0000 7f1157fff700 1 Processor -- start 2023-01-24T03:36:25.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.724+0000 7f1157fff700 1 -- start start 2023-01-24T03:36:25.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.724+0000 7f1157fff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f1158089660 0x7f11580c6190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:25.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.724+0000 7f1157fff700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1158159450 con 0x7f1158158860 2023-01-24T03:36:25.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.724+0000 7f1157fff700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1158089a80 con 0x7f1158158c30 2023-01-24T03:36:25.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.724+0000 7f1157fff700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11580c30c0 con 0x7f1158089660 2023-01-24T03:36:25.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.725+0000 7f11577fe700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f1158089660 0x7f11580c6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:25.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.725+0000 7f1156ffd700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f1158158860 0x7f1158155150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35026/0 (socket says 172.21.15.112:35026) 2023-01-24T03:36:25.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.725+0000 7f1156ffd700 1 -- 172.21.15.112:0/118178165 learned_addr learned my addr 172.21.15.112:0/118178165 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:36:25.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.725+0000 7f11577fe700 1 -- 172.21.15.112:0/118178165 >> v1:172.21.15.142:6789/0 conn(0x7f1158158c30 legacy=0x7f1158068c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:25.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.725+0000 7f11577fe700 1 -- 172.21.15.112:0/118178165 >> v1:172.21.15.112:6789/0 conn(0x7f1158158860 legacy=0x7f1158155150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:25.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.725+0000 7f11577fe700 1 -- 172.21.15.112:0/118178165 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11580c33a0 con 0x7f1158089660 2023-01-24T03:36:25.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.725+0000 7f11577fe700 1 --2- 172.21.15.112:0/118178165 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1158089660 0x7f11580c6190 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1150009760 tx=0x7f1150009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ba7c5eef8c9342a server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:25.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.725+0000 7f1155ffb700 1 -- 172.21.15.112:0/118178165 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1150011630 con 0x7f1158089660 2023-01-24T03:36:25.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.725+0000 7f1155ffb700 1 -- 172.21.15.112:0/118178165 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1150011790 con 0x7f1158089660 2023-01-24T03:36:25.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.725+0000 7f1155ffb700 1 -- 172.21.15.112:0/118178165 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1150011b10 con 0x7f1158089660 2023-01-24T03:36:25.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.726+0000 7f1157fff700 1 -- 172.21.15.112:0/118178165 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1158089660 msgr2=0x7f11580c6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:25.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.726+0000 7f1157fff700 1 --2- 172.21.15.112:0/118178165 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1158089660 0x7f11580c6190 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1150009760 tx=0x7f1150009060 comp rx=0 tx=0).stop 2023-01-24T03:36:25.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.726+0000 7f1157fff700 1 -- 172.21.15.112:0/118178165 shutdown_connections 2023-01-24T03:36:25.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.726+0000 7f1157fff700 1 --2- 172.21.15.112:0/118178165 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1158089660 0x7f11580c6190 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:25.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.726+0000 7f1157fff700 1 -- 172.21.15.112:0/118178165 >> 172.21.15.112:0/118178165 conn(0x7f115805e2b0 msgr2=0x7f11580cb8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:25.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.726+0000 7f1157fff700 1 -- 172.21.15.112:0/118178165 shutdown_connections 2023-01-24T03:36:25.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.726+0000 7f1157fff700 1 -- 172.21.15.112:0/118178165 wait complete. 2023-01-24T03:36:25.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.727+0000 7f1157fff700 1 Processor -- start 2023-01-24T03:36:25.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.727+0000 7f1157fff700 1 -- start start 2023-01-24T03:36:25.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.728+0000 7f1157fff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1158089660 0x7f1158166d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:25.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.728+0000 7f1157fff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1158158860 0x7f1158167280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:25.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.728+0000 7f1157fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1158158c30 0x7f115814db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:25.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.728+0000 7f1157fff700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1158166010 con 0x7f1158158c30 2023-01-24T03:36:25.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.728+0000 7f1157fff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1158152a40 con 0x7f1158158860 2023-01-24T03:36:25.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.728+0000 7f1157fff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1158152d20 con 0x7f1158089660 2023-01-24T03:36:25.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.728+0000 7f11577fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1158158c30 0x7f115814db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:25.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.728+0000 7f11577fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1158158c30 0x7f115814db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54048/0 (socket says 172.21.15.112:54048) 2023-01-24T03:36:25.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.728+0000 7f11577fe700 1 -- 172.21.15.112:0/2054545440 learned_addr learned my addr 172.21.15.112:0/2054545440 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:36:25.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.728+0000 7f1156ffd700 1 --2- 172.21.15.112:0/2054545440 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1158089660 0x7f1158166d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:25.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.728+0000 7f11567fc700 1 --2- 172.21.15.112:0/2054545440 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1158158860 0x7f1158167280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:25.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.728+0000 7f11577fe700 1 -- 172.21.15.112:0/2054545440 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1158089660 msgr2=0x7f1158166d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:25.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.728+0000 7f11577fe700 1 --2- 172.21.15.112:0/2054545440 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1158089660 0x7f1158166d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:25.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.728+0000 7f11577fe700 1 -- 172.21.15.112:0/2054545440 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1158158860 msgr2=0x7f1158167280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:25.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.728+0000 7f11577fe700 1 --2- 172.21.15.112:0/2054545440 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1158158860 0x7f1158167280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:25.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.728+0000 7f11577fe700 1 -- 172.21.15.112:0/2054545440 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11500092a0 con 0x7f1158158c30 2023-01-24T03:36:25.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.728+0000 7f11577fe700 1 --2- 172.21.15.112:0/2054545440 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1158158c30 0x7f115814db10 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f1150009eb0 tx=0x7f1150017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:25.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.729+0000 7f114effd700 1 -- 172.21.15.112:0/2054545440 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f115001b430 con 0x7f1158158c30 2023-01-24T03:36:25.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.729+0000 7f114effd700 1 -- 172.21.15.112:0/2054545440 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f115001f070 con 0x7f1158158c30 2023-01-24T03:36:25.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.729+0000 7f1157fff700 1 -- 172.21.15.112:0/2054545440 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1158166260 con 0x7f1158158c30 2023-01-24T03:36:25.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.729+0000 7f114effd700 1 -- 172.21.15.112:0/2054545440 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f115001b800 con 0x7f1158158c30 2023-01-24T03:36:25.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.729+0000 7f1157fff700 1 -- 172.21.15.112:0/2054545440 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1158166820 con 0x7f1158158c30 2023-01-24T03:36:25.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.730+0000 7f1157fff700 1 -- 172.21.15.112:0/2054545440 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1158166a10 con 0x7f1158158c30 2023-01-24T03:36:25.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.730+0000 7f114effd700 1 -- 172.21.15.112:0/2054545440 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f115000f050 con 0x7f1158158c30 2023-01-24T03:36:25.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.730+0000 7f114effd700 1 --2- 172.21.15.112:0/2054545440 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f113c0611c0 0x7f113c063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:25.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.730+0000 7f114effd700 1 -- 172.21.15.112:0/2054545440 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 7211+0+0 (secure 0 0 0) 0x7f115007ef50 con 0x7f1158158c30 2023-01-24T03:36:25.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.730+0000 7f1156ffd700 1 --2- 172.21.15.112:0/2054545440 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f113c0611c0 0x7f113c063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:25.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.731+0000 7f1156ffd700 1 --2- 172.21.15.112:0/2054545440 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f113c0611c0 0x7f113c063670 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1148001a90 tx=0x7f1148001560 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:25.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.735+0000 7f114effd700 1 -- 172.21.15.112:0/2054545440 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f115004ed00 con 0x7f1158158c30 2023-01-24T03:36:25.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.952+0000 7f1157fff700 1 -- 172.21.15.112:0/2054545440 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f115805ea20 con 0x7f1158158c30 2023-01-24T03:36:25.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.953+0000 7f114effd700 1 -- 172.21.15.112:0/2054545440 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+5357 (secure 0 0 0) 0x7f1150015090 con 0x7f1158158c30 2023-01-24T03:36:25.957 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:36:25.957 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:36:23.647869+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4907},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4907":{"gid":4907,"name":"d","rank":0,"incarnation":27,"state":"up:replay","state_seq":1,"addr":"172.21.15.142:6835/824478388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":824478388},{"type":"v1","addr":"172.21.15.142:6835","nonce":824478388}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:36:25.958 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.955+0000 7f1157fff700 1 -- 172.21.15.112:0/2054545440 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f113c0611c0 msgr2=0x7f113c063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:25.959 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.955+0000 7f1157fff700 1 --2- 172.21.15.112:0/2054545440 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f113c0611c0 0x7f113c063670 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1148001a90 tx=0x7f1148001560 comp rx=0 tx=0).stop 2023-01-24T03:36:25.959 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.955+0000 7f1157fff700 1 -- 172.21.15.112:0/2054545440 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1158158c30 msgr2=0x7f115814db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:25.959 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.955+0000 7f1157fff700 1 --2- 172.21.15.112:0/2054545440 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1158158c30 0x7f115814db10 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f1150009eb0 tx=0x7f1150017000 comp rx=0 tx=0).stop 2023-01-24T03:36:25.959 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.955+0000 7f1157fff700 1 -- 172.21.15.112:0/2054545440 shutdown_connections 2023-01-24T03:36:25.959 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.955+0000 7f1157fff700 1 --2- 172.21.15.112:0/2054545440 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f113c0611c0 0x7f113c063670 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:25.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.955+0000 7f1157fff700 1 --2- 172.21.15.112:0/2054545440 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1158158c30 0x7f115814db10 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:25.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.955+0000 7f1157fff700 1 --2- 172.21.15.112:0/2054545440 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1158089660 0x7f1158166d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:25.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.955+0000 7f1157fff700 1 --2- 172.21.15.112:0/2054545440 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1158158860 0x7f1158167280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:25.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.955+0000 7f1157fff700 1 -- 172.21.15.112:0/2054545440 >> 172.21.15.112:0/2054545440 conn(0x7f115805e2b0 msgr2=0x7f1158067ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:25.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.955+0000 7f1157fff700 1 -- 172.21.15.112:0/2054545440 shutdown_connections 2023-01-24T03:36:25.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:25.955+0000 7f1157fff700 1 -- 172.21.15.112:0/2054545440 wait complete. 2023-01-24T03:36:25.961 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 27 2023-01-24T03:36:25.970 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2023-01-24T03:36:25.970 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-01-24T03:36:25.970 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:36:26.083 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.080+0000 7f0e18f96700 1 Processor -- start 2023-01-24T03:36:26.083 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.080+0000 7f0e18f96700 1 -- start start 2023-01-24T03:36:26.085 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.081+0000 7f0e18f96700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f0e140b0aa0 0x7f0e14155da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:26.085 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.081+0000 7f0e18f96700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e140b0f40 con 0x7f0e140c7e40 2023-01-24T03:36:26.086 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.081+0000 7f0e18f96700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e140b1170 con 0x7f0e140ca3e0 2023-01-24T03:36:26.086 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.081+0000 7f0e18f96700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e140b1450 con 0x7f0e140b0aa0 2023-01-24T03:36:26.086 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.081+0000 7f0e11d9b700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f0e140b0aa0 0x7f0e14155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:26.087 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.081+0000 7f0e1259c700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f0e140c7e40 0x7f0e140ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35040/0 (socket says 172.21.15.112:35040) 2023-01-24T03:36:26.087 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.081+0000 7f0e1259c700 1 -- 172.21.15.112:0/1848393045 learned_addr learned my addr 172.21.15.112:0/1848393045 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:36:26.087 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.081+0000 7f0e11d9b700 1 -- 172.21.15.112:0/1848393045 >> v1:172.21.15.142:6789/0 conn(0x7f0e140ca3e0 legacy=0x7f0e140ca7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:36:26.088 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.081+0000 7f0e11d9b700 1 -- 172.21.15.112:0/1848393045 >> v1:172.21.15.112:6789/0 conn(0x7f0e140c7e40 legacy=0x7f0e140ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:26.088 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.081+0000 7f0e11d9b700 1 -- 172.21.15.112:0/1848393045 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e140b1730 con 0x7f0e140b0aa0 2023-01-24T03:36:26.088 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.081+0000 7f0e1159a700 1 -- 172.21.15.112:0/1848393045 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3887327647 0 0) 0x7f0e140b0f40 con 0x7f0e140c7e40 2023-01-24T03:36:26.088 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.081+0000 7f0e11d9b700 1 --2- 172.21.15.112:0/1848393045 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0e140b0aa0 0x7f0e14155da0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f0e08009760 tx=0x7f0e08009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1871204d743b59a0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:26.088 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.081+0000 7f0e1159a700 1 -- 172.21.15.112:0/1848393045 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e08011630 con 0x7f0e140b0aa0 2023-01-24T03:36:26.089 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.081+0000 7f0e1159a700 1 -- 172.21.15.112:0/1848393045 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e08011790 con 0x7f0e140b0aa0 2023-01-24T03:36:26.089 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.081+0000 7f0e1159a700 1 -- 172.21.15.112:0/1848393045 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e08011b30 con 0x7f0e140b0aa0 2023-01-24T03:36:26.089 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.082+0000 7f0e18f96700 1 -- 172.21.15.112:0/1848393045 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0e140b0aa0 msgr2=0x7f0e14155da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:26.089 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.082+0000 7f0e18f96700 1 --2- 172.21.15.112:0/1848393045 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0e140b0aa0 0x7f0e14155da0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f0e08009760 tx=0x7f0e08009060 comp rx=0 tx=0).stop 2023-01-24T03:36:26.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.082+0000 7f0e18f96700 1 -- 172.21.15.112:0/1848393045 shutdown_connections 2023-01-24T03:36:26.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.082+0000 7f0e18f96700 1 --2- 172.21.15.112:0/1848393045 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0e140b0aa0 0x7f0e14155da0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:26.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.082+0000 7f0e18f96700 1 -- 172.21.15.112:0/1848393045 >> 172.21.15.112:0/1848393045 conn(0x7f0e1405e2c0 msgr2=0x7f0e140cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:26.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.082+0000 7f0e18f96700 1 -- 172.21.15.112:0/1848393045 shutdown_connections 2023-01-24T03:36:26.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.082+0000 7f0e18f96700 1 -- 172.21.15.112:0/1848393045 wait complete. 2023-01-24T03:36:26.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.083+0000 7f0e18f96700 1 Processor -- start 2023-01-24T03:36:26.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.083+0000 7f0e18f96700 1 -- start start 2023-01-24T03:36:26.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.084+0000 7f0e18f96700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0e140b0aa0 0x7f0e140c53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:26.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.084+0000 7f0e18f96700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0e140c7e40 0x7f0e140bfdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:26.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.084+0000 7f0e18f96700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0e140ca3e0 0x7f0e140c0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:26.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.084+0000 7f0e18f96700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e140bf970 con 0x7f0e140b0aa0 2023-01-24T03:36:26.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.084+0000 7f0e18f96700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e14165960 con 0x7f0e140ca3e0 2023-01-24T03:36:26.095 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.084+0000 7f0e18f96700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e14165c40 con 0x7f0e140c7e40 2023-01-24T03:36:26.095 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.084+0000 7f0e1259c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0e140b0aa0 0x7f0e140c53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:26.095 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.084+0000 7f0e1259c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0e140b0aa0 0x7f0e140c53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54058/0 (socket says 172.21.15.112:54058) 2023-01-24T03:36:26.095 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.084+0000 7f0e1259c700 1 -- 172.21.15.112:0/1667846649 learned_addr learned my addr 172.21.15.112:0/1667846649 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:36:26.095 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.084+0000 7f0e11d9b700 1 --2- 172.21.15.112:0/1667846649 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0e140c7e40 0x7f0e140bfdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:26.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.084+0000 7f0e12d9d700 1 --2- 172.21.15.112:0/1667846649 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0e140ca3e0 0x7f0e140c0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:26.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.084+0000 7f0e1259c700 1 -- 172.21.15.112:0/1667846649 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0e140c7e40 msgr2=0x7f0e140bfdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:26.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.084+0000 7f0e1259c700 1 --2- 172.21.15.112:0/1667846649 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0e140c7e40 0x7f0e140bfdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:26.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.084+0000 7f0e1259c700 1 -- 172.21.15.112:0/1667846649 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0e140ca3e0 msgr2=0x7f0e140c0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:26.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.084+0000 7f0e1259c700 1 --2- 172.21.15.112:0/1667846649 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0e140ca3e0 0x7f0e140c0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:26.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.084+0000 7f0e1259c700 1 -- 172.21.15.112:0/1667846649 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e080092a0 con 0x7f0e140b0aa0 2023-01-24T03:36:26.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.084+0000 7f0e12d9d700 1 --2- 172.21.15.112:0/1667846649 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0e140ca3e0 0x7f0e140c0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:36:26.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.084+0000 7f0e1259c700 1 --2- 172.21.15.112:0/1667846649 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0e140b0aa0 0x7f0e140c53d0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f0e04001eb0 tx=0x7f0e04007e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:26.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.084+0000 7f0e027fc700 1 -- 172.21.15.112:0/1667846649 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e0400ce90 con 0x7f0e140b0aa0 2023-01-24T03:36:26.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.084+0000 7f0e027fc700 1 -- 172.21.15.112:0/1667846649 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e040072e0 con 0x7f0e140b0aa0 2023-01-24T03:36:26.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.084+0000 7f0e027fc700 1 -- 172.21.15.112:0/1667846649 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e04005780 con 0x7f0e140b0aa0 2023-01-24T03:36:26.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.084+0000 7f0e18f96700 1 -- 172.21.15.112:0/1667846649 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e14165f20 con 0x7f0e140b0aa0 2023-01-24T03:36:26.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.084+0000 7f0e18f96700 1 -- 172.21.15.112:0/1667846649 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e141664e0 con 0x7f0e140b0aa0 2023-01-24T03:36:26.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.085+0000 7f0e18f96700 1 -- 172.21.15.112:0/1667846649 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e140bfc10 con 0x7f0e140b0aa0 2023-01-24T03:36:26.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.086+0000 7f0e027fc700 1 -- 172.21.15.112:0/1667846649 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f0e04019030 con 0x7f0e140b0aa0 2023-01-24T03:36:26.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.086+0000 7f0e027fc700 1 --2- 172.21.15.112:0/1667846649 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f0df00611c0 0x7f0df0063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:26.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.086+0000 7f0e027fc700 1 -- 172.21.15.112:0/1667846649 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 7211+0+0 (secure 0 0 0) 0x7f0e0407cc30 con 0x7f0e140b0aa0 2023-01-24T03:36:26.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.086+0000 7f0e11d9b700 1 --2- 172.21.15.112:0/1667846649 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f0df00611c0 0x7f0df0063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:26.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.086+0000 7f0e11d9b700 1 --2- 172.21.15.112:0/1667846649 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f0df00611c0 0x7f0df0063670 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0e08011fd0 tx=0x7f0e08016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:26.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.090+0000 7f0e027fc700 1 -- 172.21.15.112:0/1667846649 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f0e0404c8e0 con 0x7f0e140b0aa0 2023-01-24T03:36:26.314 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.311+0000 7f0e18f96700 1 -- 172.21.15.112:0/1667846649 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e1404bb80 con 0x7f0e140b0aa0 2023-01-24T03:36:26.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.312+0000 7f0e027fc700 1 -- 172.21.15.112:0/1667846649 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+5357 (secure 0 0 0) 0x7f0e0404ff00 con 0x7f0e140b0aa0 2023-01-24T03:36:26.315 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:36:26.316 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:36:23.647869+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4907},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4907":{"gid":4907,"name":"d","rank":0,"incarnation":27,"state":"up:replay","state_seq":1,"addr":"172.21.15.142:6835/824478388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":824478388},{"type":"v1","addr":"172.21.15.142:6835","nonce":824478388}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:36:26.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.314+0000 7f0e18f96700 1 -- 172.21.15.112:0/1667846649 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f0df00611c0 msgr2=0x7f0df0063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:26.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.314+0000 7f0e18f96700 1 --2- 172.21.15.112:0/1667846649 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f0df00611c0 0x7f0df0063670 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0e08011fd0 tx=0x7f0e08016000 comp rx=0 tx=0).stop 2023-01-24T03:36:26.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.314+0000 7f0e18f96700 1 -- 172.21.15.112:0/1667846649 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0e140b0aa0 msgr2=0x7f0e140c53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:26.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.314+0000 7f0e18f96700 1 --2- 172.21.15.112:0/1667846649 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0e140b0aa0 0x7f0e140c53d0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f0e04001eb0 tx=0x7f0e04007e90 comp rx=0 tx=0).stop 2023-01-24T03:36:26.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.314+0000 7f0e18f96700 1 -- 172.21.15.112:0/1667846649 shutdown_connections 2023-01-24T03:36:26.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.314+0000 7f0e18f96700 1 --2- 172.21.15.112:0/1667846649 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f0df00611c0 0x7f0df0063670 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:26.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.314+0000 7f0e18f96700 1 --2- 172.21.15.112:0/1667846649 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0e140b0aa0 0x7f0e140c53d0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:26.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.314+0000 7f0e18f96700 1 --2- 172.21.15.112:0/1667846649 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0e140ca3e0 0x7f0e140c0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:26.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.314+0000 7f0e18f96700 1 --2- 172.21.15.112:0/1667846649 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0e140c7e40 0x7f0e140bfdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:26.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.314+0000 7f0e18f96700 1 -- 172.21.15.112:0/1667846649 >> 172.21.15.112:0/1667846649 conn(0x7f0e1405e2c0 msgr2=0x7f0e140680c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:26.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.314+0000 7f0e18f96700 1 -- 172.21.15.112:0/1667846649 shutdown_connections 2023-01-24T03:36:26.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:26.314+0000 7f0e18f96700 1 -- 172.21.15.112:0/1667846649 wait complete. 2023-01-24T03:36:26.320 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 27 2023-01-24T03:36:26.329 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 27, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-01-24T03:30:20.971710+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 27, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4907": { "addr": "172.21.15.142:6835/824478388", "addrs": { "addrvec": [ { "addr": "172.21.15.142:6833", "nonce": 824478388, "type": "v2" }, { "addr": "172.21.15.142:6835", "nonce": 824478388, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4907, "incarnation": 27, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 46, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-01-24T03:36:23.647869+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4907 } } } ], "standbys": [ { "addr": "172.21.15.142:6837/2181345869", "addrs": { "addrvec": [ { "addr": "172.21.15.142:6836", "nonce": 2181345869, "type": "v2" }, { "addr": "172.21.15.142:6837", "nonce": 2181345869, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6835/330343927", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6834", "nonce": 330343927, "type": "v2" }, { "addr": "172.21.15.112:6835", "nonce": 330343927, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6837/12176817", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6836", "nonce": 12176817, "type": "v2" }, { "addr": "172.21.15.112:6837", "nonce": 12176817, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4276, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6839/1110983841", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6838", "nonce": 1110983841, "type": "v2" }, { "addr": "172.21.15.112:6839", "nonce": 1110983841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4281, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-01-24T03:36:28.330 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:36:28.456 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.454+0000 7f15beaf0700 1 Processor -- start 2023-01-24T03:36:28.456 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.454+0000 7f15beaf0700 1 -- start start 2023-01-24T03:36:28.462 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.454+0000 7f15beaf0700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f15b80c7e40 0x7f15b80b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:28.462 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.454+0000 7f15beaf0700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15b8156240 con 0x7f15b80ca3e0 2023-01-24T03:36:28.462 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.454+0000 7f15beaf0700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15b80b1170 con 0x7f15b8155da0 2023-01-24T03:36:28.463 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.454+0000 7f15beaf0700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15b80b1450 con 0x7f15b80c7e40 2023-01-24T03:36:28.463 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.454+0000 7f15bd08d700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f15b80ca3e0 0x7f15b80ca7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35060/0 (socket says 172.21.15.112:35060) 2023-01-24T03:36:28.463 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.454+0000 7f15bd08d700 1 -- 172.21.15.112:0/3786949126 learned_addr learned my addr 172.21.15.112:0/3786949126 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:36:28.463 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.455+0000 7f15bc88c700 1 --2- 172.21.15.112:0/3786949126 >> v2:172.21.15.142:3300/0 conn(0x7f15b80c7e40 0x7f15b80b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:28.464 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.455+0000 7f15b77fe700 1 -- 172.21.15.112:0/3786949126 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1886742314 0 0) 0x7f15b8156240 con 0x7f15b80ca3e0 2023-01-24T03:36:28.464 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.455+0000 7f15b77fe700 1 -- 172.21.15.112:0/3786949126 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15a40033f0 con 0x7f15b80ca3e0 2023-01-24T03:36:28.464 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.455+0000 7f15b77fe700 1 -- 172.21.15.112:0/3786949126 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 191746354 0 0) 0x7f15a40033f0 con 0x7f15b80ca3e0 2023-01-24T03:36:28.464 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.455+0000 7f15b77fe700 1 -- 172.21.15.112:0/3786949126 >> v2:172.21.15.142:3300/0 conn(0x7f15b80c7e40 msgr2=0x7f15b80b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:36:28.464 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.455+0000 7f15b77fe700 1 --2- 172.21.15.112:0/3786949126 >> v2:172.21.15.142:3300/0 conn(0x7f15b80c7e40 0x7f15b80b0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:28.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.455+0000 7f15b77fe700 1 -- 172.21.15.112:0/3786949126 >> v1:172.21.15.142:6789/0 conn(0x7f15b8155da0 legacy=0x7f15b80ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:28.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.455+0000 7f15b77fe700 1 -- 172.21.15.112:0/3786949126 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b80b1730 con 0x7f15b80ca3e0 2023-01-24T03:36:28.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.455+0000 7f15b77fe700 1 -- 172.21.15.112:0/3786949126 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7f15b00017f0 con 0x7f15b80ca3e0 2023-01-24T03:36:28.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.455+0000 7f15b77fe700 1 -- 172.21.15.112:0/3786949126 >> v1:172.21.15.112:6789/0 conn(0x7f15b80ca3e0 legacy=0x7f15b80ca7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:28.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.455+0000 7f15b77fe700 1 --2- 172.21.15.112:0/3786949126 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15a40038e0 0x7f15a4003cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:28.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.455+0000 7f15b77fe700 1 --2- 172.21.15.112:0/3786949126 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f15a4004690 0x7f15a4004b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:28.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.455+0000 7f15b77fe700 1 --2- 172.21.15.112:0/3786949126 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15a4005040 0x7f15a400b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:28.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.455+0000 7f15b77fe700 1 -- 172.21.15.112:0/3786949126 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b80b1450 con 0x7f15a4004690 2023-01-24T03:36:28.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.455+0000 7f15b77fe700 1 -- 172.21.15.112:0/3786949126 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15a400bfe0 con 0x7f15a40038e0 2023-01-24T03:36:28.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.455+0000 7f15b77fe700 1 -- 172.21.15.112:0/3786949126 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15a400c1a0 con 0x7f15a4005040 2023-01-24T03:36:28.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.455+0000 7f15b7fff700 1 --2- 172.21.15.112:0/3786949126 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f15a4004690 0x7f15a4004b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:28.467 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.455+0000 7f15bc88c700 1 --2- 172.21.15.112:0/3786949126 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15a40038e0 0x7f15a4003cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:28.467 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.455+0000 7f15bd08d700 1 --2- 172.21.15.112:0/3786949126 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15a4005040 0x7f15a400b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:28.467 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.455+0000 7f15b7fff700 1 -- 172.21.15.112:0/3786949126 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15a4005040 msgr2=0x7f15a400b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:36:28.467 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.455+0000 7f15b7fff700 1 --2- 172.21.15.112:0/3786949126 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15a4005040 0x7f15a400b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:28.468 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.455+0000 7f15b7fff700 1 -- 172.21.15.112:0/3786949126 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15a40038e0 msgr2=0x7f15a4003cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:36:28.468 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.455+0000 7f15b7fff700 1 --2- 172.21.15.112:0/3786949126 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15a40038e0 0x7f15a4003cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:28.468 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.455+0000 7f15b7fff700 1 -- 172.21.15.112:0/3786949126 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a40033f0 con 0x7f15a4004690 2023-01-24T03:36:28.468 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.456+0000 7f15bd08d700 1 --2- 172.21.15.112:0/3786949126 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15a4005040 0x7f15a400b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:36:28.468 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.456+0000 7f15bd08d700 1 -- 172.21.15.112:0/3786949126 reap_dead start 2023-01-24T03:36:28.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.456+0000 7f15b7fff700 1 --2- 172.21.15.112:0/3786949126 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f15a4004690 0x7f15a4004b00 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f15ac001730 tx=0x7f15ac007bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d102873bb5f1f094 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:28.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.456+0000 7f15b77fe700 1 -- 172.21.15.112:0/3786949126 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15ac012440 con 0x7f15a4004690 2023-01-24T03:36:28.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.456+0000 7f15b77fe700 1 -- 172.21.15.112:0/3786949126 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15ac0125a0 con 0x7f15a4004690 2023-01-24T03:36:28.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.456+0000 7f15b77fe700 1 -- 172.21.15.112:0/3786949126 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15ac012960 con 0x7f15a4004690 2023-01-24T03:36:28.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.456+0000 7f15beaf0700 1 -- 172.21.15.112:0/3786949126 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f15a4004690 msgr2=0x7f15a4004b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:28.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.456+0000 7f15beaf0700 1 --2- 172.21.15.112:0/3786949126 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f15a4004690 0x7f15a4004b00 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f15ac001730 tx=0x7f15ac007bb0 comp rx=0 tx=0).stop 2023-01-24T03:36:28.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.456+0000 7f15beaf0700 1 -- 172.21.15.112:0/3786949126 shutdown_connections 2023-01-24T03:36:28.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.456+0000 7f15beaf0700 1 --2- 172.21.15.112:0/3786949126 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f15a4004690 0x7f15a4004b00 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:28.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.456+0000 7f15beaf0700 1 -- 172.21.15.112:0/3786949126 >> 172.21.15.112:0/3786949126 conn(0x7f15b805e2c0 msgr2=0x7f15b80cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:28.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.456+0000 7f15beaf0700 1 -- 172.21.15.112:0/3786949126 shutdown_connections 2023-01-24T03:36:28.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.456+0000 7f15beaf0700 1 -- 172.21.15.112:0/3786949126 wait complete. 2023-01-24T03:36:28.471 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.457+0000 7f15beaf0700 1 Processor -- start 2023-01-24T03:36:28.471 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.458+0000 7f15beaf0700 1 -- start start 2023-01-24T03:36:28.471 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.458+0000 7f15beaf0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15a4004690 0x7f15b8157030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:28.471 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.458+0000 7f15beaf0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f15b80bf840 0x7f15b8165500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:28.471 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.458+0000 7f15beaf0700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15b815bc40 0x7f15b814ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:28.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.458+0000 7f15beaf0700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b8150eb0 con 0x7f15b80bf840 2023-01-24T03:36:28.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.458+0000 7f15beaf0700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b8150370 con 0x7f15b815bc40 2023-01-24T03:36:28.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.458+0000 7f15beaf0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15b80af420 con 0x7f15a4004690 2023-01-24T03:36:28.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.458+0000 7f15b7fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f15b80bf840 0x7f15b8165500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:28.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.458+0000 7f15b7fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f15b80bf840 0x7f15b8165500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54084/0 (socket says 172.21.15.112:54084) 2023-01-24T03:36:28.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.458+0000 7f15b7fff700 1 -- 172.21.15.112:0/1033255370 learned_addr learned my addr 172.21.15.112:0/1033255370 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:36:28.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.459+0000 7f15bc88c700 1 --2- 172.21.15.112:0/1033255370 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15a4004690 0x7f15b8157030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:28.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.459+0000 7f15bd08d700 1 --2- 172.21.15.112:0/1033255370 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15b815bc40 0x7f15b814ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:28.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.459+0000 7f15b7fff700 1 -- 172.21.15.112:0/1033255370 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15a4004690 msgr2=0x7f15b8157030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:28.474 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.459+0000 7f15b7fff700 1 --2- 172.21.15.112:0/1033255370 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15a4004690 0x7f15b8157030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:28.474 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.459+0000 7f15b7fff700 1 -- 172.21.15.112:0/1033255370 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15b815bc40 msgr2=0x7f15b814ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:28.474 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.459+0000 7f15b7fff700 1 --2- 172.21.15.112:0/1033255370 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15b815bc40 0x7f15b814ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:28.474 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.459+0000 7f15b7fff700 1 -- 172.21.15.112:0/1033255370 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b814fdb0 con 0x7f15b80bf840 2023-01-24T03:36:28.474 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.459+0000 7f15b7fff700 1 --2- 172.21.15.112:0/1033255370 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f15b80bf840 0x7f15b8165500 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f15a400bfb0 tx=0x7f15ac015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:28.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.459+0000 7f15b4ff9700 1 -- 172.21.15.112:0/1033255370 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15ac0116b0 con 0x7f15b80bf840 2023-01-24T03:36:28.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.459+0000 7f15b4ff9700 1 -- 172.21.15.112:0/1033255370 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15ac011830 con 0x7f15b80bf840 2023-01-24T03:36:28.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.459+0000 7f15b4ff9700 1 -- 172.21.15.112:0/1033255370 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15ac011bf0 con 0x7f15b80bf840 2023-01-24T03:36:28.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.459+0000 7f15beaf0700 1 -- 172.21.15.112:0/1033255370 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15b8150090 con 0x7f15b80bf840 2023-01-24T03:36:28.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.459+0000 7f15beaf0700 1 -- 172.21.15.112:0/1033255370 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15b8150b40 con 0x7f15b80bf840 2023-01-24T03:36:28.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.460+0000 7f15beaf0700 1 -- 172.21.15.112:0/1033255370 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15b804bb80 con 0x7f15b80bf840 2023-01-24T03:36:28.476 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.460+0000 7f15b4ff9700 1 -- 172.21.15.112:0/1033255370 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f15ac010070 con 0x7f15b80bf840 2023-01-24T03:36:28.477 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.461+0000 7f15b4ff9700 1 --2- 172.21.15.112:0/1033255370 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f15980611f0 0x7f15980636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:28.477 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.461+0000 7f15b4ff9700 1 -- 172.21.15.112:0/1033255370 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 7211+0+0 (secure 0 0 0) 0x7f15ac07efa0 con 0x7f15b80bf840 2023-01-24T03:36:28.477 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.461+0000 7f15bc88c700 1 --2- 172.21.15.112:0/1033255370 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f15980611f0 0x7f15980636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:28.477 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.461+0000 7f15bc88c700 1 --2- 172.21.15.112:0/1033255370 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f15980611f0 0x7f15980636a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f15a8006d80 tx=0x7f15a8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:28.477 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.465+0000 7f15b4ff9700 1 -- 172.21.15.112:0/1033255370 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f15ac04ec50 con 0x7f15b80bf840 2023-01-24T03:36:28.700 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.698+0000 7f15beaf0700 1 -- 172.21.15.112:0/1033255370 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15b80ae550 con 0x7f15b80bf840 2023-01-24T03:36:28.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.698+0000 7f15b4ff9700 1 -- 172.21.15.112:0/1033255370 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+5357 (secure 0 0 0) 0x7f15ac017090 con 0x7f15b80bf840 2023-01-24T03:36:28.701 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:36:28.701 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:36:23.647869+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4907},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4907":{"gid":4907,"name":"d","rank":0,"incarnation":27,"state":"up:replay","state_seq":1,"addr":"172.21.15.142:6835/824478388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":824478388},{"type":"v1","addr":"172.21.15.142:6835","nonce":824478388}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:36:28.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.701+0000 7f15beaf0700 1 -- 172.21.15.112:0/1033255370 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f15980611f0 msgr2=0x7f15980636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:28.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.701+0000 7f15beaf0700 1 --2- 172.21.15.112:0/1033255370 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f15980611f0 0x7f15980636a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f15a8006d80 tx=0x7f15a8008040 comp rx=0 tx=0).stop 2023-01-24T03:36:28.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.701+0000 7f15beaf0700 1 -- 172.21.15.112:0/1033255370 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f15b80bf840 msgr2=0x7f15b8165500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:28.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.701+0000 7f15beaf0700 1 --2- 172.21.15.112:0/1033255370 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f15b80bf840 0x7f15b8165500 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f15a400bfb0 tx=0x7f15ac015000 comp rx=0 tx=0).stop 2023-01-24T03:36:28.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.701+0000 7f15beaf0700 1 -- 172.21.15.112:0/1033255370 shutdown_connections 2023-01-24T03:36:28.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.701+0000 7f15beaf0700 1 --2- 172.21.15.112:0/1033255370 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f15980611f0 0x7f15980636a0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:28.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.701+0000 7f15beaf0700 1 --2- 172.21.15.112:0/1033255370 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f15b815bc40 0x7f15b814ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:28.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.701+0000 7f15beaf0700 1 --2- 172.21.15.112:0/1033255370 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f15a4004690 0x7f15b8157030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:28.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.701+0000 7f15beaf0700 1 --2- 172.21.15.112:0/1033255370 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f15b80bf840 0x7f15b8165500 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:28.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.701+0000 7f15beaf0700 1 -- 172.21.15.112:0/1033255370 >> 172.21.15.112:0/1033255370 conn(0x7f15b805e2c0 msgr2=0x7f15b8069360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:28.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.701+0000 7f15beaf0700 1 -- 172.21.15.112:0/1033255370 shutdown_connections 2023-01-24T03:36:28.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:28.701+0000 7f15beaf0700 1 -- 172.21.15.112:0/1033255370 wait complete. 2023-01-24T03:36:28.706 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 27 2023-01-24T03:36:28.715 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:36:29.647 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:36:29.650 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:36:30.715 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:36:30.840 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.838+0000 7f0156d01700 1 Processor -- start 2023-01-24T03:36:30.840 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.838+0000 7f0156d01700 1 -- start start 2023-01-24T03:36:30.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.839+0000 7f0156d01700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f01500c7e40 0x7f01500b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:30.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.839+0000 7f0156d01700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f01500ca3e0 0x7f01500ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:30.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.839+0000 7f0156d01700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0150156240 con 0x7f0150155da0 2023-01-24T03:36:30.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.839+0000 7f0156d01700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01500b1170 con 0x7f01500c7e40 2023-01-24T03:36:30.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.839+0000 7f0156d01700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01500b1450 con 0x7f01500ca3e0 2023-01-24T03:36:30.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.839+0000 7f0154a9d700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f01500c7e40 0x7f01500b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:30.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.839+0000 7f014ffff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f0150155da0 0x7f01500ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35080/0 (socket says 172.21.15.112:35080) 2023-01-24T03:36:30.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.839+0000 7f014ffff700 1 -- 172.21.15.112:0/2167011114 learned_addr learned my addr 172.21.15.112:0/2167011114 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:36:30.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.839+0000 7f015529e700 1 --2- 172.21.15.112:0/2167011114 >> v2:172.21.15.142:3300/0 conn(0x7f01500ca3e0 0x7f01500ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:30.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.839+0000 7f0154a9d700 1 -- 172.21.15.112:0/2167011114 >> v2:172.21.15.142:3300/0 conn(0x7f01500ca3e0 msgr2=0x7f01500ca7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:36:30.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.839+0000 7f0154a9d700 1 --2- 172.21.15.112:0/2167011114 >> v2:172.21.15.142:3300/0 conn(0x7f01500ca3e0 0x7f01500ca7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:30.851 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.839+0000 7f0154a9d700 1 -- 172.21.15.112:0/2167011114 >> v1:172.21.15.112:6789/0 conn(0x7f0150155da0 legacy=0x7f01500ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:30.851 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.839+0000 7f0154a9d700 1 -- 172.21.15.112:0/2167011114 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01500b1730 con 0x7f01500c7e40 2023-01-24T03:36:30.851 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.839+0000 7f014f7fe700 1 -- 172.21.15.112:0/2167011114 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 674933655 0 0) 0x7f0150156240 con 0x7f0150155da0 2023-01-24T03:36:30.851 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.840+0000 7f0154a9d700 1 --2- 172.21.15.112:0/2167011114 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01500c7e40 0x7f01500b0aa0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f015005e810 tx=0x7f0148009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=421aff7b6c8b14f5 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:30.851 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.840+0000 7f014f7fe700 1 -- 172.21.15.112:0/2167011114 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0148011630 con 0x7f01500c7e40 2023-01-24T03:36:30.852 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.840+0000 7f014f7fe700 1 -- 172.21.15.112:0/2167011114 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0148011790 con 0x7f01500c7e40 2023-01-24T03:36:30.852 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.840+0000 7f014f7fe700 1 -- 172.21.15.112:0/2167011114 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0148011b30 con 0x7f01500c7e40 2023-01-24T03:36:30.852 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.840+0000 7f0156d01700 1 -- 172.21.15.112:0/2167011114 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01500c7e40 msgr2=0x7f01500b0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:30.852 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.840+0000 7f0156d01700 1 --2- 172.21.15.112:0/2167011114 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01500c7e40 0x7f01500b0aa0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f015005e810 tx=0x7f0148009060 comp rx=0 tx=0).stop 2023-01-24T03:36:30.853 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.840+0000 7f0156d01700 1 -- 172.21.15.112:0/2167011114 shutdown_connections 2023-01-24T03:36:30.853 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.840+0000 7f0156d01700 1 --2- 172.21.15.112:0/2167011114 >> v2:172.21.15.142:3300/0 conn(0x7f01500ca3e0 0x7f01500ca7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:30.853 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.840+0000 7f0156d01700 1 --2- 172.21.15.112:0/2167011114 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01500c7e40 0x7f01500b0aa0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:30.853 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.840+0000 7f0156d01700 1 -- 172.21.15.112:0/2167011114 >> 172.21.15.112:0/2167011114 conn(0x7f015005e2c0 msgr2=0x7f01500cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:30.853 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.840+0000 7f0156d01700 1 -- 172.21.15.112:0/2167011114 shutdown_connections 2023-01-24T03:36:30.854 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.840+0000 7f0156d01700 1 -- 172.21.15.112:0/2167011114 wait complete. 2023-01-24T03:36:30.854 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.841+0000 7f0156d01700 1 Processor -- start 2023-01-24T03:36:30.854 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.841+0000 7f0156d01700 1 -- start start 2023-01-24T03:36:30.854 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.842+0000 7f0156d01700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01500c7e40 0x7f015014c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:30.854 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.842+0000 7f0156d01700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f01500ca3e0 0x7f01500c0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:30.855 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.842+0000 7f0156d01700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0150155da0 0x7f01500c0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:30.855 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.842+0000 7f0156d01700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01500c2840 con 0x7f01500c7e40 2023-01-24T03:36:30.855 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.842+0000 7f0156d01700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01500c19c0 con 0x7f01500ca3e0 2023-01-24T03:36:30.855 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.842+0000 7f0156d01700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01500c0fc0 con 0x7f0150155da0 2023-01-24T03:36:30.855 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.842+0000 7f0154a9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01500c7e40 0x7f015014c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:30.855 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.842+0000 7f0154a9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01500c7e40 0x7f015014c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54096/0 (socket says 172.21.15.112:54096) 2023-01-24T03:36:30.856 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.842+0000 7f0154a9d700 1 -- 172.21.15.112:0/3565054883 learned_addr learned my addr 172.21.15.112:0/3565054883 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:36:30.856 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.842+0000 7f014ffff700 1 --2- 172.21.15.112:0/3565054883 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f01500ca3e0 0x7f01500c0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:30.856 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.842+0000 7f015529e700 1 --2- 172.21.15.112:0/3565054883 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0150155da0 0x7f01500c0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:30.856 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.842+0000 7f0154a9d700 1 -- 172.21.15.112:0/3565054883 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0150155da0 msgr2=0x7f01500c0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:30.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.842+0000 7f0154a9d700 1 --2- 172.21.15.112:0/3565054883 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0150155da0 0x7f01500c0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:30.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.842+0000 7f0154a9d700 1 -- 172.21.15.112:0/3565054883 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f01500ca3e0 msgr2=0x7f01500c0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:30.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.842+0000 7f0154a9d700 1 --2- 172.21.15.112:0/3565054883 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f01500ca3e0 0x7f01500c0250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:30.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.842+0000 7f0154a9d700 1 -- 172.21.15.112:0/3565054883 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01480092a0 con 0x7f01500c7e40 2023-01-24T03:36:30.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.842+0000 7f015529e700 1 --2- 172.21.15.112:0/3565054883 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0150155da0 0x7f01500c0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:36:30.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.843+0000 7f0154a9d700 1 --2- 172.21.15.112:0/3565054883 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01500c7e40 0x7f015014c1c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f0150089340 tx=0x7f0148017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:30.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.843+0000 7f014cff9700 1 -- 172.21.15.112:0/3565054883 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0148011900 con 0x7f01500c7e40 2023-01-24T03:36:30.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.843+0000 7f014cff9700 1 -- 172.21.15.112:0/3565054883 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f014801f070 con 0x7f01500c7e40 2023-01-24T03:36:30.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.843+0000 7f014cff9700 1 -- 172.21.15.112:0/3565054883 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014801b430 con 0x7f01500c7e40 2023-01-24T03:36:30.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.843+0000 7f0156d01700 1 -- 172.21.15.112:0/3565054883 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01500c0030 con 0x7f01500c7e40 2023-01-24T03:36:30.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.843+0000 7f0156d01700 1 -- 172.21.15.112:0/3565054883 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01500c2280 con 0x7f01500c7e40 2023-01-24T03:36:30.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.844+0000 7f0156d01700 1 -- 172.21.15.112:0/3565054883 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01500caf00 con 0x7f01500c7e40 2023-01-24T03:36:30.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.844+0000 7f014cff9700 1 -- 172.21.15.112:0/3565054883 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f014800f050 con 0x7f01500c7e40 2023-01-24T03:36:30.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.844+0000 7f014cff9700 1 --2- 172.21.15.112:0/3565054883 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f01340611c0 0x7f0134063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:30.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.844+0000 7f014cff9700 1 -- 172.21.15.112:0/3565054883 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 7211+0+0 (secure 0 0 0) 0x7f014807eb20 con 0x7f01500c7e40 2023-01-24T03:36:30.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.844+0000 7f014ffff700 1 --2- 172.21.15.112:0/3565054883 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f01340611c0 0x7f0134063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:30.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.845+0000 7f014ffff700 1 --2- 172.21.15.112:0/3565054883 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f01340611c0 0x7f0134063670 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f01500c9b20 tx=0x7f0140007f30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:30.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:30.849+0000 7f014cff9700 1 -- 172.21.15.112:0/3565054883 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f014804e770 con 0x7f01500c7e40 2023-01-24T03:36:31.071 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:31.069+0000 7f0156d01700 1 -- 172.21.15.112:0/3565054883 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f015004bb80 con 0x7f01500c7e40 2023-01-24T03:36:31.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:31.070+0000 7f014cff9700 1 -- 172.21.15.112:0/3565054883 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+5357 (secure 0 0 0) 0x7f0148051d90 con 0x7f01500c7e40 2023-01-24T03:36:31.072 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:36:31.072 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:36:23.647869+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4907},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4907":{"gid":4907,"name":"d","rank":0,"incarnation":27,"state":"up:replay","state_seq":1,"addr":"172.21.15.142:6835/824478388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":824478388},{"type":"v1","addr":"172.21.15.142:6835","nonce":824478388}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:36:31.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:31.071+0000 7f0156d01700 1 -- 172.21.15.112:0/3565054883 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f01340611c0 msgr2=0x7f0134063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:31.075 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:31.071+0000 7f0156d01700 1 --2- 172.21.15.112:0/3565054883 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f01340611c0 0x7f0134063670 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f01500c9b20 tx=0x7f0140007f30 comp rx=0 tx=0).stop 2023-01-24T03:36:31.075 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:31.071+0000 7f0156d01700 1 -- 172.21.15.112:0/3565054883 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01500c7e40 msgr2=0x7f015014c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:31.075 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:31.071+0000 7f0156d01700 1 --2- 172.21.15.112:0/3565054883 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01500c7e40 0x7f015014c1c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f0150089340 tx=0x7f0148017000 comp rx=0 tx=0).stop 2023-01-24T03:36:31.075 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:31.072+0000 7f0156d01700 1 -- 172.21.15.112:0/3565054883 shutdown_connections 2023-01-24T03:36:31.075 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:31.072+0000 7f0156d01700 1 --2- 172.21.15.112:0/3565054883 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f01340611c0 0x7f0134063670 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:31.076 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:31.072+0000 7f0156d01700 1 --2- 172.21.15.112:0/3565054883 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01500c7e40 0x7f015014c1c0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:31.076 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:31.072+0000 7f0156d01700 1 --2- 172.21.15.112:0/3565054883 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0150155da0 0x7f01500c0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:31.076 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:31.072+0000 7f0156d01700 1 --2- 172.21.15.112:0/3565054883 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f01500ca3e0 0x7f01500c0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:31.076 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:31.072+0000 7f0156d01700 1 -- 172.21.15.112:0/3565054883 >> 172.21.15.112:0/3565054883 conn(0x7f015005e2c0 msgr2=0x7f01500c8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:31.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:31.072+0000 7f0156d01700 1 -- 172.21.15.112:0/3565054883 shutdown_connections 2023-01-24T03:36:31.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:31.072+0000 7f0156d01700 1 -- 172.21.15.112:0/3565054883 wait complete. 2023-01-24T03:36:31.077 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 27 2023-01-24T03:36:31.086 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:36:33.087 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:36:33.212 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.210+0000 7fcb754c3700 1 Processor -- start 2023-01-24T03:36:33.212 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.210+0000 7fcb754c3700 1 -- start start 2023-01-24T03:36:33.218 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.211+0000 7fcb754c3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcb700c7e40 0x7fcb700b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:33.218 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.211+0000 7fcb754c3700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fcb700ca3e0 0x7fcb700ca820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:33.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.211+0000 7fcb754c3700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb70156210 con 0x7fcb70155da0 2023-01-24T03:36:33.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.211+0000 7fcb754c3700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb700cad60 con 0x7fcb700ca3e0 2023-01-24T03:36:33.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.211+0000 7fcb754c3700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb700b1170 con 0x7fcb700c7e40 2023-01-24T03:36:33.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.211+0000 7fcb6e7fc700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fcb70155da0 0x7fcb700ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35094/0 (socket says 172.21.15.112:35094) 2023-01-24T03:36:33.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.211+0000 7fcb6e7fc700 1 -- 172.21.15.112:0/405809423 learned_addr learned my addr 172.21.15.112:0/405809423 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:36:33.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.211+0000 7fcb6effd700 1 --2- 172.21.15.112:0/405809423 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcb700c7e40 0x7fcb700b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:33.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.211+0000 7fcb6f7fe700 1 --2- 172.21.15.112:0/405809423 >> v2:172.21.15.142:3300/0 conn(0x7fcb700ca3e0 0x7fcb700ca820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:33.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.211+0000 7fcb6dffb700 1 -- 172.21.15.112:0/405809423 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3437557553 0 0) 0x7fcb70156210 con 0x7fcb70155da0 2023-01-24T03:36:33.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.211+0000 7fcb6dffb700 1 -- 172.21.15.112:0/405809423 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb5c0033f0 con 0x7fcb70155da0 2023-01-24T03:36:33.221 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.211+0000 7fcb6dffb700 1 -- 172.21.15.112:0/405809423 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1772286739 0 0) 0x7fcb5c0033f0 con 0x7fcb70155da0 2023-01-24T03:36:33.221 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.211+0000 7fcb6dffb700 1 -- 172.21.15.112:0/405809423 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcb700c7e40 msgr2=0x7fcb700b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:36:33.221 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.211+0000 7fcb6dffb700 1 --2- 172.21.15.112:0/405809423 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcb700c7e40 0x7fcb700b0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:33.221 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.211+0000 7fcb6dffb700 1 -- 172.21.15.112:0/405809423 >> v2:172.21.15.142:3300/0 conn(0x7fcb700ca3e0 msgr2=0x7fcb700ca820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:36:33.221 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.211+0000 7fcb6dffb700 1 --2- 172.21.15.112:0/405809423 >> v2:172.21.15.142:3300/0 conn(0x7fcb700ca3e0 0x7fcb700ca820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:33.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.211+0000 7fcb6dffb700 1 -- 172.21.15.112:0/405809423 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb700b1450 con 0x7fcb70155da0 2023-01-24T03:36:33.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.211+0000 7fcb6dffb700 1 -- 172.21.15.112:0/405809423 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7fcb64001470 con 0x7fcb70155da0 2023-01-24T03:36:33.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.211+0000 7fcb6dffb700 1 -- 172.21.15.112:0/405809423 >> v1:172.21.15.112:6789/0 conn(0x7fcb70155da0 legacy=0x7fcb700ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:33.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.211+0000 7fcb6dffb700 1 --2- 172.21.15.112:0/405809423 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb5c0038e0 0x7fcb5c003cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:33.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.211+0000 7fcb6dffb700 1 --2- 172.21.15.112:0/405809423 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcb5c004690 0x7fcb5c004b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:33.223 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.212+0000 7fcb6dffb700 1 --2- 172.21.15.112:0/405809423 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcb700c7e40 0x7fcb5c005040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:33.223 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.212+0000 7fcb6dffb700 1 -- 172.21.15.112:0/405809423 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb700cad60 con 0x7fcb5c004690 2023-01-24T03:36:33.223 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.212+0000 7fcb6dffb700 1 -- 172.21.15.112:0/405809423 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb700b1170 con 0x7fcb5c0038e0 2023-01-24T03:36:33.223 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.212+0000 7fcb6dffb700 1 -- 172.21.15.112:0/405809423 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5c00b8e0 con 0x7fcb700c7e40 2023-01-24T03:36:33.223 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.212+0000 7fcb6e7fc700 1 --2- 172.21.15.112:0/405809423 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcb5c004690 0x7fcb5c004b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:33.223 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.212+0000 7fcb6effd700 1 --2- 172.21.15.112:0/405809423 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb5c0038e0 0x7fcb5c003cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:33.224 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.212+0000 7fcb6f7fe700 1 --2- 172.21.15.112:0/405809423 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcb700c7e40 0x7fcb5c005040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:33.224 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.212+0000 7fcb6e7fc700 1 -- 172.21.15.112:0/405809423 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcb700c7e40 msgr2=0x7fcb5c005040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:36:33.224 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.212+0000 7fcb6e7fc700 1 --2- 172.21.15.112:0/405809423 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcb700c7e40 0x7fcb5c005040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:33.224 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.212+0000 7fcb6e7fc700 1 -- 172.21.15.112:0/405809423 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb5c0038e0 msgr2=0x7fcb5c003cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:36:33.224 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.212+0000 7fcb6e7fc700 1 --2- 172.21.15.112:0/405809423 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb5c0038e0 0x7fcb5c003cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:33.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.212+0000 7fcb6e7fc700 1 -- 172.21.15.112:0/405809423 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb5c0033f0 con 0x7fcb5c004690 2023-01-24T03:36:33.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.212+0000 7fcb6f7fe700 1 --2- 172.21.15.112:0/405809423 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcb700c7e40 0x7fcb5c005040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:36:33.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.212+0000 7fcb6e7fc700 1 --2- 172.21.15.112:0/405809423 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcb5c004690 0x7fcb5c004b00 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fcb6400c240 tx=0x7fcb64001e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef486033b429f0f7 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:33.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.212+0000 7fcb6dffb700 1 -- 172.21.15.112:0/405809423 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb6400cad0 con 0x7fcb5c004690 2023-01-24T03:36:33.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.212+0000 7fcb6dffb700 1 -- 172.21.15.112:0/405809423 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb6400cc30 con 0x7fcb5c004690 2023-01-24T03:36:33.226 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.212+0000 7fcb6dffb700 1 -- 172.21.15.112:0/405809423 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb640070d0 con 0x7fcb5c004690 2023-01-24T03:36:33.226 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.213+0000 7fcb754c3700 1 -- 172.21.15.112:0/405809423 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcb5c004690 msgr2=0x7fcb5c004b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:33.226 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.213+0000 7fcb754c3700 1 --2- 172.21.15.112:0/405809423 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcb5c004690 0x7fcb5c004b00 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fcb6400c240 tx=0x7fcb64001e90 comp rx=0 tx=0).stop 2023-01-24T03:36:33.226 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.213+0000 7fcb6f7fe700 1 -- 172.21.15.112:0/405809423 reap_dead start 2023-01-24T03:36:33.226 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.213+0000 7fcb754c3700 1 -- 172.21.15.112:0/405809423 shutdown_connections 2023-01-24T03:36:33.226 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.213+0000 7fcb754c3700 1 -- 172.21.15.112:0/405809423 >> 172.21.15.112:0/405809423 conn(0x7fcb7005e2c0 msgr2=0x7fcb700cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:33.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.213+0000 7fcb754c3700 1 -- 172.21.15.112:0/405809423 shutdown_connections 2023-01-24T03:36:33.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.213+0000 7fcb754c3700 1 -- 172.21.15.112:0/405809423 wait complete. 2023-01-24T03:36:33.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.214+0000 7fcb754c3700 1 Processor -- start 2023-01-24T03:36:33.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.214+0000 7fcb754c3700 1 -- start start 2023-01-24T03:36:33.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.215+0000 7fcb754c3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcb5c004690 0x7fcb70164ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:33.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.215+0000 7fcb754c3700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcb700c4a70 0x7fcb7014ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:33.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.215+0000 7fcb754c3700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb700c4ec0 0x7fcb7014f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:33.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.215+0000 7fcb754c3700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb700c2c00 con 0x7fcb700c4a70 2023-01-24T03:36:33.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.215+0000 7fcb754c3700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb700c5b40 con 0x7fcb700c4ec0 2023-01-24T03:36:33.229 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.215+0000 7fcb754c3700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb700c58a0 con 0x7fcb5c004690 2023-01-24T03:36:33.229 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.215+0000 7fcb6e7fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcb700c4a70 0x7fcb7014ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:33.229 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.215+0000 7fcb6e7fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcb700c4a70 0x7fcb7014ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54118/0 (socket says 172.21.15.112:54118) 2023-01-24T03:36:33.229 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.215+0000 7fcb6e7fc700 1 -- 172.21.15.112:0/509428268 learned_addr learned my addr 172.21.15.112:0/509428268 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:36:33.229 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.215+0000 7fcb6f7fe700 1 --2- 172.21.15.112:0/509428268 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb700c4ec0 0x7fcb7014f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:33.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.215+0000 7fcb6effd700 1 --2- 172.21.15.112:0/509428268 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcb5c004690 0x7fcb70164ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:33.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.215+0000 7fcb6e7fc700 1 -- 172.21.15.112:0/509428268 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcb5c004690 msgr2=0x7fcb70164ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:33.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.215+0000 7fcb6e7fc700 1 --2- 172.21.15.112:0/509428268 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcb5c004690 0x7fcb70164ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:33.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.215+0000 7fcb6e7fc700 1 -- 172.21.15.112:0/509428268 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb700c4ec0 msgr2=0x7fcb7014f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:33.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.215+0000 7fcb6e7fc700 1 --2- 172.21.15.112:0/509428268 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb700c4ec0 0x7fcb7014f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:33.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.215+0000 7fcb6e7fc700 1 -- 172.21.15.112:0/509428268 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb700c2230 con 0x7fcb700c4a70 2023-01-24T03:36:33.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.215+0000 7fcb6e7fc700 1 --2- 172.21.15.112:0/509428268 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcb700c4a70 0x7fcb7014ed60 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fcb64011000 tx=0x7fcb64015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:33.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.215+0000 7fcb56ffd700 1 -- 172.21.15.112:0/509428268 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb64000d00 con 0x7fcb700c4a70 2023-01-24T03:36:33.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.215+0000 7fcb56ffd700 1 -- 172.21.15.112:0/509428268 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb640076f0 con 0x7fcb700c4a70 2023-01-24T03:36:33.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.215+0000 7fcb754c3700 1 -- 172.21.15.112:0/509428268 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb700c2510 con 0x7fcb700c4a70 2023-01-24T03:36:33.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.215+0000 7fcb56ffd700 1 -- 172.21.15.112:0/509428268 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb640049c0 con 0x7fcb700c4a70 2023-01-24T03:36:33.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.215+0000 7fcb754c3700 1 -- 172.21.15.112:0/509428268 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb70152be0 con 0x7fcb700c4a70 2023-01-24T03:36:33.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.216+0000 7fcb754c3700 1 -- 172.21.15.112:0/509428268 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb700c2d70 con 0x7fcb700c4a70 2023-01-24T03:36:33.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.216+0000 7fcb56ffd700 1 -- 172.21.15.112:0/509428268 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fcb64025030 con 0x7fcb700c4a70 2023-01-24T03:36:33.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.217+0000 7fcb56ffd700 1 --2- 172.21.15.112:0/509428268 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fcb500611f0 0x7fcb500636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:33.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.217+0000 7fcb56ffd700 1 -- 172.21.15.112:0/509428268 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 7211+0+0 (secure 0 0 0) 0x7fcb6407e4d0 con 0x7fcb700c4a70 2023-01-24T03:36:33.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.217+0000 7fcb6effd700 1 --2- 172.21.15.112:0/509428268 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fcb500611f0 0x7fcb500636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:33.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.217+0000 7fcb6effd700 1 --2- 172.21.15.112:0/509428268 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fcb500611f0 0x7fcb500636a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fcb600094d0 tx=0x7fcb6000b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:33.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.221+0000 7fcb56ffd700 1 -- 172.21.15.112:0/509428268 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fcb6404e180 con 0x7fcb700c4a70 2023-01-24T03:36:33.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.442+0000 7fcb754c3700 1 -- 172.21.15.112:0/509428268 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb7004bb80 con 0x7fcb700c4a70 2023-01-24T03:36:33.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.442+0000 7fcb56ffd700 1 -- 172.21.15.112:0/509428268 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5361 (secure 0 0 0) 0x7fcb640517a0 con 0x7fcb700c4a70 2023-01-24T03:36:33.445 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:36:33.445 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:36:31.726255+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4907},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4907":{"gid":4907,"name":"d","rank":0,"incarnation":27,"state":"up:reconnect","state_seq":25,"addr":"172.21.15.142:6835/824478388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":824478388},{"type":"v1","addr":"172.21.15.142:6835","nonce":824478388}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:36:33.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.444+0000 7fcb754c3700 1 -- 172.21.15.112:0/509428268 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fcb500611f0 msgr2=0x7fcb500636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:33.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.444+0000 7fcb754c3700 1 --2- 172.21.15.112:0/509428268 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fcb500611f0 0x7fcb500636a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fcb600094d0 tx=0x7fcb6000b040 comp rx=0 tx=0).stop 2023-01-24T03:36:33.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.444+0000 7fcb754c3700 1 -- 172.21.15.112:0/509428268 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcb700c4a70 msgr2=0x7fcb7014ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:33.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.444+0000 7fcb754c3700 1 --2- 172.21.15.112:0/509428268 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcb700c4a70 0x7fcb7014ed60 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fcb64011000 tx=0x7fcb64015000 comp rx=0 tx=0).stop 2023-01-24T03:36:33.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.444+0000 7fcb754c3700 1 -- 172.21.15.112:0/509428268 shutdown_connections 2023-01-24T03:36:33.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.444+0000 7fcb754c3700 1 --2- 172.21.15.112:0/509428268 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fcb500611f0 0x7fcb500636a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:33.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.444+0000 7fcb754c3700 1 --2- 172.21.15.112:0/509428268 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb700c4ec0 0x7fcb7014f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:33.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.444+0000 7fcb754c3700 1 --2- 172.21.15.112:0/509428268 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcb5c004690 0x7fcb70164ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:33.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.444+0000 7fcb754c3700 1 --2- 172.21.15.112:0/509428268 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcb700c4a70 0x7fcb7014ed60 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:33.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.444+0000 7fcb754c3700 1 -- 172.21.15.112:0/509428268 >> 172.21.15.112:0/509428268 conn(0x7fcb7005e2c0 msgr2=0x7fcb70156700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:33.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.444+0000 7fcb754c3700 1 -- 172.21.15.112:0/509428268 shutdown_connections 2023-01-24T03:36:33.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:33.444+0000 7fcb754c3700 1 -- 172.21.15.112:0/509428268 wait complete. 2023-01-24T03:36:33.449 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 28 2023-01-24T03:36:33.458 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:36:35.459 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:36:35.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.583+0000 7f47ce168700 1 Processor -- start 2023-01-24T03:36:35.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.583+0000 7f47ce168700 1 -- start start 2023-01-24T03:36:35.588 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.584+0000 7f47ce168700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f47c80c7e40 0x7f47c80b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:35.588 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.584+0000 7f47ce168700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f47c80ca3e0 0x7f47c80ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:35.589 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.584+0000 7f47ce168700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47c8156240 con 0x7f47c8155da0 2023-01-24T03:36:35.589 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.584+0000 7f47ce168700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47c80b1170 con 0x7f47c80ca3e0 2023-01-24T03:36:35.589 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.584+0000 7f47ce168700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47c80b1450 con 0x7f47c80c7e40 2023-01-24T03:36:35.589 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.584+0000 7f47c7fff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f47c80ca3e0 0x7f47c80ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:35.590 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.584+0000 7f47c7fff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f47c80ca3e0 0x7f47c80ca7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54126/0 (socket says 172.21.15.112:54126) 2023-01-24T03:36:35.590 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.584+0000 7f47c7fff700 1 -- 172.21.15.112:0/2349552778 learned_addr learned my addr 172.21.15.112:0/2349552778 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:36:35.591 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.584+0000 7f47c77fe700 1 --2- 172.21.15.112:0/2349552778 >> v2:172.21.15.142:3300/0 conn(0x7f47c80c7e40 0x7f47c80b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:35.591 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.584+0000 7f47c7fff700 1 -- 172.21.15.112:0/2349552778 >> v2:172.21.15.142:3300/0 conn(0x7f47c80c7e40 msgr2=0x7f47c80b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:36:35.591 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.584+0000 7f47c7fff700 1 --2- 172.21.15.112:0/2349552778 >> v2:172.21.15.142:3300/0 conn(0x7f47c80c7e40 0x7f47c80b0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:35.591 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.584+0000 7f47c7fff700 1 -- 172.21.15.112:0/2349552778 >> v1:172.21.15.142:6789/0 conn(0x7f47c8155da0 legacy=0x7f47c80ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:36:35.592 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.584+0000 7f47c7fff700 1 -- 172.21.15.112:0/2349552778 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c80b1730 con 0x7f47c80ca3e0 2023-01-24T03:36:35.592 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.584+0000 7f47c7fff700 1 --2- 172.21.15.112:0/2349552778 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47c80ca3e0 0x7f47c80ca7e0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f47c8089cf0 tx=0x7f47c0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8efbddb43743d1f server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:35.592 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.584+0000 7f47c67fc700 1 -- 172.21.15.112:0/2349552778 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c000e070 con 0x7f47c80ca3e0 2023-01-24T03:36:35.592 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.585+0000 7f47c67fc700 1 -- 172.21.15.112:0/2349552778 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47c000ba10 con 0x7f47c80ca3e0 2023-01-24T03:36:35.592 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.585+0000 7f47c67fc700 1 -- 172.21.15.112:0/2349552778 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c00092e0 con 0x7f47c80ca3e0 2023-01-24T03:36:35.593 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.585+0000 7f47ce168700 1 -- 172.21.15.112:0/2349552778 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47c80ca3e0 msgr2=0x7f47c80ca7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:35.593 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.585+0000 7f47ce168700 1 --2- 172.21.15.112:0/2349552778 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47c80ca3e0 0x7f47c80ca7e0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f47c8089cf0 tx=0x7f47c0009840 comp rx=0 tx=0).stop 2023-01-24T03:36:35.593 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.585+0000 7f47ce168700 1 -- 172.21.15.112:0/2349552778 shutdown_connections 2023-01-24T03:36:35.593 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.585+0000 7f47ce168700 1 --2- 172.21.15.112:0/2349552778 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47c80ca3e0 0x7f47c80ca7e0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:35.594 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.585+0000 7f47ce168700 1 --2- 172.21.15.112:0/2349552778 >> v2:172.21.15.142:3300/0 conn(0x7f47c80c7e40 0x7f47c80b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:35.594 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.585+0000 7f47ce168700 1 -- 172.21.15.112:0/2349552778 >> 172.21.15.112:0/2349552778 conn(0x7f47c805e2c0 msgr2=0x7f47c80cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:35.594 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.585+0000 7f47ce168700 1 -- 172.21.15.112:0/2349552778 shutdown_connections 2023-01-24T03:36:35.594 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.585+0000 7f47ce168700 1 -- 172.21.15.112:0/2349552778 wait complete. 2023-01-24T03:36:35.594 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.586+0000 7f47ce168700 1 Processor -- start 2023-01-24T03:36:35.595 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.586+0000 7f47ce168700 1 -- start start 2023-01-24T03:36:35.597 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.587+0000 7f47ce168700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f47c80c7e40 0x7f47c80c2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:35.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.587+0000 7f47ce168700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f47c80ca3e0 0x7f47c8166e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:35.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.587+0000 7f47ce168700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47c8155da0 0x7f47c8151f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:35.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.587+0000 7f47ce168700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c80c47c0 con 0x7f47c8155da0 2023-01-24T03:36:35.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.587+0000 7f47ce168700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c80c3de0 con 0x7f47c80c7e40 2023-01-24T03:36:35.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.587+0000 7f47ce168700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47c8165a10 con 0x7f47c80ca3e0 2023-01-24T03:36:35.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.587+0000 7f47c7fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47c8155da0 0x7f47c8151f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:35.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.587+0000 7f47c7fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47c8155da0 0x7f47c8151f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54132/0 (socket says 172.21.15.112:54132) 2023-01-24T03:36:35.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.587+0000 7f47c7fff700 1 -- 172.21.15.112:0/2061105559 learned_addr learned my addr 172.21.15.112:0/2061105559 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:36:35.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.587+0000 7f47c77fe700 1 --2- 172.21.15.112:0/2061105559 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f47c80c7e40 0x7f47c80c2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:35.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.587+0000 7f47c6ffd700 1 --2- 172.21.15.112:0/2061105559 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f47c80ca3e0 0x7f47c8166e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:35.600 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.587+0000 7f47c7fff700 1 -- 172.21.15.112:0/2061105559 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f47c80ca3e0 msgr2=0x7f47c8166e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:35.600 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.587+0000 7f47c7fff700 1 --2- 172.21.15.112:0/2061105559 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f47c80ca3e0 0x7f47c8166e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:35.600 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.587+0000 7f47c7fff700 1 -- 172.21.15.112:0/2061105559 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f47c80c7e40 msgr2=0x7f47c80c2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:35.600 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.587+0000 7f47c7fff700 1 --2- 172.21.15.112:0/2061105559 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f47c80c7e40 0x7f47c80c2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:35.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.587+0000 7f47c7fff700 1 -- 172.21.15.112:0/2061105559 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c0009ae0 con 0x7f47c8155da0 2023-01-24T03:36:35.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.588+0000 7f47c7fff700 1 --2- 172.21.15.112:0/2061105559 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47c8155da0 0x7f47c8151f90 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f47c0009fd0 tx=0x7f47c0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:35.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.588+0000 7f47af7fe700 1 -- 172.21.15.112:0/2061105559 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c00032f0 con 0x7f47c8155da0 2023-01-24T03:36:35.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.588+0000 7f47af7fe700 1 -- 172.21.15.112:0/2061105559 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47c0009940 con 0x7f47c8155da0 2023-01-24T03:36:35.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.588+0000 7f47ce168700 1 -- 172.21.15.112:0/2061105559 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47c8166270 con 0x7f47c8155da0 2023-01-24T03:36:35.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.588+0000 7f47af7fe700 1 -- 172.21.15.112:0/2061105559 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c000be80 con 0x7f47c8155da0 2023-01-24T03:36:35.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.588+0000 7f47ce168700 1 -- 172.21.15.112:0/2061105559 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47c80c3b00 con 0x7f47c8155da0 2023-01-24T03:36:35.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.589+0000 7f47ce168700 1 -- 172.21.15.112:0/2061105559 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47c8166400 con 0x7f47c8155da0 2023-01-24T03:36:35.603 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.589+0000 7f47af7fe700 1 -- 172.21.15.112:0/2061105559 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f47c0023070 con 0x7f47c8155da0 2023-01-24T03:36:35.603 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.589+0000 7f47af7fe700 1 --2- 172.21.15.112:0/2061105559 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f47a80611c0 0x7f47a8063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:35.603 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.589+0000 7f47af7fe700 1 -- 172.21.15.112:0/2061105559 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 7211+0+0 (secure 0 0 0) 0x7f47c007f2f0 con 0x7f47c8155da0 2023-01-24T03:36:35.603 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.589+0000 7f47c77fe700 1 --2- 172.21.15.112:0/2061105559 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f47a80611c0 0x7f47a8063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:35.603 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.590+0000 7f47c77fe700 1 --2- 172.21.15.112:0/2061105559 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f47a80611c0 0x7f47a8063670 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f47b800bb10 tx=0x7f47b8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:35.604 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.594+0000 7f47af7fe700 1 -- 172.21.15.112:0/2061105559 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f47c004efa0 con 0x7f47c8155da0 2023-01-24T03:36:35.818 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.816+0000 7f47ce168700 1 -- 172.21.15.112:0/2061105559 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47c80c5df0 con 0x7f47c8155da0 2023-01-24T03:36:35.819 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.817+0000 7f47af7fe700 1 -- 172.21.15.112:0/2061105559 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5361 (secure 0 0 0) 0x7f47c00525c0 con 0x7f47c8155da0 2023-01-24T03:36:35.819 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:36:35.820 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:36:31.726255+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4907},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4907":{"gid":4907,"name":"d","rank":0,"incarnation":27,"state":"up:reconnect","state_seq":25,"addr":"172.21.15.142:6835/824478388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":824478388},{"type":"v1","addr":"172.21.15.142:6835","nonce":824478388}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:36:35.821 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.819+0000 7f47ce168700 1 -- 172.21.15.112:0/2061105559 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f47a80611c0 msgr2=0x7f47a8063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:35.821 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.819+0000 7f47ce168700 1 --2- 172.21.15.112:0/2061105559 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f47a80611c0 0x7f47a8063670 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f47b800bb10 tx=0x7f47b8008040 comp rx=0 tx=0).stop 2023-01-24T03:36:35.821 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.819+0000 7f47ce168700 1 -- 172.21.15.112:0/2061105559 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47c8155da0 msgr2=0x7f47c8151f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:35.822 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.819+0000 7f47ce168700 1 --2- 172.21.15.112:0/2061105559 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47c8155da0 0x7f47c8151f90 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f47c0009fd0 tx=0x7f47c0017000 comp rx=0 tx=0).stop 2023-01-24T03:36:35.822 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.819+0000 7f47ce168700 1 -- 172.21.15.112:0/2061105559 shutdown_connections 2023-01-24T03:36:35.822 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.819+0000 7f47ce168700 1 --2- 172.21.15.112:0/2061105559 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f47a80611c0 0x7f47a8063670 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:35.822 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.819+0000 7f47ce168700 1 --2- 172.21.15.112:0/2061105559 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47c8155da0 0x7f47c8151f90 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:35.822 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.819+0000 7f47ce168700 1 --2- 172.21.15.112:0/2061105559 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f47c80c7e40 0x7f47c80c2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:35.823 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.819+0000 7f47ce168700 1 --2- 172.21.15.112:0/2061105559 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f47c80ca3e0 0x7f47c8166e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:35.823 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.819+0000 7f47ce168700 1 -- 172.21.15.112:0/2061105559 >> 172.21.15.112:0/2061105559 conn(0x7f47c805e2c0 msgr2=0x7f47c80c8210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:35.823 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.819+0000 7f47ce168700 1 -- 172.21.15.112:0/2061105559 shutdown_connections 2023-01-24T03:36:35.823 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:35.819+0000 7f47ce168700 1 -- 172.21.15.112:0/2061105559 wait complete. 2023-01-24T03:36:35.824 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 28 2023-01-24T03:36:35.833 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:36:37.833 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:36:37.957 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.955+0000 7fed8ffff700 1 Processor -- start 2023-01-24T03:36:37.957 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.955+0000 7fed8ffff700 1 -- start start 2023-01-24T03:36:37.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.956+0000 7fed8ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed900c5680 0x7fed90156730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:37.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.956+0000 7fed8ffff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fed900c5a50 0x7fed90156c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:37.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.956+0000 7fed8ffff700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed900c3d10 con 0x7fed90155e20 2023-01-24T03:36:37.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.956+0000 7fed8ffff700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed900c4eb0 con 0x7fed900c5a50 2023-01-24T03:36:37.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.956+0000 7fed8ffff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed900c2e20 con 0x7fed900c5680 2023-01-24T03:36:37.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.956+0000 7fed8effd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed900c5680 0x7fed90156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:37.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.956+0000 7fed8f7fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fed900c5a50 0x7fed90156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:37.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.956+0000 7fed8effd700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed900c5680 0x7fed90156730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.112:41442/0 (socket says 172.21.15.112:41442) 2023-01-24T03:36:37.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.956+0000 7fed8f7fe700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fed900c5a50 0x7fed90156c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.112:46636/0 (socket says 172.21.15.112:46636) 2023-01-24T03:36:37.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.956+0000 7fed8effd700 1 -- 172.21.15.112:0/3381248208 learned_addr learned my addr 172.21.15.112:0/3381248208 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:36:37.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.956+0000 7fed8effd700 1 -- 172.21.15.112:0/3381248208 >> v2:172.21.15.142:3300/0 conn(0x7fed900c5a50 msgr2=0x7fed90156c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:36:37.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.956+0000 7fed8effd700 1 --2- 172.21.15.112:0/3381248208 >> v2:172.21.15.142:3300/0 conn(0x7fed900c5a50 0x7fed90156c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:37.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.956+0000 7fed8effd700 1 -- 172.21.15.112:0/3381248208 >> v1:172.21.15.142:6789/0 conn(0x7fed90155e20 legacy=0x7fed900cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:37.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.956+0000 7fed8effd700 1 -- 172.21.15.112:0/3381248208 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed900c3100 con 0x7fed900c5680 2023-01-24T03:36:37.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.956+0000 7fed8f7fe700 1 --2- 172.21.15.112:0/3381248208 >> v2:172.21.15.142:3300/0 conn(0x7fed900c5a50 0x7fed90156c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:36:37.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.957+0000 7fed8effd700 1 --2- 172.21.15.112:0/3381248208 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed900c5680 0x7fed90156730 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fed80009830 tx=0x7fed800090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=794852cb1148dca server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:37.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.957+0000 7fed8dffb700 1 -- 172.21.15.112:0/3381248208 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed80011630 con 0x7fed900c5680 2023-01-24T03:36:37.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.957+0000 7fed8dffb700 1 -- 172.21.15.112:0/3381248208 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed80011790 con 0x7fed900c5680 2023-01-24T03:36:37.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.957+0000 7fed8dffb700 1 -- 172.21.15.112:0/3381248208 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed80011b50 con 0x7fed900c5680 2023-01-24T03:36:37.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.957+0000 7fed8ffff700 1 -- 172.21.15.112:0/3381248208 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed900c5680 msgr2=0x7fed90156730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:37.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.957+0000 7fed8ffff700 1 --2- 172.21.15.112:0/3381248208 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed900c5680 0x7fed90156730 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fed80009830 tx=0x7fed800090f0 comp rx=0 tx=0).stop 2023-01-24T03:36:37.965 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.957+0000 7fed8ffff700 1 -- 172.21.15.112:0/3381248208 shutdown_connections 2023-01-24T03:36:37.965 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.957+0000 7fed8ffff700 1 --2- 172.21.15.112:0/3381248208 >> v2:172.21.15.142:3300/0 conn(0x7fed900c5a50 0x7fed90156c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:37.965 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.957+0000 7fed8ffff700 1 --2- 172.21.15.112:0/3381248208 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed900c5680 0x7fed90156730 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:37.965 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.957+0000 7fed8ffff700 1 -- 172.21.15.112:0/3381248208 >> 172.21.15.112:0/3381248208 conn(0x7fed900685b0 msgr2=0x7fed900689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:37.966 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.957+0000 7fed8ffff700 1 -- 172.21.15.112:0/3381248208 shutdown_connections 2023-01-24T03:36:37.966 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.957+0000 7fed8ffff700 1 -- 172.21.15.112:0/3381248208 wait complete. 2023-01-24T03:36:37.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.959+0000 7fed8ffff700 1 Processor -- start 2023-01-24T03:36:37.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.959+0000 7fed8ffff700 1 -- start start 2023-01-24T03:36:37.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.959+0000 7fed8ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed900c5680 0x7fed90154e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:37.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.959+0000 7fed8ffff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed900c5a50 0x7fed90165870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:37.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.959+0000 7fed8ffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fed90155e20 0x7fed90152280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:37.970 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.959+0000 7fed8ffff700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed900bf450 con 0x7fed900c5680 2023-01-24T03:36:37.970 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.959+0000 7fed8ffff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed90162da0 con 0x7fed90155e20 2023-01-24T03:36:37.970 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.959+0000 7fed8ffff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed90167550 con 0x7fed900c5a50 2023-01-24T03:36:37.970 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.959+0000 7fed8effd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed900c5680 0x7fed90154e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:37.970 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.959+0000 7fed8effd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed900c5680 0x7fed90154e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54146/0 (socket says 172.21.15.112:54146) 2023-01-24T03:36:37.970 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.960+0000 7fed8effd700 1 -- 172.21.15.112:0/677660336 learned_addr learned my addr 172.21.15.112:0/677660336 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:36:37.971 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.960+0000 7fed8f7fe700 1 --2- 172.21.15.112:0/677660336 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fed90155e20 0x7fed90152280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:37.971 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.960+0000 7fed8e7fc700 1 --2- 172.21.15.112:0/677660336 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed900c5a50 0x7fed90165870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:37.971 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.960+0000 7fed8effd700 1 -- 172.21.15.112:0/677660336 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed900c5a50 msgr2=0x7fed90165870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:37.971 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.960+0000 7fed8effd700 1 --2- 172.21.15.112:0/677660336 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed900c5a50 0x7fed90165870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:37.971 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.960+0000 7fed8effd700 1 -- 172.21.15.112:0/677660336 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fed90155e20 msgr2=0x7fed90152280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:37.971 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.960+0000 7fed8effd700 1 --2- 172.21.15.112:0/677660336 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fed90155e20 0x7fed90152280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:37.972 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.960+0000 7fed8effd700 1 -- 172.21.15.112:0/677660336 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed88009a20 con 0x7fed900c5680 2023-01-24T03:36:37.972 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.960+0000 7fed8effd700 1 --2- 172.21.15.112:0/677660336 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed900c5680 0x7fed90154e30 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fed80002000 tx=0x7fed80016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:37.972 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.960+0000 7fed76ffd700 1 -- 172.21.15.112:0/677660336 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed80011900 con 0x7fed900c5680 2023-01-24T03:36:37.972 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.960+0000 7fed76ffd700 1 -- 172.21.15.112:0/677660336 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed80020070 con 0x7fed900c5680 2023-01-24T03:36:37.973 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.960+0000 7fed8ffff700 1 -- 172.21.15.112:0/677660336 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed80009370 con 0x7fed900c5680 2023-01-24T03:36:37.973 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.960+0000 7fed76ffd700 1 -- 172.21.15.112:0/677660336 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed80027480 con 0x7fed900c5680 2023-01-24T03:36:37.973 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.960+0000 7fed8ffff700 1 -- 172.21.15.112:0/677660336 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed90166940 con 0x7fed900c5680 2023-01-24T03:36:37.973 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.961+0000 7fed8ffff700 1 -- 172.21.15.112:0/677660336 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed90165db0 con 0x7fed900c5680 2023-01-24T03:36:37.973 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.961+0000 7fed76ffd700 1 -- 172.21.15.112:0/677660336 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fed80017020 con 0x7fed900c5680 2023-01-24T03:36:37.974 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.962+0000 7fed76ffd700 1 --2- 172.21.15.112:0/677660336 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fed700611c0 0x7fed70063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:37.974 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.962+0000 7fed76ffd700 1 -- 172.21.15.112:0/677660336 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 7211+0+0 (secure 0 0 0) 0x7fed80022070 con 0x7fed900c5680 2023-01-24T03:36:37.974 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.962+0000 7fed8e7fc700 1 --2- 172.21.15.112:0/677660336 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fed700611c0 0x7fed70063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:37.974 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.962+0000 7fed8e7fc700 1 --2- 172.21.15.112:0/677660336 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fed700611c0 0x7fed70063670 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fed9014d6b0 tx=0x7fed84007d40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:37.974 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:37.966+0000 7fed76ffd700 1 -- 172.21.15.112:0/677660336 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fed8004f460 con 0x7fed900c5680 2023-01-24T03:36:38.187 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:38.185+0000 7fed8ffff700 1 -- 172.21.15.112:0/677660336 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed9005ea20 con 0x7fed900c5680 2023-01-24T03:36:38.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:38.186+0000 7fed76ffd700 1 -- 172.21.15.112:0/677660336 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+5358 (secure 0 0 0) 0x7fed80052a80 con 0x7fed900c5680 2023-01-24T03:36:38.188 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:36:38.188 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:36:36.738759+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4907},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4907":{"gid":4907,"name":"d","rank":0,"incarnation":27,"state":"up:rejoin","state_seq":27,"addr":"172.21.15.142:6835/824478388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":824478388},{"type":"v1","addr":"172.21.15.142:6835","nonce":824478388}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:36:38.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:38.187+0000 7fed8ffff700 1 -- 172.21.15.112:0/677660336 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fed700611c0 msgr2=0x7fed70063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:38.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:38.187+0000 7fed8ffff700 1 --2- 172.21.15.112:0/677660336 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fed700611c0 0x7fed70063670 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fed9014d6b0 tx=0x7fed84007d40 comp rx=0 tx=0).stop 2023-01-24T03:36:38.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:38.187+0000 7fed8ffff700 1 -- 172.21.15.112:0/677660336 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed900c5680 msgr2=0x7fed90154e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:38.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:38.187+0000 7fed8ffff700 1 --2- 172.21.15.112:0/677660336 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed900c5680 0x7fed90154e30 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fed80002000 tx=0x7fed80016000 comp rx=0 tx=0).stop 2023-01-24T03:36:38.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:38.188+0000 7fed8ffff700 1 -- 172.21.15.112:0/677660336 shutdown_connections 2023-01-24T03:36:38.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:38.188+0000 7fed8ffff700 1 --2- 172.21.15.112:0/677660336 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fed700611c0 0x7fed70063670 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:38.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:38.188+0000 7fed8ffff700 1 --2- 172.21.15.112:0/677660336 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed900c5680 0x7fed90154e30 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:38.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:38.188+0000 7fed8ffff700 1 --2- 172.21.15.112:0/677660336 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fed90155e20 0x7fed90152280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:38.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:38.188+0000 7fed8ffff700 1 --2- 172.21.15.112:0/677660336 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fed900c5a50 0x7fed90165870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:38.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:38.188+0000 7fed8ffff700 1 -- 172.21.15.112:0/677660336 >> 172.21.15.112:0/677660336 conn(0x7fed900685b0 msgr2=0x7fed90153b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:38.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:38.188+0000 7fed8ffff700 1 -- 172.21.15.112:0/677660336 shutdown_connections 2023-01-24T03:36:38.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:38.188+0000 7fed8ffff700 1 -- 172.21.15.112:0/677660336 wait complete. 2023-01-24T03:36:38.192 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 29 2023-01-24T03:36:38.202 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 29, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-01-24T03:30:20.971710+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 29, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4907": { "addr": "172.21.15.142:6835/824478388", "addrs": { "addrvec": [ { "addr": "172.21.15.142:6833", "nonce": 824478388, "type": "v2" }, { "addr": "172.21.15.142:6835", "nonce": 824478388, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4907, "incarnation": 27, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:rejoin", "state_seq": 27 } }, "last_failure": 0, "last_failure_osd_epoch": 46, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-01-24T03:36:36.738759+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4907 } } } ], "standbys": [ { "addr": "172.21.15.142:6837/2181345869", "addrs": { "addrvec": [ { "addr": "172.21.15.142:6836", "nonce": 2181345869, "type": "v2" }, { "addr": "172.21.15.142:6837", "nonce": 2181345869, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6835/330343927", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6834", "nonce": 330343927, "type": "v2" }, { "addr": "172.21.15.112:6835", "nonce": 330343927, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6837/12176817", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6836", "nonce": 12176817, "type": "v2" }, { "addr": "172.21.15.112:6837", "nonce": 12176817, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4276, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6839/1110983841", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6838", "nonce": 1110983841, "type": "v2" }, { "addr": "172.21.15.112:6839", "nonce": 1110983841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4281, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-01-24T03:36:40.203 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:36:40.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.326+0000 7f4dadb3f700 1 Processor -- start 2023-01-24T03:36:40.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.326+0000 7f4dadb3f700 1 -- start start 2023-01-24T03:36:40.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.327+0000 7f4dadb3f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4da80b0aa0 0x7f4da8155da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:40.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.327+0000 7f4dadb3f700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f4da80ca3e0 0x7f4da80ca840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:40.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.327+0000 7f4dadb3f700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4da80b0f40 con 0x7f4da80c7e40 2023-01-24T03:36:40.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.327+0000 7f4dadb3f700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4da80cad80 con 0x7f4da80ca3e0 2023-01-24T03:36:40.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.327+0000 7f4dadb3f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4da80b1170 con 0x7f4da80b0aa0 2023-01-24T03:36:40.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.327+0000 7f4da7fff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f4da80ca3e0 0x7f4da80ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:40.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.327+0000 7f4da77fe700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f4da80c7e40 0x7f4da80ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35146/0 (socket says 172.21.15.112:35146) 2023-01-24T03:36:40.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.327+0000 7f4da77fe700 1 -- 172.21.15.112:0/1625177840 learned_addr learned my addr 172.21.15.112:0/1625177840 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:36:40.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.327+0000 7f4da7fff700 1 -- 172.21.15.112:0/1625177840 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4da80b0aa0 msgr2=0x7f4da8155da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:36:40.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.327+0000 7f4da7fff700 1 --2- 172.21.15.112:0/1625177840 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4da80b0aa0 0x7f4da8155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:40.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.327+0000 7f4da7fff700 1 -- 172.21.15.112:0/1625177840 >> v1:172.21.15.112:6789/0 conn(0x7f4da80c7e40 legacy=0x7f4da80ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:40.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.327+0000 7f4da7fff700 1 -- 172.21.15.112:0/1625177840 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da80b1450 con 0x7f4da80ca3e0 2023-01-24T03:36:40.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.327+0000 7f4da67fc700 1 -- 172.21.15.112:0/1625177840 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3474458752 0 0) 0x7f4da80b0f40 con 0x7f4da80c7e40 2023-01-24T03:36:40.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.327+0000 7f4da7fff700 1 --2- 172.21.15.112:0/1625177840 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4da80ca3e0 0x7f4da80ca840 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f4da0009e60 tx=0x7f4da0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=584c45108d679ba server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:40.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.327+0000 7f4da67fc700 1 -- 172.21.15.112:0/1625177840 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da000e070 con 0x7f4da80ca3e0 2023-01-24T03:36:40.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.327+0000 7f4da67fc700 1 -- 172.21.15.112:0/1625177840 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4da000bc30 con 0x7f4da80ca3e0 2023-01-24T03:36:40.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.327+0000 7f4da67fc700 1 -- 172.21.15.112:0/1625177840 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da0012430 con 0x7f4da80ca3e0 2023-01-24T03:36:40.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.328+0000 7f4dadb3f700 1 -- 172.21.15.112:0/1625177840 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4da80ca3e0 msgr2=0x7f4da80ca840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:40.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.328+0000 7f4dadb3f700 1 --2- 172.21.15.112:0/1625177840 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4da80ca3e0 0x7f4da80ca840 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f4da0009e60 tx=0x7f4da0009780 comp rx=0 tx=0).stop 2023-01-24T03:36:40.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.328+0000 7f4dadb3f700 1 -- 172.21.15.112:0/1625177840 shutdown_connections 2023-01-24T03:36:40.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.328+0000 7f4dadb3f700 1 --2- 172.21.15.112:0/1625177840 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4da80ca3e0 0x7f4da80ca840 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:40.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.328+0000 7f4dadb3f700 1 --2- 172.21.15.112:0/1625177840 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4da80b0aa0 0x7f4da8155da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:40.335 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.328+0000 7f4dadb3f700 1 -- 172.21.15.112:0/1625177840 >> 172.21.15.112:0/1625177840 conn(0x7f4da805e2c0 msgr2=0x7f4da80cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:40.335 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.328+0000 7f4dadb3f700 1 -- 172.21.15.112:0/1625177840 shutdown_connections 2023-01-24T03:36:40.335 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.328+0000 7f4dadb3f700 1 -- 172.21.15.112:0/1625177840 wait complete. 2023-01-24T03:36:40.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.329+0000 7f4dadb3f700 1 Processor -- start 2023-01-24T03:36:40.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.329+0000 7f4dadb3f700 1 -- start start 2023-01-24T03:36:40.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.330+0000 7f4dadb3f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4da80b0aa0 0x7f4da8141eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:40.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.330+0000 7f4dadb3f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4da80c7e40 0x7f4da814c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:40.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.330+0000 7f4dadb3f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4da80ca3e0 0x7f4da814c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:40.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.330+0000 7f4dadb3f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da81693f0 con 0x7f4da80c7e40 2023-01-24T03:36:40.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.330+0000 7f4dadb3f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da8168840 con 0x7f4da80ca3e0 2023-01-24T03:36:40.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.330+0000 7f4dadb3f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4da814b180 con 0x7f4da80b0aa0 2023-01-24T03:36:40.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.330+0000 7f4da6ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4da80c7e40 0x7f4da814c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:40.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.330+0000 7f4da6ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4da80c7e40 0x7f4da814c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54166/0 (socket says 172.21.15.112:54166) 2023-01-24T03:36:40.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.330+0000 7f4da6ffd700 1 -- 172.21.15.112:0/2422771531 learned_addr learned my addr 172.21.15.112:0/2422771531 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:36:40.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.330+0000 7f4da7fff700 1 --2- 172.21.15.112:0/2422771531 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4da80ca3e0 0x7f4da814c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:40.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.330+0000 7f4da77fe700 1 --2- 172.21.15.112:0/2422771531 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4da80b0aa0 0x7f4da8141eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:40.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.330+0000 7f4da6ffd700 1 -- 172.21.15.112:0/2422771531 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4da80b0aa0 msgr2=0x7f4da8141eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:40.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.330+0000 7f4da6ffd700 1 --2- 172.21.15.112:0/2422771531 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4da80b0aa0 0x7f4da8141eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:40.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.330+0000 7f4da6ffd700 1 -- 172.21.15.112:0/2422771531 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4da80ca3e0 msgr2=0x7f4da814c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:40.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.330+0000 7f4da6ffd700 1 --2- 172.21.15.112:0/2422771531 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4da80ca3e0 0x7f4da814c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:40.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.330+0000 7f4da6ffd700 1 -- 172.21.15.112:0/2422771531 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da0009a20 con 0x7f4da80c7e40 2023-01-24T03:36:40.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.330+0000 7f4da6ffd700 1 --2- 172.21.15.112:0/2422771531 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4da80c7e40 0x7f4da814c4b0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f4d9800b490 tx=0x7f4d98009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:40.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.330+0000 7f4d8f7fe700 1 -- 172.21.15.112:0/2422771531 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9800f510 con 0x7f4da80c7e40 2023-01-24T03:36:40.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.331+0000 7f4d8f7fe700 1 -- 172.21.15.112:0/2422771531 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d9800f670 con 0x7f4da80c7e40 2023-01-24T03:36:40.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.331+0000 7f4dadb3f700 1 -- 172.21.15.112:0/2422771531 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4da8142f60 con 0x7f4da80c7e40 2023-01-24T03:36:40.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.331+0000 7f4d8f7fe700 1 -- 172.21.15.112:0/2422771531 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9800fa30 con 0x7f4da80c7e40 2023-01-24T03:36:40.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.331+0000 7f4dadb3f700 1 -- 172.21.15.112:0/2422771531 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4da8168d30 con 0x7f4da80c7e40 2023-01-24T03:36:40.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.332+0000 7f4dadb3f700 1 -- 172.21.15.112:0/2422771531 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4da80cab80 con 0x7f4da80c7e40 2023-01-24T03:36:40.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.332+0000 7f4d8f7fe700 1 -- 172.21.15.112:0/2422771531 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f4d9800f070 con 0x7f4da80c7e40 2023-01-24T03:36:40.343 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.332+0000 7f4d8f7fe700 1 --2- 172.21.15.112:0/2422771531 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f4d880611c0 0x7f4d88063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:40.343 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.332+0000 7f4d8f7fe700 1 -- 172.21.15.112:0/2422771531 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 7211+0+0 (secure 0 0 0) 0x7f4d9807d320 con 0x7f4da80c7e40 2023-01-24T03:36:40.343 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.332+0000 7f4da77fe700 1 --2- 172.21.15.112:0/2422771531 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f4d880611c0 0x7f4d88063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:40.343 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.333+0000 7f4da77fe700 1 --2- 172.21.15.112:0/2422771531 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f4d880611c0 0x7f4d88063670 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c0011e0 tx=0x7f4d9c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:40.344 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.336+0000 7f4d8f7fe700 1 -- 172.21.15.112:0/2422771531 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f4d9804d050 con 0x7f4da80c7e40 2023-01-24T03:36:40.558 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.556+0000 7f4dadb3f700 1 -- 172.21.15.112:0/2422771531 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4da80caf50 con 0x7f4da80c7e40 2023-01-24T03:36:40.559 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.557+0000 7f4d8f7fe700 1 -- 172.21.15.112:0/2422771531 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+5358 (secure 0 0 0) 0x7f4d98050670 con 0x7f4da80c7e40 2023-01-24T03:36:40.559 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:36:40.559 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:36:36.738759+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4907},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4907":{"gid":4907,"name":"d","rank":0,"incarnation":27,"state":"up:rejoin","state_seq":27,"addr":"172.21.15.142:6835/824478388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":824478388},{"type":"v1","addr":"172.21.15.142:6835","nonce":824478388}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:36:40.561 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.559+0000 7f4dadb3f700 1 -- 172.21.15.112:0/2422771531 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f4d880611c0 msgr2=0x7f4d88063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:40.561 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.559+0000 7f4dadb3f700 1 --2- 172.21.15.112:0/2422771531 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f4d880611c0 0x7f4d88063670 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c0011e0 tx=0x7f4d9c009040 comp rx=0 tx=0).stop 2023-01-24T03:36:40.561 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.559+0000 7f4dadb3f700 1 -- 172.21.15.112:0/2422771531 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4da80c7e40 msgr2=0x7f4da814c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:40.562 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.559+0000 7f4dadb3f700 1 --2- 172.21.15.112:0/2422771531 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4da80c7e40 0x7f4da814c4b0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f4d9800b490 tx=0x7f4d98009060 comp rx=0 tx=0).stop 2023-01-24T03:36:40.562 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.559+0000 7f4dadb3f700 1 -- 172.21.15.112:0/2422771531 shutdown_connections 2023-01-24T03:36:40.562 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.559+0000 7f4dadb3f700 1 --2- 172.21.15.112:0/2422771531 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f4d880611c0 0x7f4d88063670 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:40.562 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.559+0000 7f4dadb3f700 1 --2- 172.21.15.112:0/2422771531 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4da80ca3e0 0x7f4da814c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:40.562 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.559+0000 7f4dadb3f700 1 --2- 172.21.15.112:0/2422771531 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4da80b0aa0 0x7f4da8141eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:40.563 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.559+0000 7f4dadb3f700 1 --2- 172.21.15.112:0/2422771531 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4da80c7e40 0x7f4da814c4b0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:40.563 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.559+0000 7f4dadb3f700 1 -- 172.21.15.112:0/2422771531 >> 172.21.15.112:0/2422771531 conn(0x7f4da805e2c0 msgr2=0x7f4da8068b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:40.563 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.559+0000 7f4dadb3f700 1 -- 172.21.15.112:0/2422771531 shutdown_connections 2023-01-24T03:36:40.563 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:40.559+0000 7f4dadb3f700 1 -- 172.21.15.112:0/2422771531 wait complete. 2023-01-24T03:36:40.564 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 29 2023-01-24T03:36:40.573 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:36:42.573 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:36:42.698 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.696+0000 7f5a7be42700 1 Processor -- start 2023-01-24T03:36:42.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.696+0000 7f5a7be42700 1 -- start start 2023-01-24T03:36:42.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.697+0000 7f5a7be42700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f5a7406eec0 0x7f5a740bdf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:42.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.697+0000 7f5a7be42700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a7406a000 con 0x7f5a7406e040 2023-01-24T03:36:42.702 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.697+0000 7f5a7be42700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a7406f420 con 0x7f5a74067fe0 2023-01-24T03:36:42.702 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.697+0000 7f5a7be42700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a740c47b0 con 0x7f5a7406eec0 2023-01-24T03:36:42.702 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.697+0000 7f5a793dd700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f5a7406e040 0x7f5a740bd850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35164/0 (socket says 172.21.15.112:35164) 2023-01-24T03:36:42.702 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.697+0000 7f5a793dd700 1 -- 172.21.15.112:0/784929909 learned_addr learned my addr 172.21.15.112:0/784929909 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:36:42.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.697+0000 7f5a7a3df700 1 --2- 172.21.15.112:0/784929909 >> v2:172.21.15.142:3300/0 conn(0x7f5a7406eec0 0x7f5a740bdf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:42.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.697+0000 7f5a78bdc700 1 -- 172.21.15.112:0/784929909 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2337015708 0 0) 0x7f5a7406a000 con 0x7f5a7406e040 2023-01-24T03:36:42.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.698+0000 7f5a78bdc700 1 -- 172.21.15.112:0/784929909 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a600033f0 con 0x7f5a7406e040 2023-01-24T03:36:42.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.698+0000 7f5a7a3df700 1 -- 172.21.15.112:0/784929909 >> v1:172.21.15.142:6789/0 conn(0x7f5a74067fe0 legacy=0x7f5a7406d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:42.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.698+0000 7f5a78bdc700 1 -- 172.21.15.112:0/784929909 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3609732159 0 0) 0x7f5a7406f420 con 0x7f5a74067fe0 2023-01-24T03:36:42.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.698+0000 7f5a7a3df700 1 -- 172.21.15.112:0/784929909 >> v1:172.21.15.112:6789/0 conn(0x7f5a7406e040 legacy=0x7f5a740bd850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:42.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.698+0000 7f5a7a3df700 1 -- 172.21.15.112:0/784929909 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a740693f0 con 0x7f5a7406eec0 2023-01-24T03:36:42.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.698+0000 7f5a7a3df700 1 --2- 172.21.15.112:0/784929909 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a7406eec0 0x7f5a740bdf60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c009760 tx=0x7f5a6c009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a3101cea618d5f3 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:42.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.698+0000 7f5a78bdc700 1 -- 172.21.15.112:0/784929909 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a68001470 con 0x7f5a7406eec0 2023-01-24T03:36:42.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.698+0000 7f5a78bdc700 1 -- 172.21.15.112:0/784929909 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a6c011440 con 0x7f5a7406eec0 2023-01-24T03:36:42.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.698+0000 7f5a78bdc700 1 -- 172.21.15.112:0/784929909 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6c0117c0 con 0x7f5a7406eec0 2023-01-24T03:36:42.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.699+0000 7f5a7be42700 1 -- 172.21.15.112:0/784929909 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a7406eec0 msgr2=0x7f5a740bdf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:42.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.699+0000 7f5a7be42700 1 --2- 172.21.15.112:0/784929909 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a7406eec0 0x7f5a740bdf60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c009760 tx=0x7f5a6c009060 comp rx=0 tx=0).stop 2023-01-24T03:36:42.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.699+0000 7f5a7be42700 1 -- 172.21.15.112:0/784929909 shutdown_connections 2023-01-24T03:36:42.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.699+0000 7f5a7be42700 1 --2- 172.21.15.112:0/784929909 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a7406eec0 0x7f5a740bdf60 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:42.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.699+0000 7f5a7be42700 1 -- 172.21.15.112:0/784929909 >> 172.21.15.112:0/784929909 conn(0x7f5a740c8940 msgr2=0x7f5a740c4910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:42.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.699+0000 7f5a7be42700 1 -- 172.21.15.112:0/784929909 shutdown_connections 2023-01-24T03:36:42.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.699+0000 7f5a7be42700 1 -- 172.21.15.112:0/784929909 wait complete. 2023-01-24T03:36:42.709 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.700+0000 7f5a7be42700 1 Processor -- start 2023-01-24T03:36:42.709 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.700+0000 7f5a7be42700 1 -- start start 2023-01-24T03:36:42.710 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.700+0000 7f5a7be42700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5a74067fe0 0x7f5a740ba9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:42.710 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.700+0000 7f5a7be42700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a7406e040 0x7f5a740baf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:42.710 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.700+0000 7f5a7be42700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a7406eec0 0x7f5a74147f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:42.710 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.700+0000 7f5a7be42700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a74147c60 con 0x7f5a74067fe0 2023-01-24T03:36:42.710 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.700+0000 7f5a7be42700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a741466b0 con 0x7f5a7406eec0 2023-01-24T03:36:42.711 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.700+0000 7f5a7be42700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a740c24b0 con 0x7f5a7406e040 2023-01-24T03:36:42.711 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.701+0000 7f5a79bde700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5a74067fe0 0x7f5a740ba9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:42.711 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.701+0000 7f5a79bde700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5a74067fe0 0x7f5a740ba9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54180/0 (socket says 172.21.15.112:54180) 2023-01-24T03:36:42.711 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.701+0000 7f5a79bde700 1 -- 172.21.15.112:0/545764278 learned_addr learned my addr 172.21.15.112:0/545764278 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:36:42.712 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.701+0000 7f5a793dd700 1 --2- 172.21.15.112:0/545764278 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a7406e040 0x7f5a740baf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:42.712 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.701+0000 7f5a7a3df700 1 --2- 172.21.15.112:0/545764278 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a7406eec0 0x7f5a74147f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:42.712 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.701+0000 7f5a79bde700 1 -- 172.21.15.112:0/545764278 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a7406e040 msgr2=0x7f5a740baf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:42.712 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.701+0000 7f5a79bde700 1 --2- 172.21.15.112:0/545764278 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a7406e040 0x7f5a740baf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:42.712 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.701+0000 7f5a79bde700 1 -- 172.21.15.112:0/545764278 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a7406eec0 msgr2=0x7f5a74147f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:42.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.701+0000 7f5a79bde700 1 --2- 172.21.15.112:0/545764278 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a7406eec0 0x7f5a74147f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:42.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.701+0000 7f5a79bde700 1 -- 172.21.15.112:0/545764278 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a6c0092a0 con 0x7f5a74067fe0 2023-01-24T03:36:42.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.701+0000 7f5a79bde700 1 --2- 172.21.15.112:0/545764278 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5a74067fe0 0x7f5a740ba9f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f5a6400c790 tx=0x7f5a640015b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:42.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.701+0000 7f5a71ffb700 1 -- 172.21.15.112:0/545764278 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a64012630 con 0x7f5a74067fe0 2023-01-24T03:36:42.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.701+0000 7f5a71ffb700 1 -- 172.21.15.112:0/545764278 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a640127b0 con 0x7f5a74067fe0 2023-01-24T03:36:42.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.701+0000 7f5a7be42700 1 -- 172.21.15.112:0/545764278 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a741457c0 con 0x7f5a74067fe0 2023-01-24T03:36:42.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.701+0000 7f5a71ffb700 1 -- 172.21.15.112:0/545764278 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a64012b70 con 0x7f5a74067fe0 2023-01-24T03:36:42.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.701+0000 7f5a7be42700 1 -- 172.21.15.112:0/545764278 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a74145d80 con 0x7f5a74067fe0 2023-01-24T03:36:42.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.702+0000 7f5a7be42700 1 -- 172.21.15.112:0/545764278 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a74146990 con 0x7f5a74067fe0 2023-01-24T03:36:42.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.702+0000 7f5a71ffb700 1 -- 172.21.15.112:0/545764278 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f5a64018020 con 0x7f5a74067fe0 2023-01-24T03:36:42.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.703+0000 7f5a71ffb700 1 --2- 172.21.15.112:0/545764278 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5a540611c0 0x7f5a54063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:42.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.703+0000 7f5a71ffb700 1 -- 172.21.15.112:0/545764278 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 7211+0+0 (secure 0 0 0) 0x7f5a6407db70 con 0x7f5a74067fe0 2023-01-24T03:36:42.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.703+0000 7f5a793dd700 1 --2- 172.21.15.112:0/545764278 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5a540611c0 0x7f5a54063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:42.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.703+0000 7f5a793dd700 1 --2- 172.21.15.112:0/545764278 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5a540611c0 0x7f5a54063670 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5a68001ed0 tx=0x7f5a680019f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:42.716 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.707+0000 7f5a71ffb700 1 -- 172.21.15.112:0/545764278 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f5a6404d820 con 0x7f5a74067fe0 2023-01-24T03:36:42.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.927+0000 7f5a7be42700 1 -- 172.21.15.112:0/545764278 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a7404bb80 con 0x7f5a74067fe0 2023-01-24T03:36:42.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.927+0000 7f5a71ffb700 1 -- 172.21.15.112:0/545764278 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5358 (secure 0 0 0) 0x7f5a64050e40 con 0x7f5a74067fe0 2023-01-24T03:36:42.930 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:36:42.930 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:36:42.756007+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4907},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4907":{"gid":4907,"name":"d","rank":0,"incarnation":27,"state":"up:active","state_seq":29,"addr":"172.21.15.142:6835/824478388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":824478388},{"type":"v1","addr":"172.21.15.142:6835","nonce":824478388}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:36:42.931 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.929+0000 7f5a7be42700 1 -- 172.21.15.112:0/545764278 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5a540611c0 msgr2=0x7f5a54063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:42.932 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.929+0000 7f5a7be42700 1 --2- 172.21.15.112:0/545764278 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5a540611c0 0x7f5a54063670 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5a68001ed0 tx=0x7f5a680019f0 comp rx=0 tx=0).stop 2023-01-24T03:36:42.932 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.929+0000 7f5a7be42700 1 -- 172.21.15.112:0/545764278 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5a74067fe0 msgr2=0x7f5a740ba9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:42.932 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.929+0000 7f5a7be42700 1 --2- 172.21.15.112:0/545764278 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5a74067fe0 0x7f5a740ba9f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f5a6400c790 tx=0x7f5a640015b0 comp rx=0 tx=0).stop 2023-01-24T03:36:42.932 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.929+0000 7f5a7be42700 1 -- 172.21.15.112:0/545764278 shutdown_connections 2023-01-24T03:36:42.932 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.929+0000 7f5a7be42700 1 --2- 172.21.15.112:0/545764278 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5a540611c0 0x7f5a54063670 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:42.933 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.929+0000 7f5a7be42700 1 --2- 172.21.15.112:0/545764278 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5a74067fe0 0x7f5a740ba9f0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:42.933 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.929+0000 7f5a7be42700 1 --2- 172.21.15.112:0/545764278 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a7406eec0 0x7f5a74147f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:42.933 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.929+0000 7f5a7be42700 1 --2- 172.21.15.112:0/545764278 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5a7406e040 0x7f5a740baf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:42.933 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.929+0000 7f5a7be42700 1 -- 172.21.15.112:0/545764278 >> 172.21.15.112:0/545764278 conn(0x7f5a740c8940 msgr2=0x7f5a740c2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:42.933 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.929+0000 7f5a7be42700 1 -- 172.21.15.112:0/545764278 shutdown_connections 2023-01-24T03:36:42.934 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:42.929+0000 7f5a7be42700 1 -- 172.21.15.112:0/545764278 wait complete. 2023-01-24T03:36:42.934 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 30 2023-01-24T03:36:42.943 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=0, replacing gid=5041 2023-01-24T03:36:42.943 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.b 2023-01-24T03:36:47.912 INFO:tasks.workunit.client.0.smithi112.stderr:error: RPC failed; curl 18 transfer closed with outstanding read data remaining 2023-01-24T03:36:47.913 INFO:tasks.workunit.client.0.smithi112.stderr:fatal: the remote end hung up unexpectedly 2023-01-24T03:36:47.913 INFO:tasks.workunit.client.0.smithi112.stderr:fatal: early EOF 2023-01-24T03:36:47.916 INFO:tasks.workunit.client.0.smithi112.stderr:fatal: index-pack failed 2023-01-24T03:36:47.944 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2023-01-24T03:36:47.944 INFO:tasks.ceph.mds.b:Restarting daemon 2023-01-24T03:36:47.944 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-01-24T03:36:47.947 INFO:tasks.ceph.mds.b:Started 2023-01-24T03:36:47.947 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:36:48.023 INFO:tasks.ceph.mds.b.smithi142.stdout:starting mds.b at 2023-01-24T03:36:48.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.060+0000 7f69ca1a1700 1 Processor -- start 2023-01-24T03:36:48.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.060+0000 7f69ca1a1700 1 -- start start 2023-01-24T03:36:48.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.061+0000 7f69ca1a1700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f69c40c5680 0x7f69c4156730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:48.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.061+0000 7f69ca1a1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f69c40c5a50 0x7f69c4156c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:48.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.061+0000 7f69ca1a1700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69c40c3d10 con 0x7f69c4155e20 2023-01-24T03:36:48.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.061+0000 7f69ca1a1700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69c40c4eb0 con 0x7f69c40c5680 2023-01-24T03:36:48.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.061+0000 7f69ca1a1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69c40c2e20 con 0x7f69c40c5a50 2023-01-24T03:36:48.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.061+0000 7f69c899e700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f69c4155e20 0x7f69c40cb870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35180/0 (socket says 172.21.15.112:35180) 2023-01-24T03:36:48.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.061+0000 7f69c899e700 1 -- 172.21.15.112:0/4076809129 learned_addr learned my addr 172.21.15.112:0/4076809129 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:36:48.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.061+0000 7f69c99a0700 1 --2- 172.21.15.112:0/4076809129 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f69c40c5a50 0x7f69c4156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:48.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.061+0000 7f69c919f700 1 --2- 172.21.15.112:0/4076809129 >> v2:172.21.15.142:3300/0 conn(0x7f69c40c5680 0x7f69c4156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:48.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.061+0000 7f69c3fff700 1 -- 172.21.15.112:0/4076809129 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2853081686 0 0) 0x7f69c40c3d10 con 0x7f69c4155e20 2023-01-24T03:36:48.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.062+0000 7f69c3fff700 1 -- 172.21.15.112:0/4076809129 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69b00033f0 con 0x7f69c4155e20 2023-01-24T03:36:48.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.062+0000 7f69c99a0700 1 -- 172.21.15.112:0/4076809129 >> v2:172.21.15.142:3300/0 conn(0x7f69c40c5680 msgr2=0x7f69c4156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:36:48.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.062+0000 7f69c99a0700 1 --2- 172.21.15.112:0/4076809129 >> v2:172.21.15.142:3300/0 conn(0x7f69c40c5680 0x7f69c4156730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:48.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.062+0000 7f69c99a0700 1 -- 172.21.15.112:0/4076809129 >> v1:172.21.15.112:6789/0 conn(0x7f69c4155e20 legacy=0x7f69c40cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:48.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.062+0000 7f69c99a0700 1 -- 172.21.15.112:0/4076809129 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c40c3100 con 0x7f69c40c5a50 2023-01-24T03:36:48.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.062+0000 7f69c919f700 1 --2- 172.21.15.112:0/4076809129 >> v2:172.21.15.142:3300/0 conn(0x7f69c40c5680 0x7f69c4156730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:36:48.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.062+0000 7f69c99a0700 1 --2- 172.21.15.112:0/4076809129 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f69c40c5a50 0x7f69c4156c70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f69bc009760 tx=0x7f69bc009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=971ea219dbd09012 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:48.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.062+0000 7f69c3fff700 1 -- 172.21.15.112:0/4076809129 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69bc0105b0 con 0x7f69c40c5a50 2023-01-24T03:36:48.069 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.062+0000 7f69c3fff700 1 -- 172.21.15.112:0/4076809129 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69bc010710 con 0x7f69c40c5a50 2023-01-24T03:36:48.069 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.062+0000 7f69c3fff700 1 -- 172.21.15.112:0/4076809129 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69bc010ab0 con 0x7f69c40c5a50 2023-01-24T03:36:48.069 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.062+0000 7f69ca1a1700 1 -- 172.21.15.112:0/4076809129 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f69c40c5a50 msgr2=0x7f69c4156c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:48.069 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.062+0000 7f69ca1a1700 1 --2- 172.21.15.112:0/4076809129 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f69c40c5a50 0x7f69c4156c70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f69bc009760 tx=0x7f69bc009060 comp rx=0 tx=0).stop 2023-01-24T03:36:48.069 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.062+0000 7f69ca1a1700 1 -- 172.21.15.112:0/4076809129 shutdown_connections 2023-01-24T03:36:48.069 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.062+0000 7f69ca1a1700 1 --2- 172.21.15.112:0/4076809129 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f69c40c5a50 0x7f69c4156c70 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:48.070 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.062+0000 7f69ca1a1700 1 --2- 172.21.15.112:0/4076809129 >> v2:172.21.15.142:3300/0 conn(0x7f69c40c5680 0x7f69c4156730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:48.070 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.062+0000 7f69ca1a1700 1 -- 172.21.15.112:0/4076809129 >> 172.21.15.112:0/4076809129 conn(0x7f69c40685b0 msgr2=0x7f69c40689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:48.070 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.062+0000 7f69ca1a1700 1 -- 172.21.15.112:0/4076809129 shutdown_connections 2023-01-24T03:36:48.070 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.062+0000 7f69ca1a1700 1 -- 172.21.15.112:0/4076809129 wait complete. 2023-01-24T03:36:48.073 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.063+0000 7f69ca1a1700 1 Processor -- start 2023-01-24T03:36:48.073 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.063+0000 7f69ca1a1700 1 -- start start 2023-01-24T03:36:48.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.064+0000 7f69ca1a1700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f69c40c5680 0x7f69c4150f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:48.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.064+0000 7f69ca1a1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f69c40c5a50 0x7f69c4151470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:48.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.064+0000 7f69ca1a1700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69c4155e20 0x7f69c41685b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:48.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.064+0000 7f69ca1a1700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c40cbe00 con 0x7f69c4155e20 2023-01-24T03:36:48.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.064+0000 7f69ca1a1700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c40cb240 con 0x7f69c40c5680 2023-01-24T03:36:48.075 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.064+0000 7f69ca1a1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69c414b890 con 0x7f69c40c5a50 2023-01-24T03:36:48.075 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.064+0000 7f69c99a0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69c4155e20 0x7f69c41685b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:48.075 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.064+0000 7f69c99a0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69c4155e20 0x7f69c41685b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54198/0 (socket says 172.21.15.112:54198) 2023-01-24T03:36:48.075 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.064+0000 7f69c99a0700 1 -- 172.21.15.112:0/335996421 learned_addr learned my addr 172.21.15.112:0/335996421 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:36:48.075 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.065+0000 7f69c899e700 1 --2- 172.21.15.112:0/335996421 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f69c40c5a50 0x7f69c4151470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:48.076 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.065+0000 7f69c919f700 1 --2- 172.21.15.112:0/335996421 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f69c40c5680 0x7f69c4150f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:48.076 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.065+0000 7f69c99a0700 1 -- 172.21.15.112:0/335996421 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f69c40c5a50 msgr2=0x7f69c4151470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:48.076 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.065+0000 7f69c99a0700 1 --2- 172.21.15.112:0/335996421 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f69c40c5a50 0x7f69c4151470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:48.076 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.065+0000 7f69c99a0700 1 -- 172.21.15.112:0/335996421 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f69c40c5680 msgr2=0x7f69c4150f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:48.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.065+0000 7f69c99a0700 1 --2- 172.21.15.112:0/335996421 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f69c40c5680 0x7f69c4150f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:48.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.065+0000 7f69c99a0700 1 -- 172.21.15.112:0/335996421 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b4009a20 con 0x7f69c4155e20 2023-01-24T03:36:48.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.065+0000 7f69c99a0700 1 --2- 172.21.15.112:0/335996421 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69c4155e20 0x7f69c41685b0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f69bc009d50 tx=0x7f69bc016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:48.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.065+0000 7f69c17fa700 1 -- 172.21.15.112:0/335996421 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69bc010880 con 0x7f69c4155e20 2023-01-24T03:36:48.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.065+0000 7f69c17fa700 1 -- 172.21.15.112:0/335996421 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69bc020070 con 0x7f69c4155e20 2023-01-24T03:36:48.078 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.065+0000 7f69ca1a1700 1 -- 172.21.15.112:0/335996421 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69bc0092a0 con 0x7f69c4155e20 2023-01-24T03:36:48.078 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.065+0000 7f69c17fa700 1 -- 172.21.15.112:0/335996421 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69bc025430 con 0x7f69c4155e20 2023-01-24T03:36:48.078 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.065+0000 7f69ca1a1700 1 -- 172.21.15.112:0/335996421 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69c40cb7b0 con 0x7f69c4155e20 2023-01-24T03:36:48.078 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.066+0000 7f69ca1a1700 1 -- 172.21.15.112:0/335996421 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69c40cc190 con 0x7f69c4155e20 2023-01-24T03:36:48.078 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.066+0000 7f69c17fa700 1 -- 172.21.15.112:0/335996421 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f69bc02b020 con 0x7f69c4155e20 2023-01-24T03:36:48.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.067+0000 7f69c17fa700 1 --2- 172.21.15.112:0/335996421 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f69a40611c0 0x7f69a4063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:48.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.067+0000 7f69c17fa700 1 -- 172.21.15.112:0/335996421 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 7211+0+0 (secure 0 0 0) 0x7f69bc07f720 con 0x7f69c4155e20 2023-01-24T03:36:48.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.067+0000 7f69c919f700 1 --2- 172.21.15.112:0/335996421 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f69a40611c0 0x7f69a4063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:48.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.067+0000 7f69c919f700 1 --2- 172.21.15.112:0/335996421 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f69a40611c0 0x7f69a4063670 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f69b40099f0 tx=0x7f69b400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:48.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.071+0000 7f69c17fa700 1 -- 172.21.15.112:0/335996421 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f69c40cc190 con 0x7f69c4155e20 2023-01-24T03:36:48.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.288+0000 7f69ca1a1700 1 -- 172.21.15.112:0/335996421 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69c40cc190 con 0x7f69c4155e20 2023-01-24T03:36:48.292 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.289+0000 7f69c17fa700 1 -- 172.21.15.112:0/335996421 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5358 (secure 0 0 0) 0x7f69c40cc190 con 0x7f69c4155e20 2023-01-24T03:36:48.292 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:36:48.293 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:36:42.756007+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4907},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4907":{"gid":4907,"name":"d","rank":0,"incarnation":27,"state":"up:active","state_seq":29,"addr":"172.21.15.142:6835/824478388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":824478388},{"type":"v1","addr":"172.21.15.142:6835","nonce":824478388}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:36:48.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.291+0000 7f69ca1a1700 1 -- 172.21.15.112:0/335996421 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f69a40611c0 msgr2=0x7f69a4063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:48.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.291+0000 7f69ca1a1700 1 --2- 172.21.15.112:0/335996421 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f69a40611c0 0x7f69a4063670 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f69b40099f0 tx=0x7f69b400b040 comp rx=0 tx=0).stop 2023-01-24T03:36:48.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.291+0000 7f69ca1a1700 1 -- 172.21.15.112:0/335996421 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69c4155e20 msgr2=0x7f69c41685b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:48.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.291+0000 7f69ca1a1700 1 --2- 172.21.15.112:0/335996421 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69c4155e20 0x7f69c41685b0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f69bc009d50 tx=0x7f69bc016000 comp rx=0 tx=0).stop 2023-01-24T03:36:48.295 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.291+0000 7f69ca1a1700 1 -- 172.21.15.112:0/335996421 shutdown_connections 2023-01-24T03:36:48.295 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.291+0000 7f69ca1a1700 1 --2- 172.21.15.112:0/335996421 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f69a40611c0 0x7f69a4063670 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:48.295 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.291+0000 7f69ca1a1700 1 --2- 172.21.15.112:0/335996421 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f69c4155e20 0x7f69c41685b0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:48.295 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.291+0000 7f69ca1a1700 1 --2- 172.21.15.112:0/335996421 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f69c40c5680 0x7f69c4150f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:48.295 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.291+0000 7f69ca1a1700 1 --2- 172.21.15.112:0/335996421 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f69c40c5a50 0x7f69c4151470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:48.296 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.291+0000 7f69ca1a1700 1 -- 172.21.15.112:0/335996421 >> 172.21.15.112:0/335996421 conn(0x7f69c40685b0 msgr2=0x7f69c414bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:48.296 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.291+0000 7f69ca1a1700 1 -- 172.21.15.112:0/335996421 shutdown_connections 2023-01-24T03:36:48.296 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:48.291+0000 7f69ca1a1700 1 -- 172.21.15.112:0/335996421 wait complete. 2023-01-24T03:36:48.296 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 30 2023-01-24T03:36:48.305 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2023-01-24T03:36:48.408 INFO:tasks.workunit.client.0.smithi112.stderr:+ retry 2023-01-24T03:36:48.408 INFO:tasks.workunit.client.0.smithi112.stderr:+ rm -rf ceph 2023-01-24T03:36:48.409 INFO:tasks.workunit.client.0.smithi112.stderr:+ timeout 3600 git clone https://git.ceph.com/ceph.git 2023-01-24T03:36:48.411 INFO:tasks.workunit.client.0.smithi112.stderr:Cloning into 'ceph'... 2023-01-24T03:36:50.306 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:36:50.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.428+0000 7fd1c70d1700 1 Processor -- start 2023-01-24T03:36:50.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.428+0000 7fd1c70d1700 1 -- start start 2023-01-24T03:36:50.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.429+0000 7fd1c70d1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1c00caaf0 0x7fd1c0155da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:50.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.429+0000 7fd1c70d1700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1c00b0f40 con 0x7fd1c00b0aa0 2023-01-24T03:36:50.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.429+0000 7fd1c70d1700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1c00b1170 con 0x7fd1c00c7e40 2023-01-24T03:36:50.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.429+0000 7fd1c70d1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c00b1450 con 0x7fd1c00caaf0 2023-01-24T03:36:50.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.429+0000 7fd1bffff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fd1c00b0aa0 0x7fd1c00ca3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35196/0 (socket says 172.21.15.112:35196) 2023-01-24T03:36:50.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.429+0000 7fd1bffff700 1 -- 172.21.15.112:0/1674312498 learned_addr learned my addr 172.21.15.112:0/1674312498 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:36:50.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.429+0000 7fd1c566e700 1 --2- 172.21.15.112:0/1674312498 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1c00caaf0 0x7fd1c0155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:50.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.430+0000 7fd1bf7fe700 1 -- 172.21.15.112:0/1674312498 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3671772996 0 0) 0x7fd1c00b0f40 con 0x7fd1c00b0aa0 2023-01-24T03:36:50.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.430+0000 7fd1bf7fe700 1 -- 172.21.15.112:0/1674312498 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1ac0033f0 con 0x7fd1c00b0aa0 2023-01-24T03:36:50.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.430+0000 7fd1bf7fe700 1 -- 172.21.15.112:0/1674312498 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4213577326 0 0) 0x7fd1ac0033f0 con 0x7fd1c00b0aa0 2023-01-24T03:36:50.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.430+0000 7fd1bf7fe700 1 -- 172.21.15.112:0/1674312498 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1c00caaf0 msgr2=0x7fd1c0155da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:36:50.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.430+0000 7fd1bf7fe700 1 --2- 172.21.15.112:0/1674312498 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1c00caaf0 0x7fd1c0155da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:50.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.430+0000 7fd1bf7fe700 1 -- 172.21.15.112:0/1674312498 >> v1:172.21.15.142:6789/0 conn(0x7fd1c00c7e40 legacy=0x7fd1c00ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:50.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.430+0000 7fd1bf7fe700 1 -- 172.21.15.112:0/1674312498 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1c00b1730 con 0x7fd1c00b0aa0 2023-01-24T03:36:50.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.430+0000 7fd1bf7fe700 1 -- 172.21.15.112:0/1674312498 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7fd1b4001470 con 0x7fd1c00b0aa0 2023-01-24T03:36:50.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.430+0000 7fd1bf7fe700 1 -- 172.21.15.112:0/1674312498 >> v1:172.21.15.112:6789/0 conn(0x7fd1c00b0aa0 legacy=0x7fd1c00ca3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:50.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.430+0000 7fd1bf7fe700 1 --2- 172.21.15.112:0/1674312498 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1c00caaf0 0x7fd1ac003910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:50.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.430+0000 7fd1bf7fe700 1 --2- 172.21.15.112:0/1674312498 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd1ac003e50 0x7fd1ac004690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:50.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.430+0000 7fd1bf7fe700 1 --2- 172.21.15.112:0/1674312498 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1ac004bd0 0x7fd1ac005040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:50.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.430+0000 7fd1bf7fe700 1 -- 172.21.15.112:0/1674312498 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c00b1450 con 0x7fd1ac004bd0 2023-01-24T03:36:50.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.430+0000 7fd1bf7fe700 1 -- 172.21.15.112:0/1674312498 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ac00ba20 con 0x7fd1ac003e50 2023-01-24T03:36:50.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.430+0000 7fd1bf7fe700 1 -- 172.21.15.112:0/1674312498 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ac00bbe0 con 0x7fd1c00caaf0 2023-01-24T03:36:50.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.430+0000 7fd1c4e6d700 1 --2- 172.21.15.112:0/1674312498 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1c00caaf0 0x7fd1ac003910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:50.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.430+0000 7fd1bffff700 1 --2- 172.21.15.112:0/1674312498 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd1ac003e50 0x7fd1ac004690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:50.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.430+0000 7fd1c566e700 1 --2- 172.21.15.112:0/1674312498 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1ac004bd0 0x7fd1ac005040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:50.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.430+0000 7fd1c566e700 1 -- 172.21.15.112:0/1674312498 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1c00caaf0 msgr2=0x7fd1ac003910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:36:50.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.430+0000 7fd1c566e700 1 --2- 172.21.15.112:0/1674312498 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1c00caaf0 0x7fd1ac003910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:50.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.430+0000 7fd1c566e700 1 -- 172.21.15.112:0/1674312498 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd1ac003e50 msgr2=0x7fd1ac004690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:36:50.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.430+0000 7fd1c566e700 1 --2- 172.21.15.112:0/1674312498 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd1ac003e50 0x7fd1ac004690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:50.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.430+0000 7fd1c566e700 1 -- 172.21.15.112:0/1674312498 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1ac0033f0 con 0x7fd1ac004bd0 2023-01-24T03:36:50.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.430+0000 7fd1c4e6d700 1 --2- 172.21.15.112:0/1674312498 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1c00caaf0 0x7fd1ac003910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:36:50.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.431+0000 7fd1c566e700 1 --2- 172.21.15.112:0/1674312498 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1ac004bd0 0x7fd1ac005040 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8009a80 tx=0x7fd1b8009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd9f6c0a646270d3 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:50.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.431+0000 7fd1bf7fe700 1 -- 172.21.15.112:0/1674312498 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b8009430 con 0x7fd1ac004bd0 2023-01-24T03:36:50.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.431+0000 7fd1bf7fe700 1 -- 172.21.15.112:0/1674312498 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1b800e640 con 0x7fd1ac004bd0 2023-01-24T03:36:50.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.431+0000 7fd1bf7fe700 1 -- 172.21.15.112:0/1674312498 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b800ea00 con 0x7fd1ac004bd0 2023-01-24T03:36:50.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.431+0000 7fd1c70d1700 1 -- 172.21.15.112:0/1674312498 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1ac004bd0 msgr2=0x7fd1ac005040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:50.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.431+0000 7fd1c70d1700 1 --2- 172.21.15.112:0/1674312498 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1ac004bd0 0x7fd1ac005040 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8009a80 tx=0x7fd1b8009060 comp rx=0 tx=0).stop 2023-01-24T03:36:50.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.431+0000 7fd1c566e700 1 -- 172.21.15.112:0/1674312498 reap_dead start 2023-01-24T03:36:50.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.431+0000 7fd1c70d1700 1 -- 172.21.15.112:0/1674312498 shutdown_connections 2023-01-24T03:36:50.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.431+0000 7fd1c70d1700 1 -- 172.21.15.112:0/1674312498 >> 172.21.15.112:0/1674312498 conn(0x7fd1c005e2c0 msgr2=0x7fd1c00cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:50.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.431+0000 7fd1c70d1700 1 -- 172.21.15.112:0/1674312498 shutdown_connections 2023-01-24T03:36:50.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.431+0000 7fd1c70d1700 1 -- 172.21.15.112:0/1674312498 wait complete. 2023-01-24T03:36:50.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.432+0000 7fd1c70d1700 1 Processor -- start 2023-01-24T03:36:50.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.433+0000 7fd1c70d1700 1 -- start start 2023-01-24T03:36:50.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.433+0000 7fd1c70d1700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd1ac004bd0 0x7fd1c00c1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:50.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.433+0000 7fd1c70d1700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1c00bf2a0 0x7fd1c0157f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:50.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.433+0000 7fd1c70d1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1c00c4c20 0x7fd1c00c5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:50.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.433+0000 7fd1c70d1700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c01547f0 con 0x7fd1c00bf2a0 2023-01-24T03:36:50.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.433+0000 7fd1c70d1700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c0169800 con 0x7fd1ac004bd0 2023-01-24T03:36:50.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.433+0000 7fd1c70d1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c0165ca0 con 0x7fd1c00c4c20 2023-01-24T03:36:50.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.433+0000 7fd1bffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1c00bf2a0 0x7fd1c0157f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:50.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.433+0000 7fd1bffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1c00bf2a0 0x7fd1c0157f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54220/0 (socket says 172.21.15.112:54220) 2023-01-24T03:36:50.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.433+0000 7fd1bffff700 1 -- 172.21.15.112:0/1516151443 learned_addr learned my addr 172.21.15.112:0/1516151443 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:36:50.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.434+0000 7fd1c4e6d700 1 --2- 172.21.15.112:0/1516151443 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd1ac004bd0 0x7fd1c00c1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:50.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.434+0000 7fd1c566e700 1 --2- 172.21.15.112:0/1516151443 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1c00c4c20 0x7fd1c00c5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:50.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.434+0000 7fd1bffff700 1 -- 172.21.15.112:0/1516151443 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1c00c4c20 msgr2=0x7fd1c00c5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:50.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.434+0000 7fd1bffff700 1 --2- 172.21.15.112:0/1516151443 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1c00c4c20 0x7fd1c00c5070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:50.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.434+0000 7fd1bffff700 1 -- 172.21.15.112:0/1516151443 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd1ac004bd0 msgr2=0x7fd1c00c1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:50.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.434+0000 7fd1bffff700 1 --2- 172.21.15.112:0/1516151443 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd1ac004bd0 0x7fd1c00c1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:50.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.434+0000 7fd1bffff700 1 -- 172.21.15.112:0/1516151443 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1c01653b0 con 0x7fd1c00bf2a0 2023-01-24T03:36:50.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.434+0000 7fd1c4e6d700 1 --2- 172.21.15.112:0/1516151443 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd1ac004bd0 0x7fd1c00c1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:36:50.450 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.434+0000 7fd1bffff700 1 --2- 172.21.15.112:0/1516151443 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1c00bf2a0 0x7fd1c0157f50 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fd1b40076f0 tx=0x7fd1b4001da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:50.450 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.434+0000 7fd1bcff9700 1 -- 172.21.15.112:0/1516151443 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b4018030 con 0x7fd1c00bf2a0 2023-01-24T03:36:50.450 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.434+0000 7fd1bcff9700 1 -- 172.21.15.112:0/1516151443 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1b4011660 con 0x7fd1c00bf2a0 2023-01-24T03:36:50.450 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.434+0000 7fd1c70d1700 1 -- 172.21.15.112:0/1516151443 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1c0168a80 con 0x7fd1c00bf2a0 2023-01-24T03:36:50.450 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.434+0000 7fd1bcff9700 1 -- 172.21.15.112:0/1516151443 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b4011a00 con 0x7fd1c00bf2a0 2023-01-24T03:36:50.451 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.434+0000 7fd1c70d1700 1 -- 172.21.15.112:0/1516151443 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1c0165930 con 0x7fd1c00bf2a0 2023-01-24T03:36:50.452 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.435+0000 7fd1c70d1700 1 -- 172.21.15.112:0/1516151443 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1c0165af0 con 0x7fd1c00bf2a0 2023-01-24T03:36:50.452 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.435+0000 7fd1bcff9700 1 -- 172.21.15.112:0/1516151443 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fd1b401e470 con 0x7fd1c00bf2a0 2023-01-24T03:36:50.452 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.436+0000 7fd1bcff9700 1 --2- 172.21.15.112:0/1516151443 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd1a00611f0 0x7fd1a00636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:36:50.452 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.436+0000 7fd1bcff9700 1 -- 172.21.15.112:0/1516151443 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 7211+0+0 (secure 0 0 0) 0x7fd1b4019070 con 0x7fd1c00bf2a0 2023-01-24T03:36:50.453 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.436+0000 7fd1c4e6d700 1 --2- 172.21.15.112:0/1516151443 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd1a00611f0 0x7fd1a00636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:36:50.453 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.436+0000 7fd1c4e6d700 1 --2- 172.21.15.112:0/1516151443 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd1a00611f0 0x7fd1a00636a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0000f80 tx=0x7fd1b0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:36:50.453 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.440+0000 7fd1bcff9700 1 -- 172.21.15.112:0/1516151443 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fd1b404d8d0 con 0x7fd1c00bf2a0 2023-01-24T03:36:50.660 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.658+0000 7fd1c70d1700 1 -- 172.21.15.112:0/1516151443 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1c00bf680 con 0x7fd1c00bf2a0 2023-01-24T03:36:50.661 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.658+0000 7fd1bcff9700 1 -- 172.21.15.112:0/1516151443 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+6118 (secure 0 0 0) 0x7fd1b4011db0 con 0x7fd1c00bf2a0 2023-01-24T03:36:50.661 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:36:50.662 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5296,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/2838716685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2838716685},{"type":"v1","addr":"172.21.15.142:6834","nonce":2838716685}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:36:42.756007+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4907},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4907":{"gid":4907,"name":"d","rank":0,"incarnation":27,"state":"up:active","state_seq":29,"addr":"172.21.15.142:6835/824478388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":824478388},{"type":"v1","addr":"172.21.15.142:6835","nonce":824478388}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:36:50.664 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.660+0000 7fd1c70d1700 1 -- 172.21.15.112:0/1516151443 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd1a00611f0 msgr2=0x7fd1a00636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:50.664 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.660+0000 7fd1c70d1700 1 --2- 172.21.15.112:0/1516151443 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd1a00611f0 0x7fd1a00636a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0000f80 tx=0x7fd1b0009040 comp rx=0 tx=0).stop 2023-01-24T03:36:50.664 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.660+0000 7fd1c70d1700 1 -- 172.21.15.112:0/1516151443 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1c00bf2a0 msgr2=0x7fd1c0157f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:36:50.664 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.660+0000 7fd1c70d1700 1 --2- 172.21.15.112:0/1516151443 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1c00bf2a0 0x7fd1c0157f50 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fd1b40076f0 tx=0x7fd1b4001da0 comp rx=0 tx=0).stop 2023-01-24T03:36:50.664 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.660+0000 7fd1c70d1700 1 -- 172.21.15.112:0/1516151443 shutdown_connections 2023-01-24T03:36:50.665 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.660+0000 7fd1c70d1700 1 --2- 172.21.15.112:0/1516151443 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd1a00611f0 0x7fd1a00636a0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:50.665 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.660+0000 7fd1c70d1700 1 --2- 172.21.15.112:0/1516151443 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd1c00c4c20 0x7fd1c00c5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:50.665 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.660+0000 7fd1c70d1700 1 --2- 172.21.15.112:0/1516151443 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd1ac004bd0 0x7fd1c00c1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:50.665 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.660+0000 7fd1c70d1700 1 --2- 172.21.15.112:0/1516151443 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd1c00bf2a0 0x7fd1c0157f50 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:36:50.665 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.660+0000 7fd1c70d1700 1 -- 172.21.15.112:0/1516151443 >> 172.21.15.112:0/1516151443 conn(0x7fd1c005e2c0 msgr2=0x7fd1c014ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:36:50.666 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.660+0000 7fd1c70d1700 1 -- 172.21.15.112:0/1516151443 shutdown_connections 2023-01-24T03:36:50.666 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:36:50.660+0000 7fd1c70d1700 1 -- 172.21.15.112:0/1516151443 wait complete. 2023-01-24T03:36:50.666 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 31 2023-01-24T03:36:50.675 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2023-01-24T03:36:50.675 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 34 secs before thrashing 2023-01-24T03:36:50.890 INFO:tasks.ceph.mds.d.smithi142.stderr:2023-01-24T03:36:50.888+0000 7ff564823700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-01-24T03:36:59.726 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:36:59.729 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:37:24.676 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:37:24.793 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.791+0000 7f642660c700 1 Processor -- start 2023-01-24T03:37:24.793 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.791+0000 7f642660c700 1 -- start start 2023-01-24T03:37:24.796 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.791+0000 7f642660c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f642005dc00 0x7f64201534b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:24.796 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.791+0000 7f642660c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f64200c4f90 0x7f64201539f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:24.796 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.791+0000 7f642660c700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6420156980 con 0x7f642006ee00 2023-01-24T03:37:24.796 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.791+0000 7f642660c700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64200c2a30 con 0x7f642005dc00 2023-01-24T03:37:24.797 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.792+0000 7f642660c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6420068760 con 0x7f64200c4f90 2023-01-24T03:37:24.797 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.792+0000 7f642560a700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f642006ee00 0x7f64200c1330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35270/0 (socket says 172.21.15.112:35270) 2023-01-24T03:37:24.797 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.792+0000 7f642560a700 1 -- 172.21.15.112:0/3966579328 learned_addr learned my addr 172.21.15.112:0/3966579328 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:37:24.797 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.792+0000 7f6425e0b700 1 --2- 172.21.15.112:0/3966579328 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f64200c4f90 0x7f64201539f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:24.797 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.792+0000 7f6424e09700 1 --2- 172.21.15.112:0/3966579328 >> v2:172.21.15.142:3300/0 conn(0x7f642005dc00 0x7f64201534b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:24.798 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.792+0000 7f640ffff700 1 -- 172.21.15.112:0/3966579328 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2828243374 0 0) 0x7f6420156980 con 0x7f642006ee00 2023-01-24T03:37:24.798 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.792+0000 7f640ffff700 1 -- 172.21.15.112:0/3966579328 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64080033f0 con 0x7f642006ee00 2023-01-24T03:37:24.798 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.792+0000 7f6425e0b700 1 -- 172.21.15.112:0/3966579328 >> v2:172.21.15.142:3300/0 conn(0x7f642005dc00 msgr2=0x7f64201534b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:37:24.798 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.792+0000 7f6425e0b700 1 --2- 172.21.15.112:0/3966579328 >> v2:172.21.15.142:3300/0 conn(0x7f642005dc00 0x7f64201534b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:24.798 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.792+0000 7f6425e0b700 1 -- 172.21.15.112:0/3966579328 >> v1:172.21.15.112:6789/0 conn(0x7f642006ee00 legacy=0x7f64200c1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:24.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.792+0000 7f6425e0b700 1 -- 172.21.15.112:0/3966579328 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6420156c60 con 0x7f64200c4f90 2023-01-24T03:37:24.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.792+0000 7f6425e0b700 1 --2- 172.21.15.112:0/3966579328 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f64200c4f90 0x7f64201539f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6418009e60 tx=0x7f6418009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63446748b2a4331 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:24.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.792+0000 7f640ffff700 1 -- 172.21.15.112:0/3966579328 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641800e070 con 0x7f64200c4f90 2023-01-24T03:37:24.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.793+0000 7f640ffff700 1 -- 172.21.15.112:0/3966579328 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f641800bab0 con 0x7f64200c4f90 2023-01-24T03:37:24.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.793+0000 7f640ffff700 1 -- 172.21.15.112:0/3966579328 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641800be70 con 0x7f64200c4f90 2023-01-24T03:37:24.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.793+0000 7f642660c700 1 -- 172.21.15.112:0/3966579328 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f64200c4f90 msgr2=0x7f64201539f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:24.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.793+0000 7f642660c700 1 --2- 172.21.15.112:0/3966579328 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f64200c4f90 0x7f64201539f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6418009e60 tx=0x7f6418009780 comp rx=0 tx=0).stop 2023-01-24T03:37:24.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.793+0000 7f642660c700 1 -- 172.21.15.112:0/3966579328 shutdown_connections 2023-01-24T03:37:24.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.793+0000 7f642660c700 1 --2- 172.21.15.112:0/3966579328 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f64200c4f90 0x7f64201539f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:24.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.793+0000 7f642660c700 1 --2- 172.21.15.112:0/3966579328 >> v2:172.21.15.142:3300/0 conn(0x7f642005dc00 0x7f64201534b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:24.801 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.793+0000 7f642660c700 1 -- 172.21.15.112:0/3966579328 >> 172.21.15.112:0/3966579328 conn(0x7f64200c87b0 msgr2=0x7f64200c4780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:37:24.801 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.793+0000 7f642660c700 1 -- 172.21.15.112:0/3966579328 shutdown_connections 2023-01-24T03:37:24.801 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.793+0000 7f642660c700 1 -- 172.21.15.112:0/3966579328 wait complete. 2023-01-24T03:37:24.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.794+0000 7f642660c700 1 Processor -- start 2023-01-24T03:37:24.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.794+0000 7f642660c700 1 -- start start 2023-01-24T03:37:24.805 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.795+0000 7f642660c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f642005dc00 0x7f64200bdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:24.805 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.795+0000 7f642660c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f642006ee00 0x7f6420158b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:24.805 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.795+0000 7f642660c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f64200c4f90 0x7f6420166310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:24.805 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.795+0000 7f642660c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6420162da0 con 0x7f64200c4f90 2023-01-24T03:37:24.805 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.795+0000 7f642660c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6420163600 con 0x7f642006ee00 2023-01-24T03:37:24.806 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.795+0000 7f642660c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6420150f10 con 0x7f642005dc00 2023-01-24T03:37:24.806 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.795+0000 7f6425e0b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f64200c4f90 0x7f6420166310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:24.806 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.795+0000 7f6425e0b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f64200c4f90 0x7f6420166310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54292/0 (socket says 172.21.15.112:54292) 2023-01-24T03:37:24.806 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.795+0000 7f6425e0b700 1 -- 172.21.15.112:0/1790872181 learned_addr learned my addr 172.21.15.112:0/1790872181 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:37:24.806 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.795+0000 7f642560a700 1 --2- 172.21.15.112:0/1790872181 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f642005dc00 0x7f64200bdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:24.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.795+0000 7f6424e09700 1 --2- 172.21.15.112:0/1790872181 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f642006ee00 0x7f6420158b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:24.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.796+0000 7f642560a700 1 -- 172.21.15.112:0/1790872181 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f642006ee00 msgr2=0x7f6420158b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:24.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.796+0000 7f642560a700 1 --2- 172.21.15.112:0/1790872181 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f642006ee00 0x7f6420158b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:24.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.796+0000 7f642560a700 1 -- 172.21.15.112:0/1790872181 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f64200c4f90 msgr2=0x7f6420166310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:24.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.796+0000 7f642560a700 1 --2- 172.21.15.112:0/1790872181 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f64200c4f90 0x7f6420166310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:24.808 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.796+0000 7f642560a700 1 -- 172.21.15.112:0/1790872181 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6418009a20 con 0x7f642005dc00 2023-01-24T03:37:24.808 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.796+0000 7f6425e0b700 1 --2- 172.21.15.112:0/1790872181 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f64200c4f90 0x7f6420166310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:37:24.808 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.796+0000 7f642560a700 1 --2- 172.21.15.112:0/1790872181 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f642005dc00 0x7f64200bdfe0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f641c001ed0 tx=0x7f641c007bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:24.808 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.796+0000 7f640d7fa700 1 -- 172.21.15.112:0/1790872181 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641c0070d0 con 0x7f642005dc00 2023-01-24T03:37:24.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.796+0000 7f640d7fa700 1 -- 172.21.15.112:0/1790872181 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f641c007230 con 0x7f642005dc00 2023-01-24T03:37:24.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.796+0000 7f642660c700 1 -- 172.21.15.112:0/1790872181 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64201591b0 con 0x7f642005dc00 2023-01-24T03:37:24.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.796+0000 7f640d7fa700 1 -- 172.21.15.112:0/1790872181 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641c007630 con 0x7f642005dc00 2023-01-24T03:37:24.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.796+0000 7f642660c700 1 -- 172.21.15.112:0/1790872181 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6420162fb0 con 0x7f642005dc00 2023-01-24T03:37:24.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.797+0000 7f642660c700 1 -- 172.21.15.112:0/1790872181 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f642005ea20 con 0x7f642005dc00 2023-01-24T03:37:24.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.798+0000 7f640d7fa700 1 -- 172.21.15.112:0/1790872181 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f641c019030 con 0x7f642005dc00 2023-01-24T03:37:24.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.798+0000 7f640d7fa700 1 --2- 172.21.15.112:0/1790872181 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f64000611c0 0x7f6400063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:24.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.798+0000 7f640d7fa700 1 -- 172.21.15.112:0/1790872181 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 7211+0+0 (secure 0 0 0) 0x7f641c07cb20 con 0x7f642005dc00 2023-01-24T03:37:24.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.798+0000 7f6424e09700 1 --2- 172.21.15.112:0/1790872181 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f64000611c0 0x7f6400063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:24.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.799+0000 7f6424e09700 1 --2- 172.21.15.112:0/1790872181 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f64000611c0 0x7f6400063670 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f64201673a0 tx=0x7f6410009060 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:24.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:24.803+0000 7f640d7fa700 1 -- 172.21.15.112:0/1790872181 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f641c04c7d0 con 0x7f642005dc00 2023-01-24T03:37:25.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.035+0000 7f642660c700 1 -- 172.21.15.112:0/1790872181 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6420158900 con 0x7f642005dc00 2023-01-24T03:37:25.038 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.036+0000 7f640d7fa700 1 -- 172.21.15.112:0/1790872181 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+6118 (secure 0 0 0) 0x7f641c04fdf0 con 0x7f642005dc00 2023-01-24T03:37:25.039 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:37:25.039 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5296,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/2838716685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2838716685},{"type":"v1","addr":"172.21.15.142:6834","nonce":2838716685}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:36:42.756007+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4907},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4907":{"gid":4907,"name":"d","rank":0,"incarnation":27,"state":"up:active","state_seq":29,"addr":"172.21.15.142:6835/824478388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":824478388},{"type":"v1","addr":"172.21.15.142:6835","nonce":824478388}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:37:25.042 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.039+0000 7f642660c700 1 -- 172.21.15.112:0/1790872181 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f64000611c0 msgr2=0x7f6400063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:25.042 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.039+0000 7f642660c700 1 --2- 172.21.15.112:0/1790872181 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f64000611c0 0x7f6400063670 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f64201673a0 tx=0x7f6410009060 comp rx=0 tx=0).stop 2023-01-24T03:37:25.042 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.039+0000 7f642660c700 1 -- 172.21.15.112:0/1790872181 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f642005dc00 msgr2=0x7f64200bdfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:25.043 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.039+0000 7f642660c700 1 --2- 172.21.15.112:0/1790872181 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f642005dc00 0x7f64200bdfe0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f641c001ed0 tx=0x7f641c007bd0 comp rx=0 tx=0).stop 2023-01-24T03:37:25.043 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.039+0000 7f642660c700 1 -- 172.21.15.112:0/1790872181 shutdown_connections 2023-01-24T03:37:25.043 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.039+0000 7f642660c700 1 --2- 172.21.15.112:0/1790872181 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f64000611c0 0x7f6400063670 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:25.044 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.039+0000 7f642660c700 1 --2- 172.21.15.112:0/1790872181 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f64200c4f90 0x7f6420166310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:25.044 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.039+0000 7f642660c700 1 --2- 172.21.15.112:0/1790872181 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f642005dc00 0x7f64200bdfe0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:25.044 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.039+0000 7f642660c700 1 --2- 172.21.15.112:0/1790872181 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f642006ee00 0x7f6420158b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:25.045 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.039+0000 7f642660c700 1 -- 172.21.15.112:0/1790872181 >> 172.21.15.112:0/1790872181 conn(0x7f64200c87b0 msgr2=0x7f642014ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:37:25.045 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.039+0000 7f642660c700 1 -- 172.21.15.112:0/1790872181 shutdown_connections 2023-01-24T03:37:25.045 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.039+0000 7f642660c700 1 -- 172.21.15.112:0/1790872181 wait complete. 2023-01-24T03:37:25.045 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 31 2023-01-24T03:37:25.056 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=0) 2023-01-24T03:37:25.057 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-01-24T03:37:25.057 INFO:teuthology.orchestra.run:waiting for 300 2023-01-24T03:37:25.069 INFO:tasks.ceph.mds.d:Stopped 2023-01-24T03:37:25.070 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:37:25.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.181+0000 7ff8f3282700 1 Processor -- start 2023-01-24T03:37:25.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.182+0000 7ff8f3282700 1 -- start start 2023-01-24T03:37:25.187 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.182+0000 7ff8f3282700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7ff8ec06dbb0 0x7ff8ec06e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:25.187 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.182+0000 7ff8f3282700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ec06c200 con 0x7ff8ec14bff0 2023-01-24T03:37:25.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.182+0000 7ff8f3282700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ec151ec0 con 0x7ff8ec06f240 2023-01-24T03:37:25.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.182+0000 7ff8f3282700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8ec06f010 con 0x7ff8ec06dbb0 2023-01-24T03:37:25.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.183+0000 7ff8f2280700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7ff8ec14bff0 0x7ff8ec0c0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35284/0 (socket says 172.21.15.112:35284) 2023-01-24T03:37:25.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.183+0000 7ff8f2280700 1 -- 172.21.15.112:0/2412281484 learned_addr learned my addr 172.21.15.112:0/2412281484 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:37:25.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.183+0000 7ff8f1a7f700 1 --2- 172.21.15.112:0/2412281484 >> v2:172.21.15.142:3300/0 conn(0x7ff8ec06dbb0 0x7ff8ec06e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:25.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.183+0000 7ff8f127e700 1 -- 172.21.15.112:0/2412281484 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3436095592 0 0) 0x7ff8ec06c200 con 0x7ff8ec14bff0 2023-01-24T03:37:25.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.183+0000 7ff8f127e700 1 -- 172.21.15.112:0/2412281484 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8d40033f0 con 0x7ff8ec14bff0 2023-01-24T03:37:25.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.183+0000 7ff8f127e700 1 -- 172.21.15.112:0/2412281484 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3827966882 0 0) 0x7ff8d40033f0 con 0x7ff8ec14bff0 2023-01-24T03:37:25.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.183+0000 7ff8f127e700 1 -- 172.21.15.112:0/2412281484 >> v2:172.21.15.142:3300/0 conn(0x7ff8ec06dbb0 msgr2=0x7ff8ec06e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:37:25.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.183+0000 7ff8f127e700 1 --2- 172.21.15.112:0/2412281484 >> v2:172.21.15.142:3300/0 conn(0x7ff8ec06dbb0 0x7ff8ec06e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:25.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.183+0000 7ff8f127e700 1 -- 172.21.15.112:0/2412281484 >> v1:172.21.15.142:6789/0 conn(0x7ff8ec06f240 legacy=0x7ff8ec15c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:25.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.183+0000 7ff8f127e700 1 -- 172.21.15.112:0/2412281484 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8ec06b8d0 con 0x7ff8ec14bff0 2023-01-24T03:37:25.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.183+0000 7ff8f127e700 1 -- 172.21.15.112:0/2412281484 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7ff8e8001470 con 0x7ff8ec14bff0 2023-01-24T03:37:25.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.183+0000 7ff8f127e700 1 -- 172.21.15.112:0/2412281484 >> v1:172.21.15.112:6789/0 conn(0x7ff8ec14bff0 legacy=0x7ff8ec0c0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:25.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.183+0000 7ff8f127e700 1 --2- 172.21.15.112:0/2412281484 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff8d40038e0 0x7ff8d4003cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:25.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.183+0000 7ff8f127e700 1 --2- 172.21.15.112:0/2412281484 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8d4004690 0x7ff8d4004b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:25.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.183+0000 7ff8f127e700 1 --2- 172.21.15.112:0/2412281484 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8d4005040 0x7ff8d400b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:25.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.183+0000 7ff8f127e700 1 -- 172.21.15.112:0/2412281484 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d400bf20 con 0x7ff8d4004690 2023-01-24T03:37:25.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.183+0000 7ff8f127e700 1 -- 172.21.15.112:0/2412281484 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d400c090 con 0x7ff8d4005040 2023-01-24T03:37:25.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.183+0000 7ff8f127e700 1 -- 172.21.15.112:0/2412281484 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d400c250 con 0x7ff8d40038e0 2023-01-24T03:37:25.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.183+0000 7ff8f1a7f700 1 --2- 172.21.15.112:0/2412281484 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8d4004690 0x7ff8d4004b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:25.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.183+0000 7ff8f2a81700 1 --2- 172.21.15.112:0/2412281484 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8d4005040 0x7ff8d400b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:25.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.184+0000 7ff8f2280700 1 --2- 172.21.15.112:0/2412281484 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff8d40038e0 0x7ff8d4003cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:25.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.184+0000 7ff8f1a7f700 1 -- 172.21.15.112:0/2412281484 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff8d40038e0 msgr2=0x7ff8d4003cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:37:25.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.184+0000 7ff8f1a7f700 1 --2- 172.21.15.112:0/2412281484 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff8d40038e0 0x7ff8d4003cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:25.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.184+0000 7ff8f1a7f700 1 -- 172.21.15.112:0/2412281484 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8d4005040 msgr2=0x7ff8d400b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:37:25.193 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.184+0000 7ff8f1a7f700 1 --2- 172.21.15.112:0/2412281484 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8d4005040 0x7ff8d400b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:25.193 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.184+0000 7ff8f1a7f700 1 -- 172.21.15.112:0/2412281484 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d40033f0 con 0x7ff8d4004690 2023-01-24T03:37:25.193 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.184+0000 7ff8f2a81700 1 -- 172.21.15.112:0/2412281484 reap_dead start 2023-01-24T03:37:25.193 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.184+0000 7ff8f1a7f700 1 --2- 172.21.15.112:0/2412281484 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8d4004690 0x7ff8d4004b00 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec06df80 tx=0x7ff8dc009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a3faa09ca6bb879 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:25.193 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.184+0000 7ff8f127e700 1 -- 172.21.15.112:0/2412281484 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc009430 con 0x7ff8d4004690 2023-01-24T03:37:25.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.184+0000 7ff8f127e700 1 -- 172.21.15.112:0/2412281484 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8dc00e640 con 0x7ff8d4004690 2023-01-24T03:37:25.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.184+0000 7ff8f127e700 1 -- 172.21.15.112:0/2412281484 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc00ea00 con 0x7ff8d4004690 2023-01-24T03:37:25.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.185+0000 7ff8f3282700 1 -- 172.21.15.112:0/2412281484 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8d4004690 msgr2=0x7ff8d4004b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:25.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.185+0000 7ff8f3282700 1 --2- 172.21.15.112:0/2412281484 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8d4004690 0x7ff8d4004b00 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec06df80 tx=0x7ff8dc009060 comp rx=0 tx=0).stop 2023-01-24T03:37:25.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.185+0000 7ff8f3282700 1 -- 172.21.15.112:0/2412281484 shutdown_connections 2023-01-24T03:37:25.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.185+0000 7ff8f3282700 1 --2- 172.21.15.112:0/2412281484 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8d4004690 0x7ff8d4004b00 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:25.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.185+0000 7ff8f3282700 1 -- 172.21.15.112:0/2412281484 >> 172.21.15.112:0/2412281484 conn(0x7ff8ec0689b0 msgr2=0x7ff8ec0c8540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:37:25.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.185+0000 7ff8f3282700 1 -- 172.21.15.112:0/2412281484 shutdown_connections 2023-01-24T03:37:25.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.185+0000 7ff8f3282700 1 -- 172.21.15.112:0/2412281484 wait complete. 2023-01-24T03:37:25.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.186+0000 7ff8f3282700 1 Processor -- start 2023-01-24T03:37:25.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.186+0000 7ff8f3282700 1 -- start start 2023-01-24T03:37:25.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.187+0000 7ff8f3282700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8d4004690 0x7ff8ec067670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:25.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.187+0000 7ff8f3282700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff8ec150f90 0x7ff8ec14b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:25.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.187+0000 7ff8f3282700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8ec14cf00 0x7ff8ec1552e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:25.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.187+0000 7ff8f3282700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ec1677a0 con 0x7ff8d4004690 2023-01-24T03:37:25.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.187+0000 7ff8f3282700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ec0671e0 con 0x7ff8ec14cf00 2023-01-24T03:37:25.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.187+0000 7ff8f3282700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ec066eb0 con 0x7ff8ec150f90 2023-01-24T03:37:25.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.187+0000 7ff8f2280700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8d4004690 0x7ff8ec067670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:25.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.187+0000 7ff8f2280700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8d4004690 0x7ff8ec067670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54308/0 (socket says 172.21.15.112:54308) 2023-01-24T03:37:25.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.187+0000 7ff8f2280700 1 -- 172.21.15.112:0/878967119 learned_addr learned my addr 172.21.15.112:0/878967119 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:37:25.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.187+0000 7ff8f2a81700 1 --2- 172.21.15.112:0/878967119 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8ec14cf00 0x7ff8ec1552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:25.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.187+0000 7ff8f1a7f700 1 --2- 172.21.15.112:0/878967119 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff8ec150f90 0x7ff8ec14b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:25.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.187+0000 7ff8f2280700 1 -- 172.21.15.112:0/878967119 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff8ec150f90 msgr2=0x7ff8ec14b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:25.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.188+0000 7ff8f2280700 1 --2- 172.21.15.112:0/878967119 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff8ec150f90 0x7ff8ec14b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:25.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.188+0000 7ff8f2280700 1 -- 172.21.15.112:0/878967119 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8ec14cf00 msgr2=0x7ff8ec1552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:25.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.188+0000 7ff8f2280700 1 --2- 172.21.15.112:0/878967119 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8ec14cf00 0x7ff8ec1552e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:25.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.188+0000 7ff8f2280700 1 -- 172.21.15.112:0/878967119 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8ec14d8d0 con 0x7ff8d4004690 2023-01-24T03:37:25.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.188+0000 7ff8f2a81700 1 --2- 172.21.15.112:0/878967119 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8ec14cf00 0x7ff8ec1552e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:37:25.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.188+0000 7ff8f2280700 1 --2- 172.21.15.112:0/878967119 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8d4004690 0x7ff8ec067670 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8004a10 tx=0x7ff8e8001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:25.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.188+0000 7ff8da7fc700 1 -- 172.21.15.112:0/878967119 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e800f870 con 0x7ff8d4004690 2023-01-24T03:37:25.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.188+0000 7ff8da7fc700 1 -- 172.21.15.112:0/878967119 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8e800fa10 con 0x7ff8d4004690 2023-01-24T03:37:25.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.188+0000 7ff8da7fc700 1 -- 172.21.15.112:0/878967119 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e800fdd0 con 0x7ff8d4004690 2023-01-24T03:37:25.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.188+0000 7ff8f3282700 1 -- 172.21.15.112:0/878967119 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8ec14db90 con 0x7ff8d4004690 2023-01-24T03:37:25.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.188+0000 7ff8f3282700 1 -- 172.21.15.112:0/878967119 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8ec14e150 con 0x7ff8d4004690 2023-01-24T03:37:25.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.189+0000 7ff8f3282700 1 -- 172.21.15.112:0/878967119 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8ec0674c0 con 0x7ff8d4004690 2023-01-24T03:37:25.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.190+0000 7ff8da7fc700 1 -- 172.21.15.112:0/878967119 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7ff8e801c030 con 0x7ff8d4004690 2023-01-24T03:37:25.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.191+0000 7ff8da7fc700 1 --2- 172.21.15.112:0/878967119 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff8cc0611c0 0x7ff8cc063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:25.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.191+0000 7ff8da7fc700 1 -- 172.21.15.112:0/878967119 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 7211+0+0 (secure 0 0 0) 0x7ff8e807df50 con 0x7ff8d4004690 2023-01-24T03:37:25.204 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.191+0000 7ff8f1a7f700 1 --2- 172.21.15.112:0/878967119 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff8cc0611c0 0x7ff8cc063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:25.204 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.191+0000 7ff8f1a7f700 1 --2- 172.21.15.112:0/878967119 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff8cc0611c0 0x7ff8cc063670 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec150ab0 tx=0x7ff8dc01a000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:25.204 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.194+0000 7ff8da7fc700 1 -- 172.21.15.112:0/878967119 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7ff8e804dc40 con 0x7ff8d4004690 2023-01-24T03:37:25.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.427+0000 7ff8f3282700 1 -- 172.21.15.112:0/878967119 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8ec05ea20 con 0x7ff8d4004690 2023-01-24T03:37:25.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.427+0000 7ff8da7fc700 1 -- 172.21.15.112:0/878967119 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+6118 (secure 0 0 0) 0x7ff8e8051260 con 0x7ff8d4004690 2023-01-24T03:37:25.431 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:37:25.431 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5296,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/2838716685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2838716685},{"type":"v1","addr":"172.21.15.142:6834","nonce":2838716685}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:36:42.756007+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4907},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4907":{"gid":4907,"name":"d","rank":0,"incarnation":27,"state":"up:active","state_seq":29,"addr":"172.21.15.142:6835/824478388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":824478388},{"type":"v1","addr":"172.21.15.142:6835","nonce":824478388}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:37:25.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.429+0000 7ff8f3282700 1 -- 172.21.15.112:0/878967119 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff8cc0611c0 msgr2=0x7ff8cc063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:25.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.429+0000 7ff8f3282700 1 --2- 172.21.15.112:0/878967119 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff8cc0611c0 0x7ff8cc063670 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec150ab0 tx=0x7ff8dc01a000 comp rx=0 tx=0).stop 2023-01-24T03:37:25.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.429+0000 7ff8f3282700 1 -- 172.21.15.112:0/878967119 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8d4004690 msgr2=0x7ff8ec067670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:25.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.429+0000 7ff8f3282700 1 --2- 172.21.15.112:0/878967119 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8d4004690 0x7ff8ec067670 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8004a10 tx=0x7ff8e8001e50 comp rx=0 tx=0).stop 2023-01-24T03:37:25.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.430+0000 7ff8f3282700 1 -- 172.21.15.112:0/878967119 shutdown_connections 2023-01-24T03:37:25.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.430+0000 7ff8f3282700 1 --2- 172.21.15.112:0/878967119 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff8cc0611c0 0x7ff8cc063670 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:25.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.430+0000 7ff8f3282700 1 --2- 172.21.15.112:0/878967119 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff8d4004690 0x7ff8ec067670 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:25.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.430+0000 7ff8f3282700 1 --2- 172.21.15.112:0/878967119 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff8ec14cf00 0x7ff8ec1552e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:25.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.430+0000 7ff8f3282700 1 --2- 172.21.15.112:0/878967119 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff8ec150f90 0x7ff8ec14b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:25.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.430+0000 7ff8f3282700 1 -- 172.21.15.112:0/878967119 >> 172.21.15.112:0/878967119 conn(0x7ff8ec0689b0 msgr2=0x7ff8ec06e190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:37:25.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.430+0000 7ff8f3282700 1 -- 172.21.15.112:0/878967119 shutdown_connections 2023-01-24T03:37:25.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:25.430+0000 7ff8f3282700 1 -- 172.21.15.112:0/878967119 wait complete. 2023-01-24T03:37:25.436 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 31 2023-01-24T03:37:25.445 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-01-24T03:37:27.446 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:37:27.561 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.558+0000 7fb681335700 1 Processor -- start 2023-01-24T03:37:27.561 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.558+0000 7fb681335700 1 -- start start 2023-01-24T03:37:27.565 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.558+0000 7fb681335700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fb67c06ee20 0x7fb67c067a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:27.565 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.558+0000 7fb681335700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb67c0c36b0 con 0x7fb67c06f1f0 2023-01-24T03:37:27.565 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.558+0000 7fb681335700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb67c159770 con 0x7fb67c0c53c0 2023-01-24T03:37:27.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.558+0000 7fb681335700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb67c0c3d80 con 0x7fb67c06ee20 2023-01-24T03:37:27.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.559+0000 7fb680b34700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fb67c06f1f0 0x7fb67c159060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35316/0 (socket says 172.21.15.112:35316) 2023-01-24T03:37:27.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.559+0000 7fb680b34700 1 -- 172.21.15.112:0/126615483 learned_addr learned my addr 172.21.15.112:0/126615483 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:37:27.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.559+0000 7fb67bfff700 1 --2- 172.21.15.112:0/126615483 >> v2:172.21.15.142:3300/0 conn(0x7fb67c06ee20 0x7fb67c067a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:27.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.559+0000 7fb67affd700 1 -- 172.21.15.112:0/126615483 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3327577890 0 0) 0x7fb67c0c36b0 con 0x7fb67c06f1f0 2023-01-24T03:37:27.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.559+0000 7fb67affd700 1 -- 172.21.15.112:0/126615483 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6680033f0 con 0x7fb67c06f1f0 2023-01-24T03:37:27.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.559+0000 7fb67affd700 1 -- 172.21.15.112:0/126615483 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4197921035 0 0) 0x7fb67c159770 con 0x7fb67c0c53c0 2023-01-24T03:37:27.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.559+0000 7fb67affd700 1 -- 172.21.15.112:0/126615483 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb67c0c36b0 con 0x7fb67c0c53c0 2023-01-24T03:37:27.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.559+0000 7fb67affd700 1 -- 172.21.15.112:0/126615483 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4185538768 0 0) 0x7fb6680033f0 con 0x7fb67c06f1f0 2023-01-24T03:37:27.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.559+0000 7fb67affd700 1 -- 172.21.15.112:0/126615483 >> v2:172.21.15.142:3300/0 conn(0x7fb67c06ee20 msgr2=0x7fb67c067a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:37:27.568 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.559+0000 7fb67affd700 1 --2- 172.21.15.112:0/126615483 >> v2:172.21.15.142:3300/0 conn(0x7fb67c06ee20 0x7fb67c067a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:27.568 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.559+0000 7fb67affd700 1 -- 172.21.15.112:0/126615483 >> v1:172.21.15.142:6789/0 conn(0x7fb67c0c53c0 legacy=0x7fb67c158950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:27.568 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.559+0000 7fb67affd700 1 -- 172.21.15.112:0/126615483 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb67c0c30f0 con 0x7fb67c06f1f0 2023-01-24T03:37:27.568 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.559+0000 7fb67affd700 1 -- 172.21.15.112:0/126615483 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7fb6740017f0 con 0x7fb67c06f1f0 2023-01-24T03:37:27.569 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.559+0000 7fb67affd700 1 -- 172.21.15.112:0/126615483 >> v1:172.21.15.112:6789/0 conn(0x7fb67c06f1f0 legacy=0x7fb67c159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:27.569 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.559+0000 7fb67affd700 1 --2- 172.21.15.112:0/126615483 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb668003a70 0x7fb668003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:27.569 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.559+0000 7fb67affd700 1 --2- 172.21.15.112:0/126615483 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb668004690 0x7fb668004b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:27.569 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.559+0000 7fb67affd700 1 --2- 172.21.15.112:0/126615483 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb668005040 0x7fb66800b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:27.569 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.559+0000 7fb67affd700 1 -- 172.21.15.112:0/126615483 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c0c3d80 con 0x7fb668003a70 2023-01-24T03:37:27.570 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.559+0000 7fb67affd700 1 -- 172.21.15.112:0/126615483 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb66800bfe0 con 0x7fb668004690 2023-01-24T03:37:27.570 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.559+0000 7fb67affd700 1 -- 172.21.15.112:0/126615483 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb66800c1a0 con 0x7fb668005040 2023-01-24T03:37:27.570 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.559+0000 7fb67bfff700 1 --2- 172.21.15.112:0/126615483 >> v2:172.21.15.142:3300/0 conn(0x7fb67c06ee20 0x7fb67c067a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:37:27.570 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.560+0000 7fb67b7fe700 1 --2- 172.21.15.112:0/126615483 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb668004690 0x7fb668004b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:27.570 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.560+0000 7fb67bfff700 1 --2- 172.21.15.112:0/126615483 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb668003a70 0x7fb668003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:27.571 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.560+0000 7fb680b34700 1 --2- 172.21.15.112:0/126615483 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb668005040 0x7fb66800b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:27.571 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.560+0000 7fb67bfff700 1 -- 172.21.15.112:0/126615483 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb668005040 msgr2=0x7fb66800b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:37:27.571 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.560+0000 7fb67bfff700 1 --2- 172.21.15.112:0/126615483 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb668005040 0x7fb66800b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:27.571 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.560+0000 7fb67bfff700 1 -- 172.21.15.112:0/126615483 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb668004690 msgr2=0x7fb668004b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:37:27.572 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.560+0000 7fb67bfff700 1 --2- 172.21.15.112:0/126615483 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb668004690 0x7fb668004b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:27.572 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.560+0000 7fb67bfff700 1 -- 172.21.15.112:0/126615483 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6680033f0 con 0x7fb668003a70 2023-01-24T03:37:27.572 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.560+0000 7fb680b34700 1 -- 172.21.15.112:0/126615483 reap_dead start 2023-01-24T03:37:27.572 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.560+0000 7fb67b7fe700 1 --2- 172.21.15.112:0/126615483 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb668004690 0x7fb668004b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:37:27.572 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.560+0000 7fb67bfff700 1 --2- 172.21.15.112:0/126615483 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb668003a70 0x7fb668003e60 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fb66c006d80 tx=0x7fb66c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6959bb01dea3c8e3 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:27.573 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.560+0000 7fb67affd700 1 -- 172.21.15.112:0/126615483 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c0095c0 con 0x7fb668003a70 2023-01-24T03:37:27.573 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.560+0000 7fb67affd700 1 -- 172.21.15.112:0/126615483 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb66c00e640 con 0x7fb668003a70 2023-01-24T03:37:27.573 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.560+0000 7fb67affd700 1 -- 172.21.15.112:0/126615483 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c00e9e0 con 0x7fb668003a70 2023-01-24T03:37:27.573 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.561+0000 7fb681335700 1 -- 172.21.15.112:0/126615483 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb668003a70 msgr2=0x7fb668003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:27.573 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.561+0000 7fb681335700 1 --2- 172.21.15.112:0/126615483 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb668003a70 0x7fb668003e60 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fb66c006d80 tx=0x7fb66c009060 comp rx=0 tx=0).stop 2023-01-24T03:37:27.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.561+0000 7fb681335700 1 -- 172.21.15.112:0/126615483 shutdown_connections 2023-01-24T03:37:27.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.561+0000 7fb681335700 1 --2- 172.21.15.112:0/126615483 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb668003a70 0x7fb668003e60 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:27.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.561+0000 7fb681335700 1 -- 172.21.15.112:0/126615483 >> 172.21.15.112:0/126615483 conn(0x7fb67c0c87b0 msgr2=0x7fb67c0c4780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:37:27.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.561+0000 7fb681335700 1 -- 172.21.15.112:0/126615483 shutdown_connections 2023-01-24T03:37:27.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.561+0000 7fb681335700 1 -- 172.21.15.112:0/126615483 wait complete. 2023-01-24T03:37:27.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.562+0000 7fb681335700 1 Processor -- start 2023-01-24T03:37:27.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.562+0000 7fb681335700 1 -- start start 2023-01-24T03:37:27.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.563+0000 7fb681335700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb668003a70 0x7fb67c158ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:27.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.563+0000 7fb681335700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb67c152c40 0x7fb67c168f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:27.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.563+0000 7fb681335700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb67c15b350 0x7fb67c1694d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:27.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.563+0000 7fb681335700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c168b00 con 0x7fb67c15b350 2023-01-24T03:37:27.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.563+0000 7fb681335700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c1500b0 con 0x7fb67c152c40 2023-01-24T03:37:27.579 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.563+0000 7fb681335700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c150390 con 0x7fb668003a70 2023-01-24T03:37:27.579 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.563+0000 7fb680b34700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb67c15b350 0x7fb67c1694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:27.579 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.563+0000 7fb680b34700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb67c15b350 0x7fb67c1694d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54340/0 (socket says 172.21.15.112:54340) 2023-01-24T03:37:27.579 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.563+0000 7fb680b34700 1 -- 172.21.15.112:0/153522391 learned_addr learned my addr 172.21.15.112:0/153522391 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:37:27.580 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.563+0000 7fb67b7fe700 1 --2- 172.21.15.112:0/153522391 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb67c152c40 0x7fb67c168f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:27.580 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.563+0000 7fb67bfff700 1 --2- 172.21.15.112:0/153522391 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb668003a70 0x7fb67c158ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:27.581 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.564+0000 7fb680b34700 1 -- 172.21.15.112:0/153522391 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb668003a70 msgr2=0x7fb67c158ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:27.581 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.564+0000 7fb680b34700 1 --2- 172.21.15.112:0/153522391 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb668003a70 0x7fb67c158ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:27.581 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.564+0000 7fb680b34700 1 -- 172.21.15.112:0/153522391 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb67c152c40 msgr2=0x7fb67c168f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:27.582 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.564+0000 7fb680b34700 1 --2- 172.21.15.112:0/153522391 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb67c152c40 0x7fb67c168f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:27.582 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.564+0000 7fb680b34700 1 -- 172.21.15.112:0/153522391 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb67c150670 con 0x7fb67c15b350 2023-01-24T03:37:27.582 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.564+0000 7fb680b34700 1 --2- 172.21.15.112:0/153522391 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb67c15b350 0x7fb67c1694d0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fb674001350 tx=0x7fb674007f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:27.583 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.564+0000 7fb663fff700 1 -- 172.21.15.112:0/153522391 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb674018030 con 0x7fb67c15b350 2023-01-24T03:37:27.583 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.564+0000 7fb663fff700 1 -- 172.21.15.112:0/153522391 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb67400f9e0 con 0x7fb67c15b350 2023-01-24T03:37:27.583 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.564+0000 7fb681335700 1 -- 172.21.15.112:0/153522391 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb67c150950 con 0x7fb67c15b350 2023-01-24T03:37:27.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.564+0000 7fb663fff700 1 -- 172.21.15.112:0/153522391 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67400fda0 con 0x7fb67c15b350 2023-01-24T03:37:27.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.564+0000 7fb681335700 1 -- 172.21.15.112:0/153522391 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb67c150c30 con 0x7fb67c15b350 2023-01-24T03:37:27.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.565+0000 7fb681335700 1 -- 172.21.15.112:0/153522391 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb67c168c70 con 0x7fb67c15b350 2023-01-24T03:37:27.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.566+0000 7fb663fff700 1 -- 172.21.15.112:0/153522391 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fb674013600 con 0x7fb67c15b350 2023-01-24T03:37:27.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.566+0000 7fb663fff700 1 --2- 172.21.15.112:0/153522391 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb65c0611f0 0x7fb65c0636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:27.586 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.566+0000 7fb663fff700 1 -- 172.21.15.112:0/153522391 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 7211+0+0 (secure 0 0 0) 0x7fb674019070 con 0x7fb67c15b350 2023-01-24T03:37:27.586 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.568+0000 7fb67bfff700 1 --2- 172.21.15.112:0/153522391 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb65c0611f0 0x7fb65c0636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:27.586 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.568+0000 7fb67bfff700 1 --2- 172.21.15.112:0/153522391 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb65c0611f0 0x7fb65c0636a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb66c006d30 tx=0x7fb66c019000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:27.587 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.574+0000 7fb663fff700 1 -- 172.21.15.112:0/153522391 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fb67404d910 con 0x7fb67c15b350 2023-01-24T03:37:27.820 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.817+0000 7fb681335700 1 -- 172.21.15.112:0/153522391 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb67c168e00 con 0x7fb67c15b350 2023-01-24T03:37:27.821 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.818+0000 7fb663fff700 1 -- 172.21.15.112:0/153522391 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+6118 (secure 0 0 0) 0x7fb674050f30 con 0x7fb67c15b350 2023-01-24T03:37:27.822 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:37:27.822 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5296,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/2838716685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2838716685},{"type":"v1","addr":"172.21.15.142:6834","nonce":2838716685}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:36:42.756007+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4907},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4907":{"gid":4907,"name":"d","rank":0,"incarnation":27,"state":"up:active","state_seq":29,"addr":"172.21.15.142:6835/824478388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":824478388},{"type":"v1","addr":"172.21.15.142:6835","nonce":824478388}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:37:27.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.820+0000 7fb681335700 1 -- 172.21.15.112:0/153522391 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb65c0611f0 msgr2=0x7fb65c0636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:27.825 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.820+0000 7fb681335700 1 --2- 172.21.15.112:0/153522391 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb65c0611f0 0x7fb65c0636a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb66c006d30 tx=0x7fb66c019000 comp rx=0 tx=0).stop 2023-01-24T03:37:27.825 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.820+0000 7fb681335700 1 -- 172.21.15.112:0/153522391 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb67c15b350 msgr2=0x7fb67c1694d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:27.826 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.820+0000 7fb681335700 1 --2- 172.21.15.112:0/153522391 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb67c15b350 0x7fb67c1694d0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fb674001350 tx=0x7fb674007f20 comp rx=0 tx=0).stop 2023-01-24T03:37:27.826 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.820+0000 7fb681335700 1 -- 172.21.15.112:0/153522391 shutdown_connections 2023-01-24T03:37:27.826 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.820+0000 7fb681335700 1 --2- 172.21.15.112:0/153522391 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb65c0611f0 0x7fb65c0636a0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:27.827 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.820+0000 7fb681335700 1 --2- 172.21.15.112:0/153522391 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb67c15b350 0x7fb67c1694d0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:27.827 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.820+0000 7fb681335700 1 --2- 172.21.15.112:0/153522391 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb668003a70 0x7fb67c158ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:27.827 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.820+0000 7fb681335700 1 --2- 172.21.15.112:0/153522391 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb67c152c40 0x7fb67c168f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:27.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.820+0000 7fb681335700 1 -- 172.21.15.112:0/153522391 >> 172.21.15.112:0/153522391 conn(0x7fb67c0c87b0 msgr2=0x7fb67c14f1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:37:27.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.821+0000 7fb681335700 1 -- 172.21.15.112:0/153522391 shutdown_connections 2023-01-24T03:37:27.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:27.821+0000 7fb681335700 1 -- 172.21.15.112:0/153522391 wait complete. 2023-01-24T03:37:27.829 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 31 2023-01-24T03:37:27.836 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-01-24T03:37:29.807 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:37:29.810 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:37:29.838 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:37:29.966 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.963+0000 7f67526a0700 1 Processor -- start 2023-01-24T03:37:29.966 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.963+0000 7f67526a0700 1 -- start start 2023-01-24T03:37:29.968 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.964+0000 7f67526a0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f674c0c4f90 0x7f674c15eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:29.968 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.964+0000 7f67526a0700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f674c156980 con 0x7f674c06ee00 2023-01-24T03:37:29.968 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.964+0000 7f67526a0700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f674c156c60 con 0x7f674c05dc00 2023-01-24T03:37:29.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.964+0000 7f67526a0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f674c0c2a30 con 0x7f674c0c4f90 2023-01-24T03:37:29.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.964+0000 7f675169e700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f674c06ee00 0x7f674c0c1330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35338/0 (socket says 172.21.15.112:35338) 2023-01-24T03:37:29.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.964+0000 7f675169e700 1 -- 172.21.15.112:0/2490266682 learned_addr learned my addr 172.21.15.112:0/2490266682 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:37:29.970 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.964+0000 7f6751e9f700 1 --2- 172.21.15.112:0/2490266682 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f674c0c4f90 0x7f674c15eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:29.970 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.964+0000 7f673bfff700 1 -- 172.21.15.112:0/2490266682 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1755789187 0 0) 0x7f674c156980 con 0x7f674c06ee00 2023-01-24T03:37:29.970 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.964+0000 7f673bfff700 1 -- 172.21.15.112:0/2490266682 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67340033f0 con 0x7f674c06ee00 2023-01-24T03:37:29.970 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.964+0000 7f673bfff700 1 -- 172.21.15.112:0/2490266682 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 385440200 0 0) 0x7f674c156c60 con 0x7f674c05dc00 2023-01-24T03:37:29.971 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.964+0000 7f673bfff700 1 -- 172.21.15.112:0/2490266682 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f674c156980 con 0x7f674c05dc00 2023-01-24T03:37:29.971 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.965+0000 7f6751e9f700 1 -- 172.21.15.112:0/2490266682 >> v1:172.21.15.142:6789/0 conn(0x7f674c05dc00 legacy=0x7f674c1534b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:29.971 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.965+0000 7f6751e9f700 1 -- 172.21.15.112:0/2490266682 >> v1:172.21.15.112:6789/0 conn(0x7f674c06ee00 legacy=0x7f674c0c1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:29.972 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.965+0000 7f6751e9f700 1 -- 172.21.15.112:0/2490266682 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f674c156f40 con 0x7f674c0c4f90 2023-01-24T03:37:29.972 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.965+0000 7f6751e9f700 1 --2- 172.21.15.112:0/2490266682 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f674c0c4f90 0x7f674c15eb30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6744009e90 tx=0x7f6744009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41d7ac78e59ace77 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:29.972 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.965+0000 7f673bfff700 1 -- 172.21.15.112:0/2490266682 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674400e070 con 0x7f674c0c4f90 2023-01-24T03:37:29.972 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.965+0000 7f673bfff700 1 -- 172.21.15.112:0/2490266682 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f674400bb80 con 0x7f674c0c4f90 2023-01-24T03:37:29.973 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.965+0000 7f673bfff700 1 -- 172.21.15.112:0/2490266682 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6744012430 con 0x7f674c0c4f90 2023-01-24T03:37:29.977 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.966+0000 7f67526a0700 1 -- 172.21.15.112:0/2490266682 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f674c0c4f90 msgr2=0x7f674c15eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:29.977 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.966+0000 7f67526a0700 1 --2- 172.21.15.112:0/2490266682 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f674c0c4f90 0x7f674c15eb30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6744009e90 tx=0x7f6744009780 comp rx=0 tx=0).stop 2023-01-24T03:37:29.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.966+0000 7f67526a0700 1 -- 172.21.15.112:0/2490266682 shutdown_connections 2023-01-24T03:37:29.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.966+0000 7f67526a0700 1 --2- 172.21.15.112:0/2490266682 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f674c0c4f90 0x7f674c15eb30 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:29.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.966+0000 7f67526a0700 1 -- 172.21.15.112:0/2490266682 >> 172.21.15.112:0/2490266682 conn(0x7f674c0c87b0 msgr2=0x7f674c0c4780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:37:29.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.966+0000 7f67526a0700 1 -- 172.21.15.112:0/2490266682 shutdown_connections 2023-01-24T03:37:29.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.966+0000 7f67526a0700 1 -- 172.21.15.112:0/2490266682 wait complete. 2023-01-24T03:37:29.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.967+0000 7f67526a0700 1 Processor -- start 2023-01-24T03:37:29.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.967+0000 7f67526a0700 1 -- start start 2023-01-24T03:37:29.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.968+0000 7f67526a0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f674c05dc00 0x7f674c0c6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:29.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.968+0000 7f67526a0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f674c06ee00 0x7f674c14b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:29.980 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.968+0000 7f67526a0700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f674c0c4f90 0x7f674c14b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:29.980 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.968+0000 7f67526a0700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f674c068760 con 0x7f674c05dc00 2023-01-24T03:37:29.980 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.968+0000 7f67526a0700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f674c166680 con 0x7f674c0c4f90 2023-01-24T03:37:29.980 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.968+0000 7f67526a0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f674c1674e0 con 0x7f674c06ee00 2023-01-24T03:37:29.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.968+0000 7f675169e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f674c05dc00 0x7f674c0c6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:29.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.968+0000 7f675169e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f674c05dc00 0x7f674c0c6c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54356/0 (socket says 172.21.15.112:54356) 2023-01-24T03:37:29.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.968+0000 7f675169e700 1 -- 172.21.15.112:0/2727490040 learned_addr learned my addr 172.21.15.112:0/2727490040 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:37:29.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.968+0000 7f6750e9d700 1 --2- 172.21.15.112:0/2727490040 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f674c06ee00 0x7f674c14b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:29.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.968+0000 7f6751e9f700 1 --2- 172.21.15.112:0/2727490040 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f674c0c4f90 0x7f674c14b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:29.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.968+0000 7f675169e700 1 -- 172.21.15.112:0/2727490040 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f674c06ee00 msgr2=0x7f674c14b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:29.982 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.968+0000 7f675169e700 1 --2- 172.21.15.112:0/2727490040 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f674c06ee00 0x7f674c14b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:29.982 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.968+0000 7f675169e700 1 -- 172.21.15.112:0/2727490040 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f674c0c4f90 msgr2=0x7f674c14b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:29.982 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.968+0000 7f675169e700 1 --2- 172.21.15.112:0/2727490040 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f674c0c4f90 0x7f674c14b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:29.982 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.968+0000 7f675169e700 1 -- 172.21.15.112:0/2727490040 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6744009a20 con 0x7f674c05dc00 2023-01-24T03:37:29.982 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.968+0000 7f675169e700 1 --2- 172.21.15.112:0/2727490040 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f674c05dc00 0x7f674c0c6c80 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f674800c590 tx=0x7f6748007d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:29.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.969+0000 7f67397fa700 1 -- 172.21.15.112:0/2727490040 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67480072e0 con 0x7f674c05dc00 2023-01-24T03:37:29.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.969+0000 7f67526a0700 1 -- 172.21.15.112:0/2727490040 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f674c163ea0 con 0x7f674c05dc00 2023-01-24T03:37:29.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.969+0000 7f67397fa700 1 -- 172.21.15.112:0/2727490040 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6748007440 con 0x7f674c05dc00 2023-01-24T03:37:29.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.969+0000 7f67397fa700 1 -- 172.21.15.112:0/2727490040 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6748005780 con 0x7f674c05dc00 2023-01-24T03:37:29.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.969+0000 7f67526a0700 1 -- 172.21.15.112:0/2727490040 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f674c166bb0 con 0x7f674c05dc00 2023-01-24T03:37:29.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.970+0000 7f67526a0700 1 -- 172.21.15.112:0/2727490040 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f674c05ea20 con 0x7f674c05dc00 2023-01-24T03:37:29.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.970+0000 7f67397fa700 1 -- 172.21.15.112:0/2727490040 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f6748019030 con 0x7f674c05dc00 2023-01-24T03:37:29.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.971+0000 7f67397fa700 1 --2- 172.21.15.112:0/2727490040 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f672c0611f0 0x7f672c0636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:29.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.971+0000 7f67397fa700 1 -- 172.21.15.112:0/2727490040 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 7211+0+0 (secure 0 0 0) 0x7f674807c990 con 0x7f674c05dc00 2023-01-24T03:37:29.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.971+0000 7f6750e9d700 1 --2- 172.21.15.112:0/2727490040 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f672c0611f0 0x7f672c0636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:29.985 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.971+0000 7f6750e9d700 1 --2- 172.21.15.112:0/2727490040 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f672c0611f0 0x7f672c0636a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f674c0bb730 tx=0x7f673c007bb0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:29.985 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:29.979+0000 7f67397fa700 1 -- 172.21.15.112:0/2727490040 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f674804c590 con 0x7f674c05dc00 2023-01-24T03:37:30.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:30.208+0000 7f67526a0700 1 -- 172.21.15.112:0/2727490040 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f674c04b340 con 0x7f674c05dc00 2023-01-24T03:37:30.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:30.209+0000 7f67397fa700 1 -- 172.21.15.112:0/2727490040 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+6118 (secure 0 0 0) 0x7f6748017090 con 0x7f674c05dc00 2023-01-24T03:37:30.212 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:37:30.212 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5296,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/2838716685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2838716685},{"type":"v1","addr":"172.21.15.142:6834","nonce":2838716685}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:36:42.756007+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4907},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4907":{"gid":4907,"name":"d","rank":0,"incarnation":27,"state":"up:active","state_seq":29,"addr":"172.21.15.142:6835/824478388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":824478388},{"type":"v1","addr":"172.21.15.142:6835","nonce":824478388}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:37:30.214 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:30.211+0000 7f67526a0700 1 -- 172.21.15.112:0/2727490040 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f672c0611f0 msgr2=0x7f672c0636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:30.214 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:30.211+0000 7f67526a0700 1 --2- 172.21.15.112:0/2727490040 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f672c0611f0 0x7f672c0636a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f674c0bb730 tx=0x7f673c007bb0 comp rx=0 tx=0).stop 2023-01-24T03:37:30.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:30.211+0000 7f67526a0700 1 -- 172.21.15.112:0/2727490040 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f674c05dc00 msgr2=0x7f674c0c6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:30.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:30.211+0000 7f67526a0700 1 --2- 172.21.15.112:0/2727490040 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f674c05dc00 0x7f674c0c6c80 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f674800c590 tx=0x7f6748007d20 comp rx=0 tx=0).stop 2023-01-24T03:37:30.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:30.211+0000 7f67526a0700 1 -- 172.21.15.112:0/2727490040 shutdown_connections 2023-01-24T03:37:30.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:30.211+0000 7f67526a0700 1 --2- 172.21.15.112:0/2727490040 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f672c0611f0 0x7f672c0636a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:30.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:30.211+0000 7f67526a0700 1 --2- 172.21.15.112:0/2727490040 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f674c05dc00 0x7f674c0c6c80 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:30.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:30.211+0000 7f67526a0700 1 --2- 172.21.15.112:0/2727490040 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f674c0c4f90 0x7f674c14b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:30.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:30.211+0000 7f67526a0700 1 --2- 172.21.15.112:0/2727490040 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f674c06ee00 0x7f674c14b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:30.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:30.211+0000 7f67526a0700 1 -- 172.21.15.112:0/2727490040 >> 172.21.15.112:0/2727490040 conn(0x7f674c0c87b0 msgr2=0x7f674c0c3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:37:30.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:30.211+0000 7f67526a0700 1 -- 172.21.15.112:0/2727490040 shutdown_connections 2023-01-24T03:37:30.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:30.211+0000 7f67526a0700 1 -- 172.21.15.112:0/2727490040 wait complete. 2023-01-24T03:37:30.217 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 31 2023-01-24T03:37:30.226 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-01-24T03:37:32.228 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:37:32.345 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.342+0000 7f2cf148c700 1 Processor -- start 2023-01-24T03:37:32.345 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.342+0000 7f2cf148c700 1 -- start start 2023-01-24T03:37:32.352 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.343+0000 7f2cf148c700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f2cec06ee00 0x7f2cec0c1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:32.353 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.343+0000 7f2cf148c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f2cec05dc00 0x7f2cec0c1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:32.353 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.343+0000 7f2cf148c700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cec156980 con 0x7f2cec0c4f90 2023-01-24T03:37:32.353 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.343+0000 7f2cf148c700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cec0c2a30 con 0x7f2cec06ee00 2023-01-24T03:37:32.353 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.343+0000 7f2cf148c700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cec068760 con 0x7f2cec05dc00 2023-01-24T03:37:32.354 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.343+0000 7f2cebfff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f2cec06ee00 0x7f2cec0c1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:32.354 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.343+0000 7f2cebfff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f2cec06ee00 0x7f2cec0c1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54370/0 (socket says 172.21.15.112:54370) 2023-01-24T03:37:32.355 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.343+0000 7f2cebfff700 1 -- 172.21.15.112:0/159420587 learned_addr learned my addr 172.21.15.112:0/159420587 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:37:32.355 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.343+0000 7f2ceb7fe700 1 --2- 172.21.15.112:0/159420587 >> v2:172.21.15.142:3300/0 conn(0x7f2cec05dc00 0x7f2cec0c1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:32.355 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.343+0000 7f2cebfff700 1 -- 172.21.15.112:0/159420587 >> v2:172.21.15.142:3300/0 conn(0x7f2cec05dc00 msgr2=0x7f2cec0c1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:37:32.355 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.343+0000 7f2cebfff700 1 --2- 172.21.15.112:0/159420587 >> v2:172.21.15.142:3300/0 conn(0x7f2cec05dc00 0x7f2cec0c1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:32.356 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.343+0000 7f2cebfff700 1 -- 172.21.15.112:0/159420587 >> v1:172.21.15.142:6789/0 conn(0x7f2cec0c4f90 legacy=0x7f2cec1534b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:32.356 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.343+0000 7f2cebfff700 1 -- 172.21.15.112:0/159420587 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cec156c60 con 0x7f2cec06ee00 2023-01-24T03:37:32.356 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.343+0000 7f2cebfff700 1 --2- 172.21.15.112:0/159420587 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2cec06ee00 0x7f2cec0c1330 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f2cec06d0c0 tx=0x7f2cdc0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57358d581cd0437d server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:32.356 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.343+0000 7f2ceaffd700 1 -- 172.21.15.112:0/159420587 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cdc011630 con 0x7f2cec06ee00 2023-01-24T03:37:32.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.343+0000 7f2ceaffd700 1 -- 172.21.15.112:0/159420587 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cdc0117d0 con 0x7f2cec06ee00 2023-01-24T03:37:32.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.343+0000 7f2ceaffd700 1 -- 172.21.15.112:0/159420587 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cdc011b90 con 0x7f2cec06ee00 2023-01-24T03:37:32.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.344+0000 7f2cf148c700 1 -- 172.21.15.112:0/159420587 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2cec06ee00 msgr2=0x7f2cec0c1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:32.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.344+0000 7f2cf148c700 1 --2- 172.21.15.112:0/159420587 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2cec06ee00 0x7f2cec0c1330 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f2cec06d0c0 tx=0x7f2cdc0090f0 comp rx=0 tx=0).stop 2023-01-24T03:37:32.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.344+0000 7f2cf148c700 1 -- 172.21.15.112:0/159420587 shutdown_connections 2023-01-24T03:37:32.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.344+0000 7f2cf148c700 1 --2- 172.21.15.112:0/159420587 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2cec06ee00 0x7f2cec0c1330 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:32.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.344+0000 7f2cf148c700 1 --2- 172.21.15.112:0/159420587 >> v2:172.21.15.142:3300/0 conn(0x7f2cec05dc00 0x7f2cec0c1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:32.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.344+0000 7f2cf148c700 1 -- 172.21.15.112:0/159420587 >> 172.21.15.112:0/159420587 conn(0x7f2cec0c87b0 msgr2=0x7f2cec0c4780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:37:32.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.344+0000 7f2cf148c700 1 -- 172.21.15.112:0/159420587 shutdown_connections 2023-01-24T03:37:32.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.344+0000 7f2cf148c700 1 -- 172.21.15.112:0/159420587 wait complete. 2023-01-24T03:37:32.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.346+0000 7f2cf148c700 1 Processor -- start 2023-01-24T03:37:32.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.346+0000 7f2cf148c700 1 -- start start 2023-01-24T03:37:32.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.347+0000 7f2cf148c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2cec05dc00 0x7f2cec16a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:32.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.347+0000 7f2cf148c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2cec06ee00 0x7f2cec0c0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:32.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.347+0000 7f2cf148c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2cec0c4f90 0x7f2cec154590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:32.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.347+0000 7f2cf148c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cec158150 con 0x7f2cec06ee00 2023-01-24T03:37:32.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.347+0000 7f2cf148c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cec1586d0 con 0x7f2cec05dc00 2023-01-24T03:37:32.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.347+0000 7f2cf148c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cec1577e0 con 0x7f2cec0c4f90 2023-01-24T03:37:32.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.347+0000 7f2ceb7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2cec06ee00 0x7f2cec0c0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:32.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.347+0000 7f2ceb7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2cec06ee00 0x7f2cec0c0460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54378/0 (socket says 172.21.15.112:54378) 2023-01-24T03:37:32.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.347+0000 7f2ceb7fe700 1 -- 172.21.15.112:0/1059666537 learned_addr learned my addr 172.21.15.112:0/1059666537 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:37:32.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.347+0000 7f2cebfff700 1 --2- 172.21.15.112:0/1059666537 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2cec05dc00 0x7f2cec16a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:32.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.347+0000 7f2cf0c8b700 1 --2- 172.21.15.112:0/1059666537 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2cec0c4f90 0x7f2cec154590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:32.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.347+0000 7f2ceb7fe700 1 -- 172.21.15.112:0/1059666537 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2cec0c4f90 msgr2=0x7f2cec154590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:32.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.347+0000 7f2ceb7fe700 1 --2- 172.21.15.112:0/1059666537 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2cec0c4f90 0x7f2cec154590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:32.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.347+0000 7f2ceb7fe700 1 -- 172.21.15.112:0/1059666537 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2cec05dc00 msgr2=0x7f2cec16a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:32.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.347+0000 7f2ceb7fe700 1 --2- 172.21.15.112:0/1059666537 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2cec05dc00 0x7f2cec16a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:32.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.347+0000 7f2ceb7fe700 1 -- 172.21.15.112:0/1059666537 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cdc009370 con 0x7f2cec06ee00 2023-01-24T03:37:32.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.348+0000 7f2ceb7fe700 1 --2- 172.21.15.112:0/1059666537 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2cec06ee00 0x7f2cec0c0460 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0006980 tx=0x7f2ce0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:32.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.348+0000 7f2cd3fff700 1 -- 172.21.15.112:0/1059666537 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce00125f0 con 0x7f2cec06ee00 2023-01-24T03:37:32.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.348+0000 7f2cd3fff700 1 -- 172.21.15.112:0/1059666537 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ce0012770 con 0x7f2cec06ee00 2023-01-24T03:37:32.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.348+0000 7f2cd3fff700 1 -- 172.21.15.112:0/1059666537 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce0012b30 con 0x7f2cec06ee00 2023-01-24T03:37:32.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.348+0000 7f2cf148c700 1 -- 172.21.15.112:0/1059666537 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cec157aa0 con 0x7f2cec06ee00 2023-01-24T03:37:32.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.348+0000 7f2cf148c700 1 -- 172.21.15.112:0/1059666537 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cec158360 con 0x7f2cec06ee00 2023-01-24T03:37:32.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.349+0000 7f2cf148c700 1 -- 172.21.15.112:0/1059666537 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cec157fa0 con 0x7f2cec06ee00 2023-01-24T03:37:32.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.350+0000 7f2cd3fff700 1 -- 172.21.15.112:0/1059666537 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f2ce001e460 con 0x7f2cec06ee00 2023-01-24T03:37:32.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.350+0000 7f2cd3fff700 1 --2- 172.21.15.112:0/1059666537 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2ccc0611f0 0x7f2ccc0636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:32.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.350+0000 7f2cebfff700 1 --2- 172.21.15.112:0/1059666537 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2ccc0611f0 0x7f2ccc0636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:32.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.350+0000 7f2cd3fff700 1 -- 172.21.15.112:0/1059666537 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 7211+0+0 (secure 0 0 0) 0x7f2ce007e950 con 0x7f2cec06ee00 2023-01-24T03:37:32.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.351+0000 7f2cebfff700 1 --2- 172.21.15.112:0/1059666537 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2ccc0611f0 0x7f2ccc0636a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc009340 tx=0x7f2cdc016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:32.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.357+0000 7f2cd3fff700 1 -- 172.21.15.112:0/1059666537 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f2ce004e5a0 con 0x7f2cec06ee00 2023-01-24T03:37:32.591 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.589+0000 7f2cf148c700 1 -- 172.21.15.112:0/1059666537 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2cec05ea20 con 0x7f2cec06ee00 2023-01-24T03:37:32.592 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.589+0000 7f2cd3fff700 1 -- 172.21.15.112:0/1059666537 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+6118 (secure 0 0 0) 0x7f2ce0051bc0 con 0x7f2cec06ee00 2023-01-24T03:37:32.593 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:37:32.593 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5296,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/2838716685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2838716685},{"type":"v1","addr":"172.21.15.142:6834","nonce":2838716685}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:36:42.756007+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4907},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4907":{"gid":4907,"name":"d","rank":0,"incarnation":27,"state":"up:active","state_seq":29,"addr":"172.21.15.142:6835/824478388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":824478388},{"type":"v1","addr":"172.21.15.142:6835","nonce":824478388}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:37:32.595 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.592+0000 7f2cf148c700 1 -- 172.21.15.112:0/1059666537 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2ccc0611f0 msgr2=0x7f2ccc0636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:32.595 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.592+0000 7f2cf148c700 1 --2- 172.21.15.112:0/1059666537 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2ccc0611f0 0x7f2ccc0636a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc009340 tx=0x7f2cdc016000 comp rx=0 tx=0).stop 2023-01-24T03:37:32.595 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.592+0000 7f2cf148c700 1 -- 172.21.15.112:0/1059666537 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2cec06ee00 msgr2=0x7f2cec0c0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:32.596 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.592+0000 7f2cf148c700 1 --2- 172.21.15.112:0/1059666537 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2cec06ee00 0x7f2cec0c0460 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0006980 tx=0x7f2ce0008040 comp rx=0 tx=0).stop 2023-01-24T03:37:32.596 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.592+0000 7f2cf148c700 1 -- 172.21.15.112:0/1059666537 shutdown_connections 2023-01-24T03:37:32.596 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.592+0000 7f2cf148c700 1 --2- 172.21.15.112:0/1059666537 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2ccc0611f0 0x7f2ccc0636a0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:32.596 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.592+0000 7f2cf148c700 1 --2- 172.21.15.112:0/1059666537 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2cec0c4f90 0x7f2cec154590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:32.597 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.592+0000 7f2cf148c700 1 --2- 172.21.15.112:0/1059666537 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2cec05dc00 0x7f2cec16a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:32.597 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.592+0000 7f2cf148c700 1 --2- 172.21.15.112:0/1059666537 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2cec06ee00 0x7f2cec0c0460 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:32.597 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.592+0000 7f2cf148c700 1 -- 172.21.15.112:0/1059666537 >> 172.21.15.112:0/1059666537 conn(0x7f2cec0c87b0 msgr2=0x7f2cec06b320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:37:32.597 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.592+0000 7f2cf148c700 1 -- 172.21.15.112:0/1059666537 shutdown_connections 2023-01-24T03:37:32.597 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:32.592+0000 7f2cf148c700 1 -- 172.21.15.112:0/1059666537 wait complete. 2023-01-24T03:37:32.598 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 31 2023-01-24T03:37:32.607 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-01-24T03:37:34.609 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:37:34.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.721+0000 7fa93759e700 1 Processor -- start 2023-01-24T03:37:34.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.721+0000 7fa93759e700 1 -- start start 2023-01-24T03:37:34.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.722+0000 7fa93759e700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fa93806dbb0 0x7fa93806e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:34.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.722+0000 7fa93759e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa93806f240 0x7fa93806e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:34.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.722+0000 7fa93759e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa93806c200 con 0x7fa93814bff0 2023-01-24T03:37:34.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.722+0000 7fa93759e700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa93806f010 con 0x7fa93806dbb0 2023-01-24T03:37:34.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.722+0000 7fa93759e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa938151ec0 con 0x7fa93806f240 2023-01-24T03:37:34.727 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.722+0000 7fa935d9b700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fa93806dbb0 0x7fa93806e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:34.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.722+0000 7fa93659c700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fa93814bff0 0x7fa9380c0a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35374/0 (socket says 172.21.15.112:35374) 2023-01-24T03:37:34.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.722+0000 7fa93659c700 1 -- 172.21.15.112:0/1334676263 learned_addr learned my addr 172.21.15.112:0/1334676263 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:37:34.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.722+0000 7fa936d9d700 1 --2- 172.21.15.112:0/1334676263 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa93806f240 0x7fa93806e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:34.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.722+0000 7fa935d9b700 1 -- 172.21.15.112:0/1334676263 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa93806f240 msgr2=0x7fa93806e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:37:34.728 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.722+0000 7fa935d9b700 1 --2- 172.21.15.112:0/1334676263 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa93806f240 0x7fa93806e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:34.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.722+0000 7fa935d9b700 1 -- 172.21.15.112:0/1334676263 >> v1:172.21.15.112:6789/0 conn(0x7fa93814bff0 legacy=0x7fa9380c0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:34.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.722+0000 7fa935d9b700 1 -- 172.21.15.112:0/1334676263 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa93806b8d0 con 0x7fa93806dbb0 2023-01-24T03:37:34.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.722+0000 7fa93559a700 1 -- 172.21.15.112:0/1334676263 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3510190115 0 0) 0x7fa93806c200 con 0x7fa93814bff0 2023-01-24T03:37:34.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.722+0000 7fa936d9d700 1 --2- 172.21.15.112:0/1334676263 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa93806f240 0x7fa93806e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-01-24T03:37:34.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.722+0000 7fa935d9b700 1 --2- 172.21.15.112:0/1334676263 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa93806dbb0 0x7fa93806e1b0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fa92c009760 tx=0x7fa92c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adc39655507d4a2 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:34.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.723+0000 7fa93559a700 1 -- 172.21.15.112:0/1334676263 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa92c011630 con 0x7fa93806dbb0 2023-01-24T03:37:34.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.723+0000 7fa93559a700 1 -- 172.21.15.112:0/1334676263 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa92c011790 con 0x7fa93806dbb0 2023-01-24T03:37:34.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.723+0000 7fa93559a700 1 -- 172.21.15.112:0/1334676263 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa92c011b30 con 0x7fa93806dbb0 2023-01-24T03:37:34.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.723+0000 7fa93759e700 1 -- 172.21.15.112:0/1334676263 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa93806dbb0 msgr2=0x7fa93806e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:34.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.723+0000 7fa93759e700 1 --2- 172.21.15.112:0/1334676263 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa93806dbb0 0x7fa93806e1b0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fa92c009760 tx=0x7fa92c009060 comp rx=0 tx=0).stop 2023-01-24T03:37:34.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.723+0000 7fa93759e700 1 -- 172.21.15.112:0/1334676263 shutdown_connections 2023-01-24T03:37:34.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.723+0000 7fa93759e700 1 --2- 172.21.15.112:0/1334676263 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa93806f240 0x7fa93806e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:34.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.723+0000 7fa93759e700 1 --2- 172.21.15.112:0/1334676263 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa93806dbb0 0x7fa93806e1b0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:34.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.723+0000 7fa93759e700 1 -- 172.21.15.112:0/1334676263 >> 172.21.15.112:0/1334676263 conn(0x7fa9380689b0 msgr2=0x7fa9380c8540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:37:34.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.723+0000 7fa93759e700 1 -- 172.21.15.112:0/1334676263 shutdown_connections 2023-01-24T03:37:34.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.723+0000 7fa93759e700 1 -- 172.21.15.112:0/1334676263 wait complete. 2023-01-24T03:37:34.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.725+0000 7fa93759e700 1 Processor -- start 2023-01-24T03:37:34.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.725+0000 7fa93759e700 1 -- start start 2023-01-24T03:37:34.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.726+0000 7fa93759e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa93806dbb0 0x7fa9381618c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:34.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.726+0000 7fa93759e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa93806f240 0x7fa938149d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:34.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.726+0000 7fa93759e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa93814bff0 0x7fa9380c1b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:34.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.726+0000 7fa93759e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9380c7250 con 0x7fa93806dbb0 2023-01-24T03:37:34.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.726+0000 7fa93759e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa938164490 con 0x7fa93814bff0 2023-01-24T03:37:34.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.726+0000 7fa93759e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa938164770 con 0x7fa93806f240 2023-01-24T03:37:34.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.726+0000 7fa93659c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa93806dbb0 0x7fa9381618c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:34.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.726+0000 7fa93659c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa93806dbb0 0x7fa9381618c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54392/0 (socket says 172.21.15.112:54392) 2023-01-24T03:37:34.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.726+0000 7fa93659c700 1 -- 172.21.15.112:0/995062199 learned_addr learned my addr 172.21.15.112:0/995062199 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:37:34.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.726+0000 7fa936d9d700 1 --2- 172.21.15.112:0/995062199 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa93814bff0 0x7fa9380c1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:34.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.727+0000 7fa935d9b700 1 --2- 172.21.15.112:0/995062199 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa93806f240 0x7fa938149d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:34.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.727+0000 7fa93659c700 1 -- 172.21.15.112:0/995062199 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa93806f240 msgr2=0x7fa938149d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:34.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.727+0000 7fa93659c700 1 --2- 172.21.15.112:0/995062199 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa93806f240 0x7fa938149d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:34.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.727+0000 7fa93659c700 1 -- 172.21.15.112:0/995062199 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa93814bff0 msgr2=0x7fa9380c1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:34.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.727+0000 7fa93659c700 1 --2- 172.21.15.112:0/995062199 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa93814bff0 0x7fa9380c1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:34.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.727+0000 7fa93659c700 1 -- 172.21.15.112:0/995062199 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa92c0092a0 con 0x7fa93806dbb0 2023-01-24T03:37:34.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.727+0000 7fa93659c700 1 --2- 172.21.15.112:0/995062199 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa93806dbb0 0x7fa9381618c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa928001eb0 tx=0x7fa928007e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:34.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.727+0000 7fa9267fc700 1 -- 172.21.15.112:0/995062199 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa92800ce90 con 0x7fa93806dbb0 2023-01-24T03:37:34.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.727+0000 7fa9267fc700 1 -- 172.21.15.112:0/995062199 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9280072e0 con 0x7fa93806dbb0 2023-01-24T03:37:34.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.727+0000 7fa93759e700 1 -- 172.21.15.112:0/995062199 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa938164a50 con 0x7fa93806dbb0 2023-01-24T03:37:34.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.727+0000 7fa9267fc700 1 -- 172.21.15.112:0/995062199 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa928005780 con 0x7fa93806dbb0 2023-01-24T03:37:34.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.727+0000 7fa93759e700 1 -- 172.21.15.112:0/995062199 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa938165010 con 0x7fa93806dbb0 2023-01-24T03:37:34.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.728+0000 7fa93759e700 1 -- 172.21.15.112:0/995062199 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9380c1960 con 0x7fa93806dbb0 2023-01-24T03:37:34.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.728+0000 7fa9267fc700 1 -- 172.21.15.112:0/995062199 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fa928019030 con 0x7fa93806dbb0 2023-01-24T03:37:34.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.729+0000 7fa9267fc700 1 --2- 172.21.15.112:0/995062199 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa9140611f0 0x7fa9140636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:34.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.729+0000 7fa9267fc700 1 -- 172.21.15.112:0/995062199 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 7211+0+0 (secure 0 0 0) 0x7fa92807cbd0 con 0x7fa93806dbb0 2023-01-24T03:37:34.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.729+0000 7fa935d9b700 1 --2- 172.21.15.112:0/995062199 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa9140611f0 0x7fa9140636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:34.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.729+0000 7fa935d9b700 1 --2- 172.21.15.112:0/995062199 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa9140611f0 0x7fa9140636a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa93806df80 tx=0x7fa92c017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:34.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.733+0000 7fa9267fc700 1 -- 172.21.15.112:0/995062199 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fa92804c880 con 0x7fa93806dbb0 2023-01-24T03:37:34.966 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.963+0000 7fa93759e700 1 -- 172.21.15.112:0/995062199 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa93805ea20 con 0x7fa93806dbb0 2023-01-24T03:37:34.967 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.964+0000 7fa9267fc700 1 -- 172.21.15.112:0/995062199 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+6118 (secure 0 0 0) 0x7fa92804fea0 con 0x7fa93806dbb0 2023-01-24T03:37:34.967 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:37:34.967 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5296,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/2838716685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2838716685},{"type":"v1","addr":"172.21.15.142:6834","nonce":2838716685}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:36:42.756007+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4907},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4907":{"gid":4907,"name":"d","rank":0,"incarnation":27,"state":"up:active","state_seq":29,"addr":"172.21.15.142:6835/824478388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":824478388},{"type":"v1","addr":"172.21.15.142:6835","nonce":824478388}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:37:34.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.966+0000 7fa93759e700 1 -- 172.21.15.112:0/995062199 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa9140611f0 msgr2=0x7fa9140636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:34.970 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.966+0000 7fa93759e700 1 --2- 172.21.15.112:0/995062199 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa9140611f0 0x7fa9140636a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa93806df80 tx=0x7fa92c017000 comp rx=0 tx=0).stop 2023-01-24T03:37:34.970 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.966+0000 7fa93759e700 1 -- 172.21.15.112:0/995062199 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa93806dbb0 msgr2=0x7fa9381618c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:34.970 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.966+0000 7fa93759e700 1 --2- 172.21.15.112:0/995062199 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa93806dbb0 0x7fa9381618c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa928001eb0 tx=0x7fa928007e90 comp rx=0 tx=0).stop 2023-01-24T03:37:34.970 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.966+0000 7fa93759e700 1 -- 172.21.15.112:0/995062199 shutdown_connections 2023-01-24T03:37:34.971 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.966+0000 7fa93759e700 1 --2- 172.21.15.112:0/995062199 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa9140611f0 0x7fa9140636a0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:34.971 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.966+0000 7fa93759e700 1 --2- 172.21.15.112:0/995062199 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa93806dbb0 0x7fa9381618c0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:34.971 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.966+0000 7fa93759e700 1 --2- 172.21.15.112:0/995062199 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa93814bff0 0x7fa9380c1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:34.971 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.966+0000 7fa93759e700 1 --2- 172.21.15.112:0/995062199 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa93806f240 0x7fa938149d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:34.971 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.966+0000 7fa93759e700 1 -- 172.21.15.112:0/995062199 >> 172.21.15.112:0/995062199 conn(0x7fa9380689b0 msgr2=0x7fa9380c66e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:37:34.972 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.966+0000 7fa93759e700 1 -- 172.21.15.112:0/995062199 shutdown_connections 2023-01-24T03:37:34.972 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:34.966+0000 7fa93759e700 1 -- 172.21.15.112:0/995062199 wait complete. 2023-01-24T03:37:34.972 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 31 2023-01-24T03:37:34.984 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-01-24T03:37:36.986 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:37:37.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.103+0000 7f87469a0700 1 Processor -- start 2023-01-24T03:37:37.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.103+0000 7f87469a0700 1 -- start start 2023-01-24T03:37:37.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.104+0000 7f87469a0700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f87400680e0 0x7f87400c4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:37.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.104+0000 7f87469a0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f87400684b0 0x7f87400c3c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:37.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.104+0000 7f87469a0700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87400c3800 con 0x7f87400c41d0 2023-01-24T03:37:37.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.104+0000 7f87469a0700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8740068880 con 0x7f87400680e0 2023-01-24T03:37:37.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.104+0000 7f87469a0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87400c53d0 con 0x7f87400684b0 2023-01-24T03:37:37.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.104+0000 7f874619f700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f87400c41d0 0x7f874006d990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35388/0 (socket says 172.21.15.112:35388) 2023-01-24T03:37:37.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.104+0000 7f874619f700 1 -- 172.21.15.112:0/1239941146 learned_addr learned my addr 172.21.15.112:0/1239941146 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:37:37.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.104+0000 7f874599e700 1 --2- 172.21.15.112:0/1239941146 >> v2:172.21.15.142:3300/0 conn(0x7f87400680e0 0x7f87400c4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:37.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.104+0000 7f874519d700 1 --2- 172.21.15.112:0/1239941146 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f87400684b0 0x7f87400c3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:37.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.104+0000 7f874499c700 1 -- 172.21.15.112:0/1239941146 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3561511413 0 0) 0x7f87400c3800 con 0x7f87400c41d0 2023-01-24T03:37:37.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.104+0000 7f874499c700 1 -- 172.21.15.112:0/1239941146 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87280033f0 con 0x7f87400c41d0 2023-01-24T03:37:37.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.104+0000 7f874499c700 1 -- 172.21.15.112:0/1239941146 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4169212178 0 0) 0x7f87280033f0 con 0x7f87400c41d0 2023-01-24T03:37:37.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.104+0000 7f874499c700 1 -- 172.21.15.112:0/1239941146 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f87400684b0 msgr2=0x7f87400c3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:37:37.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.104+0000 7f874499c700 1 --2- 172.21.15.112:0/1239941146 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f87400684b0 0x7f87400c3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:37.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.104+0000 7f874499c700 1 -- 172.21.15.112:0/1239941146 >> v2:172.21.15.142:3300/0 conn(0x7f87400680e0 msgr2=0x7f87400c4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:37:37.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.104+0000 7f874499c700 1 --2- 172.21.15.112:0/1239941146 >> v2:172.21.15.142:3300/0 conn(0x7f87400680e0 0x7f87400c4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:37.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.104+0000 7f874499c700 1 -- 172.21.15.112:0/1239941146 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87400c2650 con 0x7f87400c41d0 2023-01-24T03:37:37.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.104+0000 7f874499c700 1 -- 172.21.15.112:0/1239941146 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7f87380017f0 con 0x7f87400c41d0 2023-01-24T03:37:37.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.105+0000 7f874499c700 1 -- 172.21.15.112:0/1239941146 >> v1:172.21.15.112:6789/0 conn(0x7f87400c41d0 legacy=0x7f874006d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:37.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.105+0000 7f874499c700 1 --2- 172.21.15.112:0/1239941146 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f87280038e0 0x7f8728003cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:37.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.105+0000 7f874499c700 1 --2- 172.21.15.112:0/1239941146 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8728004690 0x7f8728004b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:37.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.105+0000 7f874499c700 1 --2- 172.21.15.112:0/1239941146 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f87400684b0 0x7f8728005040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:37.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.105+0000 7f874499c700 1 -- 172.21.15.112:0/1239941146 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8740068880 con 0x7f87280038e0 2023-01-24T03:37:37.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.105+0000 7f874499c700 1 -- 172.21.15.112:0/1239941146 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87400c53d0 con 0x7f8728004690 2023-01-24T03:37:37.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.105+0000 7f874499c700 1 -- 172.21.15.112:0/1239941146 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f872800b8e0 con 0x7f87400684b0 2023-01-24T03:37:37.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.105+0000 7f874599e700 1 --2- 172.21.15.112:0/1239941146 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f87280038e0 0x7f8728003cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:37.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.105+0000 7f874519d700 1 --2- 172.21.15.112:0/1239941146 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8728004690 0x7f8728004b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:37.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.105+0000 7f874619f700 1 --2- 172.21.15.112:0/1239941146 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f87400684b0 0x7f8728005040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:37.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.105+0000 7f874599e700 1 -- 172.21.15.112:0/1239941146 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f87400684b0 msgr2=0x7f8728005040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:37:37.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.105+0000 7f874599e700 1 --2- 172.21.15.112:0/1239941146 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f87400684b0 0x7f8728005040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:37.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.105+0000 7f874599e700 1 -- 172.21.15.112:0/1239941146 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8728004690 msgr2=0x7f8728004b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:37:37.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.105+0000 7f874599e700 1 --2- 172.21.15.112:0/1239941146 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8728004690 0x7f8728004b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:37.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.105+0000 7f874599e700 1 -- 172.21.15.112:0/1239941146 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87280033f0 con 0x7f87280038e0 2023-01-24T03:37:37.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.105+0000 7f874599e700 1 --2- 172.21.15.112:0/1239941146 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f87280038e0 0x7f8728003cf0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f8730009ab0 tx=0x7f8730009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25744f6ed8452b28 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:37.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.105+0000 7f874499c700 1 -- 172.21.15.112:0/1239941146 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8730009430 con 0x7f87280038e0 2023-01-24T03:37:37.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.105+0000 7f874499c700 1 -- 172.21.15.112:0/1239941146 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f873000e640 con 0x7f87280038e0 2023-01-24T03:37:37.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.105+0000 7f874499c700 1 -- 172.21.15.112:0/1239941146 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873000ea00 con 0x7f87280038e0 2023-01-24T03:37:37.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.106+0000 7f87469a0700 1 -- 172.21.15.112:0/1239941146 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f87280038e0 msgr2=0x7f8728003cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:37.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.106+0000 7f87469a0700 1 --2- 172.21.15.112:0/1239941146 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f87280038e0 0x7f8728003cf0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f8730009ab0 tx=0x7f8730009060 comp rx=0 tx=0).stop 2023-01-24T03:37:37.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.106+0000 7f874619f700 1 -- 172.21.15.112:0/1239941146 reap_dead start 2023-01-24T03:37:37.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.106+0000 7f87469a0700 1 -- 172.21.15.112:0/1239941146 shutdown_connections 2023-01-24T03:37:37.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.106+0000 7f87469a0700 1 -- 172.21.15.112:0/1239941146 >> 172.21.15.112:0/1239941146 conn(0x7f87400c87b0 msgr2=0x7f87400c4780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:37:37.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.106+0000 7f87469a0700 1 -- 172.21.15.112:0/1239941146 shutdown_connections 2023-01-24T03:37:37.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.106+0000 7f87469a0700 1 -- 172.21.15.112:0/1239941146 wait complete. 2023-01-24T03:37:37.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.107+0000 7f87469a0700 1 Processor -- start 2023-01-24T03:37:37.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.107+0000 7f87469a0700 1 -- start start 2023-01-24T03:37:37.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.108+0000 7f87469a0700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f87280038e0 0x7f8740153fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:37.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.108+0000 7f87469a0700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f874014f9d0 0x7f87400bdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:37.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.108+0000 7f87469a0700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f87400be030 0x7f8740150130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:37.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.108+0000 7f87469a0700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87400bd660 con 0x7f874014f9d0 2023-01-24T03:37:37.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.108+0000 7f87469a0700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87400bd0a0 con 0x7f87280038e0 2023-01-24T03:37:37.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.108+0000 7f87469a0700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87400be4a0 con 0x7f87400be030 2023-01-24T03:37:37.124 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.108+0000 7f874519d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f874014f9d0 0x7f87400bdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:37.124 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.108+0000 7f874519d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f874014f9d0 0x7f87400bdaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54412/0 (socket says 172.21.15.112:54412) 2023-01-24T03:37:37.124 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.108+0000 7f874519d700 1 -- 172.21.15.112:0/1320698346 learned_addr learned my addr 172.21.15.112:0/1320698346 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:37:37.124 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.108+0000 7f874599e700 1 --2- 172.21.15.112:0/1320698346 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f87280038e0 0x7f8740153fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:37.124 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.108+0000 7f874619f700 1 --2- 172.21.15.112:0/1320698346 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f87400be030 0x7f8740150130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:37.125 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.108+0000 7f874519d700 1 -- 172.21.15.112:0/1320698346 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f87400be030 msgr2=0x7f8740150130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:37.125 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.108+0000 7f874519d700 1 --2- 172.21.15.112:0/1320698346 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f87400be030 0x7f8740150130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:37.125 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.108+0000 7f874519d700 1 -- 172.21.15.112:0/1320698346 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f87280038e0 msgr2=0x7f8740153fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:37.125 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.108+0000 7f874519d700 1 --2- 172.21.15.112:0/1320698346 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f87280038e0 0x7f8740153fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:37.125 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.108+0000 7f874519d700 1 -- 172.21.15.112:0/1320698346 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87400bc780 con 0x7f874014f9d0 2023-01-24T03:37:37.126 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.108+0000 7f874519d700 1 --2- 172.21.15.112:0/1320698346 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f874014f9d0 0x7f87400bdaf0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f873c0119f0 tx=0x7f873c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:37.126 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.108+0000 7f872dffb700 1 -- 172.21.15.112:0/1320698346 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873c00f790 con 0x7f874014f9d0 2023-01-24T03:37:37.126 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.108+0000 7f872dffb700 1 -- 172.21.15.112:0/1320698346 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f873c00f8f0 con 0x7f874014f9d0 2023-01-24T03:37:37.126 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.108+0000 7f87469a0700 1 -- 172.21.15.112:0/1320698346 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87400bca20 con 0x7f874014f9d0 2023-01-24T03:37:37.126 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.108+0000 7f872dffb700 1 -- 172.21.15.112:0/1320698346 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873c00fc90 con 0x7f874014f9d0 2023-01-24T03:37:37.127 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.109+0000 7f87469a0700 1 -- 172.21.15.112:0/1320698346 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87400bd2f0 con 0x7f874014f9d0 2023-01-24T03:37:37.127 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.110+0000 7f87469a0700 1 -- 172.21.15.112:0/1320698346 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87400bcec0 con 0x7f874014f9d0 2023-01-24T03:37:37.127 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.110+0000 7f872dffb700 1 -- 172.21.15.112:0/1320698346 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f873c024070 con 0x7f874014f9d0 2023-01-24T03:37:37.127 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.110+0000 7f872dffb700 1 --2- 172.21.15.112:0/1320698346 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f87200611c0 0x7f8720063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:37.127 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.110+0000 7f872dffb700 1 -- 172.21.15.112:0/1320698346 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 7211+0+0 (secure 0 0 0) 0x7f873c018030 con 0x7f874014f9d0 2023-01-24T03:37:37.128 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.111+0000 7f874599e700 1 --2- 172.21.15.112:0/1320698346 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f87200611c0 0x7f8720063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:37.128 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.111+0000 7f874599e700 1 --2- 172.21.15.112:0/1320698346 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f87200611c0 0x7f8720063670 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8730006d50 tx=0x7f873001a000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:37.128 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.115+0000 7f872dffb700 1 -- 172.21.15.112:0/1320698346 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f87400bcec0 con 0x7f874014f9d0 2023-01-24T03:37:37.343 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.340+0000 7f87469a0700 1 -- 172.21.15.112:0/1320698346 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87400bcec0 con 0x7f874014f9d0 2023-01-24T03:37:37.344 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.340+0000 7f872dffb700 1 -- 172.21.15.112:0/1320698346 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+6118 (secure 0 0 0) 0x7f87400bcec0 con 0x7f874014f9d0 2023-01-24T03:37:37.344 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:37:37.344 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5296,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/2838716685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2838716685},{"type":"v1","addr":"172.21.15.142:6834","nonce":2838716685}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:36:42.756007+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4907},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4907":{"gid":4907,"name":"d","rank":0,"incarnation":27,"state":"up:active","state_seq":29,"addr":"172.21.15.142:6835/824478388","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":824478388},{"type":"v1","addr":"172.21.15.142:6835","nonce":824478388}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:37:37.346 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.343+0000 7f87469a0700 1 -- 172.21.15.112:0/1320698346 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f87200611c0 msgr2=0x7f8720063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:37.346 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.343+0000 7f87469a0700 1 --2- 172.21.15.112:0/1320698346 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f87200611c0 0x7f8720063670 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8730006d50 tx=0x7f873001a000 comp rx=0 tx=0).stop 2023-01-24T03:37:37.346 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.343+0000 7f87469a0700 1 -- 172.21.15.112:0/1320698346 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f874014f9d0 msgr2=0x7f87400bdaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:37.346 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.343+0000 7f87469a0700 1 --2- 172.21.15.112:0/1320698346 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f874014f9d0 0x7f87400bdaf0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f873c0119f0 tx=0x7f873c009060 comp rx=0 tx=0).stop 2023-01-24T03:37:37.347 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.343+0000 7f87469a0700 1 -- 172.21.15.112:0/1320698346 shutdown_connections 2023-01-24T03:37:37.347 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.343+0000 7f87469a0700 1 --2- 172.21.15.112:0/1320698346 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f87200611c0 0x7f8720063670 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:37.347 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.343+0000 7f87469a0700 1 --2- 172.21.15.112:0/1320698346 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f87400be030 0x7f8740150130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:37.347 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.343+0000 7f87469a0700 1 --2- 172.21.15.112:0/1320698346 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f87280038e0 0x7f8740153fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:37.348 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.343+0000 7f87469a0700 1 --2- 172.21.15.112:0/1320698346 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f874014f9d0 0x7f87400bdaf0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:37.348 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.343+0000 7f87469a0700 1 -- 172.21.15.112:0/1320698346 >> 172.21.15.112:0/1320698346 conn(0x7f87400c87b0 msgr2=0x7f87400c6670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:37:37.348 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.343+0000 7f87469a0700 1 -- 172.21.15.112:0/1320698346 shutdown_connections 2023-01-24T03:37:37.348 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:37.343+0000 7f87469a0700 1 -- 172.21.15.112:0/1320698346 wait complete. 2023-01-24T03:37:37.349 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 31 2023-01-24T03:37:37.358 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-01-24T03:37:39.359 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:37:39.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.481+0000 7f9ba5465700 1 Processor -- start 2023-01-24T03:37:39.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.481+0000 7f9ba5465700 1 -- start start 2023-01-24T03:37:39.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.482+0000 7f9ba5465700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f9ba0156e40 0x7f9ba00cb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:39.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.482+0000 7f9ba5465700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f9ba0155e20 0x7f9ba00cbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:39.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.482+0000 7f9ba5465700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ba00c3d10 con 0x7f9ba00c5680 2023-01-24T03:37:39.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.482+0000 7f9ba5465700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ba00c4eb0 con 0x7f9ba0156e40 2023-01-24T03:37:39.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.482+0000 7f9ba5465700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ba00cc2f0 con 0x7f9ba0155e20 2023-01-24T03:37:39.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.482+0000 7f9b97fff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f9ba0156e40 0x7f9ba00cb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:39.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.482+0000 7f9b97fff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f9ba0156e40 0x7f9ba00cb870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54430/0 (socket says 172.21.15.112:54430) 2023-01-24T03:37:39.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.482+0000 7f9b97fff700 1 -- 172.21.15.112:0/2573578595 learned_addr learned my addr 172.21.15.112:0/2573578595 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:37:39.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.482+0000 7f9ba4c64700 1 --2- 172.21.15.112:0/2573578595 >> v2:172.21.15.142:3300/0 conn(0x7f9ba0155e20 0x7f9ba00cbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:39.489 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.482+0000 7f9b97fff700 1 -- 172.21.15.112:0/2573578595 >> v2:172.21.15.142:3300/0 conn(0x7f9ba0155e20 msgr2=0x7f9ba00cbdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:37:39.489 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.482+0000 7f9b97fff700 1 --2- 172.21.15.112:0/2573578595 >> v2:172.21.15.142:3300/0 conn(0x7f9ba0155e20 0x7f9ba00cbdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:39.489 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.482+0000 7f9b97fff700 1 -- 172.21.15.112:0/2573578595 >> v1:172.21.15.142:6789/0 conn(0x7f9ba00c5680 legacy=0x7f9ba0156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:39.489 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.482+0000 7f9b97fff700 1 -- 172.21.15.112:0/2573578595 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba00c2e20 con 0x7f9ba0156e40 2023-01-24T03:37:39.489 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.483+0000 7f9b97fff700 1 --2- 172.21.15.112:0/2573578595 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ba0156e40 0x7f9ba00cb870 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f9ba00894c0 tx=0x7f9b880090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de60236379e9584 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:39.490 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.483+0000 7f9b9f7fe700 1 -- 172.21.15.112:0/2573578595 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b88011630 con 0x7f9ba0156e40 2023-01-24T03:37:39.490 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.483+0000 7f9b9f7fe700 1 -- 172.21.15.112:0/2573578595 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b88011790 con 0x7f9ba0156e40 2023-01-24T03:37:39.490 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.483+0000 7f9b9f7fe700 1 -- 172.21.15.112:0/2573578595 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b88011b50 con 0x7f9ba0156e40 2023-01-24T03:37:39.490 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.483+0000 7f9ba5465700 1 -- 172.21.15.112:0/2573578595 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ba0156e40 msgr2=0x7f9ba00cb870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:39.490 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.483+0000 7f9ba5465700 1 --2- 172.21.15.112:0/2573578595 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ba0156e40 0x7f9ba00cb870 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f9ba00894c0 tx=0x7f9b880090f0 comp rx=0 tx=0).stop 2023-01-24T03:37:39.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.483+0000 7f9ba5465700 1 -- 172.21.15.112:0/2573578595 shutdown_connections 2023-01-24T03:37:39.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.483+0000 7f9ba5465700 1 --2- 172.21.15.112:0/2573578595 >> v2:172.21.15.142:3300/0 conn(0x7f9ba0155e20 0x7f9ba00cbdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:39.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.483+0000 7f9ba5465700 1 --2- 172.21.15.112:0/2573578595 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ba0156e40 0x7f9ba00cb870 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:39.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.483+0000 7f9ba5465700 1 -- 172.21.15.112:0/2573578595 >> 172.21.15.112:0/2573578595 conn(0x7f9ba00685b0 msgr2=0x7f9ba00689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:37:39.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.483+0000 7f9ba5465700 1 -- 172.21.15.112:0/2573578595 shutdown_connections 2023-01-24T03:37:39.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.483+0000 7f9ba5465700 1 -- 172.21.15.112:0/2573578595 wait complete. 2023-01-24T03:37:39.494 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.484+0000 7f9ba5465700 1 Processor -- start 2023-01-24T03:37:39.494 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.484+0000 7f9ba5465700 1 -- start start 2023-01-24T03:37:39.494 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.485+0000 7f9ba5465700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ba00c5680 0x7f9ba0152e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:39.495 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.485+0000 7f9ba5465700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9ba0155e20 0x7f9ba01655d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:39.495 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.485+0000 7f9ba5465700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ba0156e40 0x7f9ba00bf8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:39.495 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.485+0000 7f9ba5465700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba00c21d0 con 0x7f9ba00c5680 2023-01-24T03:37:39.495 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.485+0000 7f9ba5465700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba00c08c0 con 0x7f9ba0156e40 2023-01-24T03:37:39.495 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.485+0000 7f9ba5465700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba00c0ba0 con 0x7f9ba0155e20 2023-01-24T03:37:39.496 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.485+0000 7f9b9ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ba00c5680 0x7f9ba0152e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:39.496 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.485+0000 7f9b9ffff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ba00c5680 0x7f9ba0152e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54434/0 (socket says 172.21.15.112:54434) 2023-01-24T03:37:39.496 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.485+0000 7f9b9ffff700 1 -- 172.21.15.112:0/2437686205 learned_addr learned my addr 172.21.15.112:0/2437686205 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:37:39.496 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.485+0000 7f9ba4c64700 1 --2- 172.21.15.112:0/2437686205 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ba0156e40 0x7f9ba00bf8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:39.496 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.485+0000 7f9b97fff700 1 --2- 172.21.15.112:0/2437686205 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9ba0155e20 0x7f9ba01655d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:39.497 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.486+0000 7f9b9ffff700 1 -- 172.21.15.112:0/2437686205 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9ba0155e20 msgr2=0x7f9ba01655d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:39.497 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.486+0000 7f9b9ffff700 1 --2- 172.21.15.112:0/2437686205 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9ba0155e20 0x7f9ba01655d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:39.497 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.486+0000 7f9b9ffff700 1 -- 172.21.15.112:0/2437686205 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ba0156e40 msgr2=0x7f9ba00bf8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:39.497 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.486+0000 7f9b9ffff700 1 --2- 172.21.15.112:0/2437686205 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ba0156e40 0x7f9ba00bf8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:39.498 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.486+0000 7f9b9ffff700 1 -- 172.21.15.112:0/2437686205 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b88009370 con 0x7f9ba00c5680 2023-01-24T03:37:39.498 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.486+0000 7f9b9ffff700 1 --2- 172.21.15.112:0/2437686205 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ba00c5680 0x7f9ba0152e70 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f9b9000c4c0 tx=0x7f9b90007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:39.498 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.486+0000 7f9b9cff9700 1 -- 172.21.15.112:0/2437686205 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b90011590 con 0x7f9ba00c5680 2023-01-24T03:37:39.498 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.486+0000 7f9b9cff9700 1 -- 172.21.15.112:0/2437686205 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b90011710 con 0x7f9ba00c5680 2023-01-24T03:37:39.498 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.486+0000 7f9b9cff9700 1 -- 172.21.15.112:0/2437686205 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b90011ad0 con 0x7f9ba00c5680 2023-01-24T03:37:39.499 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.486+0000 7f9ba5465700 1 -- 172.21.15.112:0/2437686205 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ba00c0e80 con 0x7f9ba00c5680 2023-01-24T03:37:39.499 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.486+0000 7f9ba5465700 1 -- 172.21.15.112:0/2437686205 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ba00c1440 con 0x7f9ba00c5680 2023-01-24T03:37:39.499 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.487+0000 7f9ba5465700 1 -- 172.21.15.112:0/2437686205 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ba005ea20 con 0x7f9ba00c5680 2023-01-24T03:37:39.499 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.487+0000 7f9b9cff9700 1 -- 172.21.15.112:0/2437686205 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f9b90019030 con 0x7f9ba00c5680 2023-01-24T03:37:39.499 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.488+0000 7f9b9cff9700 1 --2- 172.21.15.112:0/2437686205 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9b840611c0 0x7f9b84063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:39.499 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.488+0000 7f9b9cff9700 1 -- 172.21.15.112:0/2437686205 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 7297+0+0 (secure 0 0 0) 0x7f9b9007cdf0 con 0x7f9ba00c5680 2023-01-24T03:37:39.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.488+0000 7f9b97fff700 1 --2- 172.21.15.112:0/2437686205 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9b840611c0 0x7f9b84063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:39.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.488+0000 7f9b97fff700 1 --2- 172.21.15.112:0/2437686205 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9b840611c0 0x7f9b84063670 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9ba00894c0 tx=0x7f9b88016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:39.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.492+0000 7f9b9cff9700 1 -- 172.21.15.112:0/2437686205 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f9b9004c8f0 con 0x7f9ba00c5680 2023-01-24T03:37:39.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.712+0000 7f9ba5465700 1 -- 172.21.15.112:0/2437686205 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ba00c3100 con 0x7f9ba00c5680 2023-01-24T03:37:39.716 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.713+0000 7f9b9cff9700 1 -- 172.21.15.112:0/2437686205 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5360 (secure 0 0 0) 0x7f9b90017020 con 0x7f9ba00c5680 2023-01-24T03:37:39.716 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:37:39.717 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:37:38.677965+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5296":{"gid":5296,"name":"b","rank":0,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.142:6834/2838716685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2838716685},{"type":"v1","addr":"172.21.15.142:6834","nonce":2838716685}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:37:39.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.715+0000 7f9ba5465700 1 -- 172.21.15.112:0/2437686205 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9b840611c0 msgr2=0x7f9b84063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:39.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.715+0000 7f9ba5465700 1 --2- 172.21.15.112:0/2437686205 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9b840611c0 0x7f9b84063670 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9ba00894c0 tx=0x7f9b88016000 comp rx=0 tx=0).stop 2023-01-24T03:37:39.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.715+0000 7f9ba5465700 1 -- 172.21.15.112:0/2437686205 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ba00c5680 msgr2=0x7f9ba0152e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:39.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.715+0000 7f9ba5465700 1 --2- 172.21.15.112:0/2437686205 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ba00c5680 0x7f9ba0152e70 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f9b9000c4c0 tx=0x7f9b90007e00 comp rx=0 tx=0).stop 2023-01-24T03:37:39.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.715+0000 7f9ba5465700 1 -- 172.21.15.112:0/2437686205 shutdown_connections 2023-01-24T03:37:39.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.715+0000 7f9ba5465700 1 --2- 172.21.15.112:0/2437686205 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9b840611c0 0x7f9b84063670 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:39.720 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.715+0000 7f9ba5465700 1 --2- 172.21.15.112:0/2437686205 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ba00c5680 0x7f9ba0152e70 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:39.720 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.715+0000 7f9ba5465700 1 --2- 172.21.15.112:0/2437686205 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ba0156e40 0x7f9ba00bf8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:39.720 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.715+0000 7f9ba5465700 1 --2- 172.21.15.112:0/2437686205 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9ba0155e20 0x7f9ba01655d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:39.720 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.715+0000 7f9ba5465700 1 -- 172.21.15.112:0/2437686205 >> 172.21.15.112:0/2437686205 conn(0x7f9ba00685b0 msgr2=0x7f9ba014db90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:37:39.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.715+0000 7f9ba5465700 1 -- 172.21.15.112:0/2437686205 shutdown_connections 2023-01-24T03:37:39.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.715+0000 7f9ba5465700 1 -- 172.21.15.112:0/2437686205 wait complete. 2023-01-24T03:37:39.722 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 32 2023-01-24T03:37:39.730 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2023-01-24T03:37:39.730 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-01-24T03:37:39.731 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:37:39.844 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.841+0000 7f033a143700 1 Processor -- start 2023-01-24T03:37:39.844 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.841+0000 7f033a143700 1 -- start start 2023-01-24T03:37:39.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.841+0000 7f033a143700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03340c5680 0x7f0334156730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:39.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.841+0000 7f033a143700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f0334155e20 0x7f0334156c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:39.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.841+0000 7f033a143700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03340c3d10 con 0x7f03340c5a50 2023-01-24T03:37:39.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.841+0000 7f033a143700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03340c4eb0 con 0x7f0334155e20 2023-01-24T03:37:39.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.841+0000 7f033a143700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03340c2e20 con 0x7f03340c5680 2023-01-24T03:37:39.851 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.842+0000 7f0339942700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f03340c5a50 0x7f03340cb870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35432/0 (socket says 172.21.15.112:35432) 2023-01-24T03:37:39.851 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.842+0000 7f0339942700 1 -- 172.21.15.112:0/1023860414 learned_addr learned my addr 172.21.15.112:0/1023860414 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:37:39.851 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.842+0000 7f0338940700 1 --2- 172.21.15.112:0/1023860414 >> v2:172.21.15.142:3300/0 conn(0x7f0334155e20 0x7f0334156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:39.851 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.842+0000 7f0339141700 1 --2- 172.21.15.112:0/1023860414 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03340c5680 0x7f0334156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:39.852 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.842+0000 7f0333fff700 1 -- 172.21.15.112:0/1023860414 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1516640478 0 0) 0x7f03340c3d10 con 0x7f03340c5a50 2023-01-24T03:37:39.852 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.842+0000 7f0333fff700 1 -- 172.21.15.112:0/1023860414 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03200033f0 con 0x7f03340c5a50 2023-01-24T03:37:39.852 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.842+0000 7f0333fff700 1 -- 172.21.15.112:0/1023860414 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1415040048 0 0) 0x7f03200033f0 con 0x7f03340c5a50 2023-01-24T03:37:39.852 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.842+0000 7f0333fff700 1 -- 172.21.15.112:0/1023860414 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03340c5680 msgr2=0x7f0334156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:37:39.852 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.842+0000 7f0333fff700 1 --2- 172.21.15.112:0/1023860414 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03340c5680 0x7f0334156730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:39.853 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.842+0000 7f0333fff700 1 -- 172.21.15.112:0/1023860414 >> v2:172.21.15.142:3300/0 conn(0x7f0334155e20 msgr2=0x7f0334156c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:37:39.853 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.842+0000 7f0333fff700 1 --2- 172.21.15.112:0/1023860414 >> v2:172.21.15.142:3300/0 conn(0x7f0334155e20 0x7f0334156c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:39.853 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.842+0000 7f0333fff700 1 -- 172.21.15.112:0/1023860414 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03340c3100 con 0x7f03340c5a50 2023-01-24T03:37:39.853 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.842+0000 7f0333fff700 1 -- 172.21.15.112:0/1023860414 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7f032c001470 con 0x7f03340c5a50 2023-01-24T03:37:39.853 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.842+0000 7f0333fff700 1 -- 172.21.15.112:0/1023860414 >> v1:172.21.15.112:6789/0 conn(0x7f03340c5a50 legacy=0x7f03340cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:39.854 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.842+0000 7f0333fff700 1 --2- 172.21.15.112:0/1023860414 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03200038e0 0x7f0320003cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:39.854 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.842+0000 7f0333fff700 1 --2- 172.21.15.112:0/1023860414 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03340c5680 0x7f0320004690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:39.854 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.842+0000 7f0333fff700 1 --2- 172.21.15.112:0/1023860414 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0320004bd0 0x7f0320005040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:39.854 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.842+0000 7f0333fff700 1 -- 172.21.15.112:0/1023860414 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03340c4eb0 con 0x7f0320004bd0 2023-01-24T03:37:39.855 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.842+0000 7f0333fff700 1 -- 172.21.15.112:0/1023860414 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03340c2e20 con 0x7f03200038e0 2023-01-24T03:37:39.855 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.842+0000 7f0333fff700 1 -- 172.21.15.112:0/1023860414 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f032000b8e0 con 0x7f03340c5680 2023-01-24T03:37:39.855 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.842+0000 7f0339942700 1 --2- 172.21.15.112:0/1023860414 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0320004bd0 0x7f0320005040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:39.855 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.842+0000 7f0339141700 1 --2- 172.21.15.112:0/1023860414 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03200038e0 0x7f0320003cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:39.856 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.842+0000 7f0338940700 1 --2- 172.21.15.112:0/1023860414 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03340c5680 0x7f0320004690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:39.856 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.843+0000 7f0339942700 1 -- 172.21.15.112:0/1023860414 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03340c5680 msgr2=0x7f0320004690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:37:39.856 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.843+0000 7f0339942700 1 --2- 172.21.15.112:0/1023860414 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f03340c5680 0x7f0320004690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:39.856 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.843+0000 7f0339942700 1 -- 172.21.15.112:0/1023860414 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03200038e0 msgr2=0x7f0320003cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:37:39.856 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.843+0000 7f0339942700 1 --2- 172.21.15.112:0/1023860414 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f03200038e0 0x7f0320003cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:39.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.843+0000 7f0339942700 1 -- 172.21.15.112:0/1023860414 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03200033f0 con 0x7f0320004bd0 2023-01-24T03:37:39.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.843+0000 7f0339942700 1 --2- 172.21.15.112:0/1023860414 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0320004bd0 0x7f0320005040 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f032c00c190 tx=0x7f032c001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=302d45ed97df73de server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:39.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.843+0000 7f0333fff700 1 -- 172.21.15.112:0/1023860414 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f032c00ca60 con 0x7f0320004bd0 2023-01-24T03:37:39.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.843+0000 7f0333fff700 1 -- 172.21.15.112:0/1023860414 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f032c00cbc0 con 0x7f0320004bd0 2023-01-24T03:37:39.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.843+0000 7f0333fff700 1 -- 172.21.15.112:0/1023860414 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f032c0070d0 con 0x7f0320004bd0 2023-01-24T03:37:39.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.843+0000 7f033a143700 1 -- 172.21.15.112:0/1023860414 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0320004bd0 msgr2=0x7f0320005040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:39.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.843+0000 7f033a143700 1 --2- 172.21.15.112:0/1023860414 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0320004bd0 0x7f0320005040 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f032c00c190 tx=0x7f032c001e50 comp rx=0 tx=0).stop 2023-01-24T03:37:39.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.843+0000 7f0339942700 1 -- 172.21.15.112:0/1023860414 reap_dead start 2023-01-24T03:37:39.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.843+0000 7f033a143700 1 -- 172.21.15.112:0/1023860414 shutdown_connections 2023-01-24T03:37:39.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.843+0000 7f033a143700 1 -- 172.21.15.112:0/1023860414 >> 172.21.15.112:0/1023860414 conn(0x7f03340685b0 msgr2=0x7f03340689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:37:39.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.844+0000 7f033a143700 1 -- 172.21.15.112:0/1023860414 shutdown_connections 2023-01-24T03:37:39.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.844+0000 7f033a143700 1 -- 172.21.15.112:0/1023860414 wait complete. 2023-01-24T03:37:39.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.845+0000 7f033a143700 1 Processor -- start 2023-01-24T03:37:39.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.845+0000 7f033a143700 1 -- start start 2023-01-24T03:37:39.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.845+0000 7f033a143700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0320004bd0 0x7f0334149ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:39.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.845+0000 7f033a143700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f03340c88d0 0x7f03340c8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:39.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.845+0000 7f033a143700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0334151340 0x7f03341517b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:39.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.845+0000 7f033a143700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03340c6220 con 0x7f03340c88d0 2023-01-24T03:37:39.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.845+0000 7f033a143700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03340c64d0 con 0x7f0320004bd0 2023-01-24T03:37:39.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.845+0000 7f033a143700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03340c7e20 con 0x7f0334151340 2023-01-24T03:37:39.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.846+0000 7f0338940700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f03340c88d0 0x7f03340c8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:39.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.846+0000 7f0338940700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f03340c88d0 0x7f03340c8d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54456/0 (socket says 172.21.15.112:54456) 2023-01-24T03:37:39.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.846+0000 7f0338940700 1 -- 172.21.15.112:0/1485095689 learned_addr learned my addr 172.21.15.112:0/1485095689 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:37:39.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.846+0000 7f0339141700 1 --2- 172.21.15.112:0/1485095689 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0320004bd0 0x7f0334149ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:39.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.846+0000 7f0339942700 1 --2- 172.21.15.112:0/1485095689 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0334151340 0x7f03341517b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:39.862 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.846+0000 7f0338940700 1 -- 172.21.15.112:0/1485095689 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0334151340 msgr2=0x7f03341517b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:39.862 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.846+0000 7f0338940700 1 --2- 172.21.15.112:0/1485095689 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0334151340 0x7f03341517b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:39.862 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.846+0000 7f0338940700 1 -- 172.21.15.112:0/1485095689 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0320004bd0 msgr2=0x7f0334149ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:39.862 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.846+0000 7f0338940700 1 --2- 172.21.15.112:0/1485095689 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0320004bd0 0x7f0334149ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:39.862 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.846+0000 7f0338940700 1 -- 172.21.15.112:0/1485095689 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0334156d10 con 0x7f03340c88d0 2023-01-24T03:37:39.862 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.846+0000 7f0338940700 1 --2- 172.21.15.112:0/1485095689 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f03340c88d0 0x7f03340c8d40 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f0328011940 tx=0x7f0328009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:39.863 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.846+0000 7f03317fa700 1 -- 172.21.15.112:0/1485095689 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f032800f790 con 0x7f03340c88d0 2023-01-24T03:37:39.863 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.846+0000 7f03317fa700 1 -- 172.21.15.112:0/1485095689 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f032800f8f0 con 0x7f03340c88d0 2023-01-24T03:37:39.863 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.846+0000 7f033a143700 1 -- 172.21.15.112:0/1485095689 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0334154aa0 con 0x7f03340c88d0 2023-01-24T03:37:39.863 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.846+0000 7f03317fa700 1 -- 172.21.15.112:0/1485095689 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f032800fcb0 con 0x7f03340c88d0 2023-01-24T03:37:39.863 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.846+0000 7f033a143700 1 -- 172.21.15.112:0/1485095689 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0334156f60 con 0x7f03340c88d0 2023-01-24T03:37:39.864 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.847+0000 7f033a143700 1 -- 172.21.15.112:0/1485095689 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f033405ea20 con 0x7f03340c88d0 2023-01-24T03:37:39.865 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.847+0000 7f03317fa700 1 -- 172.21.15.112:0/1485095689 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f0328014440 con 0x7f03340c88d0 2023-01-24T03:37:39.865 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.848+0000 7f03317fa700 1 --2- 172.21.15.112:0/1485095689 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f03140611f0 0x7f03140636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:39.865 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.848+0000 7f03317fa700 1 -- 172.21.15.112:0/1485095689 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 7297+0+0 (secure 0 0 0) 0x7f0328018070 con 0x7f03340c88d0 2023-01-24T03:37:39.866 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.848+0000 7f0339141700 1 --2- 172.21.15.112:0/1485095689 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f03140611f0 0x7f03140636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:39.866 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.848+0000 7f0339141700 1 --2- 172.21.15.112:0/1485095689 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f03140611f0 0x7f03140636a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f03240094d0 tx=0x7f032400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:39.866 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:39.852+0000 7f03317fa700 1 -- 172.21.15.112:0/1485095689 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f032804e330 con 0x7f03340c88d0 2023-01-24T03:37:40.076 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:40.073+0000 7f033a143700 1 -- 172.21.15.112:0/1485095689 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f033404b340 con 0x7f03340c88d0 2023-01-24T03:37:40.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:40.074+0000 7f03317fa700 1 -- 172.21.15.112:0/1485095689 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5360 (secure 0 0 0) 0x7f0328051950 con 0x7f03340c88d0 2023-01-24T03:37:40.077 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:37:40.078 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:37:38.677965+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5296":{"gid":5296,"name":"b","rank":0,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.142:6834/2838716685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2838716685},{"type":"v1","addr":"172.21.15.142:6834","nonce":2838716685}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:37:40.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:40.076+0000 7f033a143700 1 -- 172.21.15.112:0/1485095689 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f03140611f0 msgr2=0x7f03140636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:40.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:40.076+0000 7f033a143700 1 --2- 172.21.15.112:0/1485095689 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f03140611f0 0x7f03140636a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f03240094d0 tx=0x7f032400b040 comp rx=0 tx=0).stop 2023-01-24T03:37:40.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:40.076+0000 7f033a143700 1 -- 172.21.15.112:0/1485095689 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f03340c88d0 msgr2=0x7f03340c8d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:40.080 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:40.076+0000 7f033a143700 1 --2- 172.21.15.112:0/1485095689 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f03340c88d0 0x7f03340c8d40 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f0328011940 tx=0x7f0328009060 comp rx=0 tx=0).stop 2023-01-24T03:37:40.080 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:40.076+0000 7f033a143700 1 -- 172.21.15.112:0/1485095689 shutdown_connections 2023-01-24T03:37:40.080 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:40.076+0000 7f033a143700 1 --2- 172.21.15.112:0/1485095689 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f03140611f0 0x7f03140636a0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:40.080 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:40.076+0000 7f033a143700 1 --2- 172.21.15.112:0/1485095689 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f0334151340 0x7f03341517b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:40.080 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:40.076+0000 7f033a143700 1 --2- 172.21.15.112:0/1485095689 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0320004bd0 0x7f0334149ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:40.081 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:40.076+0000 7f033a143700 1 --2- 172.21.15.112:0/1485095689 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f03340c88d0 0x7f03340c8d40 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:40.081 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:40.076+0000 7f033a143700 1 -- 172.21.15.112:0/1485095689 >> 172.21.15.112:0/1485095689 conn(0x7f03340685b0 msgr2=0x7f0334153a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:37:40.081 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:40.076+0000 7f033a143700 1 -- 172.21.15.112:0/1485095689 shutdown_connections 2023-01-24T03:37:40.081 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:40.076+0000 7f033a143700 1 -- 172.21.15.112:0/1485095689 wait complete. 2023-01-24T03:37:40.082 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 32 2023-01-24T03:37:40.091 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 32, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-01-24T03:30:20.971710+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 32, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5296": { "addr": "172.21.15.142:6834/2838716685", "addrs": { "addrvec": [ { "addr": "172.21.15.142:6832", "nonce": 2838716685, "type": "v2" }, { "addr": "172.21.15.142:6834", "nonce": 2838716685, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5296, "incarnation": 32, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 47, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-01-24T03:37:38.677965+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5296 } } } ], "standbys": [ { "addr": "172.21.15.142:6837/2181345869", "addrs": { "addrvec": [ { "addr": "172.21.15.142:6836", "nonce": 2181345869, "type": "v2" }, { "addr": "172.21.15.142:6837", "nonce": 2181345869, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6835/330343927", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6834", "nonce": 330343927, "type": "v2" }, { "addr": "172.21.15.112:6835", "nonce": 330343927, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6837/12176817", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6836", "nonce": 12176817, "type": "v2" }, { "addr": "172.21.15.112:6837", "nonce": 12176817, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4276, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6839/1110983841", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6838", "nonce": 1110983841, "type": "v2" }, { "addr": "172.21.15.112:6839", "nonce": 1110983841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4281, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-01-24T03:37:42.093 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:37:42.221 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.219+0000 7f663df5f700 1 Processor -- start 2023-01-24T03:37:42.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.219+0000 7f663df5f700 1 -- start start 2023-01-24T03:37:42.224 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.220+0000 7f663df5f700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f66380c7e40 0x7f66380b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:42.224 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.220+0000 7f663df5f700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f66380ca3e0 0x7f66380ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:42.224 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.220+0000 7f663df5f700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6638156240 con 0x7f6638155da0 2023-01-24T03:37:42.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.220+0000 7f663df5f700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66380b1170 con 0x7f66380c7e40 2023-01-24T03:37:42.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.220+0000 7f663df5f700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66380b1450 con 0x7f66380ca3e0 2023-01-24T03:37:42.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.220+0000 7f66377fe700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f66380c7e40 0x7f66380b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:42.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.220+0000 7f6636ffd700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f6638155da0 0x7f66380ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35456/0 (socket says 172.21.15.112:35456) 2023-01-24T03:37:42.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.220+0000 7f6636ffd700 1 -- 172.21.15.112:0/550859019 learned_addr learned my addr 172.21.15.112:0/550859019 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:37:42.226 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.221+0000 7f6637fff700 1 --2- 172.21.15.112:0/550859019 >> v2:172.21.15.142:3300/0 conn(0x7f66380ca3e0 0x7f66380ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:42.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.221+0000 7f66377fe700 1 -- 172.21.15.112:0/550859019 >> v2:172.21.15.142:3300/0 conn(0x7f66380ca3e0 msgr2=0x7f66380ca7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:37:42.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.221+0000 7f66377fe700 1 --2- 172.21.15.112:0/550859019 >> v2:172.21.15.142:3300/0 conn(0x7f66380ca3e0 0x7f66380ca7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:42.229 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.221+0000 7f66377fe700 1 -- 172.21.15.112:0/550859019 >> v1:172.21.15.112:6789/0 conn(0x7f6638155da0 legacy=0x7f66380ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:42.229 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.221+0000 7f66367fc700 1 -- 172.21.15.112:0/550859019 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2536768273 0 0) 0x7f6638156240 con 0x7f6638155da0 2023-01-24T03:37:42.229 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.221+0000 7f66377fe700 1 -- 172.21.15.112:0/550859019 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66380b1730 con 0x7f66380c7e40 2023-01-24T03:37:42.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.221+0000 7f66377fe700 1 --2- 172.21.15.112:0/550859019 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66380c7e40 0x7f66380b0aa0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f663805e810 tx=0x7f6628009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4e890b76d4b1603 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:42.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.221+0000 7f66367fc700 1 -- 172.21.15.112:0/550859019 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6628011630 con 0x7f66380c7e40 2023-01-24T03:37:42.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.221+0000 7f66367fc700 1 -- 172.21.15.112:0/550859019 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6628011790 con 0x7f66380c7e40 2023-01-24T03:37:42.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.221+0000 7f66367fc700 1 -- 172.21.15.112:0/550859019 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6628011b50 con 0x7f66380c7e40 2023-01-24T03:37:42.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.221+0000 7f663df5f700 1 -- 172.21.15.112:0/550859019 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66380c7e40 msgr2=0x7f66380b0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:42.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.221+0000 7f663df5f700 1 --2- 172.21.15.112:0/550859019 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66380c7e40 0x7f66380b0aa0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f663805e810 tx=0x7f6628009060 comp rx=0 tx=0).stop 2023-01-24T03:37:42.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.221+0000 7f663df5f700 1 -- 172.21.15.112:0/550859019 shutdown_connections 2023-01-24T03:37:42.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.221+0000 7f663df5f700 1 --2- 172.21.15.112:0/550859019 >> v2:172.21.15.142:3300/0 conn(0x7f66380ca3e0 0x7f66380ca7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:42.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.221+0000 7f663df5f700 1 --2- 172.21.15.112:0/550859019 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66380c7e40 0x7f66380b0aa0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:42.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.221+0000 7f663df5f700 1 -- 172.21.15.112:0/550859019 >> 172.21.15.112:0/550859019 conn(0x7f663805e2c0 msgr2=0x7f66380cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:37:42.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.221+0000 7f663df5f700 1 -- 172.21.15.112:0/550859019 shutdown_connections 2023-01-24T03:37:42.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.221+0000 7f663df5f700 1 -- 172.21.15.112:0/550859019 wait complete. 2023-01-24T03:37:42.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.223+0000 7f663df5f700 1 Processor -- start 2023-01-24T03:37:42.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.223+0000 7f663df5f700 1 -- start start 2023-01-24T03:37:42.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.223+0000 7f663df5f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66380c7e40 0x7f6638150630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:42.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.223+0000 7f663df5f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66380ca3e0 0x7f6638169140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:42.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.223+0000 7f663df5f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6638155da0 0x7f66380c1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:42.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.223+0000 7f663df5f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6638168cb0 con 0x7f66380c7e40 2023-01-24T03:37:42.238 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.223+0000 7f663df5f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6638168980 con 0x7f6638155da0 2023-01-24T03:37:42.238 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.223+0000 7f663df5f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66381547b0 con 0x7f66380ca3e0 2023-01-24T03:37:42.238 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.223+0000 7f66377fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66380c7e40 0x7f6638150630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:42.238 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.223+0000 7f66377fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66380c7e40 0x7f6638150630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54470/0 (socket says 172.21.15.112:54470) 2023-01-24T03:37:42.238 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.223+0000 7f66377fe700 1 -- 172.21.15.112:0/198366189 learned_addr learned my addr 172.21.15.112:0/198366189 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:37:42.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.224+0000 7f66377fe700 1 -- 172.21.15.112:0/198366189 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66380ca3e0 msgr2=0x7f6638169140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:42.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.224+0000 7f66377fe700 1 --2- 172.21.15.112:0/198366189 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66380ca3e0 0x7f6638169140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:42.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.224+0000 7f6637fff700 1 --2- 172.21.15.112:0/198366189 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6638155da0 0x7f66380c1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:42.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.224+0000 7f66377fe700 1 -- 172.21.15.112:0/198366189 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6638155da0 msgr2=0x7f66380c1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:42.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.224+0000 7f66377fe700 1 --2- 172.21.15.112:0/198366189 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6638155da0 0x7f66380c1aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:42.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.224+0000 7f66377fe700 1 -- 172.21.15.112:0/198366189 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66280092a0 con 0x7f66380c7e40 2023-01-24T03:37:42.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.224+0000 7f66377fe700 1 --2- 172.21.15.112:0/198366189 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66380c7e40 0x7f6638150630 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f6628009e80 tx=0x7f6628017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:42.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.224+0000 7f661f7fe700 1 -- 172.21.15.112:0/198366189 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6628000d10 con 0x7f66380c7e40 2023-01-24T03:37:42.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.224+0000 7f661f7fe700 1 -- 172.21.15.112:0/198366189 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6628011900 con 0x7f66380c7e40 2023-01-24T03:37:42.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.224+0000 7f663df5f700 1 -- 172.21.15.112:0/198366189 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6638154a90 con 0x7f66380c7e40 2023-01-24T03:37:42.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.224+0000 7f661f7fe700 1 -- 172.21.15.112:0/198366189 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6628025420 con 0x7f66380c7e40 2023-01-24T03:37:42.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.224+0000 7f663df5f700 1 -- 172.21.15.112:0/198366189 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6638155050 con 0x7f66380c7e40 2023-01-24T03:37:42.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.225+0000 7f663df5f700 1 -- 172.21.15.112:0/198366189 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6638168af0 con 0x7f66380c7e40 2023-01-24T03:37:42.242 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.225+0000 7f661f7fe700 1 -- 172.21.15.112:0/198366189 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f662802c030 con 0x7f66380c7e40 2023-01-24T03:37:42.242 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.226+0000 7f661f7fe700 1 --2- 172.21.15.112:0/198366189 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f66180611c0 0x7f6618063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:42.242 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.226+0000 7f661f7fe700 1 -- 172.21.15.112:0/198366189 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 7297+0+0 (secure 0 0 0) 0x7f662807f020 con 0x7f66380c7e40 2023-01-24T03:37:42.242 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.226+0000 7f6636ffd700 1 --2- 172.21.15.112:0/198366189 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f66180611c0 0x7f6618063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:42.242 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.226+0000 7f6636ffd700 1 --2- 172.21.15.112:0/198366189 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f66180611c0 0x7f6618063670 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f662c0011e0 tx=0x7f662c007850 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:42.243 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.230+0000 7f661f7fe700 1 -- 172.21.15.112:0/198366189 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f6638168af0 con 0x7f66380c7e40 2023-01-24T03:37:42.452 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.450+0000 7f663df5f700 1 -- 172.21.15.112:0/198366189 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6638168af0 con 0x7f66380c7e40 2023-01-24T03:37:42.453 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.451+0000 7f661f7fe700 1 -- 172.21.15.112:0/198366189 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5360 (secure 0 0 0) 0x7f6638168af0 con 0x7f66380c7e40 2023-01-24T03:37:42.454 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:37:42.454 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:37:38.677965+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5296":{"gid":5296,"name":"b","rank":0,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.142:6834/2838716685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2838716685},{"type":"v1","addr":"172.21.15.142:6834","nonce":2838716685}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:37:42.456 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.452+0000 7f663df5f700 1 -- 172.21.15.112:0/198366189 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f66180611c0 msgr2=0x7f6618063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:42.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.452+0000 7f663df5f700 1 --2- 172.21.15.112:0/198366189 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f66180611c0 0x7f6618063670 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f662c0011e0 tx=0x7f662c007850 comp rx=0 tx=0).stop 2023-01-24T03:37:42.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.452+0000 7f663df5f700 1 -- 172.21.15.112:0/198366189 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66380c7e40 msgr2=0x7f6638150630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:42.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.452+0000 7f663df5f700 1 --2- 172.21.15.112:0/198366189 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66380c7e40 0x7f6638150630 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f6628009e80 tx=0x7f6628017000 comp rx=0 tx=0).stop 2023-01-24T03:37:42.458 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.453+0000 7f663df5f700 1 -- 172.21.15.112:0/198366189 shutdown_connections 2023-01-24T03:37:42.458 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.453+0000 7f663df5f700 1 --2- 172.21.15.112:0/198366189 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f66180611c0 0x7f6618063670 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:42.459 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.453+0000 7f663df5f700 1 --2- 172.21.15.112:0/198366189 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66380c7e40 0x7f6638150630 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:42.459 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.453+0000 7f663df5f700 1 --2- 172.21.15.112:0/198366189 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6638155da0 0x7f66380c1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:42.459 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.453+0000 7f663df5f700 1 --2- 172.21.15.112:0/198366189 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66380ca3e0 0x7f6638169140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:42.460 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.453+0000 7f663df5f700 1 -- 172.21.15.112:0/198366189 >> 172.21.15.112:0/198366189 conn(0x7f663805e2c0 msgr2=0x7f6638069930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:37:42.460 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.453+0000 7f663df5f700 1 -- 172.21.15.112:0/198366189 shutdown_connections 2023-01-24T03:37:42.461 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:42.453+0000 7f663df5f700 1 -- 172.21.15.112:0/198366189 wait complete. 2023-01-24T03:37:42.461 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 32 2023-01-24T03:37:42.467 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:37:44.468 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:37:44.589 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.587+0000 7fa6b7818700 1 Processor -- start 2023-01-24T03:37:44.590 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.588+0000 7fa6b7818700 1 -- start start 2023-01-24T03:37:44.592 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.588+0000 7fa6b7818700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6b006f000 0x7fa6b00b8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:44.592 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.588+0000 7fa6b7818700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fa6b00b8850 0x7fa6b006c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:44.592 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.588+0000 7fa6b7818700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b006b900 con 0x7fa6b014f6e0 2023-01-24T03:37:44.593 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.588+0000 7fa6b7818700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6b00c3390 con 0x7fa6b00b8850 2023-01-24T03:37:44.593 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.588+0000 7fa6b7818700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b00c3d70 con 0x7fa6b006f000 2023-01-24T03:37:44.593 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.589+0000 7fa6b5db5700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fa6b00b8850 0x7fa6b006c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:44.593 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.589+0000 7fa6b5db5700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fa6b00b8850 0x7fa6b006c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54482/0 (socket says 172.21.15.112:54482) 2023-01-24T03:37:44.594 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.589+0000 7fa6b5db5700 1 -- 172.21.15.112:0/2737346872 learned_addr learned my addr 172.21.15.112:0/2737346872 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:37:44.594 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.589+0000 7fa6b4db3700 1 --2- 172.21.15.112:0/2737346872 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6b006f000 0x7fa6b00b8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:44.594 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.589+0000 7fa6b5db5700 1 -- 172.21.15.112:0/2737346872 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6b006f000 msgr2=0x7fa6b00b8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:37:44.594 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.589+0000 7fa6b5db5700 1 --2- 172.21.15.112:0/2737346872 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6b006f000 0x7fa6b00b8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:44.594 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.589+0000 7fa6b5db5700 1 -- 172.21.15.112:0/2737346872 >> v1:172.21.15.142:6789/0 conn(0x7fa6b014f6e0 legacy=0x7fa6b0155380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:44.595 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.589+0000 7fa6b5db5700 1 -- 172.21.15.112:0/2737346872 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b006bec0 con 0x7fa6b00b8850 2023-01-24T03:37:44.595 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.589+0000 7fa6b5db5700 1 --2- 172.21.15.112:0/2737346872 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa6b00b8850 0x7fa6b006c350 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fa6a8006fd0 tx=0x7fa6a80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=595d03d9eddd0d47 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:44.595 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.589+0000 7fa6affff700 1 -- 172.21.15.112:0/2737346872 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a8011630 con 0x7fa6b00b8850 2023-01-24T03:37:44.595 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.589+0000 7fa6affff700 1 -- 172.21.15.112:0/2737346872 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6a80117d0 con 0x7fa6b00b8850 2023-01-24T03:37:44.595 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.589+0000 7fa6affff700 1 -- 172.21.15.112:0/2737346872 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a8011b90 con 0x7fa6b00b8850 2023-01-24T03:37:44.596 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.590+0000 7fa6b7818700 1 -- 172.21.15.112:0/2737346872 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa6b00b8850 msgr2=0x7fa6b006c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:44.596 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.590+0000 7fa6b7818700 1 --2- 172.21.15.112:0/2737346872 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa6b00b8850 0x7fa6b006c350 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fa6a8006fd0 tx=0x7fa6a80090f0 comp rx=0 tx=0).stop 2023-01-24T03:37:44.596 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.590+0000 7fa6b7818700 1 -- 172.21.15.112:0/2737346872 shutdown_connections 2023-01-24T03:37:44.596 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.590+0000 7fa6b7818700 1 --2- 172.21.15.112:0/2737346872 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa6b00b8850 0x7fa6b006c350 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:44.596 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.590+0000 7fa6b7818700 1 --2- 172.21.15.112:0/2737346872 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6b006f000 0x7fa6b00b8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:44.597 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.590+0000 7fa6b7818700 1 -- 172.21.15.112:0/2737346872 >> 172.21.15.112:0/2737346872 conn(0x7fa6b0067f20 msgr2=0x7fa6b0068320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:37:44.597 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.590+0000 7fa6b7818700 1 -- 172.21.15.112:0/2737346872 shutdown_connections 2023-01-24T03:37:44.597 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.590+0000 7fa6b7818700 1 -- 172.21.15.112:0/2737346872 wait complete. 2023-01-24T03:37:44.600 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.591+0000 7fa6b7818700 1 Processor -- start 2023-01-24T03:37:44.600 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.591+0000 7fa6b7818700 1 -- start start 2023-01-24T03:37:44.600 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.591+0000 7fa6b7818700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6b006f000 0x7fa6b0166000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:44.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.592+0000 7fa6b7818700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa6b00b8850 0x7fa6b00c1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:44.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.592+0000 7fa6b7818700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6b014f6e0 0x7fa6b00c1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:44.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.592+0000 7fa6b7818700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b015b0d0 con 0x7fa6b00b8850 2023-01-24T03:37:44.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.592+0000 7fa6b7818700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b00c83d0 con 0x7fa6b006f000 2023-01-24T03:37:44.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.592+0000 7fa6b7818700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b0157c70 con 0x7fa6b014f6e0 2023-01-24T03:37:44.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.592+0000 7fa6b4db3700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa6b00b8850 0x7fa6b00c1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:44.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.592+0000 7fa6b4db3700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa6b00b8850 0x7fa6b00c1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54486/0 (socket says 172.21.15.112:54486) 2023-01-24T03:37:44.603 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.592+0000 7fa6b4db3700 1 -- 172.21.15.112:0/1228671803 learned_addr learned my addr 172.21.15.112:0/1228671803 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:37:44.603 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.592+0000 7fa6b55b4700 1 --2- 172.21.15.112:0/1228671803 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6b006f000 0x7fa6b0166000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:44.603 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.592+0000 7fa6b5db5700 1 --2- 172.21.15.112:0/1228671803 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6b014f6e0 0x7fa6b00c1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:44.603 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.592+0000 7fa6b4db3700 1 -- 172.21.15.112:0/1228671803 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6b014f6e0 msgr2=0x7fa6b00c1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:44.603 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.592+0000 7fa6b4db3700 1 --2- 172.21.15.112:0/1228671803 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6b014f6e0 0x7fa6b00c1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:44.604 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.592+0000 7fa6b4db3700 1 -- 172.21.15.112:0/1228671803 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6b006f000 msgr2=0x7fa6b0166000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:44.604 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.592+0000 7fa6b4db3700 1 --2- 172.21.15.112:0/1228671803 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6b006f000 0x7fa6b0166000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:44.604 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.592+0000 7fa6b4db3700 1 -- 172.21.15.112:0/1228671803 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a8009370 con 0x7fa6b00b8850 2023-01-24T03:37:44.604 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.592+0000 7fa6b55b4700 1 --2- 172.21.15.112:0/1228671803 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6b006f000 0x7fa6b0166000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:37:44.604 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.592+0000 7fa6b5db5700 1 --2- 172.21.15.112:0/1228671803 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6b014f6e0 0x7fa6b00c1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:37:44.605 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.592+0000 7fa6b4db3700 1 --2- 172.21.15.112:0/1228671803 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa6b00b8850 0x7fa6b00c1a50 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fa6a400e900 tx=0x7fa6a4009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:44.605 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.592+0000 7fa6ad7fa700 1 -- 172.21.15.112:0/1228671803 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a400c6d0 con 0x7fa6b00b8850 2023-01-24T03:37:44.605 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.592+0000 7fa6ad7fa700 1 -- 172.21.15.112:0/1228671803 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6a400c830 con 0x7fa6b00b8850 2023-01-24T03:37:44.605 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.592+0000 7fa6b7818700 1 -- 172.21.15.112:0/1228671803 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6b0158230 con 0x7fa6b00b8850 2023-01-24T03:37:44.605 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.592+0000 7fa6ad7fa700 1 -- 172.21.15.112:0/1228671803 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a400cbf0 con 0x7fa6b00b8850 2023-01-24T03:37:44.606 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.592+0000 7fa6b7818700 1 -- 172.21.15.112:0/1228671803 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6b0166fc0 con 0x7fa6b00b8850 2023-01-24T03:37:44.606 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.593+0000 7fa6b7818700 1 -- 172.21.15.112:0/1228671803 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6b004bb80 con 0x7fa6b00b8850 2023-01-24T03:37:44.606 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.593+0000 7fa6ad7fa700 1 -- 172.21.15.112:0/1228671803 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fa6a4019030 con 0x7fa6b00b8850 2023-01-24T03:37:44.606 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.594+0000 7fa6ad7fa700 1 --2- 172.21.15.112:0/1228671803 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa6900611c0 0x7fa690063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:44.606 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.594+0000 7fa6ad7fa700 1 -- 172.21.15.112:0/1228671803 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 7297+0+0 (secure 0 0 0) 0x7fa6a407d9e0 con 0x7fa6b00b8850 2023-01-24T03:37:44.607 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.594+0000 7fa6b55b4700 1 --2- 172.21.15.112:0/1228671803 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa6900611c0 0x7fa690063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:44.607 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.594+0000 7fa6b55b4700 1 --2- 172.21.15.112:0/1228671803 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa6900611c0 0x7fa690063670 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa6a0001e30 tx=0x7fa6a0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:44.607 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.598+0000 7fa6ad7fa700 1 -- 172.21.15.112:0/1228671803 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fa6a404d630 con 0x7fa6b00b8850 2023-01-24T03:37:44.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.828+0000 7fa6b7818700 1 -- 172.21.15.112:0/1228671803 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6b0067a30 con 0x7fa6b00b8850 2023-01-24T03:37:44.831 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.829+0000 7fa6ad7fa700 1 -- 172.21.15.112:0/1228671803 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5360 (secure 0 0 0) 0x7fa6a4050c50 con 0x7fa6b00b8850 2023-01-24T03:37:44.831 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:37:44.831 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:37:38.677965+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5296":{"gid":5296,"name":"b","rank":0,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.142:6834/2838716685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2838716685},{"type":"v1","addr":"172.21.15.142:6834","nonce":2838716685}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:37:44.833 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.831+0000 7fa6b7818700 1 -- 172.21.15.112:0/1228671803 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa6900611c0 msgr2=0x7fa690063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:44.833 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.831+0000 7fa6b7818700 1 --2- 172.21.15.112:0/1228671803 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa6900611c0 0x7fa690063670 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa6a0001e30 tx=0x7fa6a0009040 comp rx=0 tx=0).stop 2023-01-24T03:37:44.834 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.831+0000 7fa6b7818700 1 -- 172.21.15.112:0/1228671803 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa6b00b8850 msgr2=0x7fa6b00c1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:44.834 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.831+0000 7fa6b7818700 1 --2- 172.21.15.112:0/1228671803 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa6b00b8850 0x7fa6b00c1a50 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fa6a400e900 tx=0x7fa6a4009060 comp rx=0 tx=0).stop 2023-01-24T03:37:44.834 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.831+0000 7fa6b7818700 1 -- 172.21.15.112:0/1228671803 shutdown_connections 2023-01-24T03:37:44.834 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.831+0000 7fa6b7818700 1 --2- 172.21.15.112:0/1228671803 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa6900611c0 0x7fa690063670 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:44.834 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.831+0000 7fa6b7818700 1 --2- 172.21.15.112:0/1228671803 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa6b014f6e0 0x7fa6b00c1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:44.835 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.831+0000 7fa6b7818700 1 --2- 172.21.15.112:0/1228671803 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa6b006f000 0x7fa6b0166000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:44.835 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.831+0000 7fa6b7818700 1 --2- 172.21.15.112:0/1228671803 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa6b00b8850 0x7fa6b00c1a50 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:44.835 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.831+0000 7fa6b7818700 1 -- 172.21.15.112:0/1228671803 >> 172.21.15.112:0/1228671803 conn(0x7fa6b0067f20 msgr2=0x7fa6b014c460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:37:44.835 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.831+0000 7fa6b7818700 1 -- 172.21.15.112:0/1228671803 shutdown_connections 2023-01-24T03:37:44.835 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:44.831+0000 7fa6b7818700 1 -- 172.21.15.112:0/1228671803 wait complete. 2023-01-24T03:37:44.836 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 32 2023-01-24T03:37:44.845 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:37:46.845 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:37:46.971 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.969+0000 7f5fb564c700 1 Processor -- start 2023-01-24T03:37:46.971 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.969+0000 7f5fb564c700 1 -- start start 2023-01-24T03:37:46.974 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.970+0000 7f5fb564c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fb00caaf0 0x7f5fb0155da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:46.974 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.970+0000 7f5fb564c700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fb00b0f40 con 0x7f5fb00c7e40 2023-01-24T03:37:46.975 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.970+0000 7f5fb564c700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fb00b1170 con 0x7f5fb00b0aa0 2023-01-24T03:37:46.975 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.970+0000 7f5fb564c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb00b1450 con 0x7f5fb00caaf0 2023-01-24T03:37:46.975 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.970+0000 7f5faeffd700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f5fb00c7e40 0x7f5fb00ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35484/0 (socket says 172.21.15.112:35484) 2023-01-24T03:37:46.975 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.970+0000 7f5faeffd700 1 -- 172.21.15.112:0/2117715839 learned_addr learned my addr 172.21.15.112:0/2117715839 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:37:46.975 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.970+0000 7f5faf7fe700 1 --2- 172.21.15.112:0/2117715839 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fb00caaf0 0x7f5fb0155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:46.976 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.970+0000 7f5fadffb700 1 -- 172.21.15.112:0/2117715839 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2127329963 0 0) 0x7f5fb00b0f40 con 0x7f5fb00c7e40 2023-01-24T03:37:46.976 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.970+0000 7f5fadffb700 1 -- 172.21.15.112:0/2117715839 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f900033f0 con 0x7f5fb00c7e40 2023-01-24T03:37:46.976 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.970+0000 7f5fadffb700 1 -- 172.21.15.112:0/2117715839 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1381859896 0 0) 0x7f5fb00b1170 con 0x7f5fb00b0aa0 2023-01-24T03:37:46.976 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.971+0000 7f5fadffb700 1 -- 172.21.15.112:0/2117715839 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fb00b0f40 con 0x7f5fb00b0aa0 2023-01-24T03:37:46.976 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.971+0000 7f5faf7fe700 1 -- 172.21.15.112:0/2117715839 >> v1:172.21.15.142:6789/0 conn(0x7f5fb00b0aa0 legacy=0x7f5fb00ca3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:46.977 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.971+0000 7f5faf7fe700 1 -- 172.21.15.112:0/2117715839 >> v1:172.21.15.112:6789/0 conn(0x7f5fb00c7e40 legacy=0x7f5fb00ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:46.977 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.971+0000 7f5faf7fe700 1 -- 172.21.15.112:0/2117715839 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb00b1730 con 0x7f5fb00caaf0 2023-01-24T03:37:46.977 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.971+0000 7f5faf7fe700 1 --2- 172.21.15.112:0/2117715839 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fb00caaf0 0x7f5fb0155da0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8009790 tx=0x7f5fa8009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4343a73890218b6d server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:46.977 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.971+0000 7f5fadffb700 1 -- 172.21.15.112:0/2117715839 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa8011630 con 0x7f5fb00caaf0 2023-01-24T03:37:46.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.971+0000 7f5fadffb700 1 -- 172.21.15.112:0/2117715839 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa8011790 con 0x7f5fb00caaf0 2023-01-24T03:37:46.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.971+0000 7f5fadffb700 1 -- 172.21.15.112:0/2117715839 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa8011b50 con 0x7f5fb00caaf0 2023-01-24T03:37:46.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.971+0000 7f5fb564c700 1 -- 172.21.15.112:0/2117715839 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fb00caaf0 msgr2=0x7f5fb0155da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:46.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.971+0000 7f5fb564c700 1 --2- 172.21.15.112:0/2117715839 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fb00caaf0 0x7f5fb0155da0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8009790 tx=0x7f5fa8009060 comp rx=0 tx=0).stop 2023-01-24T03:37:46.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.971+0000 7f5fb564c700 1 -- 172.21.15.112:0/2117715839 shutdown_connections 2023-01-24T03:37:46.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.971+0000 7f5fb564c700 1 --2- 172.21.15.112:0/2117715839 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fb00caaf0 0x7f5fb0155da0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:46.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.971+0000 7f5fb564c700 1 -- 172.21.15.112:0/2117715839 >> 172.21.15.112:0/2117715839 conn(0x7f5fb005e2c0 msgr2=0x7f5fb00cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:37:46.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.972+0000 7f5fb564c700 1 -- 172.21.15.112:0/2117715839 shutdown_connections 2023-01-24T03:37:46.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.972+0000 7f5fb564c700 1 -- 172.21.15.112:0/2117715839 wait complete. 2023-01-24T03:37:46.982 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.973+0000 7f5fb564c700 1 Processor -- start 2023-01-24T03:37:46.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.973+0000 7f5fb564c700 1 -- start start 2023-01-24T03:37:46.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.973+0000 7f5fb564c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fb00b0aa0 0x7f5fb0165e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:46.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.973+0000 7f5fb564c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5fb00c7e40 0x7f5fb014f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:46.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.973+0000 7f5fb564c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fb00caaf0 0x7f5fb0152110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:46.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.973+0000 7f5fb564c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb0151280 con 0x7f5fb00caaf0 2023-01-24T03:37:46.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.973+0000 7f5fb564c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb00c27c0 con 0x7f5fb00c7e40 2023-01-24T03:37:46.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.973+0000 7f5fb564c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb00c4ad0 con 0x7f5fb00b0aa0 2023-01-24T03:37:46.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.974+0000 7f5faf7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fb00caaf0 0x7f5fb0152110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:46.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.974+0000 7f5faf7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fb00caaf0 0x7f5fb0152110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54506/0 (socket says 172.21.15.112:54506) 2023-01-24T03:37:46.985 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.974+0000 7f5faf7fe700 1 -- 172.21.15.112:0/2864548771 learned_addr learned my addr 172.21.15.112:0/2864548771 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:37:46.985 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.974+0000 7f5faeffd700 1 --2- 172.21.15.112:0/2864548771 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fb00b0aa0 0x7f5fb0165e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:46.985 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.974+0000 7f5fae7fc700 1 --2- 172.21.15.112:0/2864548771 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5fb00c7e40 0x7f5fb014f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:46.985 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.974+0000 7f5faf7fe700 1 -- 172.21.15.112:0/2864548771 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fb00b0aa0 msgr2=0x7f5fb0165e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:46.985 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.974+0000 7f5faf7fe700 1 --2- 172.21.15.112:0/2864548771 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fb00b0aa0 0x7f5fb0165e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:46.986 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.974+0000 7f5faf7fe700 1 -- 172.21.15.112:0/2864548771 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5fb00c7e40 msgr2=0x7f5fb014f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:46.986 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.974+0000 7f5faf7fe700 1 --2- 172.21.15.112:0/2864548771 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5fb00c7e40 0x7f5fb014f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:46.986 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.974+0000 7f5faf7fe700 1 -- 172.21.15.112:0/2864548771 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa80092a0 con 0x7f5fb00caaf0 2023-01-24T03:37:46.986 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.974+0000 7f5faf7fe700 1 --2- 172.21.15.112:0/2864548771 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fb00caaf0 0x7f5fb0152110 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f5fb0089340 tx=0x7f5fa8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:46.986 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.974+0000 7f5fa6ffd700 1 -- 172.21.15.112:0/2864548771 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa801f070 con 0x7f5fb00caaf0 2023-01-24T03:37:46.987 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.974+0000 7f5fa6ffd700 1 -- 172.21.15.112:0/2864548771 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa8011b10 con 0x7f5fb00caaf0 2023-01-24T03:37:46.987 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.974+0000 7f5fa6ffd700 1 -- 172.21.15.112:0/2864548771 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa8002f70 con 0x7f5fb00caaf0 2023-01-24T03:37:46.987 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.974+0000 7f5fb564c700 1 -- 172.21.15.112:0/2864548771 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fb01659a0 con 0x7f5fb00caaf0 2023-01-24T03:37:46.987 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.974+0000 7f5fb564c700 1 -- 172.21.15.112:0/2864548771 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fb00c4510 con 0x7f5fb00caaf0 2023-01-24T03:37:46.987 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.975+0000 7f5fb564c700 1 -- 172.21.15.112:0/2864548771 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fb0165b30 con 0x7f5fb00caaf0 2023-01-24T03:37:46.987 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.975+0000 7f5fa6ffd700 1 -- 172.21.15.112:0/2864548771 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f5fa802a030 con 0x7f5fb00caaf0 2023-01-24T03:37:46.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.976+0000 7f5fa6ffd700 1 --2- 172.21.15.112:0/2864548771 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5f940611c0 0x7f5f94063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:46.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.976+0000 7f5fa6ffd700 1 -- 172.21.15.112:0/2864548771 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 7297+0+0 (secure 0 0 0) 0x7f5fa807ebd0 con 0x7f5fb00caaf0 2023-01-24T03:37:46.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.976+0000 7f5faeffd700 1 --2- 172.21.15.112:0/2864548771 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5f940611c0 0x7f5f94063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:46.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.976+0000 7f5faeffd700 1 --2- 172.21.15.112:0/2864548771 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5f940611c0 0x7f5f94063670 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0005ad0 tx=0x7f5fa0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:46.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:46.980+0000 7f5fa6ffd700 1 -- 172.21.15.112:0/2864548771 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f5fa804e8d0 con 0x7f5fb00caaf0 2023-01-24T03:37:47.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:47.200+0000 7f5fb564c700 1 -- 172.21.15.112:0/2864548771 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5fb00c5df0 con 0x7f5fb00caaf0 2023-01-24T03:37:47.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:47.201+0000 7f5fa6ffd700 1 -- 172.21.15.112:0/2864548771 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5364 (secure 0 0 0) 0x7f5fa8051ef0 con 0x7f5fb00caaf0 2023-01-24T03:37:47.203 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:37:47.204 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:37:46.721488+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5296":{"gid":5296,"name":"b","rank":0,"incarnation":32,"state":"up:reconnect","state_seq":16,"addr":"172.21.15.142:6834/2838716685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2838716685},{"type":"v1","addr":"172.21.15.142:6834","nonce":2838716685}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:37:47.205 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:47.202+0000 7f5fb564c700 1 -- 172.21.15.112:0/2864548771 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5f940611c0 msgr2=0x7f5f94063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:47.205 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:47.202+0000 7f5fb564c700 1 --2- 172.21.15.112:0/2864548771 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5f940611c0 0x7f5f94063670 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0005ad0 tx=0x7f5fa0009040 comp rx=0 tx=0).stop 2023-01-24T03:37:47.206 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:47.202+0000 7f5fb564c700 1 -- 172.21.15.112:0/2864548771 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fb00caaf0 msgr2=0x7f5fb0152110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:47.206 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:47.202+0000 7f5fb564c700 1 --2- 172.21.15.112:0/2864548771 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fb00caaf0 0x7f5fb0152110 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f5fb0089340 tx=0x7f5fa8017000 comp rx=0 tx=0).stop 2023-01-24T03:37:47.206 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:47.203+0000 7f5fb564c700 1 -- 172.21.15.112:0/2864548771 shutdown_connections 2023-01-24T03:37:47.206 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:47.203+0000 7f5fb564c700 1 --2- 172.21.15.112:0/2864548771 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5f940611c0 0x7f5f94063670 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:47.206 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:47.203+0000 7f5fb564c700 1 --2- 172.21.15.112:0/2864548771 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fb00caaf0 0x7f5fb0152110 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:47.207 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:47.203+0000 7f5fb564c700 1 --2- 172.21.15.112:0/2864548771 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fb00b0aa0 0x7f5fb0165e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:47.207 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:47.203+0000 7f5fb564c700 1 --2- 172.21.15.112:0/2864548771 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5fb00c7e40 0x7f5fb014f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:47.207 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:47.203+0000 7f5fb564c700 1 -- 172.21.15.112:0/2864548771 >> 172.21.15.112:0/2864548771 conn(0x7f5fb005e2c0 msgr2=0x7f5fb0155160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:37:47.207 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:47.203+0000 7f5fb564c700 1 -- 172.21.15.112:0/2864548771 shutdown_connections 2023-01-24T03:37:47.207 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:47.203+0000 7f5fb564c700 1 -- 172.21.15.112:0/2864548771 wait complete. 2023-01-24T03:37:47.208 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 33 2023-01-24T03:37:47.217 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:37:49.218 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:37:49.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.340+0000 7ff34dc91700 1 Processor -- start 2023-01-24T03:37:49.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.340+0000 7ff34dc91700 1 -- start start 2023-01-24T03:37:49.348 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.341+0000 7ff34dc91700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff3480c7e40 0x7ff3480b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:49.349 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.341+0000 7ff34dc91700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff348156210 con 0x7ff3480ca3e0 2023-01-24T03:37:49.349 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.341+0000 7ff34dc91700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3480b1170 con 0x7ff348155da0 2023-01-24T03:37:49.349 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.341+0000 7ff34dc91700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3480b1450 con 0x7ff3480c7e40 2023-01-24T03:37:49.349 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.341+0000 7ff347fff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7ff3480ca3e0 0x7ff3480ca820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35508/0 (socket says 172.21.15.112:35508) 2023-01-24T03:37:49.349 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.341+0000 7ff347fff700 1 -- 172.21.15.112:0/969684812 learned_addr learned my addr 172.21.15.112:0/969684812 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:37:49.350 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.341+0000 7ff3477fe700 1 --2- 172.21.15.112:0/969684812 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff3480c7e40 0x7ff3480b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:49.350 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.341+0000 7ff3467fc700 1 -- 172.21.15.112:0/969684812 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2100448484 0 0) 0x7ff348156210 con 0x7ff3480ca3e0 2023-01-24T03:37:49.350 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.341+0000 7ff3467fc700 1 -- 172.21.15.112:0/969684812 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3340033f0 con 0x7ff3480ca3e0 2023-01-24T03:37:49.350 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.341+0000 7ff3467fc700 1 -- 172.21.15.112:0/969684812 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3529806329 0 0) 0x7ff3340033f0 con 0x7ff3480ca3e0 2023-01-24T03:37:49.351 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.341+0000 7ff3467fc700 1 -- 172.21.15.112:0/969684812 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff3480c7e40 msgr2=0x7ff3480b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:37:49.351 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.341+0000 7ff3467fc700 1 --2- 172.21.15.112:0/969684812 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff3480c7e40 0x7ff3480b0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:49.351 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.341+0000 7ff3467fc700 1 -- 172.21.15.112:0/969684812 >> v1:172.21.15.142:6789/0 conn(0x7ff348155da0 legacy=0x7ff3480ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:49.351 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.341+0000 7ff3467fc700 1 -- 172.21.15.112:0/969684812 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3480b1730 con 0x7ff3480ca3e0 2023-01-24T03:37:49.351 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.341+0000 7ff3467fc700 1 -- 172.21.15.112:0/969684812 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7ff340001470 con 0x7ff3480ca3e0 2023-01-24T03:37:49.352 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.342+0000 7ff3467fc700 1 -- 172.21.15.112:0/969684812 >> v1:172.21.15.112:6789/0 conn(0x7ff3480ca3e0 legacy=0x7ff3480ca820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:49.352 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.342+0000 7ff3467fc700 1 --2- 172.21.15.112:0/969684812 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff3480c7e40 0x7ff334003910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:49.352 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.342+0000 7ff3467fc700 1 --2- 172.21.15.112:0/969684812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff334003e50 0x7ff334004690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:49.352 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.342+0000 7ff3467fc700 1 --2- 172.21.15.112:0/969684812 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff334004bd0 0x7ff334005040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:49.353 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.342+0000 7ff3467fc700 1 -- 172.21.15.112:0/969684812 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3480b1450 con 0x7ff334003e50 2023-01-24T03:37:49.353 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.342+0000 7ff3467fc700 1 -- 172.21.15.112:0/969684812 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33400b9c0 con 0x7ff334004bd0 2023-01-24T03:37:49.353 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.342+0000 7ff3467fc700 1 -- 172.21.15.112:0/969684812 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff33400bb80 con 0x7ff3480c7e40 2023-01-24T03:37:49.353 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.342+0000 7ff346ffd700 1 --2- 172.21.15.112:0/969684812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff334003e50 0x7ff334004690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:49.353 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.342+0000 7ff3477fe700 1 --2- 172.21.15.112:0/969684812 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff3480c7e40 0x7ff334003910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:49.354 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.342+0000 7ff347fff700 1 --2- 172.21.15.112:0/969684812 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff334004bd0 0x7ff334005040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:49.354 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.342+0000 7ff346ffd700 1 -- 172.21.15.112:0/969684812 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff3480c7e40 msgr2=0x7ff334003910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:37:49.354 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.342+0000 7ff346ffd700 1 --2- 172.21.15.112:0/969684812 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff3480c7e40 0x7ff334003910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:49.354 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.342+0000 7ff346ffd700 1 -- 172.21.15.112:0/969684812 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff334004bd0 msgr2=0x7ff334005040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:37:49.354 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.342+0000 7ff346ffd700 1 --2- 172.21.15.112:0/969684812 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff334004bd0 0x7ff334005040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:49.355 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.342+0000 7ff346ffd700 1 -- 172.21.15.112:0/969684812 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3340033f0 con 0x7ff334003e50 2023-01-24T03:37:49.355 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.342+0000 7ff346ffd700 1 --2- 172.21.15.112:0/969684812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff334003e50 0x7ff334004690 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7ff33c001820 tx=0x7ff33c007b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28016dd94ba0780f server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:49.355 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.342+0000 7ff3467fc700 1 -- 172.21.15.112:0/969684812 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33c007ea0 con 0x7ff334003e50 2023-01-24T03:37:49.355 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.342+0000 7ff3467fc700 1 -- 172.21.15.112:0/969684812 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff33c012420 con 0x7ff334003e50 2023-01-24T03:37:49.355 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.342+0000 7ff3467fc700 1 -- 172.21.15.112:0/969684812 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33c0127c0 con 0x7ff334003e50 2023-01-24T03:37:49.356 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.343+0000 7ff34dc91700 1 -- 172.21.15.112:0/969684812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff334003e50 msgr2=0x7ff334004690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:49.356 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.343+0000 7ff34dc91700 1 --2- 172.21.15.112:0/969684812 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff334003e50 0x7ff334004690 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7ff33c001820 tx=0x7ff33c007b90 comp rx=0 tx=0).stop 2023-01-24T03:37:49.356 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.343+0000 7ff347fff700 1 -- 172.21.15.112:0/969684812 reap_dead start 2023-01-24T03:37:49.356 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.343+0000 7ff34dc91700 1 -- 172.21.15.112:0/969684812 shutdown_connections 2023-01-24T03:37:49.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.343+0000 7ff34dc91700 1 -- 172.21.15.112:0/969684812 >> 172.21.15.112:0/969684812 conn(0x7ff34805e2c0 msgr2=0x7ff3480cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:37:49.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.343+0000 7ff34dc91700 1 -- 172.21.15.112:0/969684812 shutdown_connections 2023-01-24T03:37:49.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.343+0000 7ff34dc91700 1 -- 172.21.15.112:0/969684812 wait complete. 2023-01-24T03:37:49.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.344+0000 7ff34dc91700 1 Processor -- start 2023-01-24T03:37:49.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.344+0000 7ff34dc91700 1 -- start start 2023-01-24T03:37:49.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.345+0000 7ff34dc91700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff334003e50 0x7ff348068630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:49.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.345+0000 7ff34dc91700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff348168e50 0x7ff3481692c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:49.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.345+0000 7ff34dc91700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34814e7b0 0x7ff34814ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:49.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.345+0000 7ff34dc91700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff348151880 con 0x7ff334003e50 2023-01-24T03:37:49.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.345+0000 7ff34dc91700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3480c5580 con 0x7ff348168e50 2023-01-24T03:37:49.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.345+0000 7ff34dc91700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3480c0e80 con 0x7ff34814e7b0 2023-01-24T03:37:49.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.345+0000 7ff3477fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff334003e50 0x7ff348068630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:49.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.345+0000 7ff3477fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff334003e50 0x7ff348068630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54528/0 (socket says 172.21.15.112:54528) 2023-01-24T03:37:49.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.345+0000 7ff3477fe700 1 -- 172.21.15.112:0/1094471419 learned_addr learned my addr 172.21.15.112:0/1094471419 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:37:49.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.345+0000 7ff347fff700 1 --2- 172.21.15.112:0/1094471419 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34814e7b0 0x7ff34814ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:49.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.345+0000 7ff346ffd700 1 --2- 172.21.15.112:0/1094471419 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff348168e50 0x7ff3481692c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:49.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.345+0000 7ff3477fe700 1 -- 172.21.15.112:0/1094471419 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34814e7b0 msgr2=0x7ff34814ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:49.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.345+0000 7ff3477fe700 1 --2- 172.21.15.112:0/1094471419 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34814e7b0 0x7ff34814ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:49.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.345+0000 7ff3477fe700 1 -- 172.21.15.112:0/1094471419 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff348168e50 msgr2=0x7ff3481692c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:49.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.345+0000 7ff3477fe700 1 --2- 172.21.15.112:0/1094471419 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff348168e50 0x7ff3481692c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:49.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.345+0000 7ff3477fe700 1 -- 172.21.15.112:0/1094471419 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3480c10f0 con 0x7ff334003e50 2023-01-24T03:37:49.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.345+0000 7ff3477fe700 1 --2- 172.21.15.112:0/1094471419 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff334003e50 0x7ff348068630 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7ff338011fb0 tx=0x7ff33800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:49.361 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.345+0000 7ff32f7fe700 1 -- 172.21.15.112:0/1094471419 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33800e040 con 0x7ff334003e50 2023-01-24T03:37:49.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.345+0000 7ff32f7fe700 1 -- 172.21.15.112:0/1094471419 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff33800f040 con 0x7ff334003e50 2023-01-24T03:37:49.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.345+0000 7ff34dc91700 1 -- 172.21.15.112:0/1094471419 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff348168930 con 0x7ff334003e50 2023-01-24T03:37:49.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.345+0000 7ff32f7fe700 1 -- 172.21.15.112:0/1094471419 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff338015430 con 0x7ff334003e50 2023-01-24T03:37:49.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.346+0000 7ff34dc91700 1 -- 172.21.15.112:0/1094471419 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3480c13d0 con 0x7ff334003e50 2023-01-24T03:37:49.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.346+0000 7ff34dc91700 1 -- 172.21.15.112:0/1094471419 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff34804bb80 con 0x7ff334003e50 2023-01-24T03:37:49.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.347+0000 7ff32f7fe700 1 -- 172.21.15.112:0/1094471419 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7ff3380157e0 con 0x7ff334003e50 2023-01-24T03:37:49.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.347+0000 7ff32f7fe700 1 --2- 172.21.15.112:0/1094471419 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff3280611c0 0x7ff328063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:49.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.347+0000 7ff32f7fe700 1 -- 172.21.15.112:0/1094471419 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 7297+0+0 (secure 0 0 0) 0x7ff33807e7b0 con 0x7ff334003e50 2023-01-24T03:37:49.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.347+0000 7ff346ffd700 1 --2- 172.21.15.112:0/1094471419 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff3280611c0 0x7ff328063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:49.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.348+0000 7ff346ffd700 1 --2- 172.21.15.112:0/1094471419 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff3280611c0 0x7ff328063670 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff3480c2a80 tx=0x7ff33c015000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:49.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.351+0000 7ff32f7fe700 1 -- 172.21.15.112:0/1094471419 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7ff33804e4c0 con 0x7ff334003e50 2023-01-24T03:37:49.571 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.570+0000 7ff34dc91700 1 -- 172.21.15.112:0/1094471419 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3480c5df0 con 0x7ff334003e50 2023-01-24T03:37:49.572 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.570+0000 7ff32f7fe700 1 -- 172.21.15.112:0/1094471419 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5364 (secure 0 0 0) 0x7ff338051ae0 con 0x7ff334003e50 2023-01-24T03:37:49.573 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:37:49.573 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:37:46.721488+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5296":{"gid":5296,"name":"b","rank":0,"incarnation":32,"state":"up:reconnect","state_seq":16,"addr":"172.21.15.142:6834/2838716685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2838716685},{"type":"v1","addr":"172.21.15.142:6834","nonce":2838716685}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:37:49.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.572+0000 7ff34dc91700 1 -- 172.21.15.112:0/1094471419 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff3280611c0 msgr2=0x7ff328063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:49.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.572+0000 7ff34dc91700 1 --2- 172.21.15.112:0/1094471419 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff3280611c0 0x7ff328063670 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff3480c2a80 tx=0x7ff33c015000 comp rx=0 tx=0).stop 2023-01-24T03:37:49.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.572+0000 7ff34dc91700 1 -- 172.21.15.112:0/1094471419 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff334003e50 msgr2=0x7ff348068630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:49.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.572+0000 7ff34dc91700 1 --2- 172.21.15.112:0/1094471419 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff334003e50 0x7ff348068630 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7ff338011fb0 tx=0x7ff33800b040 comp rx=0 tx=0).stop 2023-01-24T03:37:49.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.572+0000 7ff34dc91700 1 -- 172.21.15.112:0/1094471419 shutdown_connections 2023-01-24T03:37:49.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.572+0000 7ff34dc91700 1 --2- 172.21.15.112:0/1094471419 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff3280611c0 0x7ff328063670 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:49.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.572+0000 7ff34dc91700 1 --2- 172.21.15.112:0/1094471419 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff334003e50 0x7ff348068630 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:49.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.572+0000 7ff34dc91700 1 --2- 172.21.15.112:0/1094471419 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff34814e7b0 0x7ff34814ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:49.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.572+0000 7ff34dc91700 1 --2- 172.21.15.112:0/1094471419 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff348168e50 0x7ff3481692c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:49.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.572+0000 7ff34dc91700 1 -- 172.21.15.112:0/1094471419 >> 172.21.15.112:0/1094471419 conn(0x7ff34805e2c0 msgr2=0x7ff3480c6fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:37:49.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.572+0000 7ff34dc91700 1 -- 172.21.15.112:0/1094471419 shutdown_connections 2023-01-24T03:37:49.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:49.572+0000 7ff34dc91700 1 -- 172.21.15.112:0/1094471419 wait complete. 2023-01-24T03:37:49.577 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 33 2023-01-24T03:37:49.586 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:37:51.587 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:37:51.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.719+0000 7ff54c6e1700 1 Processor -- start 2023-01-24T03:37:51.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.719+0000 7ff54c6e1700 1 -- start start 2023-01-24T03:37:51.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.720+0000 7ff54c6e1700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7ff5440b0aa0 0x7ff544155da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:51.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.720+0000 7ff54c6e1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff5440ca3e0 0x7ff5440ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:51.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.720+0000 7ff54c6e1700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5440b0f40 con 0x7ff5440c7e40 2023-01-24T03:37:51.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.720+0000 7ff54c6e1700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5440b1170 con 0x7ff5440b0aa0 2023-01-24T03:37:51.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.720+0000 7ff54c6e1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5440b1450 con 0x7ff5440ca3e0 2023-01-24T03:37:51.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.720+0000 7ff54a47d700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7ff5440c7e40 0x7ff5440ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35524/0 (socket says 172.21.15.112:35524) 2023-01-24T03:37:51.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.720+0000 7ff54a47d700 1 -- 172.21.15.112:0/372269430 learned_addr learned my addr 172.21.15.112:0/372269430 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:37:51.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.720+0000 7ff549c7c700 1 --2- 172.21.15.112:0/372269430 >> v2:172.21.15.142:3300/0 conn(0x7ff5440b0aa0 0x7ff544155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:51.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.720+0000 7ff54ac7e700 1 --2- 172.21.15.112:0/372269430 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff5440ca3e0 0x7ff5440ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:51.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.720+0000 7ff54947b700 1 -- 172.21.15.112:0/372269430 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 117044704 0 0) 0x7ff5440b0f40 con 0x7ff5440c7e40 2023-01-24T03:37:51.724 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.720+0000 7ff54947b700 1 -- 172.21.15.112:0/372269430 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff52c0033f0 con 0x7ff5440c7e40 2023-01-24T03:37:51.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.720+0000 7ff54947b700 1 -- 172.21.15.112:0/372269430 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 728451017 0 0) 0x7ff52c0033f0 con 0x7ff5440c7e40 2023-01-24T03:37:51.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.720+0000 7ff54947b700 1 -- 172.21.15.112:0/372269430 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff5440ca3e0 msgr2=0x7ff5440ca7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:37:51.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.720+0000 7ff54947b700 1 --2- 172.21.15.112:0/372269430 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff5440ca3e0 0x7ff5440ca7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:51.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.720+0000 7ff54947b700 1 -- 172.21.15.112:0/372269430 >> v2:172.21.15.142:3300/0 conn(0x7ff5440b0aa0 msgr2=0x7ff544155da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:37:51.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.720+0000 7ff54947b700 1 --2- 172.21.15.112:0/372269430 >> v2:172.21.15.142:3300/0 conn(0x7ff5440b0aa0 0x7ff544155da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:51.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.720+0000 7ff54947b700 1 -- 172.21.15.112:0/372269430 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5440b1730 con 0x7ff5440c7e40 2023-01-24T03:37:51.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.720+0000 7ff54947b700 1 -- 172.21.15.112:0/372269430 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7ff53c001470 con 0x7ff5440c7e40 2023-01-24T03:37:51.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.720+0000 7ff54947b700 1 -- 172.21.15.112:0/372269430 >> v1:172.21.15.112:6789/0 conn(0x7ff5440c7e40 legacy=0x7ff5440ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:51.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.720+0000 7ff54947b700 1 --2- 172.21.15.112:0/372269430 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff52c0038e0 0x7ff52c003cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:51.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.720+0000 7ff54947b700 1 --2- 172.21.15.112:0/372269430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff52c004690 0x7ff52c004b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:51.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.720+0000 7ff54947b700 1 --2- 172.21.15.112:0/372269430 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff5440ca3e0 0x7ff52c005040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:51.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.720+0000 7ff54947b700 1 -- 172.21.15.112:0/372269430 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5440b1170 con 0x7ff52c004690 2023-01-24T03:37:51.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.720+0000 7ff54947b700 1 -- 172.21.15.112:0/372269430 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5440b1450 con 0x7ff52c0038e0 2023-01-24T03:37:51.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.720+0000 7ff54947b700 1 -- 172.21.15.112:0/372269430 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff52c00b8e0 con 0x7ff5440ca3e0 2023-01-24T03:37:51.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.720+0000 7ff549c7c700 1 --2- 172.21.15.112:0/372269430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff52c004690 0x7ff52c004b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:51.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.721+0000 7ff54a47d700 1 --2- 172.21.15.112:0/372269430 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff52c0038e0 0x7ff52c003cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:51.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.721+0000 7ff54ac7e700 1 --2- 172.21.15.112:0/372269430 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff5440ca3e0 0x7ff52c005040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:51.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.721+0000 7ff549c7c700 1 -- 172.21.15.112:0/372269430 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff5440ca3e0 msgr2=0x7ff52c005040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:37:51.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.721+0000 7ff549c7c700 1 --2- 172.21.15.112:0/372269430 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff5440ca3e0 0x7ff52c005040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:51.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.721+0000 7ff549c7c700 1 -- 172.21.15.112:0/372269430 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff52c0038e0 msgr2=0x7ff52c003cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:37:51.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.721+0000 7ff549c7c700 1 --2- 172.21.15.112:0/372269430 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff52c0038e0 0x7ff52c003cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:51.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.721+0000 7ff549c7c700 1 -- 172.21.15.112:0/372269430 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff52c0033f0 con 0x7ff52c004690 2023-01-24T03:37:51.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.721+0000 7ff549c7c700 1 --2- 172.21.15.112:0/372269430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff52c004690 0x7ff52c004b00 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7ff534009ab0 tx=0x7ff534009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8958f9c8bb8c25e2 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:51.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.721+0000 7ff54947b700 1 -- 172.21.15.112:0/372269430 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff534009430 con 0x7ff52c004690 2023-01-24T03:37:51.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.721+0000 7ff54947b700 1 -- 172.21.15.112:0/372269430 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff53400e640 con 0x7ff52c004690 2023-01-24T03:37:51.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.721+0000 7ff54947b700 1 -- 172.21.15.112:0/372269430 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff53400ea00 con 0x7ff52c004690 2023-01-24T03:37:51.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.721+0000 7ff54c6e1700 1 -- 172.21.15.112:0/372269430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff52c004690 msgr2=0x7ff52c004b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:51.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.721+0000 7ff54c6e1700 1 --2- 172.21.15.112:0/372269430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff52c004690 0x7ff52c004b00 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7ff534009ab0 tx=0x7ff534009060 comp rx=0 tx=0).stop 2023-01-24T03:37:51.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.721+0000 7ff54ac7e700 1 -- 172.21.15.112:0/372269430 reap_dead start 2023-01-24T03:37:51.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.722+0000 7ff54c6e1700 1 -- 172.21.15.112:0/372269430 shutdown_connections 2023-01-24T03:37:51.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.722+0000 7ff54c6e1700 1 -- 172.21.15.112:0/372269430 >> 172.21.15.112:0/372269430 conn(0x7ff54405e2c0 msgr2=0x7ff5440cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:37:51.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.722+0000 7ff54c6e1700 1 -- 172.21.15.112:0/372269430 shutdown_connections 2023-01-24T03:37:51.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.722+0000 7ff54c6e1700 1 -- 172.21.15.112:0/372269430 wait complete. 2023-01-24T03:37:51.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.723+0000 7ff54c6e1700 1 Processor -- start 2023-01-24T03:37:51.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.723+0000 7ff54c6e1700 1 -- start start 2023-01-24T03:37:51.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.723+0000 7ff54c6e1700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff52c004690 0x7ff544169770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:51.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.723+0000 7ff54c6e1700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff544169cb0 0x7ff544152400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:51.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.723+0000 7ff54c6e1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff5441546f0 0x7ff544152940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:51.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.723+0000 7ff54c6e1700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff544150520 con 0x7ff52c004690 2023-01-24T03:37:51.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.723+0000 7ff54c6e1700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff54416a4b0 con 0x7ff544169cb0 2023-01-24T03:37:51.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.723+0000 7ff54c6e1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5440c5890 con 0x7ff5441546f0 2023-01-24T03:37:51.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.723+0000 7ff54a47d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff52c004690 0x7ff544169770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:51.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.723+0000 7ff54a47d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff52c004690 0x7ff544169770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54548/0 (socket says 172.21.15.112:54548) 2023-01-24T03:37:51.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.723+0000 7ff54a47d700 1 -- 172.21.15.112:0/3861311898 learned_addr learned my addr 172.21.15.112:0/3861311898 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:37:51.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.724+0000 7ff54ac7e700 1 --2- 172.21.15.112:0/3861311898 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff5441546f0 0x7ff544152940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:51.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.724+0000 7ff549c7c700 1 --2- 172.21.15.112:0/3861311898 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff544169cb0 0x7ff544152400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:51.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.724+0000 7ff54a47d700 1 -- 172.21.15.112:0/3861311898 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff5441546f0 msgr2=0x7ff544152940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:51.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.724+0000 7ff54a47d700 1 --2- 172.21.15.112:0/3861311898 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff5441546f0 0x7ff544152940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:51.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.724+0000 7ff54a47d700 1 -- 172.21.15.112:0/3861311898 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff544169cb0 msgr2=0x7ff544152400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:51.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.724+0000 7ff54a47d700 1 --2- 172.21.15.112:0/3861311898 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff544169cb0 0x7ff544152400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:51.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.724+0000 7ff54a47d700 1 -- 172.21.15.112:0/3861311898 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff544150280 con 0x7ff52c004690 2023-01-24T03:37:51.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.724+0000 7ff54a47d700 1 --2- 172.21.15.112:0/3861311898 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff52c004690 0x7ff544169770 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7ff53c004aa0 tx=0x7ff53c001e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:51.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.724+0000 7ff5327fc700 1 -- 172.21.15.112:0/3861311898 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff53c00f7f0 con 0x7ff52c004690 2023-01-24T03:37:51.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.724+0000 7ff5327fc700 1 -- 172.21.15.112:0/3861311898 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff53c00f970 con 0x7ff52c004690 2023-01-24T03:37:51.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.724+0000 7ff54c6e1700 1 -- 172.21.15.112:0/3861311898 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff54416a1d0 con 0x7ff52c004690 2023-01-24T03:37:51.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.724+0000 7ff5327fc700 1 -- 172.21.15.112:0/3861311898 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff53c00fd30 con 0x7ff52c004690 2023-01-24T03:37:51.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.724+0000 7ff54c6e1700 1 -- 172.21.15.112:0/3861311898 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff544151c40 con 0x7ff52c004690 2023-01-24T03:37:51.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.725+0000 7ff54c6e1700 1 -- 172.21.15.112:0/3861311898 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff54404bb80 con 0x7ff52c004690 2023-01-24T03:37:51.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.725+0000 7ff5327fc700 1 -- 172.21.15.112:0/3861311898 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7ff53c01c030 con 0x7ff52c004690 2023-01-24T03:37:51.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.726+0000 7ff5327fc700 1 --2- 172.21.15.112:0/3861311898 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff5240611c0 0x7ff524063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:51.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.726+0000 7ff5327fc700 1 -- 172.21.15.112:0/3861311898 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 7297+0+0 (secure 0 0 0) 0x7ff53c07d4a0 con 0x7ff52c004690 2023-01-24T03:37:51.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.726+0000 7ff549c7c700 1 --2- 172.21.15.112:0/3861311898 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff5240611c0 0x7ff524063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:51.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.726+0000 7ff549c7c700 1 --2- 172.21.15.112:0/3861311898 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff5240611c0 0x7ff524063670 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff54416a100 tx=0x7ff53401a000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:51.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.730+0000 7ff5327fc700 1 -- 172.21.15.112:0/3861311898 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7ff53c04d0f0 con 0x7ff52c004690 2023-01-24T03:37:51.957 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.954+0000 7ff54c6e1700 1 -- 172.21.15.112:0/3861311898 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff54414f9e0 con 0x7ff52c004690 2023-01-24T03:37:51.958 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.955+0000 7ff5327fc700 1 -- 172.21.15.112:0/3861311898 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+5361 (secure 0 0 0) 0x7ff53c01a090 con 0x7ff52c004690 2023-01-24T03:37:51.958 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:37:51.959 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:37:51.743415+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5296":{"gid":5296,"name":"b","rank":0,"incarnation":32,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.142:6834/2838716685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2838716685},{"type":"v1","addr":"172.21.15.142:6834","nonce":2838716685}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:37:51.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.957+0000 7ff54c6e1700 1 -- 172.21.15.112:0/3861311898 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff5240611c0 msgr2=0x7ff524063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:51.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.957+0000 7ff54c6e1700 1 --2- 172.21.15.112:0/3861311898 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff5240611c0 0x7ff524063670 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff54416a100 tx=0x7ff53401a000 comp rx=0 tx=0).stop 2023-01-24T03:37:51.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.957+0000 7ff54c6e1700 1 -- 172.21.15.112:0/3861311898 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff52c004690 msgr2=0x7ff544169770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:51.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.957+0000 7ff54c6e1700 1 --2- 172.21.15.112:0/3861311898 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff52c004690 0x7ff544169770 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7ff53c004aa0 tx=0x7ff53c001e90 comp rx=0 tx=0).stop 2023-01-24T03:37:51.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.957+0000 7ff54c6e1700 1 -- 172.21.15.112:0/3861311898 shutdown_connections 2023-01-24T03:37:51.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.957+0000 7ff54c6e1700 1 --2- 172.21.15.112:0/3861311898 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff5240611c0 0x7ff524063670 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:51.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.957+0000 7ff54c6e1700 1 --2- 172.21.15.112:0/3861311898 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff52c004690 0x7ff544169770 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:51.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.957+0000 7ff54c6e1700 1 --2- 172.21.15.112:0/3861311898 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff5441546f0 0x7ff544152940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:51.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.957+0000 7ff54c6e1700 1 --2- 172.21.15.112:0/3861311898 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff544169cb0 0x7ff544152400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:51.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.957+0000 7ff54c6e1700 1 -- 172.21.15.112:0/3861311898 >> 172.21.15.112:0/3861311898 conn(0x7ff54405e2c0 msgr2=0x7ff5440680c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:37:51.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.957+0000 7ff54c6e1700 1 -- 172.21.15.112:0/3861311898 shutdown_connections 2023-01-24T03:37:51.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:51.957+0000 7ff54c6e1700 1 -- 172.21.15.112:0/3861311898 wait complete. 2023-01-24T03:37:51.965 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 34 2023-01-24T03:37:51.971 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 34, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-01-24T03:30:20.971710+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 34, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5296": { "addr": "172.21.15.142:6834/2838716685", "addrs": { "addrvec": [ { "addr": "172.21.15.142:6832", "nonce": 2838716685, "type": "v2" }, { "addr": "172.21.15.142:6834", "nonce": 2838716685, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5296, "incarnation": 32, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:rejoin", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 47, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-01-24T03:37:51.743415+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5296 } } } ], "standbys": [ { "addr": "172.21.15.142:6837/2181345869", "addrs": { "addrvec": [ { "addr": "172.21.15.142:6836", "nonce": 2181345869, "type": "v2" }, { "addr": "172.21.15.142:6837", "nonce": 2181345869, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6835/330343927", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6834", "nonce": 330343927, "type": "v2" }, { "addr": "172.21.15.112:6835", "nonce": 330343927, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6837/12176817", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6836", "nonce": 12176817, "type": "v2" }, { "addr": "172.21.15.112:6837", "nonce": 12176817, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4276, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6839/1110983841", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6838", "nonce": 1110983841, "type": "v2" }, { "addr": "172.21.15.112:6839", "nonce": 1110983841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4281, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-01-24T03:37:53.972 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:37:54.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.094+0000 7f5fba815700 1 Processor -- start 2023-01-24T03:37:54.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.095+0000 7f5fba815700 1 -- start start 2023-01-24T03:37:54.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.095+0000 7f5fba815700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fb40c7e40 0x7f5fb40b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:54.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.095+0000 7f5fba815700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fb4156210 con 0x7f5fb40ca3e0 2023-01-24T03:37:54.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.095+0000 7f5fba815700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fb40b1170 con 0x7f5fb4155da0 2023-01-24T03:37:54.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.095+0000 7f5fba815700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb40b1450 con 0x7f5fb40c7e40 2023-01-24T03:37:54.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.095+0000 7f5fb8db2700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f5fb40ca3e0 0x7f5fb40ca820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35548/0 (socket says 172.21.15.112:35548) 2023-01-24T03:37:54.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.095+0000 7f5fb8db2700 1 -- 172.21.15.112:0/3882423361 learned_addr learned my addr 172.21.15.112:0/3882423361 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:37:54.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.096+0000 7f5fb3fff700 1 --2- 172.21.15.112:0/3882423361 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fb40c7e40 0x7f5fb40b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:54.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.096+0000 7f5fb2ffd700 1 -- 172.21.15.112:0/3882423361 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2964848384 0 0) 0x7f5fb4156210 con 0x7f5fb40ca3e0 2023-01-24T03:37:54.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.096+0000 7f5fb2ffd700 1 -- 172.21.15.112:0/3882423361 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fa00033f0 con 0x7f5fb40ca3e0 2023-01-24T03:37:54.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.096+0000 7f5fb2ffd700 1 -- 172.21.15.112:0/3882423361 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 318829715 0 0) 0x7f5fa00033f0 con 0x7f5fb40ca3e0 2023-01-24T03:37:54.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.096+0000 7f5fb2ffd700 1 -- 172.21.15.112:0/3882423361 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fb40c7e40 msgr2=0x7f5fb40b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:37:54.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.096+0000 7f5fb2ffd700 1 --2- 172.21.15.112:0/3882423361 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fb40c7e40 0x7f5fb40b0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:54.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.096+0000 7f5fb2ffd700 1 -- 172.21.15.112:0/3882423361 >> v1:172.21.15.142:6789/0 conn(0x7f5fb4155da0 legacy=0x7f5fb40ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:54.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.096+0000 7f5fb2ffd700 1 -- 172.21.15.112:0/3882423361 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb40b1730 con 0x7f5fb40ca3e0 2023-01-24T03:37:54.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.096+0000 7f5fb2ffd700 1 -- 172.21.15.112:0/3882423361 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7f5fac0017f0 con 0x7f5fb40ca3e0 2023-01-24T03:37:54.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.096+0000 7f5fb2ffd700 1 -- 172.21.15.112:0/3882423361 >> v1:172.21.15.112:6789/0 conn(0x7f5fb40ca3e0 legacy=0x7f5fb40ca820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:54.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.096+0000 7f5fb2ffd700 1 --2- 172.21.15.112:0/3882423361 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5fa0003910 0x7f5fa0003d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:54.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.096+0000 7f5fb2ffd700 1 --2- 172.21.15.112:0/3882423361 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fa0004690 0x7f5fa0004b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:54.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.096+0000 7f5fb2ffd700 1 --2- 172.21.15.112:0/3882423361 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fb40c7e40 0x7f5fa0005060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:54.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.096+0000 7f5fb2ffd700 1 -- 172.21.15.112:0/3882423361 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb40b1450 con 0x7f5fa0004690 2023-01-24T03:37:54.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.096+0000 7f5fb2ffd700 1 -- 172.21.15.112:0/3882423361 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa000b9e0 con 0x7f5fa0003910 2023-01-24T03:37:54.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.096+0000 7f5fb2ffd700 1 -- 172.21.15.112:0/3882423361 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa000bba0 con 0x7f5fb40c7e40 2023-01-24T03:37:54.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.096+0000 7f5fb37fe700 1 --2- 172.21.15.112:0/3882423361 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fa0004690 0x7f5fa0004b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:54.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.096+0000 7f5fb3fff700 1 --2- 172.21.15.112:0/3882423361 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5fa0003910 0x7f5fa0003d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:54.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.096+0000 7f5fb8db2700 1 --2- 172.21.15.112:0/3882423361 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fb40c7e40 0x7f5fa0005060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:54.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.096+0000 7f5fb37fe700 1 -- 172.21.15.112:0/3882423361 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fb40c7e40 msgr2=0x7f5fa0005060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:37:54.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.096+0000 7f5fb37fe700 1 --2- 172.21.15.112:0/3882423361 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fb40c7e40 0x7f5fa0005060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:54.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.096+0000 7f5fb37fe700 1 -- 172.21.15.112:0/3882423361 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5fa0003910 msgr2=0x7f5fa0003d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:37:54.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.096+0000 7f5fb37fe700 1 --2- 172.21.15.112:0/3882423361 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5fa0003910 0x7f5fa0003d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:54.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.096+0000 7f5fb37fe700 1 -- 172.21.15.112:0/3882423361 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa00033f0 con 0x7f5fa0004690 2023-01-24T03:37:54.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.097+0000 7f5fb37fe700 1 --2- 172.21.15.112:0/3882423361 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fa0004690 0x7f5fa0004b00 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8001730 tx=0x7f5fa8007bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ed2a09eb11532d3 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:54.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.097+0000 7f5fb2ffd700 1 -- 172.21.15.112:0/3882423361 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa8012440 con 0x7f5fa0004690 2023-01-24T03:37:54.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.097+0000 7f5fb2ffd700 1 -- 172.21.15.112:0/3882423361 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa80125c0 con 0x7f5fa0004690 2023-01-24T03:37:54.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.097+0000 7f5fb2ffd700 1 -- 172.21.15.112:0/3882423361 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa8012980 con 0x7f5fa0004690 2023-01-24T03:37:54.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.097+0000 7f5fba815700 1 -- 172.21.15.112:0/3882423361 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fa0004690 msgr2=0x7f5fa0004b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:54.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.097+0000 7f5fba815700 1 --2- 172.21.15.112:0/3882423361 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fa0004690 0x7f5fa0004b00 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8001730 tx=0x7f5fa8007bb0 comp rx=0 tx=0).stop 2023-01-24T03:37:54.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.097+0000 7f5fb8db2700 1 -- 172.21.15.112:0/3882423361 reap_dead start 2023-01-24T03:37:54.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.097+0000 7f5fba815700 1 -- 172.21.15.112:0/3882423361 shutdown_connections 2023-01-24T03:37:54.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.097+0000 7f5fba815700 1 -- 172.21.15.112:0/3882423361 >> 172.21.15.112:0/3882423361 conn(0x7f5fb405e2c0 msgr2=0x7f5fb40cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:37:54.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.097+0000 7f5fba815700 1 -- 172.21.15.112:0/3882423361 shutdown_connections 2023-01-24T03:37:54.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.097+0000 7f5fba815700 1 -- 172.21.15.112:0/3882423361 wait complete. 2023-01-24T03:37:54.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.098+0000 7f5fba815700 1 Processor -- start 2023-01-24T03:37:54.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.099+0000 7f5fba815700 1 -- start start 2023-01-24T03:37:54.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.099+0000 7f5fba815700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fa0004690 0x7f5fb414e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:54.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.099+0000 7f5fba815700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fb40c2ad0 0x7f5fb40ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:54.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.099+0000 7f5fba815700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5fb4166450 0x7f5fb40aef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:54.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.099+0000 7f5fba815700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb41505b0 con 0x7f5fa0004690 2023-01-24T03:37:54.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.099+0000 7f5fba815700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb414e720 con 0x7f5fb4166450 2023-01-24T03:37:54.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.099+0000 7f5fba815700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb40c15c0 con 0x7f5fb40c2ad0 2023-01-24T03:37:54.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.099+0000 7f5fb3fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fa0004690 0x7f5fb414e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:54.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.099+0000 7f5fb3fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fa0004690 0x7f5fb414e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54570/0 (socket says 172.21.15.112:54570) 2023-01-24T03:37:54.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.099+0000 7f5fb3fff700 1 -- 172.21.15.112:0/2780413833 learned_addr learned my addr 172.21.15.112:0/2780413833 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:37:54.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.099+0000 7f5fb8db2700 1 --2- 172.21.15.112:0/2780413833 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5fb4166450 0x7f5fb40aef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:54.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.099+0000 7f5fb37fe700 1 --2- 172.21.15.112:0/2780413833 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fb40c2ad0 0x7f5fb40ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:54.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.100+0000 7f5fb3fff700 1 -- 172.21.15.112:0/2780413833 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fb40c2ad0 msgr2=0x7f5fb40ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:54.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.100+0000 7f5fb3fff700 1 --2- 172.21.15.112:0/2780413833 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fb40c2ad0 0x7f5fb40ae9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:54.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.100+0000 7f5fb3fff700 1 -- 172.21.15.112:0/2780413833 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5fb4166450 msgr2=0x7f5fb40aef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:54.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.100+0000 7f5fb3fff700 1 --2- 172.21.15.112:0/2780413833 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5fb4166450 0x7f5fb40aef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:54.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.100+0000 7f5fb3fff700 1 -- 172.21.15.112:0/2780413833 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb4166980 con 0x7f5fa0004690 2023-01-24T03:37:54.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.100+0000 7f5fb3fff700 1 --2- 172.21.15.112:0/2780413833 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fa0004690 0x7f5fb414e1e0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4011a20 tx=0x7f5fa4009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:54.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.100+0000 7f5f9bfff700 1 -- 172.21.15.112:0/2780413833 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa400f8e0 con 0x7f5fa0004690 2023-01-24T03:37:54.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.100+0000 7f5f9bfff700 1 -- 172.21.15.112:0/2780413833 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa400fa40 con 0x7f5fa0004690 2023-01-24T03:37:54.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.100+0000 7f5fba815700 1 -- 172.21.15.112:0/2780413833 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fb40c09b0 con 0x7f5fa0004690 2023-01-24T03:37:54.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.100+0000 7f5f9bfff700 1 -- 172.21.15.112:0/2780413833 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa400fde0 con 0x7f5fa0004690 2023-01-24T03:37:54.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.100+0000 7f5fba815700 1 -- 172.21.15.112:0/2780413833 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fb40c0f70 con 0x7f5fa0004690 2023-01-24T03:37:54.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.101+0000 7f5fba815700 1 -- 172.21.15.112:0/2780413833 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fb4089ae0 con 0x7f5fa0004690 2023-01-24T03:37:54.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.101+0000 7f5f9bfff700 1 -- 172.21.15.112:0/2780413833 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f5fa4024070 con 0x7f5fa0004690 2023-01-24T03:37:54.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.102+0000 7f5f9bfff700 1 --2- 172.21.15.112:0/2780413833 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5f940611c0 0x7f5f94063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:54.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.102+0000 7f5f9bfff700 1 -- 172.21.15.112:0/2780413833 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 7297+0+0 (secure 0 0 0) 0x7f5fa4018030 con 0x7f5fa0004690 2023-01-24T03:37:54.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.102+0000 7f5fb37fe700 1 --2- 172.21.15.112:0/2780413833 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5f940611c0 0x7f5f94063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:54.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.102+0000 7f5fb37fe700 1 --2- 172.21.15.112:0/2780413833 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5f940611c0 0x7f5f94063670 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5fb40cc3c0 tx=0x7f5fa8015000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:54.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.106+0000 7f5f9bfff700 1 -- 172.21.15.112:0/2780413833 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f5fa404ded0 con 0x7f5fa0004690 2023-01-24T03:37:54.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.327+0000 7f5fba815700 1 -- 172.21.15.112:0/2780413833 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5fb404bb80 con 0x7f5fa0004690 2023-01-24T03:37:54.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.327+0000 7f5f9bfff700 1 -- 172.21.15.112:0/2780413833 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+5361 (secure 0 0 0) 0x7f5fa40514f0 con 0x7f5fa0004690 2023-01-24T03:37:54.330 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:37:54.330 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:37:51.743415+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5296":{"gid":5296,"name":"b","rank":0,"incarnation":32,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.142:6834/2838716685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2838716685},{"type":"v1","addr":"172.21.15.142:6834","nonce":2838716685}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:37:54.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.329+0000 7f5fba815700 1 -- 172.21.15.112:0/2780413833 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5f940611c0 msgr2=0x7f5f94063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:54.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.329+0000 7f5fba815700 1 --2- 172.21.15.112:0/2780413833 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5f940611c0 0x7f5f94063670 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5fb40cc3c0 tx=0x7f5fa8015000 comp rx=0 tx=0).stop 2023-01-24T03:37:54.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.329+0000 7f5fba815700 1 -- 172.21.15.112:0/2780413833 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fa0004690 msgr2=0x7f5fb414e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:54.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.329+0000 7f5fba815700 1 --2- 172.21.15.112:0/2780413833 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fa0004690 0x7f5fb414e1e0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4011a20 tx=0x7f5fa4009060 comp rx=0 tx=0).stop 2023-01-24T03:37:54.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.329+0000 7f5fba815700 1 -- 172.21.15.112:0/2780413833 shutdown_connections 2023-01-24T03:37:54.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.329+0000 7f5fba815700 1 --2- 172.21.15.112:0/2780413833 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5f940611c0 0x7f5f94063670 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:54.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.329+0000 7f5fba815700 1 --2- 172.21.15.112:0/2780413833 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fa0004690 0x7f5fb414e1e0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:54.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.329+0000 7f5fba815700 1 --2- 172.21.15.112:0/2780413833 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5fb4166450 0x7f5fb40aef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:54.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.329+0000 7f5fba815700 1 --2- 172.21.15.112:0/2780413833 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5fb40c2ad0 0x7f5fb40ae9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:54.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.329+0000 7f5fba815700 1 -- 172.21.15.112:0/2780413833 >> 172.21.15.112:0/2780413833 conn(0x7f5fb405e2c0 msgr2=0x7f5fb414f110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:37:54.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.330+0000 7f5fba815700 1 -- 172.21.15.112:0/2780413833 shutdown_connections 2023-01-24T03:37:54.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:54.330+0000 7f5fba815700 1 -- 172.21.15.112:0/2780413833 wait complete. 2023-01-24T03:37:54.335 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 34 2023-01-24T03:37:54.343 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:37:56.345 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:37:56.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.468+0000 7f273a88e700 1 Processor -- start 2023-01-24T03:37:56.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.468+0000 7f273a88e700 1 -- start start 2023-01-24T03:37:56.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.468+0000 7f273a88e700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f27340c7e40 0x7f27340b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:56.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.468+0000 7f273a88e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2734155da0 0x7f27340ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:56.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.468+0000 7f273a88e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2734156240 con 0x7f27340aef20 2023-01-24T03:37:56.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.468+0000 7f273a88e700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27340b1170 con 0x7f27340c7e40 2023-01-24T03:37:56.474 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.468+0000 7f273a88e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27340b1450 con 0x7f2734155da0 2023-01-24T03:37:56.474 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.468+0000 7f2733fff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f27340c7e40 0x7f27340b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:56.474 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.468+0000 7f2733fff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f27340c7e40 0x7f27340b0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54578/0 (socket says 172.21.15.112:54578) 2023-01-24T03:37:56.474 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.468+0000 7f2733fff700 1 -- 172.21.15.112:0/1455617801 learned_addr learned my addr 172.21.15.112:0/1455617801 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:37:56.474 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.469+0000 7f272bfff700 1 --2- 172.21.15.112:0/1455617801 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2734155da0 0x7f27340ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:56.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.469+0000 7f2733fff700 1 -- 172.21.15.112:0/1455617801 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2734155da0 msgr2=0x7f27340ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:37:56.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.469+0000 7f2733fff700 1 --2- 172.21.15.112:0/1455617801 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2734155da0 0x7f27340ae9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:56.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.469+0000 7f2733fff700 1 -- 172.21.15.112:0/1455617801 >> v1:172.21.15.142:6789/0 conn(0x7f27340aef20 legacy=0x7f27340ca3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:56.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.469+0000 7f2733fff700 1 -- 172.21.15.112:0/1455617801 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27340b1730 con 0x7f27340c7e40 2023-01-24T03:37:56.476 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.469+0000 7f2733fff700 1 --2- 172.21.15.112:0/1455617801 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27340c7e40 0x7f27340b0aa0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f2724006fd0 tx=0x7f27240090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=182520f1fa9a889 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:56.476 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.469+0000 7f27337fe700 1 -- 172.21.15.112:0/1455617801 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2724011630 con 0x7f27340c7e40 2023-01-24T03:37:56.476 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.469+0000 7f27337fe700 1 -- 172.21.15.112:0/1455617801 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27240117d0 con 0x7f27340c7e40 2023-01-24T03:37:56.476 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.469+0000 7f27337fe700 1 -- 172.21.15.112:0/1455617801 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2724011b90 con 0x7f27340c7e40 2023-01-24T03:37:56.476 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.470+0000 7f273a88e700 1 -- 172.21.15.112:0/1455617801 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27340c7e40 msgr2=0x7f27340b0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:56.477 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.470+0000 7f273a88e700 1 --2- 172.21.15.112:0/1455617801 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27340c7e40 0x7f27340b0aa0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f2724006fd0 tx=0x7f27240090f0 comp rx=0 tx=0).stop 2023-01-24T03:37:56.477 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.470+0000 7f273a88e700 1 -- 172.21.15.112:0/1455617801 shutdown_connections 2023-01-24T03:37:56.477 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.470+0000 7f273a88e700 1 --2- 172.21.15.112:0/1455617801 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27340c7e40 0x7f27340b0aa0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:56.477 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.470+0000 7f273a88e700 1 --2- 172.21.15.112:0/1455617801 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2734155da0 0x7f27340ae9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:56.477 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.470+0000 7f273a88e700 1 -- 172.21.15.112:0/1455617801 >> 172.21.15.112:0/1455617801 conn(0x7f273405e2c0 msgr2=0x7f27340cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:37:56.478 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.470+0000 7f273a88e700 1 -- 172.21.15.112:0/1455617801 shutdown_connections 2023-01-24T03:37:56.478 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.470+0000 7f273a88e700 1 -- 172.21.15.112:0/1455617801 wait complete. 2023-01-24T03:37:56.478 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.471+0000 7f273a88e700 1 Processor -- start 2023-01-24T03:37:56.478 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.471+0000 7f273a88e700 1 -- start start 2023-01-24T03:37:56.481 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.471+0000 7f273a88e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27340aef20 0x7f273414fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:56.481 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.471+0000 7f273a88e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f27340c7e40 0x7f27340c1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:56.481 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.471+0000 7f273a88e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2734155da0 0x7f27340c1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:56.482 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.471+0000 7f273a88e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27340c5640 con 0x7f27340aef20 2023-01-24T03:37:56.482 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.471+0000 7f273a88e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27340c3280 con 0x7f27340c7e40 2023-01-24T03:37:56.482 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.471+0000 7f273a88e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27340c5830 con 0x7f2734155da0 2023-01-24T03:37:56.482 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.471+0000 7f2733fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27340aef20 0x7f273414fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:56.482 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.472+0000 7f2733fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27340aef20 0x7f273414fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54586/0 (socket says 172.21.15.112:54586) 2023-01-24T03:37:56.483 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.472+0000 7f2733fff700 1 -- 172.21.15.112:0/393743126 learned_addr learned my addr 172.21.15.112:0/393743126 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:37:56.483 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.472+0000 7f2738e2b700 1 --2- 172.21.15.112:0/393743126 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2734155da0 0x7f27340c1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:56.483 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.472+0000 7f272bfff700 1 --2- 172.21.15.112:0/393743126 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f27340c7e40 0x7f27340c1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:56.483 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.472+0000 7f2733fff700 1 -- 172.21.15.112:0/393743126 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2734155da0 msgr2=0x7f27340c1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:56.483 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.472+0000 7f2733fff700 1 --2- 172.21.15.112:0/393743126 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2734155da0 0x7f27340c1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:56.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.472+0000 7f2733fff700 1 -- 172.21.15.112:0/393743126 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f27340c7e40 msgr2=0x7f27340c1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:56.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.472+0000 7f2733fff700 1 --2- 172.21.15.112:0/393743126 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f27340c7e40 0x7f27340c1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:56.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.472+0000 7f2733fff700 1 -- 172.21.15.112:0/393743126 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2724009370 con 0x7f27340aef20 2023-01-24T03:37:56.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.472+0000 7f2733fff700 1 --2- 172.21.15.112:0/393743126 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27340aef20 0x7f273414fe40 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f2724009130 tx=0x7f2724016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:56.485 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.472+0000 7f2730ff9700 1 -- 172.21.15.112:0/393743126 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2724011940 con 0x7f27340aef20 2023-01-24T03:37:56.485 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.472+0000 7f2730ff9700 1 -- 172.21.15.112:0/393743126 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2724020070 con 0x7f27340aef20 2023-01-24T03:37:56.485 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.472+0000 7f273a88e700 1 -- 172.21.15.112:0/393743126 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27340c5b10 con 0x7f27340aef20 2023-01-24T03:37:56.485 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.472+0000 7f2730ff9700 1 -- 172.21.15.112:0/393743126 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2724027480 con 0x7f27340aef20 2023-01-24T03:37:56.485 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.472+0000 7f273a88e700 1 -- 172.21.15.112:0/393743126 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27340c0af0 con 0x7f27340aef20 2023-01-24T03:37:56.486 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.473+0000 7f273a88e700 1 -- 172.21.15.112:0/393743126 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f273404bb80 con 0x7f27340aef20 2023-01-24T03:37:56.486 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.473+0000 7f2730ff9700 1 -- 172.21.15.112:0/393743126 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f2724017020 con 0x7f27340aef20 2023-01-24T03:37:56.486 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.474+0000 7f2730ff9700 1 --2- 172.21.15.112:0/393743126 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f27180611c0 0x7f2718063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:56.486 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.474+0000 7f2730ff9700 1 -- 172.21.15.112:0/393743126 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 7297+0+0 (secure 0 0 0) 0x7f2724022070 con 0x7f27340aef20 2023-01-24T03:37:56.486 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.474+0000 7f272bfff700 1 --2- 172.21.15.112:0/393743126 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f27180611c0 0x7f2718063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:56.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.474+0000 7f272bfff700 1 --2- 172.21.15.112:0/393743126 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f27180611c0 0x7f2718063670 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f27340bf730 tx=0x7f271c009060 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:56.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.478+0000 7f2730ff9700 1 -- 172.21.15.112:0/393743126 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f272404f2f0 con 0x7f27340aef20 2023-01-24T03:37:56.700 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.698+0000 7f273a88e700 1 -- 172.21.15.112:0/393743126 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f273406efe0 con 0x7f27340aef20 2023-01-24T03:37:56.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.699+0000 7f2730ff9700 1 -- 172.21.15.112:0/393743126 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+5361 (secure 0 0 0) 0x7f2724052910 con 0x7f27340aef20 2023-01-24T03:37:56.701 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:37:56.702 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:37:51.743415+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5296":{"gid":5296,"name":"b","rank":0,"incarnation":32,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.142:6834/2838716685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2838716685},{"type":"v1","addr":"172.21.15.142:6834","nonce":2838716685}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:37:56.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.700+0000 7f273a88e700 1 -- 172.21.15.112:0/393743126 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f27180611c0 msgr2=0x7f2718063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:56.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.700+0000 7f273a88e700 1 --2- 172.21.15.112:0/393743126 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f27180611c0 0x7f2718063670 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f27340bf730 tx=0x7f271c009060 comp rx=0 tx=0).stop 2023-01-24T03:37:56.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.700+0000 7f273a88e700 1 -- 172.21.15.112:0/393743126 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27340aef20 msgr2=0x7f273414fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:56.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.700+0000 7f273a88e700 1 --2- 172.21.15.112:0/393743126 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27340aef20 0x7f273414fe40 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f2724009130 tx=0x7f2724016000 comp rx=0 tx=0).stop 2023-01-24T03:37:56.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.701+0000 7f273a88e700 1 -- 172.21.15.112:0/393743126 shutdown_connections 2023-01-24T03:37:56.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.701+0000 7f273a88e700 1 --2- 172.21.15.112:0/393743126 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f27180611c0 0x7f2718063670 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:56.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.701+0000 7f273a88e700 1 --2- 172.21.15.112:0/393743126 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27340aef20 0x7f273414fe40 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:56.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.701+0000 7f273a88e700 1 --2- 172.21.15.112:0/393743126 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2734155da0 0x7f27340c1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:56.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.701+0000 7f273a88e700 1 --2- 172.21.15.112:0/393743126 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f27340c7e40 0x7f27340c1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:56.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.701+0000 7f273a88e700 1 -- 172.21.15.112:0/393743126 >> 172.21.15.112:0/393743126 conn(0x7f273405e2c0 msgr2=0x7f2734157fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:37:56.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.701+0000 7f273a88e700 1 -- 172.21.15.112:0/393743126 shutdown_connections 2023-01-24T03:37:56.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:56.701+0000 7f273a88e700 1 -- 172.21.15.112:0/393743126 wait complete. 2023-01-24T03:37:56.706 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 34 2023-01-24T03:37:56.714 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:37:58.716 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:37:58.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.839+0000 7f043d9f6700 1 Processor -- start 2023-01-24T03:37:58.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.839+0000 7f043d9f6700 1 -- start start 2023-01-24T03:37:58.844 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.840+0000 7f043d9f6700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f0438068b10 0x7f04380cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:58.844 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.840+0000 7f043d9f6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f04380aef60 0x7f04380b14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:58.844 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.840+0000 7f043d9f6700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04381560e0 con 0x7f04380aeb90 2023-01-24T03:37:58.845 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.840+0000 7f043d9f6700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0438155530 con 0x7f0438068b10 2023-01-24T03:37:58.845 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.840+0000 7f043d9f6700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0438069ac0 con 0x7f04380aef60 2023-01-24T03:37:58.845 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.840+0000 7f0436ffd700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f04380aeb90 0x7f0438068400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35586/0 (socket says 172.21.15.112:35586) 2023-01-24T03:37:58.847 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.840+0000 7f0436ffd700 1 -- 172.21.15.112:0/2567792522 learned_addr learned my addr 172.21.15.112:0/2567792522 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:37:58.847 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.840+0000 7f04367fc700 1 --2- 172.21.15.112:0/2567792522 >> v2:172.21.15.112:3300/0 conn(0x7f0438068b10 0x7f04380cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:58.847 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.840+0000 7f04367fc700 1 -- 172.21.15.112:0/2567792522 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f04380aef60 msgr2=0x7f04380b14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:37:58.847 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.840+0000 7f04367fc700 1 --2- 172.21.15.112:0/2567792522 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f04380aef60 0x7f04380b14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:58.847 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.840+0000 7f04367fc700 1 -- 172.21.15.112:0/2567792522 >> v1:172.21.15.112:6789/0 conn(0x7f04380aeb90 legacy=0x7f0438068400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:58.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.841+0000 7f0435ffb700 1 -- 172.21.15.112:0/2567792522 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2139998554 0 0) 0x7f04381560e0 con 0x7f04380aeb90 2023-01-24T03:37:58.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.841+0000 7f04367fc700 1 -- 172.21.15.112:0/2567792522 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0438069d80 con 0x7f0438068b10 2023-01-24T03:37:58.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.841+0000 7f04367fc700 1 --2- 172.21.15.112:0/2567792522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0438068b10 0x7f04380cc3e0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f0420009760 tx=0x7f0420009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c699a0120554d768 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:58.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.841+0000 7f0435ffb700 1 -- 172.21.15.112:0/2567792522 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0420011630 con 0x7f0438068b10 2023-01-24T03:37:58.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.841+0000 7f0435ffb700 1 -- 172.21.15.112:0/2567792522 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0420011790 con 0x7f0438068b10 2023-01-24T03:37:58.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.841+0000 7f0435ffb700 1 -- 172.21.15.112:0/2567792522 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0420011b30 con 0x7f0438068b10 2023-01-24T03:37:58.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.841+0000 7f043d9f6700 1 -- 172.21.15.112:0/2567792522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0438068b10 msgr2=0x7f04380cc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:58.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.841+0000 7f043d9f6700 1 --2- 172.21.15.112:0/2567792522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0438068b10 0x7f04380cc3e0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f0420009760 tx=0x7f0420009060 comp rx=0 tx=0).stop 2023-01-24T03:37:58.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.841+0000 7f043d9f6700 1 -- 172.21.15.112:0/2567792522 shutdown_connections 2023-01-24T03:37:58.851 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.841+0000 7f043d9f6700 1 --2- 172.21.15.112:0/2567792522 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f04380aef60 0x7f04380b14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:58.851 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.841+0000 7f043d9f6700 1 --2- 172.21.15.112:0/2567792522 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0438068b10 0x7f04380cc3e0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:58.851 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.841+0000 7f043d9f6700 1 -- 172.21.15.112:0/2567792522 >> 172.21.15.112:0/2567792522 conn(0x7f043805e300 msgr2=0x7f04380cba20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:37:58.851 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.841+0000 7f043d9f6700 1 -- 172.21.15.112:0/2567792522 shutdown_connections 2023-01-24T03:37:58.852 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.841+0000 7f043d9f6700 1 -- 172.21.15.112:0/2567792522 wait complete. 2023-01-24T03:37:58.854 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.843+0000 7f043d9f6700 1 Processor -- start 2023-01-24T03:37:58.855 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.843+0000 7f043d9f6700 1 -- start start 2023-01-24T03:37:58.855 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.843+0000 7f043d9f6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0438068b10 0x7f0438149d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:58.855 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.843+0000 7f043d9f6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f04380aeb90 0x7f0438148cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:58.855 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.843+0000 7f043d9f6700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f04380aef60 0x7f0438149230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:58.856 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.843+0000 7f043d9f6700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04380c16c0 con 0x7f0438068b10 2023-01-24T03:37:58.856 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.843+0000 7f043d9f6700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04380bfa60 con 0x7f04380aef60 2023-01-24T03:37:58.856 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.843+0000 7f043d9f6700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0438166060 con 0x7f04380aeb90 2023-01-24T03:37:58.856 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.843+0000 7f0436ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0438068b10 0x7f0438149d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:58.856 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.843+0000 7f0436ffd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0438068b10 0x7f0438149d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54604/0 (socket says 172.21.15.112:54604) 2023-01-24T03:37:58.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.843+0000 7f0436ffd700 1 -- 172.21.15.112:0/1393697519 learned_addr learned my addr 172.21.15.112:0/1393697519 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:37:58.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.844+0000 7f04377fe700 1 --2- 172.21.15.112:0/1393697519 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f04380aef60 0x7f0438149230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:58.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.844+0000 7f04367fc700 1 --2- 172.21.15.112:0/1393697519 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f04380aeb90 0x7f0438148cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:58.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.844+0000 7f0436ffd700 1 -- 172.21.15.112:0/1393697519 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f04380aeb90 msgr2=0x7f0438148cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:58.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.844+0000 7f0436ffd700 1 --2- 172.21.15.112:0/1393697519 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f04380aeb90 0x7f0438148cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:58.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.844+0000 7f0436ffd700 1 -- 172.21.15.112:0/1393697519 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f04380aef60 msgr2=0x7f0438149230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:58.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.844+0000 7f0436ffd700 1 --2- 172.21.15.112:0/1393697519 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f04380aef60 0x7f0438149230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:58.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.844+0000 7f0436ffd700 1 -- 172.21.15.112:0/1393697519 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04200092a0 con 0x7f0438068b10 2023-01-24T03:37:58.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.844+0000 7f0436ffd700 1 --2- 172.21.15.112:0/1393697519 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0438068b10 0x7f0438149d80 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f042800c840 tx=0x7f0428001570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:58.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.844+0000 7f042effd700 1 -- 172.21.15.112:0/1393697519 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0428005d30 con 0x7f0438068b10 2023-01-24T03:37:58.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.844+0000 7f042effd700 1 -- 172.21.15.112:0/1393697519 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0428005e90 con 0x7f0438068b10 2023-01-24T03:37:58.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.844+0000 7f043d9f6700 1 -- 172.21.15.112:0/1393697519 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04380c49b0 con 0x7f0438068b10 2023-01-24T03:37:58.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.844+0000 7f042effd700 1 -- 172.21.15.112:0/1393697519 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0428007300 con 0x7f0438068b10 2023-01-24T03:37:58.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.844+0000 7f043d9f6700 1 -- 172.21.15.112:0/1393697519 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04380c4f70 con 0x7f0438068b10 2023-01-24T03:37:58.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.845+0000 7f043d9f6700 1 -- 172.21.15.112:0/1393697519 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04380bf830 con 0x7f0438068b10 2023-01-24T03:37:58.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.845+0000 7f042effd700 1 -- 172.21.15.112:0/1393697519 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f0428010070 con 0x7f0438068b10 2023-01-24T03:37:58.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.846+0000 7f042effd700 1 --2- 172.21.15.112:0/1393697519 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f041c0611c0 0x7f041c063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:37:58.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.846+0000 7f042effd700 1 -- 172.21.15.112:0/1393697519 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 7297+0+0 (secure 0 0 0) 0x7f042807d530 con 0x7f0438068b10 2023-01-24T03:37:58.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.846+0000 7f04367fc700 1 --2- 172.21.15.112:0/1393697519 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f041c0611c0 0x7f041c063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:37:58.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.846+0000 7f04367fc700 1 --2- 172.21.15.112:0/1393697519 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f041c0611c0 0x7f041c063670 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0420011fd0 tx=0x7f0420017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:37:58.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:58.850+0000 7f042effd700 1 -- 172.21.15.112:0/1393697519 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f04380bf830 con 0x7f0438068b10 2023-01-24T03:37:59.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:59.072+0000 7f043d9f6700 1 -- 172.21.15.112:0/1393697519 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04380bf830 con 0x7f0438068b10 2023-01-24T03:37:59.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:59.072+0000 7f042effd700 1 -- 172.21.15.112:0/1393697519 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+5361 (secure 0 0 0) 0x7f042804d200 con 0x7f0438068b10 2023-01-24T03:37:59.075 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:37:59.075 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:37:57.754547+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5296":{"gid":5296,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":20,"addr":"172.21.15.142:6834/2838716685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2838716685},{"type":"v1","addr":"172.21.15.142:6834","nonce":2838716685}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:37:59.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:59.074+0000 7f043d9f6700 1 -- 172.21.15.112:0/1393697519 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f041c0611c0 msgr2=0x7f041c063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:59.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:59.074+0000 7f043d9f6700 1 --2- 172.21.15.112:0/1393697519 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f041c0611c0 0x7f041c063670 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0420011fd0 tx=0x7f0420017000 comp rx=0 tx=0).stop 2023-01-24T03:37:59.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:59.074+0000 7f043d9f6700 1 -- 172.21.15.112:0/1393697519 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0438068b10 msgr2=0x7f0438149d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:37:59.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:59.074+0000 7f043d9f6700 1 --2- 172.21.15.112:0/1393697519 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0438068b10 0x7f0438149d80 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f042800c840 tx=0x7f0428001570 comp rx=0 tx=0).stop 2023-01-24T03:37:59.078 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:59.074+0000 7f043d9f6700 1 -- 172.21.15.112:0/1393697519 shutdown_connections 2023-01-24T03:37:59.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:59.074+0000 7f043d9f6700 1 --2- 172.21.15.112:0/1393697519 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f041c0611c0 0x7f041c063670 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:59.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:59.074+0000 7f043d9f6700 1 --2- 172.21.15.112:0/1393697519 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0438068b10 0x7f0438149d80 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:59.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:59.075+0000 7f043d9f6700 1 --2- 172.21.15.112:0/1393697519 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f04380aef60 0x7f0438149230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:59.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:59.075+0000 7f043d9f6700 1 --2- 172.21.15.112:0/1393697519 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f04380aeb90 0x7f0438148cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:37:59.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:59.075+0000 7f043d9f6700 1 -- 172.21.15.112:0/1393697519 >> 172.21.15.112:0/1393697519 conn(0x7f043805e300 msgr2=0x7f043814b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:37:59.080 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:59.075+0000 7f043d9f6700 1 -- 172.21.15.112:0/1393697519 shutdown_connections 2023-01-24T03:37:59.080 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:37:59.075+0000 7f043d9f6700 1 -- 172.21.15.112:0/1393697519 wait complete. 2023-01-24T03:37:59.080 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 35 2023-01-24T03:37:59.088 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=4907 2023-01-24T03:37:59.089 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.d 2023-01-24T03:37:59.888 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:37:59.891 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:38:03.089 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2023-01-24T03:38:03.090 INFO:tasks.ceph.mds.d:Restarting daemon 2023-01-24T03:38:03.090 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-01-24T03:38:03.092 INFO:tasks.ceph.mds.d:Started 2023-01-24T03:38:03.093 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:38:03.164 INFO:tasks.ceph.mds.d.smithi142.stdout:starting mds.d at 2023-01-24T03:38:03.208 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.205+0000 7f021c727700 1 Processor -- start 2023-01-24T03:38:03.208 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.205+0000 7f021c727700 1 -- start start 2023-01-24T03:38:03.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.206+0000 7f021c727700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f02140c7e40 0x7f02140b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:38:03.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.206+0000 7f021c727700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f0214155da0 0x7f02140ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:38:03.212 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.206+0000 7f021c727700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0214156240 con 0x7f02140aef20 2023-01-24T03:38:03.212 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.206+0000 7f021c727700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02140af320 con 0x7f0214155da0 2023-01-24T03:38:03.212 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.206+0000 7f021c727700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02140b1170 con 0x7f02140c7e40 2023-01-24T03:38:03.212 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.206+0000 7f0219cc2700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f0214155da0 0x7f02140ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:38:03.213 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.206+0000 7f0219cc2700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f0214155da0 0x7f02140ae9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54618/0 (socket says 172.21.15.112:54618) 2023-01-24T03:38:03.213 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.206+0000 7f0219cc2700 1 -- 172.21.15.112:0/3961814299 learned_addr learned my addr 172.21.15.112:0/3961814299 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:38:03.213 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.206+0000 7f021a4c3700 1 --2- 172.21.15.112:0/3961814299 >> v2:172.21.15.142:3300/0 conn(0x7f02140c7e40 0x7f02140b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:38:03.213 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.206+0000 7f0219cc2700 1 -- 172.21.15.112:0/3961814299 >> v2:172.21.15.142:3300/0 conn(0x7f02140c7e40 msgr2=0x7f02140b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:38:03.213 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.206+0000 7f0219cc2700 1 --2- 172.21.15.112:0/3961814299 >> v2:172.21.15.142:3300/0 conn(0x7f02140c7e40 0x7f02140b0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:38:03.214 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.206+0000 7f0219cc2700 1 -- 172.21.15.112:0/3961814299 >> v1:172.21.15.142:6789/0 conn(0x7f02140aef20 legacy=0x7f02140ca3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:38:03.214 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.206+0000 7f0219cc2700 1 -- 172.21.15.112:0/3961814299 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02140b1450 con 0x7f0214155da0 2023-01-24T03:38:03.214 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.207+0000 7f0219cc2700 1 --2- 172.21.15.112:0/3961814299 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0214155da0 0x7f02140ae9e0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f0214089cf0 tx=0x7f02080090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aec1715380eb3d15 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:38:03.214 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.207+0000 7f02194c1700 1 -- 172.21.15.112:0/3961814299 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0208011630 con 0x7f0214155da0 2023-01-24T03:38:03.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.207+0000 7f02194c1700 1 -- 172.21.15.112:0/3961814299 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02080117b0 con 0x7f0214155da0 2023-01-24T03:38:03.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.207+0000 7f02194c1700 1 -- 172.21.15.112:0/3961814299 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0208011b70 con 0x7f0214155da0 2023-01-24T03:38:03.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.207+0000 7f021c727700 1 -- 172.21.15.112:0/3961814299 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0214155da0 msgr2=0x7f02140ae9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:38:03.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.207+0000 7f021c727700 1 --2- 172.21.15.112:0/3961814299 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0214155da0 0x7f02140ae9e0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f0214089cf0 tx=0x7f02080090f0 comp rx=0 tx=0).stop 2023-01-24T03:38:03.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.207+0000 7f021c727700 1 -- 172.21.15.112:0/3961814299 shutdown_connections 2023-01-24T03:38:03.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.207+0000 7f021c727700 1 --2- 172.21.15.112:0/3961814299 >> v2:172.21.15.142:3300/0 conn(0x7f02140c7e40 0x7f02140b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:38:03.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.207+0000 7f021c727700 1 --2- 172.21.15.112:0/3961814299 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0214155da0 0x7f02140ae9e0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:38:03.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.207+0000 7f021c727700 1 -- 172.21.15.112:0/3961814299 >> 172.21.15.112:0/3961814299 conn(0x7f021405e2c0 msgr2=0x7f02140cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:38:03.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.207+0000 7f021c727700 1 -- 172.21.15.112:0/3961814299 shutdown_connections 2023-01-24T03:38:03.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.207+0000 7f021c727700 1 -- 172.21.15.112:0/3961814299 wait complete. 2023-01-24T03:38:03.217 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.208+0000 7f021c727700 1 Processor -- start 2023-01-24T03:38:03.217 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.208+0000 7f021c727700 1 -- start start 2023-01-24T03:38:03.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.209+0000 7f021c727700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02140aef20 0x7f02141523b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:38:03.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.209+0000 7f021c727700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02140c7e40 0x7f0214068280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:38:03.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.209+0000 7f021c727700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0214155da0 0x7f0214165e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:38:03.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.209+0000 7f021c727700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02141528f0 con 0x7f02140aef20 2023-01-24T03:38:03.221 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.209+0000 7f021c727700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02141659d0 con 0x7f0214155da0 2023-01-24T03:38:03.221 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.209+0000 7f021c727700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02140c4e10 con 0x7f02140c7e40 2023-01-24T03:38:03.221 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.209+0000 7f021a4c3700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02140aef20 0x7f02141523b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:38:03.221 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.209+0000 7f021a4c3700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02140aef20 0x7f02141523b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54620/0 (socket says 172.21.15.112:54620) 2023-01-24T03:38:03.221 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.209+0000 7f021a4c3700 1 -- 172.21.15.112:0/3304343565 learned_addr learned my addr 172.21.15.112:0/3304343565 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:38:03.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.209+0000 7f021acc4700 1 --2- 172.21.15.112:0/3304343565 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0214155da0 0x7f0214165e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:38:03.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.209+0000 7f0219cc2700 1 --2- 172.21.15.112:0/3304343565 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02140c7e40 0x7f0214068280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:38:03.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.209+0000 7f021a4c3700 1 -- 172.21.15.112:0/3304343565 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02140c7e40 msgr2=0x7f0214068280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:38:03.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.209+0000 7f021a4c3700 1 --2- 172.21.15.112:0/3304343565 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02140c7e40 0x7f0214068280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:38:03.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.209+0000 7f021a4c3700 1 -- 172.21.15.112:0/3304343565 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0214155da0 msgr2=0x7f0214165e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:38:03.223 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.209+0000 7f021a4c3700 1 --2- 172.21.15.112:0/3304343565 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0214155da0 0x7f0214165e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:38:03.223 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.209+0000 7f021a4c3700 1 -- 172.21.15.112:0/3304343565 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0208009370 con 0x7f02140aef20 2023-01-24T03:38:03.223 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.209+0000 7f021acc4700 1 --2- 172.21.15.112:0/3304343565 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0214155da0 0x7f0214165e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:38:03.223 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.210+0000 7f021a4c3700 1 --2- 172.21.15.112:0/3304343565 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02140aef20 0x7f02141523b0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f0204009fa0 tx=0x7f0204009880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:38:03.223 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.210+0000 7f02127fc700 1 -- 172.21.15.112:0/3304343565 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f020400edb0 con 0x7f02140aef20 2023-01-24T03:38:03.224 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.210+0000 7f02127fc700 1 -- 172.21.15.112:0/3304343565 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0204014420 con 0x7f02140aef20 2023-01-24T03:38:03.224 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.210+0000 7f021c727700 1 -- 172.21.15.112:0/3304343565 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02140c50f0 con 0x7f02140aef20 2023-01-24T03:38:03.224 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.210+0000 7f02127fc700 1 -- 172.21.15.112:0/3304343565 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02040147e0 con 0x7f02140aef20 2023-01-24T03:38:03.224 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.210+0000 7f021c727700 1 -- 172.21.15.112:0/3304343565 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02140c56b0 con 0x7f02140aef20 2023-01-24T03:38:03.224 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.211+0000 7f021c727700 1 -- 172.21.15.112:0/3304343565 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02140bf760 con 0x7f02140aef20 2023-01-24T03:38:03.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.211+0000 7f02127fc700 1 -- 172.21.15.112:0/3304343565 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f0204012050 con 0x7f02140aef20 2023-01-24T03:38:03.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.211+0000 7f02127fc700 1 --2- 172.21.15.112:0/3304343565 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f01f40611c0 0x7f01f4063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:38:03.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.211+0000 7f02127fc700 1 -- 172.21.15.112:0/3304343565 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 7297+0+0 (secure 0 0 0) 0x7f020407e120 con 0x7f02140aef20 2023-01-24T03:38:03.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.211+0000 7f0219cc2700 1 --2- 172.21.15.112:0/3304343565 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f01f40611c0 0x7f01f4063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:38:03.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.212+0000 7f0219cc2700 1 --2- 172.21.15.112:0/3304343565 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f01f40611c0 0x7f01f4063670 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0214089cf0 tx=0x7f0208014000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:38:03.226 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.215+0000 7f02127fc700 1 -- 172.21.15.112:0/3304343565 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f020404dd10 con 0x7f02140aef20 2023-01-24T03:38:03.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.435+0000 7f021c727700 1 -- 172.21.15.112:0/3304343565 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0214165cb0 con 0x7f02140aef20 2023-01-24T03:38:03.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.435+0000 7f02127fc700 1 -- 172.21.15.112:0/3304343565 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+5361 (secure 0 0 0) 0x7f0204051330 con 0x7f02140aef20 2023-01-24T03:38:03.440 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:38:03.441 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:37:57.754547+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5296":{"gid":5296,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":20,"addr":"172.21.15.142:6834/2838716685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2838716685},{"type":"v1","addr":"172.21.15.142:6834","nonce":2838716685}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:38:03.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.438+0000 7f021c727700 1 -- 172.21.15.112:0/3304343565 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f01f40611c0 msgr2=0x7f01f4063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:38:03.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.438+0000 7f021c727700 1 --2- 172.21.15.112:0/3304343565 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f01f40611c0 0x7f01f4063670 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0214089cf0 tx=0x7f0208014000 comp rx=0 tx=0).stop 2023-01-24T03:38:03.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.438+0000 7f021c727700 1 -- 172.21.15.112:0/3304343565 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02140aef20 msgr2=0x7f02141523b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:38:03.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.438+0000 7f021c727700 1 --2- 172.21.15.112:0/3304343565 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02140aef20 0x7f02141523b0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f0204009fa0 tx=0x7f0204009880 comp rx=0 tx=0).stop 2023-01-24T03:38:03.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.438+0000 7f021c727700 1 -- 172.21.15.112:0/3304343565 shutdown_connections 2023-01-24T03:38:03.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.438+0000 7f021c727700 1 --2- 172.21.15.112:0/3304343565 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f01f40611c0 0x7f01f4063670 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:38:03.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.438+0000 7f021c727700 1 --2- 172.21.15.112:0/3304343565 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02140aef20 0x7f02141523b0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:38:03.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.438+0000 7f021c727700 1 --2- 172.21.15.112:0/3304343565 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0214155da0 0x7f0214165e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:38:03.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.438+0000 7f021c727700 1 --2- 172.21.15.112:0/3304343565 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f02140c7e40 0x7f0214068280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:38:03.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.438+0000 7f021c727700 1 -- 172.21.15.112:0/3304343565 >> 172.21.15.112:0/3304343565 conn(0x7f021405e2c0 msgr2=0x7f0214159250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:38:03.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.438+0000 7f021c727700 1 -- 172.21.15.112:0/3304343565 shutdown_connections 2023-01-24T03:38:03.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:03.438+0000 7f021c727700 1 -- 172.21.15.112:0/3304343565 wait complete. 2023-01-24T03:38:03.445 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 35 2023-01-24T03:38:03.453 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2023-01-24T03:38:03.786 INFO:tasks.ceph.mds.b.smithi142.stderr:2023-01-24T03:38:03.783+0000 7f14ac52c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-01-24T03:38:05.454 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:38:05.570 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.567+0000 7fdf96509700 1 Processor -- start 2023-01-24T03:38:05.571 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.567+0000 7fdf96509700 1 -- start start 2023-01-24T03:38:05.573 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.568+0000 7fdf96509700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf900c5680 0x7fdf90156730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:38:05.573 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.568+0000 7fdf96509700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fdf90155e20 0x7fdf90156c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:38:05.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.568+0000 7fdf96509700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fdf900c5a50 0x7fdf900cb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:38:05.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.568+0000 7fdf96509700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf900c4eb0 con 0x7fdf900c5a50 2023-01-24T03:38:05.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.568+0000 7fdf96509700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf900c3d10 con 0x7fdf90155e20 2023-01-24T03:38:05.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.568+0000 7fdf96509700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf900c2e20 con 0x7fdf900c5680 2023-01-24T03:38:05.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.568+0000 7fdf95d08700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fdf900c5a50 0x7fdf900cb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:38:05.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.568+0000 7fdf95d08700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fdf900c5a50 0x7fdf900cb870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54640/0 (socket says 172.21.15.112:54640) 2023-01-24T03:38:05.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.568+0000 7fdf95d08700 1 -- 172.21.15.112:0/2794092164 learned_addr learned my addr 172.21.15.112:0/2794092164 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:38:05.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.568+0000 7fdf95507700 1 --2- 172.21.15.112:0/2794092164 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf900c5680 0x7fdf90156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:38:05.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.568+0000 7fdf94d06700 1 --2- 172.21.15.112:0/2794092164 >> v2:172.21.15.142:3300/0 conn(0x7fdf90155e20 0x7fdf90156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:38:05.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.568+0000 7fdf95d08700 1 -- 172.21.15.112:0/2794092164 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf900c5680 msgr2=0x7fdf90156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:38:05.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.568+0000 7fdf95d08700 1 --2- 172.21.15.112:0/2794092164 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf900c5680 0x7fdf90156730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:38:05.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.568+0000 7fdf95d08700 1 -- 172.21.15.112:0/2794092164 >> v2:172.21.15.142:3300/0 conn(0x7fdf90155e20 msgr2=0x7fdf90156c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:38:05.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.568+0000 7fdf95d08700 1 --2- 172.21.15.112:0/2794092164 >> v2:172.21.15.142:3300/0 conn(0x7fdf90155e20 0x7fdf90156c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:38:05.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.569+0000 7fdf95d08700 1 -- 172.21.15.112:0/2794092164 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf900c3100 con 0x7fdf900c5a50 2023-01-24T03:38:05.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.569+0000 7fdf95d08700 1 --2- 172.21.15.112:0/2794092164 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdf900c5a50 0x7fdf900cb870 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fdf88009f20 tx=0x7fdf88009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93b982792166050c server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:38:05.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.569+0000 7fdf7ffff700 1 -- 172.21.15.112:0/2794092164 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf8800e070 con 0x7fdf900c5a50 2023-01-24T03:38:05.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.569+0000 7fdf7ffff700 1 -- 172.21.15.112:0/2794092164 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf8800b9c0 con 0x7fdf900c5a50 2023-01-24T03:38:05.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.569+0000 7fdf7ffff700 1 -- 172.21.15.112:0/2794092164 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf880092e0 con 0x7fdf900c5a50 2023-01-24T03:38:05.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.569+0000 7fdf96509700 1 -- 172.21.15.112:0/2794092164 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdf900c5a50 msgr2=0x7fdf900cb870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:38:05.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.569+0000 7fdf96509700 1 --2- 172.21.15.112:0/2794092164 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdf900c5a50 0x7fdf900cb870 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fdf88009f20 tx=0x7fdf88009840 comp rx=0 tx=0).stop 2023-01-24T03:38:05.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.569+0000 7fdf96509700 1 -- 172.21.15.112:0/2794092164 shutdown_connections 2023-01-24T03:38:05.579 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.569+0000 7fdf96509700 1 --2- 172.21.15.112:0/2794092164 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdf900c5a50 0x7fdf900cb870 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:38:05.579 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.569+0000 7fdf96509700 1 --2- 172.21.15.112:0/2794092164 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf900c5680 0x7fdf90156730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:38:05.579 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.569+0000 7fdf96509700 1 --2- 172.21.15.112:0/2794092164 >> v2:172.21.15.142:3300/0 conn(0x7fdf90155e20 0x7fdf90156c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:38:05.579 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.569+0000 7fdf96509700 1 -- 172.21.15.112:0/2794092164 >> 172.21.15.112:0/2794092164 conn(0x7fdf900685b0 msgr2=0x7fdf900689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:38:05.579 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.569+0000 7fdf96509700 1 -- 172.21.15.112:0/2794092164 shutdown_connections 2023-01-24T03:38:05.580 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.569+0000 7fdf96509700 1 -- 172.21.15.112:0/2794092164 wait complete. 2023-01-24T03:38:05.580 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.570+0000 7fdf96509700 1 Processor -- start 2023-01-24T03:38:05.580 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.570+0000 7fdf96509700 1 -- start start 2023-01-24T03:38:05.583 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.571+0000 7fdf96509700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf900c5680 0x7fdf9014d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:38:05.583 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.571+0000 7fdf96509700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf900c5a50 0x7fdf900c7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:38:05.583 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.571+0000 7fdf96509700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdf90155e20 0x7fdf900c8340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:38:05.583 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.571+0000 7fdf96509700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf900c7970 con 0x7fdf90155e20 2023-01-24T03:38:05.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.571+0000 7fdf96509700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf90159170 con 0x7fdf900c5680 2023-01-24T03:38:05.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.571+0000 7fdf96509700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf90159410 con 0x7fdf900c5a50 2023-01-24T03:38:05.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.571+0000 7fdf95d08700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdf90155e20 0x7fdf900c8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:38:05.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.571+0000 7fdf95d08700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdf90155e20 0x7fdf900c8340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:54646/0 (socket says 172.21.15.112:54646) 2023-01-24T03:38:05.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.571+0000 7fdf95d08700 1 -- 172.21.15.112:0/3631326025 learned_addr learned my addr 172.21.15.112:0/3631326025 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:38:05.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.571+0000 7fdf94d06700 1 --2- 172.21.15.112:0/3631326025 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf900c5a50 0x7fdf900c7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:38:05.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.571+0000 7fdf95507700 1 --2- 172.21.15.112:0/3631326025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf900c5680 0x7fdf9014d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:38:05.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.572+0000 7fdf95d08700 1 -- 172.21.15.112:0/3631326025 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf900c5a50 msgr2=0x7fdf900c7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:38:05.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.572+0000 7fdf95d08700 1 --2- 172.21.15.112:0/3631326025 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf900c5a50 0x7fdf900c7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:38:05.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.572+0000 7fdf95d08700 1 -- 172.21.15.112:0/3631326025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf900c5680 msgr2=0x7fdf9014d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:38:05.586 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.572+0000 7fdf95d08700 1 --2- 172.21.15.112:0/3631326025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf900c5680 0x7fdf9014d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:38:05.586 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.572+0000 7fdf95d08700 1 -- 172.21.15.112:0/3631326025 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf88009ae0 con 0x7fdf90155e20 2023-01-24T03:38:05.586 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.572+0000 7fdf95d08700 1 --2- 172.21.15.112:0/3631326025 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdf90155e20 0x7fdf900c8340 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fdf88009880 tx=0x7fdf88018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:38:05.586 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.572+0000 7fdf7d7fa700 1 -- 172.21.15.112:0/3631326025 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf88008ad0 con 0x7fdf90155e20 2023-01-24T03:38:05.587 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.572+0000 7fdf7d7fa700 1 -- 172.21.15.112:0/3631326025 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf88008c30 con 0x7fdf90155e20 2023-01-24T03:38:05.587 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.572+0000 7fdf96509700 1 -- 172.21.15.112:0/3631326025 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf900c7320 con 0x7fdf90155e20 2023-01-24T03:38:05.587 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.572+0000 7fdf7d7fa700 1 -- 172.21.15.112:0/3631326025 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf8801d420 con 0x7fdf90155e20 2023-01-24T03:38:05.587 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.572+0000 7fdf96509700 1 -- 172.21.15.112:0/3631326025 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf900c25a0 con 0x7fdf90155e20 2023-01-24T03:38:05.588 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.573+0000 7fdf96509700 1 -- 172.21.15.112:0/3631326025 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf900c77c0 con 0x7fdf90155e20 2023-01-24T03:38:05.588 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.574+0000 7fdf7d7fa700 1 -- 172.21.15.112:0/3631326025 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fdf88024030 con 0x7fdf90155e20 2023-01-24T03:38:05.588 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.574+0000 7fdf7d7fa700 1 --2- 172.21.15.112:0/3631326025 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fdf700611c0 0x7fdf70063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:38:05.588 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.574+0000 7fdf7d7fa700 1 -- 172.21.15.112:0/3631326025 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 7297+0+0 (secure 0 0 0) 0x7fdf8807fcb0 con 0x7fdf90155e20 2023-01-24T03:38:05.588 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.574+0000 7fdf95507700 1 --2- 172.21.15.112:0/3631326025 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fdf700611c0 0x7fdf70063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:38:05.589 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.574+0000 7fdf95507700 1 --2- 172.21.15.112:0/3631326025 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fdf700611c0 0x7fdf70063670 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c009c30 tx=0x7fdf8c009060 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:38:05.589 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.578+0000 7fdf7d7fa700 1 -- 172.21.15.112:0/3631326025 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fdf8804f900 con 0x7fdf90155e20 2023-01-24T03:38:05.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.797+0000 7fdf96509700 1 -- 172.21.15.112:0/3631326025 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf900c7180 con 0x7fdf90155e20 2023-01-24T03:38:05.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.797+0000 7fdf7d7fa700 1 -- 172.21.15.112:0/3631326025 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+6121 (secure 0 0 0) 0x7fdf900c7180 con 0x7fdf90155e20 2023-01-24T03:38:05.801 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:38:05.801 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/2136880103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2136880103},{"type":"v1","addr":"172.21.15.142:6835","nonce":2136880103}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:37:57.754547+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5296":{"gid":5296,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":20,"addr":"172.21.15.142:6834/2838716685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2838716685},{"type":"v1","addr":"172.21.15.142:6834","nonce":2838716685}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:38:05.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.799+0000 7fdf96509700 1 -- 172.21.15.112:0/3631326025 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fdf700611c0 msgr2=0x7fdf70063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:38:05.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.799+0000 7fdf96509700 1 --2- 172.21.15.112:0/3631326025 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fdf700611c0 0x7fdf70063670 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c009c30 tx=0x7fdf8c009060 comp rx=0 tx=0).stop 2023-01-24T03:38:05.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.799+0000 7fdf96509700 1 -- 172.21.15.112:0/3631326025 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdf90155e20 msgr2=0x7fdf900c8340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:38:05.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.799+0000 7fdf96509700 1 --2- 172.21.15.112:0/3631326025 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdf90155e20 0x7fdf900c8340 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fdf88009880 tx=0x7fdf88018000 comp rx=0 tx=0).stop 2023-01-24T03:38:05.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.799+0000 7fdf96509700 1 -- 172.21.15.112:0/3631326025 shutdown_connections 2023-01-24T03:38:05.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.799+0000 7fdf96509700 1 --2- 172.21.15.112:0/3631326025 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fdf700611c0 0x7fdf70063670 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:38:05.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.799+0000 7fdf96509700 1 --2- 172.21.15.112:0/3631326025 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdf90155e20 0x7fdf900c8340 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:38:05.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.799+0000 7fdf96509700 1 --2- 172.21.15.112:0/3631326025 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf900c5680 0x7fdf9014d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:38:05.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.799+0000 7fdf96509700 1 --2- 172.21.15.112:0/3631326025 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdf900c5a50 0x7fdf900c7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:38:05.805 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.799+0000 7fdf96509700 1 -- 172.21.15.112:0/3631326025 >> 172.21.15.112:0/3631326025 conn(0x7fdf900685b0 msgr2=0x7fdf900cba50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:38:05.805 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.799+0000 7fdf96509700 1 -- 172.21.15.112:0/3631326025 shutdown_connections 2023-01-24T03:38:05.805 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:38:05.799+0000 7fdf96509700 1 -- 172.21.15.112:0/3631326025 wait complete. 2023-01-24T03:38:05.805 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 36 2023-01-24T03:38:05.814 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2023-01-24T03:38:05.814 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 67 secs before thrashing 2023-01-24T03:38:29.951 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:38:29.954 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:39:00.033 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:39:00.036 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:39:12.816 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:39:12.937 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.935+0000 7f13f6ed6700 1 Processor -- start 2023-01-24T03:39:12.937 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.935+0000 7f13f6ed6700 1 -- start start 2023-01-24T03:39:12.940 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.936+0000 7f13f6ed6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13f0156e40 0x7f13f00cb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:12.940 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.936+0000 7f13f6ed6700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f13f0155e20 0x7f13f00cbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:12.940 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.936+0000 7f13f6ed6700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13f00c3d10 con 0x7f13f00c5680 2023-01-24T03:39:12.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.936+0000 7f13f6ed6700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13f00c4eb0 con 0x7f13f0155e20 2023-01-24T03:39:12.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.936+0000 7f13f6ed6700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13f00cc2f0 con 0x7f13f0156e40 2023-01-24T03:39:12.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.936+0000 7f13f66d5700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f13f0155e20 0x7f13f00cbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:12.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.936+0000 7f13f56d3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13f0156e40 0x7f13f00cb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:12.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.936+0000 7f13f66d5700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f13f0155e20 0x7f13f00cbdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.112:47492/0 (socket says 172.21.15.112:47492) 2023-01-24T03:39:12.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.936+0000 7f13f66d5700 1 -- 172.21.15.112:0/3408436276 learned_addr learned my addr 172.21.15.112:0/3408436276 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:39:12.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.936+0000 7f13f5ed4700 1 --1- 172.21.15.112:0/3408436276 >> v1:172.21.15.142:6789/0 conn(0x7f13f00c5680 0x7f13f0156730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.112:49772/0 (socket says 172.21.15.112:49772) 2023-01-24T03:39:12.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.936+0000 7f13f66d5700 1 -- 172.21.15.112:0/3408436276 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13f0156e40 msgr2=0x7f13f00cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:39:12.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.936+0000 7f13f66d5700 1 --2- 172.21.15.112:0/3408436276 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13f0156e40 0x7f13f00cb870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:12.943 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.936+0000 7f13f66d5700 1 -- 172.21.15.112:0/3408436276 >> v1:172.21.15.142:6789/0 conn(0x7f13f00c5680 legacy=0x7f13f0156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:12.943 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.936+0000 7f13f66d5700 1 -- 172.21.15.112:0/3408436276 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f00c2e20 con 0x7f13f0155e20 2023-01-24T03:39:12.943 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.936+0000 7f13f56d3700 1 --2- 172.21.15.112:0/3408436276 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13f0156e40 0x7f13f00cb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:39:12.943 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.937+0000 7f13f66d5700 1 --2- 172.21.15.112:0/3408436276 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f13f0155e20 0x7f13f00cbdb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f13e8009f20 tx=0x7f13e8009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0a64fc9eaead4e7 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:12.944 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.937+0000 7f13f4ed2700 1 -- 172.21.15.112:0/3408436276 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e800e070 con 0x7f13f0155e20 2023-01-24T03:39:12.944 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.937+0000 7f13f4ed2700 1 -- 172.21.15.112:0/3408436276 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13e800ba60 con 0x7f13f0155e20 2023-01-24T03:39:12.944 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.937+0000 7f13f4ed2700 1 -- 172.21.15.112:0/3408436276 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e80092e0 con 0x7f13f0155e20 2023-01-24T03:39:12.944 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.937+0000 7f13f6ed6700 1 -- 172.21.15.112:0/3408436276 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f13f0155e20 msgr2=0x7f13f00cbdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:12.944 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.937+0000 7f13f6ed6700 1 --2- 172.21.15.112:0/3408436276 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f13f0155e20 0x7f13f00cbdb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f13e8009f20 tx=0x7f13e8009840 comp rx=0 tx=0).stop 2023-01-24T03:39:12.945 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.937+0000 7f13f6ed6700 1 -- 172.21.15.112:0/3408436276 shutdown_connections 2023-01-24T03:39:12.945 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.937+0000 7f13f6ed6700 1 --2- 172.21.15.112:0/3408436276 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f13f0155e20 0x7f13f00cbdb0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:12.945 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.937+0000 7f13f6ed6700 1 --2- 172.21.15.112:0/3408436276 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13f0156e40 0x7f13f00cb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:12.945 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.937+0000 7f13f6ed6700 1 -- 172.21.15.112:0/3408436276 >> 172.21.15.112:0/3408436276 conn(0x7f13f00685b0 msgr2=0x7f13f00689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:39:12.945 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.937+0000 7f13f6ed6700 1 -- 172.21.15.112:0/3408436276 shutdown_connections 2023-01-24T03:39:12.946 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.937+0000 7f13f6ed6700 1 -- 172.21.15.112:0/3408436276 wait complete. 2023-01-24T03:39:12.946 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.938+0000 7f13f6ed6700 1 Processor -- start 2023-01-24T03:39:12.946 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.938+0000 7f13f6ed6700 1 -- start start 2023-01-24T03:39:12.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.939+0000 7f13f6ed6700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f13f00c5680 0x7f13f00c24b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:12.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.939+0000 7f13f6ed6700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f13f0155e20 0x7f13f0153d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:12.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.939+0000 7f13f6ed6700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13f0156e40 0x7f13f014bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:12.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.939+0000 7f13f6ed6700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f01502c0 con 0x7f13f00c5680 2023-01-24T03:39:12.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.939+0000 7f13f6ed6700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f014ffe0 con 0x7f13f0155e20 2023-01-24T03:39:12.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.939+0000 7f13f6ed6700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13f00c8490 con 0x7f13f0156e40 2023-01-24T03:39:12.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.939+0000 7f13f5ed4700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f13f00c5680 0x7f13f00c24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:12.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.939+0000 7f13f56d3700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f13f0155e20 0x7f13f0153d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:12.951 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.939+0000 7f13f5ed4700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f13f00c5680 0x7f13f00c24b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55002/0 (socket says 172.21.15.112:55002) 2023-01-24T03:39:12.951 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.939+0000 7f13f56d3700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f13f0155e20 0x7f13f0153d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.112:47494/0 (socket says 172.21.15.112:47494) 2023-01-24T03:39:12.951 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.939+0000 7f13f66d5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13f0156e40 0x7f13f014bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:12.951 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.939+0000 7f13f5ed4700 1 -- 172.21.15.112:0/2665893150 learned_addr learned my addr 172.21.15.112:0/2665893150 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:39:12.951 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.939+0000 7f13f5ed4700 1 -- 172.21.15.112:0/2665893150 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13f0156e40 msgr2=0x7f13f014bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:12.952 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.939+0000 7f13f5ed4700 1 --2- 172.21.15.112:0/2665893150 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13f0156e40 0x7f13f014bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:12.952 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.939+0000 7f13f5ed4700 1 -- 172.21.15.112:0/2665893150 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f13f0155e20 msgr2=0x7f13f0153d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:12.952 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.939+0000 7f13f5ed4700 1 --2- 172.21.15.112:0/2665893150 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f13f0155e20 0x7f13f0153d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:12.952 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.939+0000 7f13f5ed4700 1 -- 172.21.15.112:0/2665893150 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e00092a0 con 0x7f13f00c5680 2023-01-24T03:39:12.952 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.939+0000 7f13f56d3700 1 --2- 172.21.15.112:0/2665893150 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f13f0155e20 0x7f13f0153d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:39:12.953 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.940+0000 7f13f5ed4700 1 --2- 172.21.15.112:0/2665893150 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f13f00c5680 0x7f13f00c24b0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f13ec007fd0 tx=0x7f13ec007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:12.953 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.940+0000 7f13ddffb700 1 -- 172.21.15.112:0/2665893150 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ec00ce10 con 0x7f13f00c5680 2023-01-24T03:39:12.953 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.940+0000 7f13ddffb700 1 -- 172.21.15.112:0/2665893150 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13ec0072e0 con 0x7f13f00c5680 2023-01-24T03:39:12.953 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.940+0000 7f13ddffb700 1 -- 172.21.15.112:0/2665893150 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ec0046b0 con 0x7f13f00c5680 2023-01-24T03:39:12.954 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.940+0000 7f13f6ed6700 1 -- 172.21.15.112:0/2665893150 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13e8009ae0 con 0x7f13f00c5680 2023-01-24T03:39:12.954 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.940+0000 7f13f6ed6700 1 -- 172.21.15.112:0/2665893150 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13f00c8a50 con 0x7f13f00c5680 2023-01-24T03:39:12.954 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.941+0000 7f13f6ed6700 1 -- 172.21.15.112:0/2665893150 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13f005ea20 con 0x7f13f00c5680 2023-01-24T03:39:12.954 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.941+0000 7f13ddffb700 1 -- 172.21.15.112:0/2665893150 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f13ec010070 con 0x7f13f00c5680 2023-01-24T03:39:12.954 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.941+0000 7f13ddffb700 1 --2- 172.21.15.112:0/2665893150 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f13d00611c0 0x7f13d0063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:12.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.941+0000 7f13ddffb700 1 -- 172.21.15.112:0/2665893150 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 7297+0+0 (secure 0 0 0) 0x7f13ec07ca20 con 0x7f13f00c5680 2023-01-24T03:39:12.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.941+0000 7f13f56d3700 1 --2- 172.21.15.112:0/2665893150 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f13d00611c0 0x7f13d0063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:12.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.942+0000 7f13f56d3700 1 --2- 172.21.15.112:0/2665893150 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f13d00611c0 0x7f13d0063670 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f13f015b650 tx=0x7f13e0009060 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:12.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:12.945+0000 7f13ddffb700 1 -- 172.21.15.112:0/2665893150 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f13ec04a1d0 con 0x7f13f00c5680 2023-01-24T03:39:13.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.169+0000 7f13f6ed6700 1 -- 172.21.15.112:0/2665893150 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13f00c3100 con 0x7f13f00c5680 2023-01-24T03:39:13.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.169+0000 7f13ddffb700 1 -- 172.21.15.112:0/2665893150 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+6121 (secure 0 0 0) 0x7f13ec04c610 con 0x7f13f00c5680 2023-01-24T03:39:13.172 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:39:13.172 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/2136880103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2136880103},{"type":"v1","addr":"172.21.15.142:6835","nonce":2136880103}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:37:57.754547+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5296":{"gid":5296,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":20,"addr":"172.21.15.142:6834/2838716685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2838716685},{"type":"v1","addr":"172.21.15.142:6834","nonce":2838716685}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:39:13.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.171+0000 7f13f6ed6700 1 -- 172.21.15.112:0/2665893150 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f13d00611c0 msgr2=0x7f13d0063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:13.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.171+0000 7f13f6ed6700 1 --2- 172.21.15.112:0/2665893150 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f13d00611c0 0x7f13d0063670 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f13f015b650 tx=0x7f13e0009060 comp rx=0 tx=0).stop 2023-01-24T03:39:13.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.171+0000 7f13f6ed6700 1 -- 172.21.15.112:0/2665893150 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f13f00c5680 msgr2=0x7f13f00c24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:13.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.171+0000 7f13f6ed6700 1 --2- 172.21.15.112:0/2665893150 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f13f00c5680 0x7f13f00c24b0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f13ec007fd0 tx=0x7f13ec007df0 comp rx=0 tx=0).stop 2023-01-24T03:39:13.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.171+0000 7f13f6ed6700 1 -- 172.21.15.112:0/2665893150 shutdown_connections 2023-01-24T03:39:13.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.171+0000 7f13f6ed6700 1 --2- 172.21.15.112:0/2665893150 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f13d00611c0 0x7f13d0063670 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:13.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.171+0000 7f13f6ed6700 1 --2- 172.21.15.112:0/2665893150 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f13f00c5680 0x7f13f00c24b0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:13.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.171+0000 7f13f6ed6700 1 --2- 172.21.15.112:0/2665893150 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f13f0156e40 0x7f13f014bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:13.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.171+0000 7f13f6ed6700 1 --2- 172.21.15.112:0/2665893150 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f13f0155e20 0x7f13f0153d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:13.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.171+0000 7f13f6ed6700 1 -- 172.21.15.112:0/2665893150 >> 172.21.15.112:0/2665893150 conn(0x7f13f00685b0 msgr2=0x7f13f0154790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:39:13.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.171+0000 7f13f6ed6700 1 -- 172.21.15.112:0/2665893150 shutdown_connections 2023-01-24T03:39:13.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.171+0000 7f13f6ed6700 1 -- 172.21.15.112:0/2665893150 wait complete. 2023-01-24T03:39:13.176 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 36 2023-01-24T03:39:13.185 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2023-01-24T03:39:13.185 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-01-24T03:39:13.185 INFO:teuthology.orchestra.run:waiting for 300 2023-01-24T03:39:13.198 INFO:tasks.ceph.mds.b:Stopped 2023-01-24T03:39:13.199 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:39:13.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.308+0000 7f3b61d65700 1 Processor -- start 2023-01-24T03:39:13.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.309+0000 7f3b61d65700 1 -- start start 2023-01-24T03:39:13.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.309+0000 7f3b61d65700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f3b5c06cf50 0x7f3b5c155db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:13.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.309+0000 7f3b61d65700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b5c0c54d0 con 0x7f3b5c067f20 2023-01-24T03:39:13.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.309+0000 7f3b61d65700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b5c1562f0 con 0x7f3b5c06d320 2023-01-24T03:39:13.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.309+0000 7f3b61d65700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b5c0c49c0 con 0x7f3b5c06cf50 2023-01-24T03:39:13.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.309+0000 7f3b5bfff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f3b5c067f20 0x7f3b5c14efd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:35998/0 (socket says 172.21.15.112:35998) 2023-01-24T03:39:13.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.309+0000 7f3b5bfff700 1 -- 172.21.15.112:0/3582987414 learned_addr learned my addr 172.21.15.112:0/3582987414 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:39:13.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.310+0000 7f3b5a7fc700 1 -- 172.21.15.112:0/3582987414 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3690345589 0 0) 0x7f3b5c0c54d0 con 0x7f3b5c067f20 2023-01-24T03:39:13.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.310+0000 7f3b5b7fe700 1 --2- 172.21.15.112:0/3582987414 >> v2:172.21.15.142:3300/0 conn(0x7f3b5c06cf50 0x7f3b5c155db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:13.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.310+0000 7f3b5a7fc700 1 -- 172.21.15.112:0/3582987414 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b480033f0 con 0x7f3b5c067f20 2023-01-24T03:39:13.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.310+0000 7f3b5a7fc700 1 -- 172.21.15.112:0/3582987414 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 121512477 0 0) 0x7f3b480033f0 con 0x7f3b5c067f20 2023-01-24T03:39:13.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.310+0000 7f3b5a7fc700 1 -- 172.21.15.112:0/3582987414 >> v2:172.21.15.142:3300/0 conn(0x7f3b5c06cf50 msgr2=0x7f3b5c155db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:39:13.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.310+0000 7f3b5a7fc700 1 --2- 172.21.15.112:0/3582987414 >> v2:172.21.15.142:3300/0 conn(0x7f3b5c06cf50 0x7f3b5c155db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:13.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.310+0000 7f3b5a7fc700 1 -- 172.21.15.112:0/3582987414 >> v1:172.21.15.142:6789/0 conn(0x7f3b5c06d320 legacy=0x7f3b5c0c5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:13.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.310+0000 7f3b5a7fc700 1 -- 172.21.15.112:0/3582987414 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b5c0c6070 con 0x7f3b5c067f20 2023-01-24T03:39:13.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.310+0000 7f3b5a7fc700 1 -- 172.21.15.112:0/3582987414 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7f3b54001470 con 0x7f3b5c067f20 2023-01-24T03:39:13.324 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.310+0000 7f3b5a7fc700 1 -- 172.21.15.112:0/3582987414 >> v1:172.21.15.112:6789/0 conn(0x7f3b5c067f20 legacy=0x7f3b5c14efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:13.324 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.310+0000 7f3b5a7fc700 1 --2- 172.21.15.112:0/3582987414 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b480038e0 0x7f3b48003cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:13.324 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.310+0000 7f3b5a7fc700 1 --2- 172.21.15.112:0/3582987414 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b48004690 0x7f3b48004b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:13.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.310+0000 7f3b5a7fc700 1 --2- 172.21.15.112:0/3582987414 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b48005040 0x7f3b4800b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:13.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.310+0000 7f3b5a7fc700 1 -- 172.21.15.112:0/3582987414 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5c0c49c0 con 0x7f3b48005040 2023-01-24T03:39:13.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.310+0000 7f3b5a7fc700 1 -- 172.21.15.112:0/3582987414 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4800bfe0 con 0x7f3b480038e0 2023-01-24T03:39:13.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.310+0000 7f3b5a7fc700 1 -- 172.21.15.112:0/3582987414 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4800c1a0 con 0x7f3b48004690 2023-01-24T03:39:13.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.310+0000 7f3b5b7fe700 1 --2- 172.21.15.112:0/3582987414 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b480038e0 0x7f3b48003cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:13.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.310+0000 7f3b5bfff700 1 --2- 172.21.15.112:0/3582987414 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b48005040 0x7f3b4800b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:13.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.310+0000 7f3b5affd700 1 --2- 172.21.15.112:0/3582987414 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b48004690 0x7f3b48004b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:13.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.310+0000 7f3b5b7fe700 1 -- 172.21.15.112:0/3582987414 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b48004690 msgr2=0x7f3b48004b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:39:13.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.310+0000 7f3b5b7fe700 1 --2- 172.21.15.112:0/3582987414 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b48004690 0x7f3b48004b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:13.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.310+0000 7f3b5b7fe700 1 -- 172.21.15.112:0/3582987414 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b48005040 msgr2=0x7f3b4800b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:39:13.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.310+0000 7f3b5b7fe700 1 --2- 172.21.15.112:0/3582987414 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b48005040 0x7f3b4800b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:13.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.310+0000 7f3b5b7fe700 1 -- 172.21.15.112:0/3582987414 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b480033f0 con 0x7f3b480038e0 2023-01-24T03:39:13.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.310+0000 7f3b5bfff700 1 --2- 172.21.15.112:0/3582987414 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b48005040 0x7f3b4800b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:39:13.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.310+0000 7f3b5bfff700 1 -- 172.21.15.112:0/3582987414 reap_dead start 2023-01-24T03:39:13.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.311+0000 7f3b5b7fe700 1 --2- 172.21.15.112:0/3582987414 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b480038e0 0x7f3b48003cf0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c00e210 tx=0x7f3b4c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8d65deb0009214e server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:13.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.311+0000 7f3b5a7fc700 1 -- 172.21.15.112:0/3582987414 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c00eac0 con 0x7f3b480038e0 2023-01-24T03:39:13.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.311+0000 7f3b5a7fc700 1 -- 172.21.15.112:0/3582987414 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b4c00f040 con 0x7f3b480038e0 2023-01-24T03:39:13.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.311+0000 7f3b5a7fc700 1 -- 172.21.15.112:0/3582987414 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c00ee70 con 0x7f3b480038e0 2023-01-24T03:39:13.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.311+0000 7f3b61d65700 1 -- 172.21.15.112:0/3582987414 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b480038e0 msgr2=0x7f3b48003cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:13.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.311+0000 7f3b61d65700 1 --2- 172.21.15.112:0/3582987414 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b480038e0 0x7f3b48003cf0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c00e210 tx=0x7f3b4c00b040 comp rx=0 tx=0).stop 2023-01-24T03:39:13.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.311+0000 7f3b61d65700 1 -- 172.21.15.112:0/3582987414 shutdown_connections 2023-01-24T03:39:13.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.311+0000 7f3b61d65700 1 --2- 172.21.15.112:0/3582987414 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b480038e0 0x7f3b48003cf0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:13.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.311+0000 7f3b61d65700 1 -- 172.21.15.112:0/3582987414 >> 172.21.15.112:0/3582987414 conn(0x7f3b5c0c9820 msgr2=0x7f3b5c0c9c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:39:13.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.311+0000 7f3b61d65700 1 -- 172.21.15.112:0/3582987414 shutdown_connections 2023-01-24T03:39:13.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.311+0000 7f3b61d65700 1 -- 172.21.15.112:0/3582987414 wait complete. 2023-01-24T03:39:13.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.313+0000 7f3b61d65700 1 Processor -- start 2023-01-24T03:39:13.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.313+0000 7f3b61d65700 1 -- start start 2023-01-24T03:39:13.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.313+0000 7f3b61d65700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b480038e0 0x7f3b5c153d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:13.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.313+0000 7f3b61d65700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b5c0bee40 0x7f3b5c14a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:13.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.313+0000 7f3b61d65700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b5c168150 0x7f3b5c1685c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:13.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.313+0000 7f3b61d65700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5c0c22c0 con 0x7f3b5c0bee40 2023-01-24T03:39:13.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.313+0000 7f3b61d65700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5c0be6a0 con 0x7f3b5c168150 2023-01-24T03:39:13.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.313+0000 7f3b61d65700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5c159d90 con 0x7f3b480038e0 2023-01-24T03:39:13.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.313+0000 7f3b5affd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b5c0bee40 0x7f3b5c14a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:13.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.313+0000 7f3b5affd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b5c0bee40 0x7f3b5c14a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55022/0 (socket says 172.21.15.112:55022) 2023-01-24T03:39:13.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.313+0000 7f3b5affd700 1 -- 172.21.15.112:0/2098551848 learned_addr learned my addr 172.21.15.112:0/2098551848 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:39:13.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.314+0000 7f3b5b7fe700 1 --2- 172.21.15.112:0/2098551848 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b480038e0 0x7f3b5c153d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:13.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.314+0000 7f3b5bfff700 1 --2- 172.21.15.112:0/2098551848 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b5c168150 0x7f3b5c1685c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:13.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.314+0000 7f3b5affd700 1 -- 172.21.15.112:0/2098551848 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b480038e0 msgr2=0x7f3b5c153d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:13.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.314+0000 7f3b5affd700 1 --2- 172.21.15.112:0/2098551848 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b480038e0 0x7f3b5c153d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:13.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.314+0000 7f3b5affd700 1 -- 172.21.15.112:0/2098551848 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b5c168150 msgr2=0x7f3b5c1685c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:13.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.314+0000 7f3b5affd700 1 --2- 172.21.15.112:0/2098551848 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b5c168150 0x7f3b5c1685c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:13.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.314+0000 7f3b5affd700 1 -- 172.21.15.112:0/2098551848 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b5c0bff90 con 0x7f3b5c0bee40 2023-01-24T03:39:13.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.314+0000 7f3b5affd700 1 --2- 172.21.15.112:0/2098551848 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b5c0bee40 0x7f3b5c14a8d0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3b5000e510 tx=0x7f3b50009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:13.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.314+0000 7f3b437fe700 1 -- 172.21.15.112:0/2098551848 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b50012590 con 0x7f3b5c0bee40 2023-01-24T03:39:13.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.314+0000 7f3b437fe700 1 -- 172.21.15.112:0/2098551848 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b50012710 con 0x7f3b5c0bee40 2023-01-24T03:39:13.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.314+0000 7f3b61d65700 1 -- 172.21.15.112:0/2098551848 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b5c0c0270 con 0x7f3b5c0bee40 2023-01-24T03:39:13.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.314+0000 7f3b437fe700 1 -- 172.21.15.112:0/2098551848 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b50012ad0 con 0x7f3b5c0bee40 2023-01-24T03:39:13.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.314+0000 7f3b61d65700 1 -- 172.21.15.112:0/2098551848 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b5c0beb30 con 0x7f3b5c0bee40 2023-01-24T03:39:13.335 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.315+0000 7f3b61d65700 1 -- 172.21.15.112:0/2098551848 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b5c0c04b0 con 0x7f3b5c0bee40 2023-01-24T03:39:13.335 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.315+0000 7f3b437fe700 1 -- 172.21.15.112:0/2098551848 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f3b5001d460 con 0x7f3b5c0bee40 2023-01-24T03:39:13.335 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.315+0000 7f3b437fe700 1 --2- 172.21.15.112:0/2098551848 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3b3c0611c0 0x7f3b3c063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:13.335 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.316+0000 7f3b437fe700 1 -- 172.21.15.112:0/2098551848 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 7297+0+0 (secure 0 0 0) 0x7f3b5007dbf0 con 0x7f3b5c0bee40 2023-01-24T03:39:13.335 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.316+0000 7f3b5b7fe700 1 --2- 172.21.15.112:0/2098551848 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3b3c0611c0 0x7f3b3c063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:13.336 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.316+0000 7f3b5b7fe700 1 --2- 172.21.15.112:0/2098551848 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3b3c0611c0 0x7f3b3c063670 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c00afd0 tx=0x7f3b4c01a000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:13.336 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.320+0000 7f3b437fe700 1 -- 172.21.15.112:0/2098551848 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f3b5c0c04b0 con 0x7f3b5c0bee40 2023-01-24T03:39:13.542 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.540+0000 7f3b61d65700 1 -- 172.21.15.112:0/2098551848 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b5c0be850 con 0x7f3b5c0bee40 2023-01-24T03:39:13.543 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.541+0000 7f3b437fe700 1 -- 172.21.15.112:0/2098551848 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+6121 (secure 0 0 0) 0x7f3b5c0be850 con 0x7f3b5c0bee40 2023-01-24T03:39:13.543 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:39:13.544 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/2136880103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2136880103},{"type":"v1","addr":"172.21.15.142:6835","nonce":2136880103}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:37:57.754547+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5296":{"gid":5296,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":20,"addr":"172.21.15.142:6834/2838716685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2838716685},{"type":"v1","addr":"172.21.15.142:6834","nonce":2838716685}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:39:13.545 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.543+0000 7f3b61d65700 1 -- 172.21.15.112:0/2098551848 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3b3c0611c0 msgr2=0x7f3b3c063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:13.546 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.543+0000 7f3b61d65700 1 --2- 172.21.15.112:0/2098551848 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3b3c0611c0 0x7f3b3c063670 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c00afd0 tx=0x7f3b4c01a000 comp rx=0 tx=0).stop 2023-01-24T03:39:13.546 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.543+0000 7f3b61d65700 1 -- 172.21.15.112:0/2098551848 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b5c0bee40 msgr2=0x7f3b5c14a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:13.546 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.543+0000 7f3b61d65700 1 --2- 172.21.15.112:0/2098551848 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b5c0bee40 0x7f3b5c14a8d0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3b5000e510 tx=0x7f3b50009040 comp rx=0 tx=0).stop 2023-01-24T03:39:13.546 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.543+0000 7f3b61d65700 1 -- 172.21.15.112:0/2098551848 shutdown_connections 2023-01-24T03:39:13.546 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.543+0000 7f3b61d65700 1 --2- 172.21.15.112:0/2098551848 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f3b3c0611c0 0x7f3b3c063670 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:13.547 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.543+0000 7f3b61d65700 1 --2- 172.21.15.112:0/2098551848 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3b5c168150 0x7f3b5c1685c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:13.547 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.543+0000 7f3b61d65700 1 --2- 172.21.15.112:0/2098551848 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f3b480038e0 0x7f3b5c153d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:13.547 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.543+0000 7f3b61d65700 1 --2- 172.21.15.112:0/2098551848 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3b5c0bee40 0x7f3b5c14a8d0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:13.547 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.543+0000 7f3b61d65700 1 -- 172.21.15.112:0/2098551848 >> 172.21.15.112:0/2098551848 conn(0x7f3b5c0c9820 msgr2=0x7f3b5c14bdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:39:13.547 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.543+0000 7f3b61d65700 1 -- 172.21.15.112:0/2098551848 shutdown_connections 2023-01-24T03:39:13.548 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:13.543+0000 7f3b61d65700 1 -- 172.21.15.112:0/2098551848 wait complete. 2023-01-24T03:39:13.548 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 36 2023-01-24T03:39:13.557 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-01-24T03:39:15.558 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:39:15.682 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.680+0000 7fd96cfe5700 1 Processor -- start 2023-01-24T03:39:15.683 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.680+0000 7fd96cfe5700 1 -- start start 2023-01-24T03:39:15.685 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.681+0000 7fd96cfe5700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd9680680e0 0x7fd96805df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:15.685 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.681+0000 7fd96cfe5700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fd9680c3750 0x7fd96806d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:15.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.681+0000 7fd96cfe5700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd96805e5d0 con 0x7fd968068b10 2023-01-24T03:39:15.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.681+0000 7fd96cfe5700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd96806dc20 con 0x7fd9680c3750 2023-01-24T03:39:15.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.681+0000 7fd96cfe5700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd968155140 con 0x7fd9680680e0 2023-01-24T03:39:15.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.681+0000 7fd967fff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fd9680c3750 0x7fd96806d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:15.687 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.681+0000 7fd967fff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fd9680c3750 0x7fd96806d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55050/0 (socket says 172.21.15.112:55050) 2023-01-24T03:39:15.687 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.681+0000 7fd967fff700 1 -- 172.21.15.112:0/860740923 learned_addr learned my addr 172.21.15.112:0/860740923 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:39:15.687 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.681+0000 7fd9677fe700 1 --2- 172.21.15.112:0/860740923 >> v2:172.21.15.142:3300/0 conn(0x7fd9680680e0 0x7fd96805df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:15.687 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.681+0000 7fd967fff700 1 -- 172.21.15.112:0/860740923 >> v2:172.21.15.142:3300/0 conn(0x7fd9680680e0 msgr2=0x7fd96805df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:39:15.687 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.681+0000 7fd967fff700 1 --2- 172.21.15.112:0/860740923 >> v2:172.21.15.142:3300/0 conn(0x7fd9680680e0 0x7fd96805df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:15.688 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.681+0000 7fd967fff700 1 -- 172.21.15.112:0/860740923 >> v1:172.21.15.142:6789/0 conn(0x7fd968068b10 legacy=0x7fd9680c3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:39:15.688 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.681+0000 7fd967fff700 1 -- 172.21.15.112:0/860740923 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd968155420 con 0x7fd9680c3750 2023-01-24T03:39:15.688 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.681+0000 7fd967fff700 1 --2- 172.21.15.112:0/860740923 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd9680c3750 0x7fd96806d6e0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fd96806ddb0 tx=0x7fd960009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=564456e5e83a2f92 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:15.688 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.682+0000 7fd9667fc700 1 -- 172.21.15.112:0/860740923 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96000e070 con 0x7fd9680c3750 2023-01-24T03:39:15.688 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.682+0000 7fd9667fc700 1 -- 172.21.15.112:0/860740923 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd96000baa0 con 0x7fd9680c3750 2023-01-24T03:39:15.689 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.682+0000 7fd9667fc700 1 -- 172.21.15.112:0/860740923 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9600092e0 con 0x7fd9680c3750 2023-01-24T03:39:15.689 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.682+0000 7fd96cfe5700 1 -- 172.21.15.112:0/860740923 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd9680c3750 msgr2=0x7fd96806d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:15.689 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.682+0000 7fd96cfe5700 1 --2- 172.21.15.112:0/860740923 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd9680c3750 0x7fd96806d6e0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fd96806ddb0 tx=0x7fd960009840 comp rx=0 tx=0).stop 2023-01-24T03:39:15.689 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.682+0000 7fd96cfe5700 1 -- 172.21.15.112:0/860740923 shutdown_connections 2023-01-24T03:39:15.689 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.682+0000 7fd96cfe5700 1 --2- 172.21.15.112:0/860740923 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd9680c3750 0x7fd96806d6e0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:15.690 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.682+0000 7fd96cfe5700 1 --2- 172.21.15.112:0/860740923 >> v2:172.21.15.142:3300/0 conn(0x7fd9680680e0 0x7fd96805df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:15.690 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.682+0000 7fd96cfe5700 1 -- 172.21.15.112:0/860740923 >> 172.21.15.112:0/860740923 conn(0x7fd96806c320 msgr2=0x7fd96805d830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:39:15.690 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.682+0000 7fd96cfe5700 1 -- 172.21.15.112:0/860740923 shutdown_connections 2023-01-24T03:39:15.690 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.682+0000 7fd96cfe5700 1 -- 172.21.15.112:0/860740923 wait complete. 2023-01-24T03:39:15.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.683+0000 7fd96cfe5700 1 Processor -- start 2023-01-24T03:39:15.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.684+0000 7fd96cfe5700 1 -- start start 2023-01-24T03:39:15.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.684+0000 7fd96cfe5700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd9680680e0 0x7fd968151af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:15.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.684+0000 7fd96cfe5700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd968068b10 0x7fd9680c8700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:15.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.684+0000 7fd96cfe5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd968165110 0x7fd968150d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:15.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.684+0000 7fd96cfe5700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9680c9970 con 0x7fd9680680e0 2023-01-24T03:39:15.695 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.684+0000 7fd96cfe5700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96814f970 con 0x7fd968068b10 2023-01-24T03:39:15.695 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.684+0000 7fd96cfe5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd968166110 con 0x7fd968165110 2023-01-24T03:39:15.695 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.684+0000 7fd9677fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd9680680e0 0x7fd968151af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:15.695 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.684+0000 7fd9677fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd9680680e0 0x7fd968151af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55052/0 (socket says 172.21.15.112:55052) 2023-01-24T03:39:15.695 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.684+0000 7fd9677fe700 1 -- 172.21.15.112:0/1967596149 learned_addr learned my addr 172.21.15.112:0/1967596149 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:39:15.696 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.684+0000 7fd967fff700 1 --2- 172.21.15.112:0/1967596149 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd968165110 0x7fd968150d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:15.696 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.684+0000 7fd966ffd700 1 --2- 172.21.15.112:0/1967596149 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd968068b10 0x7fd9680c8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:15.696 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.685+0000 7fd9677fe700 1 -- 172.21.15.112:0/1967596149 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd968165110 msgr2=0x7fd968150d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:15.696 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.685+0000 7fd9677fe700 1 --2- 172.21.15.112:0/1967596149 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd968165110 0x7fd968150d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:15.696 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.685+0000 7fd9677fe700 1 -- 172.21.15.112:0/1967596149 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd968068b10 msgr2=0x7fd9680c8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:15.697 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.685+0000 7fd9677fe700 1 --2- 172.21.15.112:0/1967596149 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd968068b10 0x7fd9680c8700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:15.697 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.685+0000 7fd9677fe700 1 -- 172.21.15.112:0/1967596149 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd960009ae0 con 0x7fd9680680e0 2023-01-24T03:39:15.697 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.685+0000 7fd967fff700 1 --2- 172.21.15.112:0/1967596149 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd968165110 0x7fd968150d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:39:15.697 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.685+0000 7fd966ffd700 1 --2- 172.21.15.112:0/1967596149 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd968068b10 0x7fd9680c8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:39:15.697 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.685+0000 7fd9677fe700 1 --2- 172.21.15.112:0/1967596149 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd9680680e0 0x7fd968151af0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fd95800d8c0 tx=0x7fd958008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:15.698 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.685+0000 7fd94f7fe700 1 -- 172.21.15.112:0/1967596149 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd958009640 con 0x7fd9680680e0 2023-01-24T03:39:15.698 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.685+0000 7fd94f7fe700 1 -- 172.21.15.112:0/1967596149 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9580097a0 con 0x7fd9680680e0 2023-01-24T03:39:15.698 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.685+0000 7fd96cfe5700 1 -- 172.21.15.112:0/1967596149 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9680c8270 con 0x7fd9680680e0 2023-01-24T03:39:15.698 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.685+0000 7fd94f7fe700 1 -- 172.21.15.112:0/1967596149 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd958009b40 con 0x7fd9680680e0 2023-01-24T03:39:15.698 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.685+0000 7fd96cfe5700 1 -- 172.21.15.112:0/1967596149 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9680c7940 con 0x7fd9680680e0 2023-01-24T03:39:15.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.686+0000 7fd96cfe5700 1 -- 172.21.15.112:0/1967596149 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9680c8550 con 0x7fd9680680e0 2023-01-24T03:39:15.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.686+0000 7fd94f7fe700 1 -- 172.21.15.112:0/1967596149 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fd958024070 con 0x7fd9680680e0 2023-01-24T03:39:15.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.687+0000 7fd94f7fe700 1 --2- 172.21.15.112:0/1967596149 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd9480611c0 0x7fd948063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:15.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.687+0000 7fd966ffd700 1 --2- 172.21.15.112:0/1967596149 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd9480611c0 0x7fd948063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:15.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.687+0000 7fd94f7fe700 1 -- 172.21.15.112:0/1967596149 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 7297+0+0 (secure 0 0 0) 0x7fd95807e0c0 con 0x7fd9680680e0 2023-01-24T03:39:15.700 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.687+0000 7fd966ffd700 1 --2- 172.21.15.112:0/1967596149 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd9480611c0 0x7fd948063670 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd9680afb20 tx=0x7fd95c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:15.700 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.691+0000 7fd94f7fe700 1 -- 172.21.15.112:0/1967596149 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fd95804dd10 con 0x7fd9680680e0 2023-01-24T03:39:15.913 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.910+0000 7fd96cfe5700 1 -- 172.21.15.112:0/1967596149 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd96804bbf0 con 0x7fd9680680e0 2023-01-24T03:39:15.913 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.911+0000 7fd94f7fe700 1 -- 172.21.15.112:0/1967596149 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+6121 (secure 0 0 0) 0x7fd958051330 con 0x7fd9680680e0 2023-01-24T03:39:15.913 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:39:15.913 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/2136880103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2136880103},{"type":"v1","addr":"172.21.15.142:6835","nonce":2136880103}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:37:57.754547+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5296":{"gid":5296,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":20,"addr":"172.21.15.142:6834/2838716685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2838716685},{"type":"v1","addr":"172.21.15.142:6834","nonce":2838716685}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:39:15.915 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.913+0000 7fd96cfe5700 1 -- 172.21.15.112:0/1967596149 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd9480611c0 msgr2=0x7fd948063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:15.915 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.913+0000 7fd96cfe5700 1 --2- 172.21.15.112:0/1967596149 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd9480611c0 0x7fd948063670 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd9680afb20 tx=0x7fd95c009040 comp rx=0 tx=0).stop 2023-01-24T03:39:15.916 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.913+0000 7fd96cfe5700 1 -- 172.21.15.112:0/1967596149 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd9680680e0 msgr2=0x7fd968151af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:15.916 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.913+0000 7fd96cfe5700 1 --2- 172.21.15.112:0/1967596149 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd9680680e0 0x7fd968151af0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fd95800d8c0 tx=0x7fd958008040 comp rx=0 tx=0).stop 2023-01-24T03:39:15.916 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.913+0000 7fd96cfe5700 1 -- 172.21.15.112:0/1967596149 shutdown_connections 2023-01-24T03:39:15.916 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.913+0000 7fd96cfe5700 1 --2- 172.21.15.112:0/1967596149 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd9480611c0 0x7fd948063670 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:15.916 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.913+0000 7fd96cfe5700 1 --2- 172.21.15.112:0/1967596149 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd9680680e0 0x7fd968151af0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:15.917 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.913+0000 7fd96cfe5700 1 --2- 172.21.15.112:0/1967596149 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd968165110 0x7fd968150d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:15.917 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.913+0000 7fd96cfe5700 1 --2- 172.21.15.112:0/1967596149 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd968068b10 0x7fd9680c8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:15.917 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.913+0000 7fd96cfe5700 1 -- 172.21.15.112:0/1967596149 >> 172.21.15.112:0/1967596149 conn(0x7fd96806c320 msgr2=0x7fd96805d830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:39:15.917 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.913+0000 7fd96cfe5700 1 -- 172.21.15.112:0/1967596149 shutdown_connections 2023-01-24T03:39:15.918 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:15.913+0000 7fd96cfe5700 1 -- 172.21.15.112:0/1967596149 wait complete. 2023-01-24T03:39:15.918 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 36 2023-01-24T03:39:15.926 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-01-24T03:39:17.928 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:39:18.052 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.049+0000 7f09b5e55700 1 Processor -- start 2023-01-24T03:39:18.053 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.049+0000 7f09b5e55700 1 -- start start 2023-01-24T03:39:18.056 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.050+0000 7f09b5e55700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f09b00b0aa0 0x7f09b0155da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:18.056 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.050+0000 7f09b5e55700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f09b00ca3e0 0x7f09b00ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:18.057 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.050+0000 7f09b5e55700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09b00b0f40 con 0x7f09b00c7e40 2023-01-24T03:39:18.057 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.050+0000 7f09b5e55700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09b00b1170 con 0x7f09b00b0aa0 2023-01-24T03:39:18.057 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.050+0000 7f09b5e55700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09b00b1450 con 0x7f09b00ca3e0 2023-01-24T03:39:18.057 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.050+0000 7f09af7fe700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f09b00c7e40 0x7f09b00ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:36048/0 (socket says 172.21.15.112:36048) 2023-01-24T03:39:18.057 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.050+0000 7f09af7fe700 1 -- 172.21.15.112:0/3613123905 learned_addr learned my addr 172.21.15.112:0/3613123905 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:39:18.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.050+0000 7f09affff700 1 --2- 172.21.15.112:0/3613123905 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f09b00ca3e0 0x7f09b00ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:18.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.050+0000 7f09aeffd700 1 --2- 172.21.15.112:0/3613123905 >> v2:172.21.15.142:3300/0 conn(0x7f09b00b0aa0 0x7f09b0155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:18.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.051+0000 7f09ae7fc700 1 -- 172.21.15.112:0/3613123905 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3607871676 0 0) 0x7f09b00b0f40 con 0x7f09b00c7e40 2023-01-24T03:39:18.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.051+0000 7f09ae7fc700 1 -- 172.21.15.112:0/3613123905 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f099c0033f0 con 0x7f09b00c7e40 2023-01-24T03:39:18.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.051+0000 7f09ae7fc700 1 -- 172.21.15.112:0/3613123905 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2398158487 0 0) 0x7f099c0033f0 con 0x7f09b00c7e40 2023-01-24T03:39:18.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.051+0000 7f09ae7fc700 1 -- 172.21.15.112:0/3613123905 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f09b00ca3e0 msgr2=0x7f09b00ca7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:39:18.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.051+0000 7f09ae7fc700 1 --2- 172.21.15.112:0/3613123905 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f09b00ca3e0 0x7f09b00ca7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:18.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.051+0000 7f09ae7fc700 1 -- 172.21.15.112:0/3613123905 >> v2:172.21.15.142:3300/0 conn(0x7f09b00b0aa0 msgr2=0x7f09b0155da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:39:18.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.051+0000 7f09ae7fc700 1 --2- 172.21.15.112:0/3613123905 >> v2:172.21.15.142:3300/0 conn(0x7f09b00b0aa0 0x7f09b0155da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:18.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.051+0000 7f09ae7fc700 1 -- 172.21.15.112:0/3613123905 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b00b1730 con 0x7f09b00c7e40 2023-01-24T03:39:18.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.051+0000 7f09ae7fc700 1 -- 172.21.15.112:0/3613123905 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7f09a00017f0 con 0x7f09b00c7e40 2023-01-24T03:39:18.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.051+0000 7f09affff700 1 --2- 172.21.15.112:0/3613123905 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f09b00ca3e0 0x7f09b00ca7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:39:18.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.051+0000 7f09ae7fc700 1 -- 172.21.15.112:0/3613123905 >> v1:172.21.15.112:6789/0 conn(0x7f09b00c7e40 legacy=0x7f09b00ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:18.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.051+0000 7f09ae7fc700 1 --2- 172.21.15.112:0/3613123905 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f099c0038e0 0x7f099c003cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:18.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.051+0000 7f09ae7fc700 1 --2- 172.21.15.112:0/3613123905 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f09b00ca3e0 0x7f099c004690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:18.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.051+0000 7f09ae7fc700 1 --2- 172.21.15.112:0/3613123905 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f099c004bd0 0x7f099c005040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:18.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.051+0000 7f09ae7fc700 1 -- 172.21.15.112:0/3613123905 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b00b1170 con 0x7f099c0038e0 2023-01-24T03:39:18.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.051+0000 7f09ae7fc700 1 -- 172.21.15.112:0/3613123905 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b00b1450 con 0x7f099c004bd0 2023-01-24T03:39:18.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.051+0000 7f09ae7fc700 1 -- 172.21.15.112:0/3613123905 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f099c00b8e0 con 0x7f09b00ca3e0 2023-01-24T03:39:18.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.051+0000 7f09aeffd700 1 --2- 172.21.15.112:0/3613123905 >> v2:172.21.15.142:3300/0 conn(0x7f09b00b0aa0 0x7f09b0155da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:39:18.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.051+0000 7f09af7fe700 1 --2- 172.21.15.112:0/3613123905 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f099c0038e0 0x7f099c003cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:18.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.051+0000 7f09affff700 1 --2- 172.21.15.112:0/3613123905 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f099c004bd0 0x7f099c005040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:18.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.051+0000 7f09aeffd700 1 --2- 172.21.15.112:0/3613123905 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f09b00ca3e0 0x7f099c004690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:18.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.051+0000 7f09af7fe700 1 -- 172.21.15.112:0/3613123905 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f09b00ca3e0 msgr2=0x7f099c004690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:39:18.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.051+0000 7f09af7fe700 1 --2- 172.21.15.112:0/3613123905 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f09b00ca3e0 0x7f099c004690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:18.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.051+0000 7f09af7fe700 1 -- 172.21.15.112:0/3613123905 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f099c004bd0 msgr2=0x7f099c005040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:39:18.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.051+0000 7f09af7fe700 1 --2- 172.21.15.112:0/3613123905 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f099c004bd0 0x7f099c005040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:18.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.051+0000 7f09af7fe700 1 -- 172.21.15.112:0/3613123905 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a80092a0 con 0x7f099c0038e0 2023-01-24T03:39:18.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.051+0000 7f09af7fe700 1 --2- 172.21.15.112:0/3613123905 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f099c0038e0 0x7f099c003cf0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f09a000c560 tx=0x7f09a0002210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51cfed100c6b1124 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:18.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.051+0000 7f09ae7fc700 1 -- 172.21.15.112:0/3613123905 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a000cdf0 con 0x7f099c0038e0 2023-01-24T03:39:18.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.051+0000 7f09ae7fc700 1 -- 172.21.15.112:0/3613123905 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a0005b20 con 0x7f099c0038e0 2023-01-24T03:39:18.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.051+0000 7f09ae7fc700 1 -- 172.21.15.112:0/3613123905 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a00070d0 con 0x7f099c0038e0 2023-01-24T03:39:18.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.052+0000 7f09b5e55700 1 -- 172.21.15.112:0/3613123905 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f099c0038e0 msgr2=0x7f099c003cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:18.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.052+0000 7f09b5e55700 1 --2- 172.21.15.112:0/3613123905 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f099c0038e0 0x7f099c003cf0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f09a000c560 tx=0x7f09a0002210 comp rx=0 tx=0).stop 2023-01-24T03:39:18.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.052+0000 7f09affff700 1 -- 172.21.15.112:0/3613123905 reap_dead start 2023-01-24T03:39:18.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.052+0000 7f09b5e55700 1 -- 172.21.15.112:0/3613123905 shutdown_connections 2023-01-24T03:39:18.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.052+0000 7f09b5e55700 1 -- 172.21.15.112:0/3613123905 >> 172.21.15.112:0/3613123905 conn(0x7f09b005e2c0 msgr2=0x7f09b00cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:39:18.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.052+0000 7f09b5e55700 1 -- 172.21.15.112:0/3613123905 shutdown_connections 2023-01-24T03:39:18.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.052+0000 7f09b5e55700 1 -- 172.21.15.112:0/3613123905 wait complete. 2023-01-24T03:39:18.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.053+0000 7f09b5e55700 1 Processor -- start 2023-01-24T03:39:18.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.053+0000 7f09b5e55700 1 -- start start 2023-01-24T03:39:18.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.054+0000 7f09b5e55700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f099c0038e0 0x7f09b0166190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:18.069 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.054+0000 7f09b5e55700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09b00c22c0 0x7f09b00c0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:18.069 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.054+0000 7f09b5e55700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f09b0154d20 0x7f09b014bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:18.069 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.054+0000 7f09b5e55700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b00c4360 con 0x7f09b0154d20 2023-01-24T03:39:18.069 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.054+0000 7f09b5e55700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b00bfe90 con 0x7f09b00c22c0 2023-01-24T03:39:18.069 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.054+0000 7f09b5e55700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09b00c57d0 con 0x7f099c0038e0 2023-01-24T03:39:18.070 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.054+0000 7f09affff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f09b0154d20 0x7f09b014bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:18.070 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.054+0000 7f09affff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f09b0154d20 0x7f09b014bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55076/0 (socket says 172.21.15.112:55076) 2023-01-24T03:39:18.070 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.054+0000 7f09affff700 1 -- 172.21.15.112:0/73061244 learned_addr learned my addr 172.21.15.112:0/73061244 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:39:18.070 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.054+0000 7f09aeffd700 1 --2- 172.21.15.112:0/73061244 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09b00c22c0 0x7f09b00c0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:18.070 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.054+0000 7f09af7fe700 1 --2- 172.21.15.112:0/73061244 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f099c0038e0 0x7f09b0166190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:18.071 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.054+0000 7f09affff700 1 -- 172.21.15.112:0/73061244 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f099c0038e0 msgr2=0x7f09b0166190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:18.071 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.054+0000 7f09affff700 1 --2- 172.21.15.112:0/73061244 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f099c0038e0 0x7f09b0166190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:18.071 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.054+0000 7f09affff700 1 -- 172.21.15.112:0/73061244 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09b00c22c0 msgr2=0x7f09b00c0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:18.071 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.054+0000 7f09affff700 1 --2- 172.21.15.112:0/73061244 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09b00c22c0 0x7f09b00c0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:18.071 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.054+0000 7f09affff700 1 -- 172.21.15.112:0/73061244 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b00c5ab0 con 0x7f09b0154d20 2023-01-24T03:39:18.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.055+0000 7f09affff700 1 --2- 172.21.15.112:0/73061244 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f09b0154d20 0x7f09b014bcc0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f09a8006d80 tx=0x7f09a8009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:18.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.055+0000 7f09977fe700 1 -- 172.21.15.112:0/73061244 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a800f8e0 con 0x7f09b0154d20 2023-01-24T03:39:18.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.055+0000 7f09977fe700 1 -- 172.21.15.112:0/73061244 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a800fa60 con 0x7f09b0154d20 2023-01-24T03:39:18.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.055+0000 7f09b5e55700 1 -- 172.21.15.112:0/73061244 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09b00bf840 con 0x7f09b0154d20 2023-01-24T03:39:18.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.055+0000 7f09977fe700 1 -- 172.21.15.112:0/73061244 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a800fe20 con 0x7f09b0154d20 2023-01-24T03:39:18.073 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.055+0000 7f09b5e55700 1 -- 172.21.15.112:0/73061244 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09b00c54d0 con 0x7f09b0154d20 2023-01-24T03:39:18.073 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.056+0000 7f09b5e55700 1 -- 172.21.15.112:0/73061244 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09b00bfce0 con 0x7f09b0154d20 2023-01-24T03:39:18.073 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.056+0000 7f09977fe700 1 -- 172.21.15.112:0/73061244 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f09a8024070 con 0x7f09b0154d20 2023-01-24T03:39:18.073 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.056+0000 7f09977fe700 1 --2- 172.21.15.112:0/73061244 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f09900611c0 0x7f0990063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:18.073 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.056+0000 7f09977fe700 1 -- 172.21.15.112:0/73061244 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 7297+0+0 (secure 0 0 0) 0x7f09a807dfc0 con 0x7f09b0154d20 2023-01-24T03:39:18.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.057+0000 7f09af7fe700 1 --2- 172.21.15.112:0/73061244 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f09900611c0 0x7f0990063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:18.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.057+0000 7f09af7fe700 1 --2- 172.21.15.112:0/73061244 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f09900611c0 0x7f0990063670 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f09a0002250 tx=0x7f09a0015000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:18.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.061+0000 7f09977fe700 1 -- 172.21.15.112:0/73061244 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f09a804dc90 con 0x7f09b0154d20 2023-01-24T03:39:18.286 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.283+0000 7f09b5e55700 1 -- 172.21.15.112:0/73061244 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09b0157880 con 0x7f09b0154d20 2023-01-24T03:39:18.286 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.283+0000 7f09977fe700 1 -- 172.21.15.112:0/73061244 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+6121 (secure 0 0 0) 0x7f09a80512b0 con 0x7f09b0154d20 2023-01-24T03:39:18.287 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:39:18.287 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/2136880103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2136880103},{"type":"v1","addr":"172.21.15.142:6835","nonce":2136880103}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:37:57.754547+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5296":{"gid":5296,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":20,"addr":"172.21.15.142:6834/2838716685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2838716685},{"type":"v1","addr":"172.21.15.142:6834","nonce":2838716685}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:39:18.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.285+0000 7f09b5e55700 1 -- 172.21.15.112:0/73061244 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f09900611c0 msgr2=0x7f0990063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:18.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.285+0000 7f09b5e55700 1 --2- 172.21.15.112:0/73061244 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f09900611c0 0x7f0990063670 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f09a0002250 tx=0x7f09a0015000 comp rx=0 tx=0).stop 2023-01-24T03:39:18.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.285+0000 7f09b5e55700 1 -- 172.21.15.112:0/73061244 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f09b0154d20 msgr2=0x7f09b014bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:18.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.285+0000 7f09b5e55700 1 --2- 172.21.15.112:0/73061244 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f09b0154d20 0x7f09b014bcc0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f09a8006d80 tx=0x7f09a8009060 comp rx=0 tx=0).stop 2023-01-24T03:39:18.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.285+0000 7f09b5e55700 1 -- 172.21.15.112:0/73061244 shutdown_connections 2023-01-24T03:39:18.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.285+0000 7f09b5e55700 1 --2- 172.21.15.112:0/73061244 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f09900611c0 0x7f0990063670 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:18.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.285+0000 7f09b5e55700 1 --2- 172.21.15.112:0/73061244 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f09b0154d20 0x7f09b014bcc0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:18.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.285+0000 7f09b5e55700 1 --2- 172.21.15.112:0/73061244 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f099c0038e0 0x7f09b0166190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:18.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.285+0000 7f09b5e55700 1 --2- 172.21.15.112:0/73061244 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f09b00c22c0 0x7f09b00c0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:18.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.285+0000 7f09b5e55700 1 -- 172.21.15.112:0/73061244 >> 172.21.15.112:0/73061244 conn(0x7f09b005e2c0 msgr2=0x7f09b0157e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:39:18.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.285+0000 7f09b5e55700 1 -- 172.21.15.112:0/73061244 shutdown_connections 2023-01-24T03:39:18.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:18.285+0000 7f09b5e55700 1 -- 172.21.15.112:0/73061244 wait complete. 2023-01-24T03:39:18.291 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 36 2023-01-24T03:39:18.300 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-01-24T03:39:20.301 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:39:20.652 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.650+0000 7ff5fc225700 1 Processor -- start 2023-01-24T03:39:20.653 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.650+0000 7ff5fc225700 1 -- start start 2023-01-24T03:39:20.654 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.651+0000 7ff5fc225700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff5f4153a40 0x7ff5f40ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:20.654 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.651+0000 7ff5fc225700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7ff5f40cb600 0x7ff5f415d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:20.654 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.651+0000 7ff5fc225700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7ff5f40b0aa0 0x7ff5f40b0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:20.655 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.651+0000 7ff5fc225700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5f40b1400 con 0x7ff5f40cb600 2023-01-24T03:39:20.655 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.651+0000 7ff5fc225700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5f415dc50 con 0x7ff5f40b0aa0 2023-01-24T03:39:20.655 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.651+0000 7ff5fc225700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f40c69b0 con 0x7ff5f4153a40 2023-01-24T03:39:20.656 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.652+0000 7ff5f97c0700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7ff5f40cb600 0x7ff5f415d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:20.656 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.652+0000 7ff5f97c0700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7ff5f40cb600 0x7ff5f415d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55176/0 (socket says 172.21.15.112:55176) 2023-01-24T03:39:20.656 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.652+0000 7ff5f97c0700 1 -- 172.21.15.112:0/1603484733 learned_addr learned my addr 172.21.15.112:0/1603484733 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:39:20.656 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.652+0000 7ff5f97c0700 1 -- 172.21.15.112:0/1603484733 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff5f4153a40 msgr2=0x7ff5f40ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:39:20.657 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.652+0000 7ff5f97c0700 1 --2- 172.21.15.112:0/1603484733 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff5f4153a40 0x7ff5f40ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:20.657 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.652+0000 7ff5f97c0700 1 -- 172.21.15.112:0/1603484733 >> v2:172.21.15.142:3300/0 conn(0x7ff5f40b0aa0 msgr2=0x7ff5f40b0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:39:20.657 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.652+0000 7ff5fa7c2700 1 --2- 172.21.15.112:0/1603484733 >> v2:172.21.15.142:3300/0 conn(0x7ff5f40b0aa0 0x7ff5f40b0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:20.658 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.654+0000 7ff5f97c0700 1 --2- 172.21.15.112:0/1603484733 >> v2:172.21.15.142:3300/0 conn(0x7ff5f40b0aa0 0x7ff5f40b0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:20.658 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.654+0000 7ff5f97c0700 1 -- 172.21.15.112:0/1603484733 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f40c7250 con 0x7ff5f40cb600 2023-01-24T03:39:20.658 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.654+0000 7ff5fa7c2700 1 --2- 172.21.15.112:0/1603484733 >> v2:172.21.15.142:3300/0 conn(0x7ff5f40b0aa0 0x7ff5f40b0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-01-24T03:39:20.658 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.655+0000 7ff5f97c0700 1 --2- 172.21.15.112:0/1603484733 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff5f40cb600 0x7ff5f415d710 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4063ef0 tx=0x7ff5e00090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8ef248134d2515b server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:20.659 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.656+0000 7ff5f8fbf700 1 -- 172.21.15.112:0/1603484733 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e0011630 con 0x7ff5f40cb600 2023-01-24T03:39:20.659 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.656+0000 7ff5f8fbf700 1 -- 172.21.15.112:0/1603484733 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5e00117d0 con 0x7ff5f40cb600 2023-01-24T03:39:20.659 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.656+0000 7ff5f8fbf700 1 -- 172.21.15.112:0/1603484733 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e0011b90 con 0x7ff5f40cb600 2023-01-24T03:39:20.660 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.657+0000 7ff5fc225700 1 -- 172.21.15.112:0/1603484733 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff5f40cb600 msgr2=0x7ff5f415d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:20.660 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.657+0000 7ff5fc225700 1 --2- 172.21.15.112:0/1603484733 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff5f40cb600 0x7ff5f415d710 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4063ef0 tx=0x7ff5e00090f0 comp rx=0 tx=0).stop 2023-01-24T03:39:20.661 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.657+0000 7ff5fc225700 1 -- 172.21.15.112:0/1603484733 shutdown_connections 2023-01-24T03:39:20.661 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.657+0000 7ff5fc225700 1 --2- 172.21.15.112:0/1603484733 >> v2:172.21.15.142:3300/0 conn(0x7ff5f40b0aa0 0x7ff5f40b0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:20.661 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.657+0000 7ff5fc225700 1 --2- 172.21.15.112:0/1603484733 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff5f4153a40 0x7ff5f40ae9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:20.661 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.657+0000 7ff5fc225700 1 --2- 172.21.15.112:0/1603484733 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff5f40cb600 0x7ff5f415d710 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:20.661 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.657+0000 7ff5fc225700 1 -- 172.21.15.112:0/1603484733 >> 172.21.15.112:0/1603484733 conn(0x7ff5f405e2c0 msgr2=0x7ff5f40cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:39:20.662 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.657+0000 7ff5fc225700 1 -- 172.21.15.112:0/1603484733 shutdown_connections 2023-01-24T03:39:20.662 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.657+0000 7ff5fc225700 1 -- 172.21.15.112:0/1603484733 wait complete. 2023-01-24T03:39:20.662 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.659+0000 7ff5fc225700 1 Processor -- start 2023-01-24T03:39:20.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.659+0000 7ff5fc225700 1 -- start start 2023-01-24T03:39:20.664 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.660+0000 7ff5fc225700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff5f40b0aa0 0x7ff5f416a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:20.664 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.660+0000 7ff5fc225700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff5f4153a40 0x7ff5f40c4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:20.664 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.660+0000 7ff5fc225700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff5f4155f60 0x7ff5f40c30d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:20.664 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.660+0000 7ff5fc225700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f416aaf0 con 0x7ff5f4153a40 2023-01-24T03:39:20.665 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.660+0000 7ff5fc225700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f40c4200 con 0x7ff5f4155f60 2023-01-24T03:39:20.665 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.660+0000 7ff5fc225700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f4156910 con 0x7ff5f40b0aa0 2023-01-24T03:39:20.665 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.660+0000 7ff5fa7c2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff5f4155f60 0x7ff5f40c30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:20.665 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.660+0000 7ff5f9fc1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff5f40b0aa0 0x7ff5f416a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:20.665 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.660+0000 7ff5f9fc1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff5f40b0aa0 0x7ff5f416a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.112:42484/0 (socket says 172.21.15.112:42484) 2023-01-24T03:39:20.666 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.660+0000 7ff5f9fc1700 1 -- 172.21.15.112:0/2373672518 learned_addr learned my addr 172.21.15.112:0/2373672518 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:39:20.666 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.661+0000 7ff5f9fc1700 1 -- 172.21.15.112:0/2373672518 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff5f4155f60 msgr2=0x7ff5f40c30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:20.666 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.661+0000 7ff5f9fc1700 1 --2- 172.21.15.112:0/2373672518 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff5f4155f60 0x7ff5f40c30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:20.666 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.661+0000 7ff5f9fc1700 1 -- 172.21.15.112:0/2373672518 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff5f4153a40 msgr2=0x7ff5f40c4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:20.666 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.661+0000 7ff5f9fc1700 1 --2- 172.21.15.112:0/2373672518 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff5f4153a40 0x7ff5f40c4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:20.667 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.661+0000 7ff5f9fc1700 1 -- 172.21.15.112:0/2373672518 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e0009370 con 0x7ff5f40b0aa0 2023-01-24T03:39:20.667 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.661+0000 7ff5f9fc1700 1 --2- 172.21.15.112:0/2373672518 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff5f40b0aa0 0x7ff5f416a5b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff5e800bc30 tx=0x7ff5e8009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:20.670 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.667+0000 7ff5f1ffb700 1 -- 172.21.15.112:0/2373672518 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e80092e0 con 0x7ff5f40b0aa0 2023-01-24T03:39:20.670 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.667+0000 7ff5f1ffb700 1 -- 172.21.15.112:0/2373672518 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5e800c7a0 con 0x7ff5f40b0aa0 2023-01-24T03:39:20.670 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.667+0000 7ff5f1ffb700 1 -- 172.21.15.112:0/2373672518 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e800cb60 con 0x7ff5f40b0aa0 2023-01-24T03:39:20.671 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.667+0000 7ff5fc225700 1 -- 172.21.15.112:0/2373672518 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5f41540b0 con 0x7ff5f40b0aa0 2023-01-24T03:39:20.674 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.668+0000 7ff5fc225700 1 -- 172.21.15.112:0/2373672518 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5f4156e70 con 0x7ff5f40b0aa0 2023-01-24T03:39:20.674 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.669+0000 7ff5fc225700 1 -- 172.21.15.112:0/2373672518 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5f41566b0 con 0x7ff5f40b0aa0 2023-01-24T03:39:20.675 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.669+0000 7ff5f1ffb700 1 -- 172.21.15.112:0/2373672518 <== mon.2 v2:172.21.15.142:3301/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7ff5e8015470 con 0x7ff5f40b0aa0 2023-01-24T03:39:20.675 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.670+0000 7ff5f1ffb700 1 --2- 172.21.15.112:0/2373672518 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff5dc061290 0x7ff5dc063740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:20.675 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.670+0000 7ff5f1ffb700 1 -- 172.21.15.112:0/2373672518 <== mon.2 v2:172.21.15.142:3301/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 7297+0+0 (secure 0 0 0) 0x7ff5e807d030 con 0x7ff5f40b0aa0 2023-01-24T03:39:20.675 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.670+0000 7ff5f97c0700 1 --2- 172.21.15.112:0/2373672518 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff5dc061290 0x7ff5dc063740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:20.675 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.671+0000 7ff5f97c0700 1 --2- 172.21.15.112:0/2373672518 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff5dc061290 0x7ff5dc063740 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0009340 tx=0x7ff5e0016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:20.679 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.676+0000 7ff5f1ffb700 1 -- 172.21.15.112:0/2373672518 <== mon.2 v2:172.21.15.142:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7ff5e804cc50 con 0x7ff5f40b0aa0 2023-01-24T03:39:20.927 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.924+0000 7ff5fc225700 1 -- 172.21.15.112:0/2373672518 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5f404bb80 con 0x7ff5f40b0aa0 2023-01-24T03:39:20.928 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.925+0000 7ff5f1ffb700 1 -- 172.21.15.112:0/2373672518 <== mon.2 v2:172.21.15.142:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+6121 (secure 0 0 0) 0x7ff5e8050270 con 0x7ff5f40b0aa0 2023-01-24T03:39:20.928 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:39:20.929 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/2136880103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2136880103},{"type":"v1","addr":"172.21.15.142:6835","nonce":2136880103}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:37:57.754547+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5296":{"gid":5296,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":20,"addr":"172.21.15.142:6834/2838716685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2838716685},{"type":"v1","addr":"172.21.15.142:6834","nonce":2838716685}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:39:20.930 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.927+0000 7ff5fc225700 1 -- 172.21.15.112:0/2373672518 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff5dc061290 msgr2=0x7ff5dc063740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:20.931 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.927+0000 7ff5fc225700 1 --2- 172.21.15.112:0/2373672518 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff5dc061290 0x7ff5dc063740 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0009340 tx=0x7ff5e0016000 comp rx=0 tx=0).stop 2023-01-24T03:39:20.931 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.927+0000 7ff5fc225700 1 -- 172.21.15.112:0/2373672518 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff5f40b0aa0 msgr2=0x7ff5f416a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:20.931 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.927+0000 7ff5fc225700 1 --2- 172.21.15.112:0/2373672518 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff5f40b0aa0 0x7ff5f416a5b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff5e800bc30 tx=0x7ff5e8009840 comp rx=0 tx=0).stop 2023-01-24T03:39:20.931 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.927+0000 7ff5fc225700 1 -- 172.21.15.112:0/2373672518 shutdown_connections 2023-01-24T03:39:20.931 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.927+0000 7ff5fc225700 1 --2- 172.21.15.112:0/2373672518 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff5dc061290 0x7ff5dc063740 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:20.932 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.927+0000 7ff5fc225700 1 --2- 172.21.15.112:0/2373672518 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff5f4155f60 0x7ff5f40c30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:20.932 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.927+0000 7ff5fc225700 1 --2- 172.21.15.112:0/2373672518 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff5f40b0aa0 0x7ff5f416a5b0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:20.932 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.927+0000 7ff5fc225700 1 --2- 172.21.15.112:0/2373672518 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff5f4153a40 0x7ff5f40c4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:20.932 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.927+0000 7ff5fc225700 1 -- 172.21.15.112:0/2373672518 >> 172.21.15.112:0/2373672518 conn(0x7ff5f405e2c0 msgr2=0x7ff5f40cb9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:39:20.932 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.927+0000 7ff5fc225700 1 -- 172.21.15.112:0/2373672518 shutdown_connections 2023-01-24T03:39:20.933 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:20.927+0000 7ff5fc225700 1 -- 172.21.15.112:0/2373672518 wait complete. 2023-01-24T03:39:20.933 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 36 2023-01-24T03:39:20.942 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-01-24T03:39:22.943 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:39:23.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.064+0000 7fbd649ee700 1 Processor -- start 2023-01-24T03:39:23.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.064+0000 7fbd649ee700 1 -- start start 2023-01-24T03:39:23.070 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.065+0000 7fbd649ee700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd600b0aa0 0x7fbd60155da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:23.071 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.065+0000 7fbd649ee700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd600b0f40 con 0x7fbd600ca3e0 2023-01-24T03:39:23.071 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.065+0000 7fbd649ee700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd600b1170 con 0x7fbd600c7e40 2023-01-24T03:39:23.071 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.065+0000 7fbd649ee700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd600b1450 con 0x7fbd600b0aa0 2023-01-24T03:39:23.071 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.065+0000 7fbd5ed9d700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fbd600ca3e0 0x7fbd600ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:36192/0 (socket says 172.21.15.112:36192) 2023-01-24T03:39:23.071 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.066+0000 7fbd5ed9d700 1 -- 172.21.15.112:0/1070060488 learned_addr learned my addr 172.21.15.112:0/1070060488 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:39:23.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.066+0000 7fbd5dd9b700 1 --2- 172.21.15.112:0/1070060488 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd600b0aa0 0x7fbd60155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:23.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.066+0000 7fbd5d59a700 1 -- 172.21.15.112:0/1070060488 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4048748571 0 0) 0x7fbd600b0f40 con 0x7fbd600ca3e0 2023-01-24T03:39:23.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.066+0000 7fbd5d59a700 1 -- 172.21.15.112:0/1070060488 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd400033f0 con 0x7fbd600ca3e0 2023-01-24T03:39:23.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.066+0000 7fbd5d59a700 1 -- 172.21.15.112:0/1070060488 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2477985402 0 0) 0x7fbd600b1170 con 0x7fbd600c7e40 2023-01-24T03:39:23.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.066+0000 7fbd5d59a700 1 -- 172.21.15.112:0/1070060488 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd600b0f40 con 0x7fbd600c7e40 2023-01-24T03:39:23.073 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.066+0000 7fbd5d59a700 1 -- 172.21.15.112:0/1070060488 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2554784289 0 0) 0x7fbd400033f0 con 0x7fbd600ca3e0 2023-01-24T03:39:23.073 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.066+0000 7fbd5dd9b700 1 -- 172.21.15.112:0/1070060488 >> v1:172.21.15.142:6789/0 conn(0x7fbd600c7e40 legacy=0x7fbd600ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:23.073 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.066+0000 7fbd5dd9b700 1 -- 172.21.15.112:0/1070060488 >> v1:172.21.15.112:6789/0 conn(0x7fbd600ca3e0 legacy=0x7fbd600ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:23.073 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.066+0000 7fbd5dd9b700 1 -- 172.21.15.112:0/1070060488 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd600b1730 con 0x7fbd600b0aa0 2023-01-24T03:39:23.073 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.066+0000 7fbd5dd9b700 1 --2- 172.21.15.112:0/1070060488 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd600b0aa0 0x7fbd60155da0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbd48009790 tx=0x7fbd48009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=754834f159f8a2db server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:23.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.066+0000 7fbd5d59a700 1 -- 172.21.15.112:0/1070060488 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd58001470 con 0x7fbd600b0aa0 2023-01-24T03:39:23.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.066+0000 7fbd5d59a700 1 -- 172.21.15.112:0/1070060488 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd48011440 con 0x7fbd600b0aa0 2023-01-24T03:39:23.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.066+0000 7fbd5d59a700 1 -- 172.21.15.112:0/1070060488 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd480117c0 con 0x7fbd600b0aa0 2023-01-24T03:39:23.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.067+0000 7fbd649ee700 1 -- 172.21.15.112:0/1070060488 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd600b0aa0 msgr2=0x7fbd60155da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:23.075 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.067+0000 7fbd649ee700 1 --2- 172.21.15.112:0/1070060488 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd600b0aa0 0x7fbd60155da0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbd48009790 tx=0x7fbd48009060 comp rx=0 tx=0).stop 2023-01-24T03:39:23.075 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.067+0000 7fbd649ee700 1 -- 172.21.15.112:0/1070060488 shutdown_connections 2023-01-24T03:39:23.075 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.067+0000 7fbd649ee700 1 --2- 172.21.15.112:0/1070060488 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd600b0aa0 0x7fbd60155da0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:23.075 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.067+0000 7fbd649ee700 1 -- 172.21.15.112:0/1070060488 >> 172.21.15.112:0/1070060488 conn(0x7fbd6005e2c0 msgr2=0x7fbd600cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:39:23.076 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.067+0000 7fbd649ee700 1 -- 172.21.15.112:0/1070060488 shutdown_connections 2023-01-24T03:39:23.076 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.067+0000 7fbd649ee700 1 -- 172.21.15.112:0/1070060488 wait complete. 2023-01-24T03:39:23.078 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.068+0000 7fbd649ee700 1 Processor -- start 2023-01-24T03:39:23.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.068+0000 7fbd649ee700 1 -- start start 2023-01-24T03:39:23.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.069+0000 7fbd649ee700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd600b0aa0 0x7fbd60152190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:23.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.069+0000 7fbd649ee700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd600c7e40 0x7fbd6014ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:23.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.069+0000 7fbd649ee700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd600ca3e0 0x7fbd601504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:23.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.069+0000 7fbd649ee700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd60152bc0 con 0x7fbd600b0aa0 2023-01-24T03:39:23.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.069+0000 7fbd649ee700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6014fa80 con 0x7fbd600ca3e0 2023-01-24T03:39:23.080 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.069+0000 7fbd649ee700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd601657d0 con 0x7fbd600c7e40 2023-01-24T03:39:23.080 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.069+0000 7fbd5e59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd600b0aa0 0x7fbd60152190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:23.080 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.069+0000 7fbd5e59c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd600b0aa0 0x7fbd60152190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55206/0 (socket says 172.21.15.112:55206) 2023-01-24T03:39:23.080 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.069+0000 7fbd5e59c700 1 -- 172.21.15.112:0/4253188655 learned_addr learned my addr 172.21.15.112:0/4253188655 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:39:23.080 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.069+0000 7fbd5dd9b700 1 --2- 172.21.15.112:0/4253188655 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd600c7e40 0x7fbd6014ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:23.081 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.069+0000 7fbd5ed9d700 1 --2- 172.21.15.112:0/4253188655 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd600ca3e0 0x7fbd601504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:23.081 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.069+0000 7fbd5e59c700 1 -- 172.21.15.112:0/4253188655 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd600c7e40 msgr2=0x7fbd6014ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:23.081 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.069+0000 7fbd5e59c700 1 --2- 172.21.15.112:0/4253188655 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd600c7e40 0x7fbd6014ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:23.081 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.069+0000 7fbd5e59c700 1 -- 172.21.15.112:0/4253188655 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd600ca3e0 msgr2=0x7fbd601504a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:23.081 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.069+0000 7fbd5e59c700 1 --2- 172.21.15.112:0/4253188655 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd600ca3e0 0x7fbd601504a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:23.082 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.069+0000 7fbd5e59c700 1 -- 172.21.15.112:0/4253188655 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd480092a0 con 0x7fbd600b0aa0 2023-01-24T03:39:23.082 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.069+0000 7fbd5e59c700 1 --2- 172.21.15.112:0/4253188655 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd600b0aa0 0x7fbd60152190 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fbd5000c810 tx=0x7fbd500026f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:23.082 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.069+0000 7fbd567fc700 1 -- 172.21.15.112:0/4253188655 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd50012630 con 0x7fbd600b0aa0 2023-01-24T03:39:23.082 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.069+0000 7fbd567fc700 1 -- 172.21.15.112:0/4253188655 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd500127b0 con 0x7fbd600b0aa0 2023-01-24T03:39:23.082 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.069+0000 7fbd649ee700 1 -- 172.21.15.112:0/4253188655 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd60165ab0 con 0x7fbd600b0aa0 2023-01-24T03:39:23.083 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.070+0000 7fbd567fc700 1 -- 172.21.15.112:0/4253188655 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd50012b70 con 0x7fbd600b0aa0 2023-01-24T03:39:23.083 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.070+0000 7fbd649ee700 1 -- 172.21.15.112:0/4253188655 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd60166070 con 0x7fbd600b0aa0 2023-01-24T03:39:23.083 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.070+0000 7fbd649ee700 1 -- 172.21.15.112:0/4253188655 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd600c0800 con 0x7fbd600b0aa0 2023-01-24T03:39:23.083 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.071+0000 7fbd567fc700 1 -- 172.21.15.112:0/4253188655 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fbd50022070 con 0x7fbd600b0aa0 2023-01-24T03:39:23.083 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.071+0000 7fbd567fc700 1 --2- 172.21.15.112:0/4253188655 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fbd440611c0 0x7fbd44063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:23.083 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.071+0000 7fbd567fc700 1 -- 172.21.15.112:0/4253188655 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 7297+0+0 (secure 0 0 0) 0x7fbd5007dc20 con 0x7fbd600b0aa0 2023-01-24T03:39:23.084 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.071+0000 7fbd5dd9b700 1 --2- 172.21.15.112:0/4253188655 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fbd440611c0 0x7fbd44063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:23.084 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.072+0000 7fbd5dd9b700 1 --2- 172.21.15.112:0/4253188655 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fbd440611c0 0x7fbd44063670 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fbd480030a0 tx=0x7fbd48015000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:23.084 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.075+0000 7fbd567fc700 1 -- 172.21.15.112:0/4253188655 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fbd5004d780 con 0x7fbd600b0aa0 2023-01-24T03:39:23.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.297+0000 7fbd649ee700 1 -- 172.21.15.112:0/4253188655 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd600c40c0 con 0x7fbd600b0aa0 2023-01-24T03:39:23.301 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.297+0000 7fbd567fc700 1 -- 172.21.15.112:0/4253188655 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+6121 (secure 0 0 0) 0x7fbd50050da0 con 0x7fbd600b0aa0 2023-01-24T03:39:23.301 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:39:23.301 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/2136880103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2136880103},{"type":"v1","addr":"172.21.15.142:6835","nonce":2136880103}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:37:57.754547+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5296":{"gid":5296,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":20,"addr":"172.21.15.142:6834/2838716685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2838716685},{"type":"v1","addr":"172.21.15.142:6834","nonce":2838716685}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:39:23.303 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.299+0000 7fbd649ee700 1 -- 172.21.15.112:0/4253188655 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fbd440611c0 msgr2=0x7fbd44063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:23.303 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.299+0000 7fbd649ee700 1 --2- 172.21.15.112:0/4253188655 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fbd440611c0 0x7fbd44063670 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fbd480030a0 tx=0x7fbd48015000 comp rx=0 tx=0).stop 2023-01-24T03:39:23.303 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.299+0000 7fbd649ee700 1 -- 172.21.15.112:0/4253188655 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd600b0aa0 msgr2=0x7fbd60152190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:23.303 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.299+0000 7fbd649ee700 1 --2- 172.21.15.112:0/4253188655 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd600b0aa0 0x7fbd60152190 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fbd5000c810 tx=0x7fbd500026f0 comp rx=0 tx=0).stop 2023-01-24T03:39:23.304 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.299+0000 7fbd649ee700 1 -- 172.21.15.112:0/4253188655 shutdown_connections 2023-01-24T03:39:23.304 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.299+0000 7fbd649ee700 1 --2- 172.21.15.112:0/4253188655 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fbd440611c0 0x7fbd44063670 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:23.304 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.299+0000 7fbd649ee700 1 --2- 172.21.15.112:0/4253188655 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbd600b0aa0 0x7fbd60152190 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:23.304 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.299+0000 7fbd649ee700 1 --2- 172.21.15.112:0/4253188655 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbd600ca3e0 0x7fbd601504a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:23.305 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.299+0000 7fbd649ee700 1 --2- 172.21.15.112:0/4253188655 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fbd600c7e40 0x7fbd6014ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:23.305 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.299+0000 7fbd649ee700 1 -- 172.21.15.112:0/4253188655 >> 172.21.15.112:0/4253188655 conn(0x7fbd6005e2c0 msgr2=0x7fbd601551d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:39:23.305 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.299+0000 7fbd649ee700 1 -- 172.21.15.112:0/4253188655 shutdown_connections 2023-01-24T03:39:23.305 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:23.299+0000 7fbd649ee700 1 -- 172.21.15.112:0/4253188655 wait complete. 2023-01-24T03:39:23.306 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 36 2023-01-24T03:39:23.314 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-01-24T03:39:25.315 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:39:25.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.429+0000 7f66efbf2700 1 Processor -- start 2023-01-24T03:39:25.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.429+0000 7f66efbf2700 1 -- start start 2023-01-24T03:39:25.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.429+0000 7f66efbf2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66e80c7e40 0x7f66e80b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:25.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.430+0000 7f66efbf2700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f66e8155da0 0x7f66e80ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:25.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.430+0000 7f66efbf2700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66e8156210 con 0x7f66e80aef20 2023-01-24T03:39:25.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.430+0000 7f66efbf2700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66e80b1170 con 0x7f66e8155da0 2023-01-24T03:39:25.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.430+0000 7f66efbf2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66e80b1450 con 0x7f66e80c7e40 2023-01-24T03:39:25.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.430+0000 7f66ed18d700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f66e8155da0 0x7f66e80ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:25.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.430+0000 7f66ed18d700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f66e8155da0 0x7f66e80ae9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55222/0 (socket says 172.21.15.112:55222) 2023-01-24T03:39:25.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.430+0000 7f66ed18d700 1 -- 172.21.15.112:0/1016419979 learned_addr learned my addr 172.21.15.112:0/1016419979 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:39:25.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.430+0000 7f66ed98e700 1 --2- 172.21.15.112:0/1016419979 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66e80c7e40 0x7f66e80b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:25.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.430+0000 7f66ed18d700 1 -- 172.21.15.112:0/1016419979 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66e80c7e40 msgr2=0x7f66e80b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:39:25.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.430+0000 7f66ed18d700 1 --2- 172.21.15.112:0/1016419979 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66e80c7e40 0x7f66e80b0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:25.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.430+0000 7f66ed18d700 1 -- 172.21.15.112:0/1016419979 >> v1:172.21.15.142:6789/0 conn(0x7f66e80aef20 legacy=0x7f66e80ca3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:25.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.430+0000 7f66ed18d700 1 -- 172.21.15.112:0/1016419979 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e80b1730 con 0x7f66e8155da0 2023-01-24T03:39:25.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.430+0000 7f66ed98e700 1 --2- 172.21.15.112:0/1016419979 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66e80c7e40 0x7f66e80b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:39:25.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.430+0000 7f66ed18d700 1 --2- 172.21.15.112:0/1016419979 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66e8155da0 0x7f66e80ae9e0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f66d8006fd0 tx=0x7f66d80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68d3dfcbb8c58718 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:25.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.430+0000 7f66ec98c700 1 -- 172.21.15.112:0/1016419979 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d8011630 con 0x7f66e8155da0 2023-01-24T03:39:25.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.430+0000 7f66ec98c700 1 -- 172.21.15.112:0/1016419979 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66d8011790 con 0x7f66e8155da0 2023-01-24T03:39:25.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.430+0000 7f66ec98c700 1 -- 172.21.15.112:0/1016419979 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d8011b50 con 0x7f66e8155da0 2023-01-24T03:39:25.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.431+0000 7f66efbf2700 1 -- 172.21.15.112:0/1016419979 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66e8155da0 msgr2=0x7f66e80ae9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:25.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.431+0000 7f66efbf2700 1 --2- 172.21.15.112:0/1016419979 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66e8155da0 0x7f66e80ae9e0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f66d8006fd0 tx=0x7f66d80090f0 comp rx=0 tx=0).stop 2023-01-24T03:39:25.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.431+0000 7f66efbf2700 1 -- 172.21.15.112:0/1016419979 shutdown_connections 2023-01-24T03:39:25.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.431+0000 7f66efbf2700 1 --2- 172.21.15.112:0/1016419979 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66e80c7e40 0x7f66e80b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:25.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.431+0000 7f66efbf2700 1 --2- 172.21.15.112:0/1016419979 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66e8155da0 0x7f66e80ae9e0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:25.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.431+0000 7f66efbf2700 1 -- 172.21.15.112:0/1016419979 >> 172.21.15.112:0/1016419979 conn(0x7f66e805e2c0 msgr2=0x7f66e80cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:39:25.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.431+0000 7f66efbf2700 1 -- 172.21.15.112:0/1016419979 shutdown_connections 2023-01-24T03:39:25.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.431+0000 7f66efbf2700 1 -- 172.21.15.112:0/1016419979 wait complete. 2023-01-24T03:39:25.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.432+0000 7f66efbf2700 1 Processor -- start 2023-01-24T03:39:25.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.432+0000 7f66efbf2700 1 -- start start 2023-01-24T03:39:25.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.433+0000 7f66efbf2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66e80aef20 0x7f66e81691b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:25.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.433+0000 7f66efbf2700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66e80c7e40 0x7f66e80c4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:25.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.433+0000 7f66efbf2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66e8155da0 0x7f66e81653c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:25.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.433+0000 7f66efbf2700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e8168d20 con 0x7f66e80c7e40 2023-01-24T03:39:25.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.433+0000 7f66efbf2700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e80c21a0 con 0x7f66e8155da0 2023-01-24T03:39:25.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.433+0000 7f66efbf2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66e80c2460 con 0x7f66e80aef20 2023-01-24T03:39:25.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.433+0000 7f66ed18d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66e80c7e40 0x7f66e80c4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:25.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.433+0000 7f66ed18d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66e80c7e40 0x7f66e80c4de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55230/0 (socket says 172.21.15.112:55230) 2023-01-24T03:39:25.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.433+0000 7f66ed18d700 1 -- 172.21.15.112:0/2018070478 learned_addr learned my addr 172.21.15.112:0/2018070478 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:39:25.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.433+0000 7f66ed98e700 1 --2- 172.21.15.112:0/2018070478 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66e80aef20 0x7f66e81691b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:25.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.433+0000 7f66ee18f700 1 --2- 172.21.15.112:0/2018070478 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66e8155da0 0x7f66e81653c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:25.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.433+0000 7f66ed18d700 1 -- 172.21.15.112:0/2018070478 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66e80aef20 msgr2=0x7f66e81691b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:25.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.433+0000 7f66ed18d700 1 --2- 172.21.15.112:0/2018070478 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66e80aef20 0x7f66e81691b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:25.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.433+0000 7f66ed18d700 1 -- 172.21.15.112:0/2018070478 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66e8155da0 msgr2=0x7f66e81653c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:25.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.433+0000 7f66ed18d700 1 --2- 172.21.15.112:0/2018070478 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66e8155da0 0x7f66e81653c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:25.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.433+0000 7f66ed18d700 1 -- 172.21.15.112:0/2018070478 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d8009370 con 0x7f66e80c7e40 2023-01-24T03:39:25.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.433+0000 7f66ed18d700 1 --2- 172.21.15.112:0/2018070478 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66e80c7e40 0x7f66e80c4de0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f66d8011fd0 tx=0x7f66d8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:25.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.433+0000 7f66d5ffb700 1 -- 172.21.15.112:0/2018070478 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d801ccb0 con 0x7f66e80c7e40 2023-01-24T03:39:25.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.433+0000 7f66d5ffb700 1 -- 172.21.15.112:0/2018070478 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66d801ce10 con 0x7f66e80c7e40 2023-01-24T03:39:25.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.433+0000 7f66d5ffb700 1 -- 172.21.15.112:0/2018070478 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d801a6a0 con 0x7f66e80c7e40 2023-01-24T03:39:25.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.433+0000 7f66efbf2700 1 -- 172.21.15.112:0/2018070478 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66e80c2740 con 0x7f66e80c7e40 2023-01-24T03:39:25.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.434+0000 7f66efbf2700 1 -- 172.21.15.112:0/2018070478 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66e81696f0 con 0x7f66e80c7e40 2023-01-24T03:39:25.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.435+0000 7f66efbf2700 1 -- 172.21.15.112:0/2018070478 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66e8168b70 con 0x7f66e80c7e40 2023-01-24T03:39:25.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.435+0000 7f66d5ffb700 1 -- 172.21.15.112:0/2018070478 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f66d8017020 con 0x7f66e80c7e40 2023-01-24T03:39:25.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.435+0000 7f66d5ffb700 1 --2- 172.21.15.112:0/2018070478 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f66c80611f0 0x7f66c80636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:25.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.435+0000 7f66d5ffb700 1 -- 172.21.15.112:0/2018070478 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 7297+0+0 (secure 0 0 0) 0x7f66d807f080 con 0x7f66e80c7e40 2023-01-24T03:39:25.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.435+0000 7f66ed98e700 1 --2- 172.21.15.112:0/2018070478 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f66c80611f0 0x7f66c80636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:25.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.436+0000 7f66ed98e700 1 --2- 172.21.15.112:0/2018070478 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f66c80611f0 0x7f66c80636a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f66e4006d80 tx=0x7f66e4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:25.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.439+0000 7f66d5ffb700 1 -- 172.21.15.112:0/2018070478 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f66d804ecd0 con 0x7f66e80c7e40 2023-01-24T03:39:25.661 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.658+0000 7f66efbf2700 1 -- 172.21.15.112:0/2018070478 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66e8168e60 con 0x7f66e80c7e40 2023-01-24T03:39:25.662 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.659+0000 7f66d5ffb700 1 -- 172.21.15.112:0/2018070478 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+6121 (secure 0 0 0) 0x7f66d8029090 con 0x7f66e80c7e40 2023-01-24T03:39:25.662 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:39:25.663 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/2136880103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2136880103},{"type":"v1","addr":"172.21.15.142:6835","nonce":2136880103}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:37:57.754547+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5296":{"gid":5296,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":20,"addr":"172.21.15.142:6834/2838716685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2838716685},{"type":"v1","addr":"172.21.15.142:6834","nonce":2838716685}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:39:25.664 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.661+0000 7f66efbf2700 1 -- 172.21.15.112:0/2018070478 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f66c80611f0 msgr2=0x7f66c80636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:25.664 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.661+0000 7f66efbf2700 1 --2- 172.21.15.112:0/2018070478 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f66c80611f0 0x7f66c80636a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f66e4006d80 tx=0x7f66e4008040 comp rx=0 tx=0).stop 2023-01-24T03:39:25.664 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.661+0000 7f66efbf2700 1 -- 172.21.15.112:0/2018070478 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66e80c7e40 msgr2=0x7f66e80c4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:25.665 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.661+0000 7f66efbf2700 1 --2- 172.21.15.112:0/2018070478 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66e80c7e40 0x7f66e80c4de0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f66d8011fd0 tx=0x7f66d8016000 comp rx=0 tx=0).stop 2023-01-24T03:39:25.665 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.661+0000 7f66efbf2700 1 -- 172.21.15.112:0/2018070478 shutdown_connections 2023-01-24T03:39:25.666 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.661+0000 7f66efbf2700 1 --2- 172.21.15.112:0/2018070478 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f66c80611f0 0x7f66c80636a0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:25.666 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.661+0000 7f66efbf2700 1 --2- 172.21.15.112:0/2018070478 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66e8155da0 0x7f66e81653c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:25.666 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.661+0000 7f66efbf2700 1 --2- 172.21.15.112:0/2018070478 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f66e80aef20 0x7f66e81691b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:25.666 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.661+0000 7f66efbf2700 1 --2- 172.21.15.112:0/2018070478 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66e80c7e40 0x7f66e80c4de0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:25.667 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.661+0000 7f66efbf2700 1 -- 172.21.15.112:0/2018070478 >> 172.21.15.112:0/2018070478 conn(0x7f66e805e2c0 msgr2=0x7f66e80c8210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:39:25.667 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.661+0000 7f66efbf2700 1 -- 172.21.15.112:0/2018070478 shutdown_connections 2023-01-24T03:39:25.667 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:25.661+0000 7f66efbf2700 1 -- 172.21.15.112:0/2018070478 wait complete. 2023-01-24T03:39:25.667 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 36 2023-01-24T03:39:25.676 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-01-24T03:39:27.677 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:39:27.802 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.799+0000 7f5c083a8700 1 Processor -- start 2023-01-24T03:39:27.802 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.799+0000 7f5c083a8700 1 -- start start 2023-01-24T03:39:27.806 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.800+0000 7f5c083a8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5c000b0aa0 0x7f5c00155da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:27.806 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.800+0000 7f5c083a8700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c000b0f40 con 0x7f5c000ca3e0 2023-01-24T03:39:27.806 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.800+0000 7f5c083a8700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c000b1170 con 0x7f5c000c7e40 2023-01-24T03:39:27.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.800+0000 7f5c083a8700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c000b1450 con 0x7f5c000b0aa0 2023-01-24T03:39:27.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.800+0000 7f5c06945700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f5c000ca3e0 0x7f5c000ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:36230/0 (socket says 172.21.15.112:36230) 2023-01-24T03:39:27.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.800+0000 7f5c06945700 1 -- 172.21.15.112:0/3641082575 learned_addr learned my addr 172.21.15.112:0/3641082575 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:39:27.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.800+0000 7f5c05943700 1 --2- 172.21.15.112:0/3641082575 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5c000b0aa0 0x7f5c00155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:27.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.801+0000 7f5c05142700 1 -- 172.21.15.112:0/3641082575 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1713406610 0 0) 0x7f5c000b0f40 con 0x7f5c000ca3e0 2023-01-24T03:39:27.808 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.801+0000 7f5c05142700 1 -- 172.21.15.112:0/3641082575 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bec0033f0 con 0x7f5c000ca3e0 2023-01-24T03:39:27.808 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.801+0000 7f5c05142700 1 -- 172.21.15.112:0/3641082575 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 853293437 0 0) 0x7f5c000b1170 con 0x7f5c000c7e40 2023-01-24T03:39:27.808 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.801+0000 7f5c05142700 1 -- 172.21.15.112:0/3641082575 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c000b0f40 con 0x7f5c000c7e40 2023-01-24T03:39:27.808 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.801+0000 7f5c05142700 1 -- 172.21.15.112:0/3641082575 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 563771420 0 0) 0x7f5bec0033f0 con 0x7f5c000ca3e0 2023-01-24T03:39:27.808 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.801+0000 7f5c05142700 1 -- 172.21.15.112:0/3641082575 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5c000b0aa0 msgr2=0x7f5c00155da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:39:27.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.801+0000 7f5c05142700 1 --2- 172.21.15.112:0/3641082575 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5c000b0aa0 0x7f5c00155da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:27.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.801+0000 7f5c05142700 1 -- 172.21.15.112:0/3641082575 >> v1:172.21.15.142:6789/0 conn(0x7f5c000c7e40 legacy=0x7f5c000ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:27.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.801+0000 7f5c05142700 1 -- 172.21.15.112:0/3641082575 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c000b1730 con 0x7f5c000ca3e0 2023-01-24T03:39:27.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.801+0000 7f5c05142700 1 -- 172.21.15.112:0/3641082575 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7f5bf8001470 con 0x7f5c000ca3e0 2023-01-24T03:39:27.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.801+0000 7f5c05142700 1 -- 172.21.15.112:0/3641082575 >> v1:172.21.15.112:6789/0 conn(0x7f5c000ca3e0 legacy=0x7f5c000ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:27.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.801+0000 7f5c05142700 1 --2- 172.21.15.112:0/3641082575 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5c000b0aa0 0x7f5bec003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:27.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.801+0000 7f5c05142700 1 --2- 172.21.15.112:0/3641082575 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5bec004000 0x7f5bec004690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:27.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.801+0000 7f5c05142700 1 --2- 172.21.15.112:0/3641082575 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5bec004bd0 0x7f5bec005040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:27.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.801+0000 7f5c05142700 1 -- 172.21.15.112:0/3641082575 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c000b1450 con 0x7f5bec004bd0 2023-01-24T03:39:27.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.801+0000 7f5c05142700 1 -- 172.21.15.112:0/3641082575 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bec00ba20 con 0x7f5bec004000 2023-01-24T03:39:27.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.801+0000 7f5c05142700 1 -- 172.21.15.112:0/3641082575 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bec00bbe0 con 0x7f5c000b0aa0 2023-01-24T03:39:27.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.801+0000 7f5c06945700 1 --2- 172.21.15.112:0/3641082575 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5bec004bd0 0x7f5bec005040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:27.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.801+0000 7f5c05943700 1 --2- 172.21.15.112:0/3641082575 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5bec004000 0x7f5bec004690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:27.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.801+0000 7f5c06144700 1 --2- 172.21.15.112:0/3641082575 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5c000b0aa0 0x7f5bec003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:27.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.801+0000 7f5c06945700 1 -- 172.21.15.112:0/3641082575 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5c000b0aa0 msgr2=0x7f5bec003aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:39:27.812 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.801+0000 7f5c06945700 1 --2- 172.21.15.112:0/3641082575 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5c000b0aa0 0x7f5bec003aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:27.812 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.801+0000 7f5c06945700 1 -- 172.21.15.112:0/3641082575 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5bec004000 msgr2=0x7f5bec004690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:39:27.812 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.801+0000 7f5c06945700 1 --2- 172.21.15.112:0/3641082575 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5bec004000 0x7f5bec004690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:27.812 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.801+0000 7f5c06945700 1 -- 172.21.15.112:0/3641082575 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bec0033f0 con 0x7f5bec004bd0 2023-01-24T03:39:27.812 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.802+0000 7f5c06945700 1 --2- 172.21.15.112:0/3641082575 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5bec004bd0 0x7f5bec005040 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f5bf800c0e0 tx=0x7f5bf8001e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=511eb2e6286cc24a server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:27.813 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.802+0000 7f5c05142700 1 -- 172.21.15.112:0/3641082575 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf800ca10 con 0x7f5bec004bd0 2023-01-24T03:39:27.813 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.802+0000 7f5c05142700 1 -- 172.21.15.112:0/3641082575 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bf800cb70 con 0x7f5bec004bd0 2023-01-24T03:39:27.813 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.802+0000 7f5c05142700 1 -- 172.21.15.112:0/3641082575 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf80070d0 con 0x7f5bec004bd0 2023-01-24T03:39:27.813 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.802+0000 7f5c083a8700 1 -- 172.21.15.112:0/3641082575 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5bec004bd0 msgr2=0x7f5bec005040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:27.813 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.802+0000 7f5c083a8700 1 --2- 172.21.15.112:0/3641082575 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5bec004bd0 0x7f5bec005040 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f5bf800c0e0 tx=0x7f5bf8001e10 comp rx=0 tx=0).stop 2023-01-24T03:39:27.813 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.802+0000 7f5c06945700 1 -- 172.21.15.112:0/3641082575 reap_dead start 2023-01-24T03:39:27.814 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.802+0000 7f5c083a8700 1 -- 172.21.15.112:0/3641082575 shutdown_connections 2023-01-24T03:39:27.814 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.802+0000 7f5c083a8700 1 -- 172.21.15.112:0/3641082575 >> 172.21.15.112:0/3641082575 conn(0x7f5c0005e2c0 msgr2=0x7f5c000cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:39:27.814 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.802+0000 7f5c083a8700 1 -- 172.21.15.112:0/3641082575 shutdown_connections 2023-01-24T03:39:27.814 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.802+0000 7f5c083a8700 1 -- 172.21.15.112:0/3641082575 wait complete. 2023-01-24T03:39:27.814 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.803+0000 7f5c083a8700 1 Processor -- start 2023-01-24T03:39:27.815 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.803+0000 7f5c083a8700 1 -- start start 2023-01-24T03:39:27.817 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.804+0000 7f5c083a8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5bec004bd0 0x7f5c0014ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:27.817 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.804+0000 7f5c083a8700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5c000c2250 0x7f5c00068520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:27.817 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.804+0000 7f5c083a8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5c000c0810 0x7f5c000c0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:27.818 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.804+0000 7f5c083a8700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c000c11c0 con 0x7f5c000c2250 2023-01-24T03:39:27.818 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.804+0000 7f5c083a8700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c00165790 con 0x7f5bec004bd0 2023-01-24T03:39:27.820 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.804+0000 7f5c083a8700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c00169a60 con 0x7f5c000c0810 2023-01-24T03:39:27.820 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.804+0000 7f5c05943700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5c000c2250 0x7f5c00068520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:27.820 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.804+0000 7f5c05943700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5c000c2250 0x7f5c00068520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55254/0 (socket says 172.21.15.112:55254) 2023-01-24T03:39:27.820 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.804+0000 7f5c05943700 1 -- 172.21.15.112:0/2002225669 learned_addr learned my addr 172.21.15.112:0/2002225669 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:39:27.821 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.804+0000 7f5c06945700 1 --2- 172.21.15.112:0/2002225669 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5c000c0810 0x7f5c000c0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:27.821 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.804+0000 7f5c06144700 1 --2- 172.21.15.112:0/2002225669 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5bec004bd0 0x7f5c0014ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:27.821 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.804+0000 7f5c05943700 1 -- 172.21.15.112:0/2002225669 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5c000c0810 msgr2=0x7f5c000c0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:27.821 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.804+0000 7f5c05943700 1 --2- 172.21.15.112:0/2002225669 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5c000c0810 0x7f5c000c0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:27.821 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.804+0000 7f5c05943700 1 -- 172.21.15.112:0/2002225669 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5bec004bd0 msgr2=0x7f5c0014ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:27.822 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.804+0000 7f5c05943700 1 --2- 172.21.15.112:0/2002225669 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5bec004bd0 0x7f5c0014ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:27.822 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.804+0000 7f5c05943700 1 -- 172.21.15.112:0/2002225669 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c00165a50 con 0x7f5c000c2250 2023-01-24T03:39:27.822 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.805+0000 7f5c05943700 1 --2- 172.21.15.112:0/2002225669 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5c000c2250 0x7f5c00068520 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f5bf40119c0 tx=0x7f5bf4009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:27.822 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.805+0000 7f5bfe7fc700 1 -- 172.21.15.112:0/2002225669 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf400f700 con 0x7f5c000c2250 2023-01-24T03:39:27.822 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.805+0000 7f5bfe7fc700 1 -- 172.21.15.112:0/2002225669 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bf400f860 con 0x7f5c000c2250 2023-01-24T03:39:27.823 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.805+0000 7f5c083a8700 1 -- 172.21.15.112:0/2002225669 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c00165d30 con 0x7f5c000c2250 2023-01-24T03:39:27.823 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.805+0000 7f5bfe7fc700 1 -- 172.21.15.112:0/2002225669 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf400fc00 con 0x7f5c000c2250 2023-01-24T03:39:27.823 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.805+0000 7f5c083a8700 1 -- 172.21.15.112:0/2002225669 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c001662f0 con 0x7f5c000c2250 2023-01-24T03:39:27.823 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.806+0000 7f5c083a8700 1 -- 172.21.15.112:0/2002225669 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c00069f10 con 0x7f5c000c2250 2023-01-24T03:39:27.823 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.806+0000 7f5bfe7fc700 1 -- 172.21.15.112:0/2002225669 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f5bf4014440 con 0x7f5c000c2250 2023-01-24T03:39:27.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.806+0000 7f5bfe7fc700 1 --2- 172.21.15.112:0/2002225669 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5be00611f0 0x7f5be00636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:27.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.806+0000 7f5bfe7fc700 1 -- 172.21.15.112:0/2002225669 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 7297+0+0 (secure 0 0 0) 0x7f5bf4018030 con 0x7f5c000c2250 2023-01-24T03:39:27.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.806+0000 7f5c06144700 1 --2- 172.21.15.112:0/2002225669 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5be00611f0 0x7f5be00636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:27.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.807+0000 7f5c06144700 1 --2- 172.21.15.112:0/2002225669 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5be00611f0 0x7f5be00636a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0002f90 tx=0x7f5bf0007f30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:27.824 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:27.811+0000 7f5bfe7fc700 1 -- 172.21.15.112:0/2002225669 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f5bf404dbf0 con 0x7f5c000c2250 2023-01-24T03:39:28.033 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:28.030+0000 7f5c083a8700 1 -- 172.21.15.112:0/2002225669 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c00166510 con 0x7f5c000c2250 2023-01-24T03:39:28.033 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:28.031+0000 7f5bfe7fc700 1 -- 172.21.15.112:0/2002225669 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+6121 (secure 0 0 0) 0x7f5bf4051210 con 0x7f5c000c2250 2023-01-24T03:39:28.034 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:39:28.034 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5078,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/2136880103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2136880103},{"type":"v1","addr":"172.21.15.142:6835","nonce":2136880103}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:37:57.754547+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5296":{"gid":5296,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":20,"addr":"172.21.15.142:6834/2838716685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2838716685},{"type":"v1","addr":"172.21.15.142:6834","nonce":2838716685}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:39:28.035 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:28.032+0000 7f5c083a8700 1 -- 172.21.15.112:0/2002225669 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5be00611f0 msgr2=0x7f5be00636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:28.036 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:28.032+0000 7f5c083a8700 1 --2- 172.21.15.112:0/2002225669 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5be00611f0 0x7f5be00636a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0002f90 tx=0x7f5bf0007f30 comp rx=0 tx=0).stop 2023-01-24T03:39:28.036 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:28.032+0000 7f5c083a8700 1 -- 172.21.15.112:0/2002225669 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5c000c2250 msgr2=0x7f5c00068520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:28.036 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:28.032+0000 7f5c083a8700 1 --2- 172.21.15.112:0/2002225669 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5c000c2250 0x7f5c00068520 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f5bf40119c0 tx=0x7f5bf4009060 comp rx=0 tx=0).stop 2023-01-24T03:39:28.036 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:28.032+0000 7f5c083a8700 1 -- 172.21.15.112:0/2002225669 shutdown_connections 2023-01-24T03:39:28.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:28.032+0000 7f5c083a8700 1 --2- 172.21.15.112:0/2002225669 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5be00611f0 0x7f5be00636a0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:28.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:28.032+0000 7f5c083a8700 1 --2- 172.21.15.112:0/2002225669 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5c000c0810 0x7f5c000c0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:28.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:28.032+0000 7f5c083a8700 1 --2- 172.21.15.112:0/2002225669 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5bec004bd0 0x7f5c0014ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:28.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:28.032+0000 7f5c083a8700 1 --2- 172.21.15.112:0/2002225669 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5c000c2250 0x7f5c00068520 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:28.037 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:28.032+0000 7f5c083a8700 1 -- 172.21.15.112:0/2002225669 >> 172.21.15.112:0/2002225669 conn(0x7f5c0005e2c0 msgr2=0x7f5c00069780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:39:28.038 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:28.032+0000 7f5c083a8700 1 -- 172.21.15.112:0/2002225669 shutdown_connections 2023-01-24T03:39:28.038 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:28.032+0000 7f5c083a8700 1 -- 172.21.15.112:0/2002225669 wait complete. 2023-01-24T03:39:28.038 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 36 2023-01-24T03:39:28.047 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-01-24T03:39:30.048 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:39:30.099 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:39:30.101 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:39:30.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.183+0000 7f12e359e700 1 Processor -- start 2023-01-24T03:39:30.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.183+0000 7f12e359e700 1 -- start start 2023-01-24T03:39:30.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.184+0000 7f12e359e700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f12e40c5680 0x7f12e4156730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:30.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.184+0000 7f12e359e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f12e40c5a50 0x7f12e4156c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:30.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.184+0000 7f12e359e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12e40c3d10 con 0x7f12e4155e20 2023-01-24T03:39:30.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.184+0000 7f12e359e700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12e40c4eb0 con 0x7f12e40c5680 2023-01-24T03:39:30.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.184+0000 7f12e359e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12e40c2e20 con 0x7f12e40c5a50 2023-01-24T03:39:30.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.184+0000 7f12e259c700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f12e40c5680 0x7f12e4156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:30.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.184+0000 7f12e259c700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f12e40c5680 0x7f12e4156730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55262/0 (socket says 172.21.15.112:55262) 2023-01-24T03:39:30.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.184+0000 7f12e259c700 1 -- 172.21.15.112:0/132325391 learned_addr learned my addr 172.21.15.112:0/132325391 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:39:30.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.184+0000 7f12e2d9d700 1 --2- 172.21.15.112:0/132325391 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f12e40c5a50 0x7f12e4156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:30.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.184+0000 7f12e259c700 1 -- 172.21.15.112:0/132325391 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f12e40c5a50 msgr2=0x7f12e4156c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:39:30.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.184+0000 7f12e259c700 1 --2- 172.21.15.112:0/132325391 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f12e40c5a50 0x7f12e4156c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:30.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.184+0000 7f12e259c700 1 -- 172.21.15.112:0/132325391 >> v1:172.21.15.142:6789/0 conn(0x7f12e4155e20 legacy=0x7f12e40cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:30.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.184+0000 7f12e259c700 1 -- 172.21.15.112:0/132325391 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e40c3100 con 0x7f12e40c5680 2023-01-24T03:39:30.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.185+0000 7f12e259c700 1 --2- 172.21.15.112:0/132325391 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f12e40c5680 0x7f12e4156730 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f12d4009f20 tx=0x7f12d4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=617cbf2603e15f3a server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:30.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.185+0000 7f12e159a700 1 -- 172.21.15.112:0/132325391 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d400e070 con 0x7f12e40c5680 2023-01-24T03:39:30.193 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.185+0000 7f12e159a700 1 -- 172.21.15.112:0/132325391 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12d400ba80 con 0x7f12e40c5680 2023-01-24T03:39:30.193 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.185+0000 7f12e159a700 1 -- 172.21.15.112:0/132325391 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d40092e0 con 0x7f12e40c5680 2023-01-24T03:39:30.193 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.185+0000 7f12e359e700 1 -- 172.21.15.112:0/132325391 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f12e40c5680 msgr2=0x7f12e4156730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:30.193 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.185+0000 7f12e359e700 1 --2- 172.21.15.112:0/132325391 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f12e40c5680 0x7f12e4156730 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f12d4009f20 tx=0x7f12d4009840 comp rx=0 tx=0).stop 2023-01-24T03:39:30.193 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.185+0000 7f12e359e700 1 -- 172.21.15.112:0/132325391 shutdown_connections 2023-01-24T03:39:30.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.185+0000 7f12e359e700 1 --2- 172.21.15.112:0/132325391 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f12e40c5a50 0x7f12e4156c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:30.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.185+0000 7f12e359e700 1 --2- 172.21.15.112:0/132325391 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f12e40c5680 0x7f12e4156730 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:30.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.185+0000 7f12e359e700 1 -- 172.21.15.112:0/132325391 >> 172.21.15.112:0/132325391 conn(0x7f12e40685b0 msgr2=0x7f12e40689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:39:30.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.185+0000 7f12e359e700 1 -- 172.21.15.112:0/132325391 shutdown_connections 2023-01-24T03:39:30.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.185+0000 7f12e359e700 1 -- 172.21.15.112:0/132325391 wait complete. 2023-01-24T03:39:30.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.186+0000 7f12e359e700 1 Processor -- start 2023-01-24T03:39:30.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.186+0000 7f12e359e700 1 -- start start 2023-01-24T03:39:30.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.187+0000 7f12e359e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f12e40c5680 0x7f12e4168000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:30.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.187+0000 7f12e359e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12e40c5a50 0x7f12e4168ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:30.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.187+0000 7f12e359e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f12e4155e20 0x7f12e4169020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:30.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.187+0000 7f12e359e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e40a9e20 con 0x7f12e40c5680 2023-01-24T03:39:30.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.187+0000 7f12e359e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e40aa610 con 0x7f12e40c5a50 2023-01-24T03:39:30.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.187+0000 7f12e359e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12e4163480 con 0x7f12e4155e20 2023-01-24T03:39:30.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.187+0000 7f12e259c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f12e40c5680 0x7f12e4168000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:30.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.187+0000 7f12e259c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f12e40c5680 0x7f12e4168000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55268/0 (socket says 172.21.15.112:55268) 2023-01-24T03:39:30.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.187+0000 7f12e259c700 1 -- 172.21.15.112:0/311568804 learned_addr learned my addr 172.21.15.112:0/311568804 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:39:30.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.187+0000 7f12e1d9b700 1 --2- 172.21.15.112:0/311568804 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12e40c5a50 0x7f12e4168ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:30.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.187+0000 7f12e2d9d700 1 --2- 172.21.15.112:0/311568804 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f12e4155e20 0x7f12e4169020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:30.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.187+0000 7f12e259c700 1 -- 172.21.15.112:0/311568804 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f12e4155e20 msgr2=0x7f12e4169020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:30.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.187+0000 7f12e259c700 1 --2- 172.21.15.112:0/311568804 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f12e4155e20 0x7f12e4169020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:30.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.187+0000 7f12e259c700 1 -- 172.21.15.112:0/311568804 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12e40c5a50 msgr2=0x7f12e4168ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:30.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.187+0000 7f12e259c700 1 --2- 172.21.15.112:0/311568804 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12e40c5a50 0x7f12e4168ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:30.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.187+0000 7f12e259c700 1 -- 172.21.15.112:0/311568804 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12d4009ae0 con 0x7f12e40c5680 2023-01-24T03:39:30.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.187+0000 7f12e2d9d700 1 --2- 172.21.15.112:0/311568804 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f12e4155e20 0x7f12e4169020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-01-24T03:39:30.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.188+0000 7f12e259c700 1 --2- 172.21.15.112:0/311568804 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f12e40c5680 0x7f12e4168000 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f12d4009430 tx=0x7f12d4018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:30.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.188+0000 7f12da7fc700 1 -- 172.21.15.112:0/311568804 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d40094e0 con 0x7f12e40c5680 2023-01-24T03:39:30.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.188+0000 7f12da7fc700 1 -- 172.21.15.112:0/311568804 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12d40089b0 con 0x7f12e40c5680 2023-01-24T03:39:30.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.188+0000 7f12da7fc700 1 -- 172.21.15.112:0/311568804 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d4008d70 con 0x7f12e40c5680 2023-01-24T03:39:30.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.188+0000 7f12e359e700 1 -- 172.21.15.112:0/311568804 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12e4167b70 con 0x7f12e40c5680 2023-01-24T03:39:30.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.188+0000 7f12e359e700 1 -- 172.21.15.112:0/311568804 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12e40c83c0 con 0x7f12e40c5680 2023-01-24T03:39:30.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.189+0000 7f12e359e700 1 -- 172.21.15.112:0/311568804 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12e405ea20 con 0x7f12e40c5680 2023-01-24T03:39:30.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.190+0000 7f12da7fc700 1 -- 172.21.15.112:0/311568804 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f12d401c410 con 0x7f12e40c5680 2023-01-24T03:39:30.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.190+0000 7f12da7fc700 1 --2- 172.21.15.112:0/311568804 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f12c80611f0 0x7f12c80636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:30.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.190+0000 7f12da7fc700 1 -- 172.21.15.112:0/311568804 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 7383+0+0 (secure 0 0 0) 0x7f12d407fa30 con 0x7f12e40c5680 2023-01-24T03:39:30.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.190+0000 7f12e1d9b700 1 --2- 172.21.15.112:0/311568804 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f12c80611f0 0x7f12c80636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:30.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.191+0000 7f12e1d9b700 1 --2- 172.21.15.112:0/311568804 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f12c80611f0 0x7f12c80636a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f12e40cb5a0 tx=0x7f12cc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:30.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.194+0000 7f12da7fc700 1 -- 172.21.15.112:0/311568804 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f12d404f610 con 0x7f12e40c5680 2023-01-24T03:39:30.418 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.415+0000 7f12e359e700 1 -- 172.21.15.112:0/311568804 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12e40b12c0 con 0x7f12e40c5680 2023-01-24T03:39:30.419 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.415+0000 7f12da7fc700 1 -- 172.21.15.112:0/311568804 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+5360 (secure 0 0 0) 0x7f12d4052c30 con 0x7f12e40c5680 2023-01-24T03:39:30.419 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:39:30.419 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:39:28.714628+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":37,"state":"up:replay","state_seq":1,"addr":"172.21.15.142:6835/2136880103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2136880103},{"type":"v1","addr":"172.21.15.142:6835","nonce":2136880103}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:39:30.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.417+0000 7f12e359e700 1 -- 172.21.15.112:0/311568804 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f12c80611f0 msgr2=0x7f12c80636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:30.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.417+0000 7f12e359e700 1 --2- 172.21.15.112:0/311568804 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f12c80611f0 0x7f12c80636a0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f12e40cb5a0 tx=0x7f12cc009040 comp rx=0 tx=0).stop 2023-01-24T03:39:30.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.417+0000 7f12e359e700 1 -- 172.21.15.112:0/311568804 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f12e40c5680 msgr2=0x7f12e4168000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:30.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.417+0000 7f12e359e700 1 --2- 172.21.15.112:0/311568804 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f12e40c5680 0x7f12e4168000 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f12d4009430 tx=0x7f12d4018000 comp rx=0 tx=0).stop 2023-01-24T03:39:30.422 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.417+0000 7f12e359e700 1 -- 172.21.15.112:0/311568804 shutdown_connections 2023-01-24T03:39:30.422 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.417+0000 7f12e359e700 1 --2- 172.21.15.112:0/311568804 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f12c80611f0 0x7f12c80636a0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:30.422 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.417+0000 7f12e359e700 1 --2- 172.21.15.112:0/311568804 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f12e40c5680 0x7f12e4168000 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:30.422 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.417+0000 7f12e359e700 1 --2- 172.21.15.112:0/311568804 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f12e4155e20 0x7f12e4169020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:30.422 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.417+0000 7f12e359e700 1 --2- 172.21.15.112:0/311568804 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f12e40c5a50 0x7f12e4168ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:30.423 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.417+0000 7f12e359e700 1 -- 172.21.15.112:0/311568804 >> 172.21.15.112:0/311568804 conn(0x7f12e40685b0 msgr2=0x7f12e4153080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:39:30.423 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.417+0000 7f12e359e700 1 -- 172.21.15.112:0/311568804 shutdown_connections 2023-01-24T03:39:30.423 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.417+0000 7f12e359e700 1 -- 172.21.15.112:0/311568804 wait complete. 2023-01-24T03:39:30.423 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 37 2023-01-24T03:39:30.432 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2023-01-24T03:39:30.432 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-01-24T03:39:30.433 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:39:30.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.581+0000 7fa7f2fdb700 1 Processor -- start 2023-01-24T03:39:30.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.581+0000 7fa7f2fdb700 1 -- start start 2023-01-24T03:39:30.587 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.582+0000 7fa7f2fdb700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7ec1522c0 0x7fa7ec0c5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:30.587 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.582+0000 7fa7f2fdb700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa7ec152690 0x7fa7ec152ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:30.588 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.582+0000 7fa7f2fdb700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7ec0c4fe0 con 0x7fa7ec0c0020 2023-01-24T03:39:30.588 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.582+0000 7fa7f2fdb700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7ec0beb60 con 0x7fa7ec152690 2023-01-24T03:39:30.588 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.582+0000 7fa7f2fdb700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ec0c40f0 con 0x7fa7ec1522c0 2023-01-24T03:39:30.588 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.583+0000 7fa7f0d77700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7ec1522c0 0x7fa7ec0c5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:30.588 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.583+0000 7fa7ebfff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa7ec152690 0x7fa7ec152ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:30.589 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.583+0000 7fa7f0d77700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7ec1522c0 0x7fa7ec0c5e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.112:42578/0 (socket says 172.21.15.112:42578) 2023-01-24T03:39:30.589 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.583+0000 7fa7f0d77700 1 -- 172.21.15.112:0/3254109775 learned_addr learned my addr 172.21.15.112:0/3254109775 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:39:30.589 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.583+0000 7fa7ebfff700 1 -- 172.21.15.112:0/3254109775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7ec1522c0 msgr2=0x7fa7ec0c5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:39:30.589 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.583+0000 7fa7ebfff700 1 --2- 172.21.15.112:0/3254109775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7ec1522c0 0x7fa7ec0c5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:30.589 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.583+0000 7fa7ebfff700 1 -- 172.21.15.112:0/3254109775 >> v1:172.21.15.142:6789/0 conn(0x7fa7ec0c0020 legacy=0x7fa7ec0c0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:30.590 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.583+0000 7fa7ebfff700 1 -- 172.21.15.112:0/3254109775 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7ec0c43d0 con 0x7fa7ec152690 2023-01-24T03:39:30.590 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.583+0000 7fa7eb7fe700 1 -- 172.21.15.112:0/3254109775 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 329462364 0 0) 0x7fa7ec0c4fe0 con 0x7fa7ec0c0020 2023-01-24T03:39:30.590 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.583+0000 7fa7f0d77700 1 --2- 172.21.15.112:0/3254109775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7ec1522c0 0x7fa7ec0c5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:39:30.590 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.583+0000 7fa7ebfff700 1 --2- 172.21.15.112:0/3254109775 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7ec152690 0x7fa7ec152ac0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0009760 tx=0x7fa7e0009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6721770bef993f80 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:30.590 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.583+0000 7fa7eb7fe700 1 -- 172.21.15.112:0/3254109775 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e0011630 con 0x7fa7ec152690 2023-01-24T03:39:30.591 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.584+0000 7fa7eb7fe700 1 -- 172.21.15.112:0/3254109775 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7e0011790 con 0x7fa7ec152690 2023-01-24T03:39:30.591 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.584+0000 7fa7eb7fe700 1 -- 172.21.15.112:0/3254109775 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e0011b50 con 0x7fa7ec152690 2023-01-24T03:39:30.591 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.584+0000 7fa7f2fdb700 1 -- 172.21.15.112:0/3254109775 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7ec152690 msgr2=0x7fa7ec152ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:30.591 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.584+0000 7fa7f2fdb700 1 --2- 172.21.15.112:0/3254109775 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7ec152690 0x7fa7ec152ac0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0009760 tx=0x7fa7e0009060 comp rx=0 tx=0).stop 2023-01-24T03:39:30.591 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.584+0000 7fa7f2fdb700 1 -- 172.21.15.112:0/3254109775 shutdown_connections 2023-01-24T03:39:30.592 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.584+0000 7fa7f2fdb700 1 --2- 172.21.15.112:0/3254109775 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7ec1522c0 0x7fa7ec0c5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:30.592 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.584+0000 7fa7f2fdb700 1 --2- 172.21.15.112:0/3254109775 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7ec152690 0x7fa7ec152ac0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:30.592 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.584+0000 7fa7f2fdb700 1 -- 172.21.15.112:0/3254109775 >> 172.21.15.112:0/3254109775 conn(0x7fa7ec066e10 msgr2=0x7fa7ec067210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:39:30.592 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.584+0000 7fa7f2fdb700 1 -- 172.21.15.112:0/3254109775 shutdown_connections 2023-01-24T03:39:30.592 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.584+0000 7fa7f2fdb700 1 -- 172.21.15.112:0/3254109775 wait complete. 2023-01-24T03:39:30.595 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.585+0000 7fa7f2fdb700 1 Processor -- start 2023-01-24T03:39:30.595 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.585+0000 7fa7f2fdb700 1 -- start start 2023-01-24T03:39:30.595 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.586+0000 7fa7f2fdb700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7ec0c0020 0x7fa7ec0b8890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:30.596 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.586+0000 7fa7f2fdb700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7ec1522c0 0x7fa7ec14f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:30.596 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.586+0000 7fa7f2fdb700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa7ec152690 0x7fa7ec06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:30.596 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.586+0000 7fa7f2fdb700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ec164f20 con 0x7fa7ec152690 2023-01-24T03:39:30.596 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.586+0000 7fa7f2fdb700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ec06edb0 con 0x7fa7ec0c0020 2023-01-24T03:39:30.596 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.586+0000 7fa7f2fdb700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ec06f050 con 0x7fa7ec1522c0 2023-01-24T03:39:30.597 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.586+0000 7fa7f1578700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa7ec152690 0x7fa7ec06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:30.597 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.586+0000 7fa7f1578700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa7ec152690 0x7fa7ec06d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55286/0 (socket says 172.21.15.112:55286) 2023-01-24T03:39:30.597 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.586+0000 7fa7ebfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7ec1522c0 0x7fa7ec14f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:30.597 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.586+0000 7fa7f0d77700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7ec0c0020 0x7fa7ec0b8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:30.597 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.586+0000 7fa7f1578700 1 -- 172.21.15.112:0/1423356991 learned_addr learned my addr 172.21.15.112:0/1423356991 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:39:30.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.586+0000 7fa7f1578700 1 -- 172.21.15.112:0/1423356991 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7ec1522c0 msgr2=0x7fa7ec14f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:30.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.586+0000 7fa7f1578700 1 --2- 172.21.15.112:0/1423356991 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7ec1522c0 0x7fa7ec14f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:30.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.586+0000 7fa7f1578700 1 -- 172.21.15.112:0/1423356991 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7ec0c0020 msgr2=0x7fa7ec0b8890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:30.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.586+0000 7fa7f1578700 1 --2- 172.21.15.112:0/1423356991 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7ec0c0020 0x7fa7ec0b8890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:30.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.586+0000 7fa7f1578700 1 -- 172.21.15.112:0/1423356991 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7dc009370 con 0x7fa7ec152690 2023-01-24T03:39:30.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.587+0000 7fa7f1578700 1 --2- 172.21.15.112:0/1423356991 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa7ec152690 0x7fa7ec06d190 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4001560 tx=0x7fa7e4007f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:30.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.587+0000 7fa7e8ff9700 1 -- 172.21.15.112:0/1423356991 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e4011590 con 0x7fa7ec152690 2023-01-24T03:39:30.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.587+0000 7fa7e8ff9700 1 -- 172.21.15.112:0/1423356991 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7e40116f0 con 0x7fa7ec152690 2023-01-24T03:39:30.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.587+0000 7fa7f2fdb700 1 -- 172.21.15.112:0/1423356991 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7e00092a0 con 0x7fa7ec152690 2023-01-24T03:39:30.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.587+0000 7fa7e8ff9700 1 -- 172.21.15.112:0/1423356991 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e4011ab0 con 0x7fa7ec152690 2023-01-24T03:39:30.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.587+0000 7fa7f2fdb700 1 -- 172.21.15.112:0/1423356991 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7ec164580 con 0x7fa7ec152690 2023-01-24T03:39:30.600 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.588+0000 7fa7f2fdb700 1 -- 172.21.15.112:0/1423356991 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7ec165060 con 0x7fa7ec152690 2023-01-24T03:39:30.600 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.588+0000 7fa7e8ff9700 1 -- 172.21.15.112:0/1423356991 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fa7e4003030 con 0x7fa7ec152690 2023-01-24T03:39:30.600 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.588+0000 7fa7e8ff9700 1 --2- 172.21.15.112:0/1423356991 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa7cc0611c0 0x7fa7cc063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:30.600 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.589+0000 7fa7e8ff9700 1 -- 172.21.15.112:0/1423356991 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 7383+0+0 (secure 0 0 0) 0x7fa7e407d670 con 0x7fa7ec152690 2023-01-24T03:39:30.600 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.589+0000 7fa7f0d77700 1 --2- 172.21.15.112:0/1423356991 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa7cc0611c0 0x7fa7cc063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:30.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.589+0000 7fa7f0d77700 1 --2- 172.21.15.112:0/1423356991 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa7cc0611c0 0x7fa7cc063670 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc006fd0 tx=0x7fa7dc0090f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:30.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.593+0000 7fa7e8ff9700 1 -- 172.21.15.112:0/1423356991 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fa7ec165060 con 0x7fa7ec152690 2023-01-24T03:39:30.814 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.811+0000 7fa7f2fdb700 1 -- 172.21.15.112:0/1423356991 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7ec165060 con 0x7fa7ec152690 2023-01-24T03:39:30.815 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.812+0000 7fa7e8ff9700 1 -- 172.21.15.112:0/1423356991 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+5360 (secure 0 0 0) 0x7fa7ec165060 con 0x7fa7ec152690 2023-01-24T03:39:30.815 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:39:30.815 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:39:28.714628+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":37,"state":"up:replay","state_seq":1,"addr":"172.21.15.142:6835/2136880103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2136880103},{"type":"v1","addr":"172.21.15.142:6835","nonce":2136880103}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:39:30.817 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.814+0000 7fa7f2fdb700 1 -- 172.21.15.112:0/1423356991 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa7cc0611c0 msgr2=0x7fa7cc063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:30.817 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.814+0000 7fa7f2fdb700 1 --2- 172.21.15.112:0/1423356991 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa7cc0611c0 0x7fa7cc063670 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc006fd0 tx=0x7fa7dc0090f0 comp rx=0 tx=0).stop 2023-01-24T03:39:30.818 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.814+0000 7fa7f2fdb700 1 -- 172.21.15.112:0/1423356991 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa7ec152690 msgr2=0x7fa7ec06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:30.818 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.814+0000 7fa7f2fdb700 1 --2- 172.21.15.112:0/1423356991 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa7ec152690 0x7fa7ec06d190 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4001560 tx=0x7fa7e4007f40 comp rx=0 tx=0).stop 2023-01-24T03:39:30.818 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.814+0000 7fa7f2fdb700 1 -- 172.21.15.112:0/1423356991 shutdown_connections 2023-01-24T03:39:30.818 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.814+0000 7fa7f2fdb700 1 --2- 172.21.15.112:0/1423356991 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa7cc0611c0 0x7fa7cc063670 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:30.818 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.814+0000 7fa7f2fdb700 1 --2- 172.21.15.112:0/1423356991 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa7ec152690 0x7fa7ec06d190 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:30.819 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.814+0000 7fa7f2fdb700 1 --2- 172.21.15.112:0/1423356991 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa7ec0c0020 0x7fa7ec0b8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:30.819 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.814+0000 7fa7f2fdb700 1 --2- 172.21.15.112:0/1423356991 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa7ec1522c0 0x7fa7ec14f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:30.819 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.814+0000 7fa7f2fdb700 1 -- 172.21.15.112:0/1423356991 >> 172.21.15.112:0/1423356991 conn(0x7fa7ec066e10 msgr2=0x7fa7ec06a900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:39:30.819 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.814+0000 7fa7f2fdb700 1 -- 172.21.15.112:0/1423356991 shutdown_connections 2023-01-24T03:39:30.819 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:30.814+0000 7fa7f2fdb700 1 -- 172.21.15.112:0/1423356991 wait complete. 2023-01-24T03:39:30.820 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 37 2023-01-24T03:39:30.829 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 37, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-01-24T03:30:20.971710+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 37, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5078": { "addr": "172.21.15.142:6835/2136880103", "addrs": { "addrvec": [ { "addr": "172.21.15.142:6833", "nonce": 2136880103, "type": "v2" }, { "addr": "172.21.15.142:6835", "nonce": 2136880103, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5078, "incarnation": 37, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 48, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-01-24T03:39:28.714628+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5078 } } } ], "standbys": [ { "addr": "172.21.15.142:6837/2181345869", "addrs": { "addrvec": [ { "addr": "172.21.15.142:6836", "nonce": 2181345869, "type": "v2" }, { "addr": "172.21.15.142:6837", "nonce": 2181345869, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6835/330343927", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6834", "nonce": 330343927, "type": "v2" }, { "addr": "172.21.15.112:6835", "nonce": 330343927, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6837/12176817", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6836", "nonce": 12176817, "type": "v2" }, { "addr": "172.21.15.112:6837", "nonce": 12176817, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4276, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6839/1110983841", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6838", "nonce": 1110983841, "type": "v2" }, { "addr": "172.21.15.112:6839", "nonce": 1110983841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4281, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-01-24T03:39:32.830 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:39:32.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.954+0000 7f9ad5b05700 1 Processor -- start 2023-01-24T03:39:32.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.954+0000 7f9ad5b05700 1 -- start start 2023-01-24T03:39:32.959 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.954+0000 7f9ad5b05700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f9ad00c7e40 0x7f9ad00b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:32.959 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.954+0000 7f9ad5b05700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9ad0155da0 0x7f9ad00ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:32.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.954+0000 7f9ad5b05700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad0156240 con 0x7f9ad00aef20 2023-01-24T03:39:32.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.954+0000 7f9ad5b05700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ad00b1170 con 0x7f9ad00c7e40 2023-01-24T03:39:32.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.954+0000 7f9ad5b05700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad00b1450 con 0x7f9ad0155da0 2023-01-24T03:39:32.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.954+0000 7f9acffff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f9ad00aef20 0x7f9ad00ca3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:36282/0 (socket says 172.21.15.112:36282) 2023-01-24T03:39:32.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.954+0000 7f9acffff700 1 -- 172.21.15.112:0/2984188949 learned_addr learned my addr 172.21.15.112:0/2984188949 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:39:32.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.954+0000 7f9acf7fe700 1 --2- 172.21.15.112:0/2984188949 >> v2:172.21.15.142:3300/0 conn(0x7f9ad00c7e40 0x7f9ad00b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:32.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.954+0000 7f9aceffd700 1 --2- 172.21.15.112:0/2984188949 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9ad0155da0 0x7f9ad00ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:32.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.954+0000 7f9ace7fc700 1 -- 172.21.15.112:0/2984188949 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1605569011 0 0) 0x7f9ad0156240 con 0x7f9ad00aef20 2023-01-24T03:39:32.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.954+0000 7f9ace7fc700 1 -- 172.21.15.112:0/2984188949 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9abc0033f0 con 0x7f9ad00aef20 2023-01-24T03:39:32.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.954+0000 7f9acf7fe700 1 -- 172.21.15.112:0/2984188949 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9ad0155da0 msgr2=0x7f9ad00ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:39:32.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.954+0000 7f9acf7fe700 1 --2- 172.21.15.112:0/2984188949 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9ad0155da0 0x7f9ad00ae9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:32.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.954+0000 7f9ace7fc700 1 -- 172.21.15.112:0/2984188949 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3949362958 0 0) 0x7f9abc0033f0 con 0x7f9ad00aef20 2023-01-24T03:39:32.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.954+0000 7f9acf7fe700 1 -- 172.21.15.112:0/2984188949 >> v1:172.21.15.112:6789/0 conn(0x7f9ad00aef20 legacy=0x7f9ad00ca3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:32.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.954+0000 7f9acf7fe700 1 -- 172.21.15.112:0/2984188949 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad00b1730 con 0x7f9ad00c7e40 2023-01-24T03:39:32.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.954+0000 7f9aceffd700 1 --2- 172.21.15.112:0/2984188949 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9ad0155da0 0x7f9ad00ae9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:39:32.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.955+0000 7f9acf7fe700 1 --2- 172.21.15.112:0/2984188949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ad00c7e40 0x7f9ad00b0aa0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9ac0009760 tx=0x7f9ac0009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70e68fb41e306215 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:32.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.955+0000 7f9ace7fc700 1 -- 172.21.15.112:0/2984188949 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac80017f0 con 0x7f9ad00c7e40 2023-01-24T03:39:32.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.955+0000 7f9ace7fc700 1 -- 172.21.15.112:0/2984188949 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ac0011440 con 0x7f9ad00c7e40 2023-01-24T03:39:32.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.955+0000 7f9ace7fc700 1 -- 172.21.15.112:0/2984188949 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac00117e0 con 0x7f9ad00c7e40 2023-01-24T03:39:32.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.955+0000 7f9ad5b05700 1 -- 172.21.15.112:0/2984188949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ad00c7e40 msgr2=0x7f9ad00b0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:32.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.955+0000 7f9ad5b05700 1 --2- 172.21.15.112:0/2984188949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ad00c7e40 0x7f9ad00b0aa0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9ac0009760 tx=0x7f9ac0009060 comp rx=0 tx=0).stop 2023-01-24T03:39:32.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.955+0000 7f9ad5b05700 1 -- 172.21.15.112:0/2984188949 shutdown_connections 2023-01-24T03:39:32.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.955+0000 7f9ad5b05700 1 --2- 172.21.15.112:0/2984188949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ad00c7e40 0x7f9ad00b0aa0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:32.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.955+0000 7f9ad5b05700 1 --2- 172.21.15.112:0/2984188949 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9ad0155da0 0x7f9ad00ae9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:32.965 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.955+0000 7f9ad5b05700 1 -- 172.21.15.112:0/2984188949 >> 172.21.15.112:0/2984188949 conn(0x7f9ad005e2c0 msgr2=0x7f9ad00cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:39:32.965 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.955+0000 7f9ad5b05700 1 -- 172.21.15.112:0/2984188949 shutdown_connections 2023-01-24T03:39:32.965 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.955+0000 7f9ad5b05700 1 -- 172.21.15.112:0/2984188949 wait complete. 2023-01-24T03:39:32.965 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.956+0000 7f9ad5b05700 1 Processor -- start 2023-01-24T03:39:32.965 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.956+0000 7f9ad5b05700 1 -- start start 2023-01-24T03:39:32.968 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.957+0000 7f9ad5b05700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ad00aef20 0x7f9ad00c4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:32.968 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.957+0000 7f9ad5b05700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9ad00c7e40 0x7f9ad00c1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:32.968 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.957+0000 7f9ad5b05700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ad0155da0 0x7f9ad0165260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:32.968 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.957+0000 7f9ad5b05700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad014e810 con 0x7f9ad00aef20 2023-01-24T03:39:32.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.957+0000 7f9ad5b05700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad00c2240 con 0x7f9ad0155da0 2023-01-24T03:39:32.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.957+0000 7f9ad5b05700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad00cb7f0 con 0x7f9ad00c7e40 2023-01-24T03:39:32.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.957+0000 7f9acf7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ad00aef20 0x7f9ad00c4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:32.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.957+0000 7f9acf7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ad00aef20 0x7f9ad00c4130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55296/0 (socket says 172.21.15.112:55296) 2023-01-24T03:39:32.970 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.957+0000 7f9acf7fe700 1 -- 172.21.15.112:0/4221786964 learned_addr learned my addr 172.21.15.112:0/4221786964 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:39:32.970 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.958+0000 7f9aceffd700 1 --2- 172.21.15.112:0/4221786964 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9ad00c7e40 0x7f9ad00c1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:32.970 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.958+0000 7f9acffff700 1 --2- 172.21.15.112:0/4221786964 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ad0155da0 0x7f9ad0165260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:32.970 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.958+0000 7f9acf7fe700 1 -- 172.21.15.112:0/4221786964 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9ad00c7e40 msgr2=0x7f9ad00c1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:32.970 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.958+0000 7f9acf7fe700 1 --2- 172.21.15.112:0/4221786964 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9ad00c7e40 0x7f9ad00c1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:32.971 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.958+0000 7f9acf7fe700 1 -- 172.21.15.112:0/4221786964 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ad0155da0 msgr2=0x7f9ad0165260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:32.971 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.958+0000 7f9acf7fe700 1 --2- 172.21.15.112:0/4221786964 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ad0155da0 0x7f9ad0165260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:32.971 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.958+0000 7f9acf7fe700 1 -- 172.21.15.112:0/4221786964 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac40092a0 con 0x7f9ad00aef20 2023-01-24T03:39:32.971 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.958+0000 7f9acf7fe700 1 --2- 172.21.15.112:0/4221786964 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ad00aef20 0x7f9ad00c4130 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f9ac0009d50 tx=0x7f9ac0015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:32.971 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.958+0000 7f9ab77fe700 1 -- 172.21.15.112:0/4221786964 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac0010a40 con 0x7f9ad00aef20 2023-01-24T03:39:32.972 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.958+0000 7f9ab77fe700 1 -- 172.21.15.112:0/4221786964 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ac0010ba0 con 0x7f9ad00aef20 2023-01-24T03:39:32.972 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.958+0000 7f9ad5b05700 1 -- 172.21.15.112:0/4221786964 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ac00092a0 con 0x7f9ad00aef20 2023-01-24T03:39:32.972 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.958+0000 7f9ab77fe700 1 -- 172.21.15.112:0/4221786964 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac001d430 con 0x7f9ad00aef20 2023-01-24T03:39:32.972 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.958+0000 7f9ad5b05700 1 -- 172.21.15.112:0/4221786964 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ad00c24e0 con 0x7f9ad00aef20 2023-01-24T03:39:32.972 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.959+0000 7f9ad5b05700 1 -- 172.21.15.112:0/4221786964 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ad004bb80 con 0x7f9ad00aef20 2023-01-24T03:39:32.973 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.959+0000 7f9ab77fe700 1 -- 172.21.15.112:0/4221786964 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f9ac000f070 con 0x7f9ad00aef20 2023-01-24T03:39:32.973 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.959+0000 7f9ab77fe700 1 --2- 172.21.15.112:0/4221786964 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9ab00611c0 0x7f9ab0063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:32.973 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.960+0000 7f9ab77fe700 1 -- 172.21.15.112:0/4221786964 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 7383+0+0 (secure 0 0 0) 0x7f9ac007f070 con 0x7f9ad00aef20 2023-01-24T03:39:32.973 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.960+0000 7f9aceffd700 1 --2- 172.21.15.112:0/4221786964 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9ab00611c0 0x7f9ab0063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:32.973 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.960+0000 7f9aceffd700 1 --2- 172.21.15.112:0/4221786964 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9ab00611c0 0x7f9ab0063670 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9ac4006d80 tx=0x7f9ac4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:32.974 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:32.964+0000 7f9ab77fe700 1 -- 172.21.15.112:0/4221786964 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f9ac004ec70 con 0x7f9ad00aef20 2023-01-24T03:39:33.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:33.190+0000 7f9ad5b05700 1 -- 172.21.15.112:0/4221786964 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ad0069c10 con 0x7f9ad00aef20 2023-01-24T03:39:33.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:33.190+0000 7f9ab77fe700 1 -- 172.21.15.112:0/4221786964 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+5360 (secure 0 0 0) 0x7f9ac0052290 con 0x7f9ad00aef20 2023-01-24T03:39:33.193 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:39:33.193 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:39:28.714628+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":37,"state":"up:replay","state_seq":1,"addr":"172.21.15.142:6835/2136880103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2136880103},{"type":"v1","addr":"172.21.15.142:6835","nonce":2136880103}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:39:33.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:33.192+0000 7f9ad5b05700 1 -- 172.21.15.112:0/4221786964 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9ab00611c0 msgr2=0x7f9ab0063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:33.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:33.192+0000 7f9ad5b05700 1 --2- 172.21.15.112:0/4221786964 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9ab00611c0 0x7f9ab0063670 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9ac4006d80 tx=0x7f9ac4008040 comp rx=0 tx=0).stop 2023-01-24T03:39:33.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:33.192+0000 7f9ad5b05700 1 -- 172.21.15.112:0/4221786964 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ad00aef20 msgr2=0x7f9ad00c4130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:33.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:33.192+0000 7f9ad5b05700 1 --2- 172.21.15.112:0/4221786964 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ad00aef20 0x7f9ad00c4130 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f9ac0009d50 tx=0x7f9ac0015000 comp rx=0 tx=0).stop 2023-01-24T03:39:33.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:33.192+0000 7f9ad5b05700 1 -- 172.21.15.112:0/4221786964 shutdown_connections 2023-01-24T03:39:33.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:33.192+0000 7f9ad5b05700 1 --2- 172.21.15.112:0/4221786964 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9ab00611c0 0x7f9ab0063670 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:33.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:33.192+0000 7f9ad5b05700 1 --2- 172.21.15.112:0/4221786964 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9ad00aef20 0x7f9ad00c4130 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:33.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:33.192+0000 7f9ad5b05700 1 --2- 172.21.15.112:0/4221786964 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9ad0155da0 0x7f9ad0165260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:33.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:33.192+0000 7f9ad5b05700 1 --2- 172.21.15.112:0/4221786964 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9ad00c7e40 0x7f9ad00c1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:33.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:33.192+0000 7f9ad5b05700 1 -- 172.21.15.112:0/4221786964 >> 172.21.15.112:0/4221786964 conn(0x7f9ad005e2c0 msgr2=0x7f9ad00680c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:39:33.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:33.192+0000 7f9ad5b05700 1 -- 172.21.15.112:0/4221786964 shutdown_connections 2023-01-24T03:39:33.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:33.192+0000 7f9ad5b05700 1 -- 172.21.15.112:0/4221786964 wait complete. 2023-01-24T03:39:33.198 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 37 2023-01-24T03:39:33.206 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:39:35.208 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:39:35.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.330+0000 7f06b307b700 1 Processor -- start 2023-01-24T03:39:35.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.331+0000 7f06b307b700 1 -- start start 2023-01-24T03:39:35.336 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.331+0000 7f06b307b700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f06ac155db0 0x7f06ac14efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:35.336 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.331+0000 7f06b307b700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06ac0c54d0 con 0x7f06ac06cf50 2023-01-24T03:39:35.336 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.331+0000 7f06b307b700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06ac0c6070 con 0x7f06ac06d320 2023-01-24T03:39:35.336 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.331+0000 7f06b307b700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06ac0c49c0 con 0x7f06ac155db0 2023-01-24T03:39:35.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.331+0000 7f06b0e17700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f06ac06cf50 0x7f06ac0c5960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:36292/0 (socket says 172.21.15.112:36292) 2023-01-24T03:39:35.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.331+0000 7f06b1618700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f06ac155db0 0x7f06ac14efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:35.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.331+0000 7f06b0e17700 1 -- 172.21.15.112:0/3076974114 learned_addr learned my addr 172.21.15.112:0/3076974114 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:39:35.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.332+0000 7f06b1618700 1 -- 172.21.15.112:0/3076974114 >> v1:172.21.15.142:6789/0 conn(0x7f06ac06d320 legacy=0x7f06ac067f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:35.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.332+0000 7f06b1618700 1 -- 172.21.15.112:0/3076974114 >> v1:172.21.15.112:6789/0 conn(0x7f06ac06cf50 legacy=0x7f06ac0c5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:35.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.332+0000 7f06b1618700 1 -- 172.21.15.112:0/3076974114 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06ac068920 con 0x7f06ac155db0 2023-01-24T03:39:35.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.332+0000 7f06ab7fe700 1 -- 172.21.15.112:0/3076974114 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1387030507 0 0) 0x7f06ac0c54d0 con 0x7f06ac06cf50 2023-01-24T03:39:35.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.332+0000 7f06b1618700 1 --2- 172.21.15.112:0/3076974114 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06ac155db0 0x7f06ac14efd0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f06a4009e60 tx=0x7f06a4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9f1260a062fdefe server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:35.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.332+0000 7f06ab7fe700 1 -- 172.21.15.112:0/3076974114 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a400e070 con 0x7f06ac155db0 2023-01-24T03:39:35.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.332+0000 7f06ab7fe700 1 -- 172.21.15.112:0/3076974114 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a400bc30 con 0x7f06ac155db0 2023-01-24T03:39:35.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.332+0000 7f06ab7fe700 1 -- 172.21.15.112:0/3076974114 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a4012430 con 0x7f06ac155db0 2023-01-24T03:39:35.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.333+0000 7f06b307b700 1 -- 172.21.15.112:0/3076974114 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06ac155db0 msgr2=0x7f06ac14efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:35.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.333+0000 7f06b307b700 1 --2- 172.21.15.112:0/3076974114 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06ac155db0 0x7f06ac14efd0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f06a4009e60 tx=0x7f06a4009780 comp rx=0 tx=0).stop 2023-01-24T03:39:35.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.333+0000 7f06b307b700 1 -- 172.21.15.112:0/3076974114 shutdown_connections 2023-01-24T03:39:35.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.333+0000 7f06b307b700 1 --2- 172.21.15.112:0/3076974114 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06ac155db0 0x7f06ac14efd0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:35.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.333+0000 7f06b307b700 1 -- 172.21.15.112:0/3076974114 >> 172.21.15.112:0/3076974114 conn(0x7f06ac0c9820 msgr2=0x7f06ac0c9c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:39:35.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.333+0000 7f06b307b700 1 -- 172.21.15.112:0/3076974114 shutdown_connections 2023-01-24T03:39:35.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.333+0000 7f06b307b700 1 -- 172.21.15.112:0/3076974114 wait complete. 2023-01-24T03:39:35.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.334+0000 7f06b307b700 1 Processor -- start 2023-01-24T03:39:35.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.334+0000 7f06b307b700 1 -- start start 2023-01-24T03:39:35.344 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.335+0000 7f06b307b700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06ac06cf50 0x7f06ac1664d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:35.344 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.335+0000 7f06b307b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06ac06d320 0x7f06ac166a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:35.345 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.335+0000 7f06b307b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f06ac155db0 0x7f06ac0c1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:35.345 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.335+0000 7f06b307b700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ac06eff0 con 0x7f06ac06d320 2023-01-24T03:39:35.345 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.335+0000 7f06b307b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ac0c12a0 con 0x7f06ac155db0 2023-01-24T03:39:35.345 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.335+0000 7f06b307b700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06ac165d10 con 0x7f06ac06cf50 2023-01-24T03:39:35.345 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.335+0000 7f06abfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06ac06d320 0x7f06ac166a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:35.346 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.335+0000 7f06abfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06ac06d320 0x7f06ac166a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55312/0 (socket says 172.21.15.112:55312) 2023-01-24T03:39:35.346 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.335+0000 7f06abfff700 1 -- 172.21.15.112:0/1085573704 learned_addr learned my addr 172.21.15.112:0/1085573704 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:39:35.346 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.335+0000 7f06b0e17700 1 --2- 172.21.15.112:0/1085573704 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06ac06cf50 0x7f06ac1664d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:35.346 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.335+0000 7f06b1618700 1 --2- 172.21.15.112:0/1085573704 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f06ac155db0 0x7f06ac0c1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:35.346 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.335+0000 7f06abfff700 1 -- 172.21.15.112:0/1085573704 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06ac06cf50 msgr2=0x7f06ac1664d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:35.347 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.335+0000 7f06abfff700 1 --2- 172.21.15.112:0/1085573704 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06ac06cf50 0x7f06ac1664d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:35.347 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.335+0000 7f06abfff700 1 -- 172.21.15.112:0/1085573704 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f06ac155db0 msgr2=0x7f06ac0c1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:35.347 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.335+0000 7f06abfff700 1 --2- 172.21.15.112:0/1085573704 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f06ac155db0 0x7f06ac0c1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:35.347 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.335+0000 7f06abfff700 1 -- 172.21.15.112:0/1085573704 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a4009a20 con 0x7f06ac06d320 2023-01-24T03:39:35.347 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.335+0000 7f06abfff700 1 --2- 172.21.15.112:0/1085573704 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06ac06d320 0x7f06ac166a10 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f06a000c440 tx=0x7f06a0007dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:35.348 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.335+0000 7f06a8ff9700 1 -- 172.21.15.112:0/1085573704 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a0012420 con 0x7f06ac06d320 2023-01-24T03:39:35.348 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.335+0000 7f06a8ff9700 1 -- 172.21.15.112:0/1085573704 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a0012580 con 0x7f06ac06d320 2023-01-24T03:39:35.348 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.335+0000 7f06b307b700 1 -- 172.21.15.112:0/1085573704 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06ac0c0f70 con 0x7f06ac06d320 2023-01-24T03:39:35.348 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.335+0000 7f06a8ff9700 1 -- 172.21.15.112:0/1085573704 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a0012940 con 0x7f06ac06d320 2023-01-24T03:39:35.349 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.335+0000 7f06b307b700 1 -- 172.21.15.112:0/1085573704 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06ac0c3a00 con 0x7f06ac06d320 2023-01-24T03:39:35.349 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.336+0000 7f06b307b700 1 -- 172.21.15.112:0/1085573704 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06ac0c1410 con 0x7f06ac06d320 2023-01-24T03:39:35.349 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.337+0000 7f06a8ff9700 1 -- 172.21.15.112:0/1085573704 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f06a0012d70 con 0x7f06ac06d320 2023-01-24T03:39:35.349 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.337+0000 7f06a8ff9700 1 --2- 172.21.15.112:0/1085573704 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f068c0611f0 0x7f068c0636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:35.349 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.337+0000 7f06b0e17700 1 --2- 172.21.15.112:0/1085573704 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f068c0611f0 0x7f068c0636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:35.350 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.337+0000 7f06a8ff9700 1 -- 172.21.15.112:0/1085573704 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 7383+0+0 (secure 0 0 0) 0x7f06a007cde0 con 0x7f06ac06d320 2023-01-24T03:39:35.350 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.338+0000 7f06b0e17700 1 --2- 172.21.15.112:0/1085573704 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f068c0611f0 0x7f068c0636a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f069c007fd0 tx=0x7f069c007dd0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:35.350 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.341+0000 7f06a8ff9700 1 -- 172.21.15.112:0/1085573704 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f06a004c9e0 con 0x7f06ac06d320 2023-01-24T03:39:35.560 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.558+0000 7f06b307b700 1 -- 172.21.15.112:0/1085573704 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06ac0c4ba0 con 0x7f06ac06d320 2023-01-24T03:39:35.561 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.559+0000 7f06a8ff9700 1 -- 172.21.15.112:0/1085573704 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+5360 (secure 0 0 0) 0x7f06a0050000 con 0x7f06ac06d320 2023-01-24T03:39:35.561 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:39:35.561 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:39:28.714628+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":37,"state":"up:replay","state_seq":1,"addr":"172.21.15.142:6835/2136880103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2136880103},{"type":"v1","addr":"172.21.15.142:6835","nonce":2136880103}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:39:35.562 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.560+0000 7f06b307b700 1 -- 172.21.15.112:0/1085573704 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f068c0611f0 msgr2=0x7f068c0636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:35.563 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.560+0000 7f06b307b700 1 --2- 172.21.15.112:0/1085573704 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f068c0611f0 0x7f068c0636a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f069c007fd0 tx=0x7f069c007dd0 comp rx=0 tx=0).stop 2023-01-24T03:39:35.563 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.560+0000 7f06b307b700 1 -- 172.21.15.112:0/1085573704 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06ac06d320 msgr2=0x7f06ac166a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:35.563 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.560+0000 7f06b307b700 1 --2- 172.21.15.112:0/1085573704 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06ac06d320 0x7f06ac166a10 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f06a000c440 tx=0x7f06a0007dc0 comp rx=0 tx=0).stop 2023-01-24T03:39:35.563 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.560+0000 7f06b307b700 1 -- 172.21.15.112:0/1085573704 shutdown_connections 2023-01-24T03:39:35.564 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.560+0000 7f06b307b700 1 --2- 172.21.15.112:0/1085573704 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f068c0611f0 0x7f068c0636a0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:35.564 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.560+0000 7f06b307b700 1 --2- 172.21.15.112:0/1085573704 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f06ac155db0 0x7f06ac0c1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:35.564 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.560+0000 7f06b307b700 1 --2- 172.21.15.112:0/1085573704 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f06ac06cf50 0x7f06ac1664d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:35.564 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.560+0000 7f06b307b700 1 --2- 172.21.15.112:0/1085573704 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f06ac06d320 0x7f06ac166a10 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:35.564 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.560+0000 7f06b307b700 1 -- 172.21.15.112:0/1085573704 >> 172.21.15.112:0/1085573704 conn(0x7f06ac0c9820 msgr2=0x7f06ac0c5960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:39:35.565 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.560+0000 7f06b307b700 1 -- 172.21.15.112:0/1085573704 shutdown_connections 2023-01-24T03:39:35.565 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:35.560+0000 7f06b307b700 1 -- 172.21.15.112:0/1085573704 wait complete. 2023-01-24T03:39:35.565 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 37 2023-01-24T03:39:35.574 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:39:37.575 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:39:37.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.697+0000 7f3941d14700 1 Processor -- start 2023-01-24T03:39:37.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.697+0000 7f3941d14700 1 -- start start 2023-01-24T03:39:37.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.698+0000 7f3941d14700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f393c0c5680 0x7f393c156730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:37.702 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.698+0000 7f3941d14700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f393c0c5a50 0x7f393c156c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:37.702 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.698+0000 7f3941d14700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f393c0c3d10 con 0x7f393c155e20 2023-01-24T03:39:37.702 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.698+0000 7f3941d14700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f393c0c4eb0 con 0x7f393c0c5680 2023-01-24T03:39:37.702 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.698+0000 7f3941d14700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f393c0c2e20 con 0x7f393c0c5a50 2023-01-24T03:39:37.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.698+0000 7f3940d12700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f393c0c5680 0x7f393c156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:37.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.698+0000 7f3940d12700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f393c0c5680 0x7f393c156730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55320/0 (socket says 172.21.15.112:55320) 2023-01-24T03:39:37.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.698+0000 7f3940d12700 1 -- 172.21.15.112:0/4158122661 learned_addr learned my addr 172.21.15.112:0/4158122661 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:39:37.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.698+0000 7f3941513700 1 --2- 172.21.15.112:0/4158122661 >> v2:172.21.15.142:3300/0 conn(0x7f393c0c5a50 0x7f393c156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:37.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.698+0000 7f3940d12700 1 -- 172.21.15.112:0/4158122661 >> v2:172.21.15.142:3300/0 conn(0x7f393c0c5a50 msgr2=0x7f393c156c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:39:37.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.698+0000 7f3940d12700 1 --2- 172.21.15.112:0/4158122661 >> v2:172.21.15.142:3300/0 conn(0x7f393c0c5a50 0x7f393c156c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:37.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.698+0000 7f3940d12700 1 -- 172.21.15.112:0/4158122661 >> v1:172.21.15.142:6789/0 conn(0x7f393c155e20 legacy=0x7f393c0cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:37.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.698+0000 7f3940d12700 1 -- 172.21.15.112:0/4158122661 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f393c0c3100 con 0x7f393c0c5680 2023-01-24T03:39:37.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.698+0000 7f3941513700 1 --2- 172.21.15.112:0/4158122661 >> v2:172.21.15.142:3300/0 conn(0x7f393c0c5a50 0x7f393c156c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:39:37.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.698+0000 7f3940d12700 1 --2- 172.21.15.112:0/4158122661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f393c0c5680 0x7f393c156730 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f393c0894c0 tx=0x7f3934009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fcc6699e9983170 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:37.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.698+0000 7f393b7fe700 1 -- 172.21.15.112:0/4158122661 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393400e070 con 0x7f393c0c5680 2023-01-24T03:39:37.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.698+0000 7f393b7fe700 1 -- 172.21.15.112:0/4158122661 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f393400bae0 con 0x7f393c0c5680 2023-01-24T03:39:37.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.698+0000 7f393b7fe700 1 -- 172.21.15.112:0/4158122661 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39340092e0 con 0x7f393c0c5680 2023-01-24T03:39:37.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.699+0000 7f3941d14700 1 -- 172.21.15.112:0/4158122661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f393c0c5680 msgr2=0x7f393c156730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:37.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.699+0000 7f3941d14700 1 --2- 172.21.15.112:0/4158122661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f393c0c5680 0x7f393c156730 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f393c0894c0 tx=0x7f3934009840 comp rx=0 tx=0).stop 2023-01-24T03:39:37.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.699+0000 7f3941d14700 1 -- 172.21.15.112:0/4158122661 shutdown_connections 2023-01-24T03:39:37.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.699+0000 7f3941d14700 1 --2- 172.21.15.112:0/4158122661 >> v2:172.21.15.142:3300/0 conn(0x7f393c0c5a50 0x7f393c156c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:37.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.699+0000 7f3941d14700 1 --2- 172.21.15.112:0/4158122661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f393c0c5680 0x7f393c156730 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:37.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.699+0000 7f3941d14700 1 -- 172.21.15.112:0/4158122661 >> 172.21.15.112:0/4158122661 conn(0x7f393c0685b0 msgr2=0x7f393c0689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:39:37.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.699+0000 7f3941d14700 1 -- 172.21.15.112:0/4158122661 shutdown_connections 2023-01-24T03:39:37.707 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.699+0000 7f3941d14700 1 -- 172.21.15.112:0/4158122661 wait complete. 2023-01-24T03:39:37.707 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.700+0000 7f3941d14700 1 Processor -- start 2023-01-24T03:39:37.710 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.700+0000 7f3941d14700 1 -- start start 2023-01-24T03:39:37.710 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.701+0000 7f3941d14700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f393c0c5680 0x7f393c169310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:37.710 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.701+0000 7f3941d14700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f393c0c5a50 0x7f393c151bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:37.710 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.701+0000 7f3941d14700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f393c155e20 0x7f393c0cad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:37.710 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.701+0000 7f3941d14700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393c1653b0 con 0x7f393c0c5680 2023-01-24T03:39:37.711 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.701+0000 7f3941d14700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393c0a8110 con 0x7f393c0c5a50 2023-01-24T03:39:37.711 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.701+0000 7f3941d14700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f393c152bf0 con 0x7f393c155e20 2023-01-24T03:39:37.711 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.701+0000 7f3940d12700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f393c0c5680 0x7f393c169310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:37.711 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.701+0000 7f3940d12700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f393c0c5680 0x7f393c169310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55326/0 (socket says 172.21.15.112:55326) 2023-01-24T03:39:37.711 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.701+0000 7f3940d12700 1 -- 172.21.15.112:0/4180144249 learned_addr learned my addr 172.21.15.112:0/4180144249 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:39:37.712 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.701+0000 7f393bfff700 1 --2- 172.21.15.112:0/4180144249 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f393c0c5a50 0x7f393c151bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:37.712 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.701+0000 7f3941513700 1 --2- 172.21.15.112:0/4180144249 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f393c155e20 0x7f393c0cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:37.712 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.701+0000 7f3940d12700 1 -- 172.21.15.112:0/4180144249 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f393c155e20 msgr2=0x7f393c0cad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:37.712 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.701+0000 7f3940d12700 1 --2- 172.21.15.112:0/4180144249 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f393c155e20 0x7f393c0cad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:37.712 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.701+0000 7f3940d12700 1 -- 172.21.15.112:0/4180144249 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f393c0c5a50 msgr2=0x7f393c151bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:37.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.701+0000 7f3940d12700 1 --2- 172.21.15.112:0/4180144249 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f393c0c5a50 0x7f393c151bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:37.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.701+0000 7f3940d12700 1 -- 172.21.15.112:0/4180144249 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3934009ae0 con 0x7f393c0c5680 2023-01-24T03:39:37.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.701+0000 7f3941513700 1 --2- 172.21.15.112:0/4180144249 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f393c155e20 0x7f393c0cad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:39:37.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.701+0000 7f3940d12700 1 --2- 172.21.15.112:0/4180144249 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f393c0c5680 0x7f393c169310 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f39340033a0 tx=0x7f3934018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:37.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.701+0000 7f3938ff9700 1 -- 172.21.15.112:0/4180144249 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39340088b0 con 0x7f393c0c5680 2023-01-24T03:39:37.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.701+0000 7f3938ff9700 1 -- 172.21.15.112:0/4180144249 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3934008a10 con 0x7f393c0c5680 2023-01-24T03:39:37.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.701+0000 7f3938ff9700 1 -- 172.21.15.112:0/4180144249 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3934008dd0 con 0x7f393c0c5680 2023-01-24T03:39:37.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.701+0000 7f3941d14700 1 -- 172.21.15.112:0/4180144249 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f393c0a78d0 con 0x7f393c0c5680 2023-01-24T03:39:37.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.702+0000 7f3941d14700 1 -- 172.21.15.112:0/4180144249 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f393c0a7b70 con 0x7f393c0c5680 2023-01-24T03:39:37.714 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.702+0000 7f3941d14700 1 -- 172.21.15.112:0/4180144249 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f393c152160 con 0x7f393c0c5680 2023-01-24T03:39:37.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.703+0000 7f3938ff9700 1 -- 172.21.15.112:0/4180144249 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f393402e070 con 0x7f393c0c5680 2023-01-24T03:39:37.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.703+0000 7f3938ff9700 1 --2- 172.21.15.112:0/4180144249 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f39200611c0 0x7f3920063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:37.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.703+0000 7f3938ff9700 1 -- 172.21.15.112:0/4180144249 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 7383+0+0 (secure 0 0 0) 0x7f393407f570 con 0x7f393c0c5680 2023-01-24T03:39:37.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.703+0000 7f393bfff700 1 --2- 172.21.15.112:0/4180144249 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f39200611c0 0x7f3920063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:37.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.704+0000 7f393bfff700 1 --2- 172.21.15.112:0/4180144249 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f39200611c0 0x7f3920063670 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f393c0cb720 tx=0x7f392c007d20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:37.716 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.707+0000 7f3938ff9700 1 -- 172.21.15.112:0/4180144249 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f393404f170 con 0x7f393c0c5680 2023-01-24T03:39:37.934 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.932+0000 7f3941d14700 1 -- 172.21.15.112:0/4180144249 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f393c05ea20 con 0x7f393c0c5680 2023-01-24T03:39:37.934 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.932+0000 7f3938ff9700 1 -- 172.21.15.112:0/4180144249 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+5364 (secure 0 0 0) 0x7f3934052790 con 0x7f393c0c5680 2023-01-24T03:39:37.935 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:39:37.935 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:39:37.870779+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":37,"state":"up:reconnect","state_seq":25,"addr":"172.21.15.142:6835/2136880103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2136880103},{"type":"v1","addr":"172.21.15.142:6835","nonce":2136880103}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:39:37.936 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.934+0000 7f3941d14700 1 -- 172.21.15.112:0/4180144249 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f39200611c0 msgr2=0x7f3920063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:37.936 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.934+0000 7f3941d14700 1 --2- 172.21.15.112:0/4180144249 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f39200611c0 0x7f3920063670 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f393c0cb720 tx=0x7f392c007d20 comp rx=0 tx=0).stop 2023-01-24T03:39:37.937 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.934+0000 7f3941d14700 1 -- 172.21.15.112:0/4180144249 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f393c0c5680 msgr2=0x7f393c169310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:37.937 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.934+0000 7f3941d14700 1 --2- 172.21.15.112:0/4180144249 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f393c0c5680 0x7f393c169310 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f39340033a0 tx=0x7f3934018000 comp rx=0 tx=0).stop 2023-01-24T03:39:37.937 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.934+0000 7f3941d14700 1 -- 172.21.15.112:0/4180144249 shutdown_connections 2023-01-24T03:39:37.937 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.934+0000 7f3941d14700 1 --2- 172.21.15.112:0/4180144249 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f39200611c0 0x7f3920063670 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:37.937 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.934+0000 7f3941d14700 1 --2- 172.21.15.112:0/4180144249 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f393c0c5680 0x7f393c169310 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:37.938 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.934+0000 7f3941d14700 1 --2- 172.21.15.112:0/4180144249 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f393c155e20 0x7f393c0cad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:37.938 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.934+0000 7f3941d14700 1 --2- 172.21.15.112:0/4180144249 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f393c0c5a50 0x7f393c151bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:37.938 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.934+0000 7f3941d14700 1 -- 172.21.15.112:0/4180144249 >> 172.21.15.112:0/4180144249 conn(0x7f393c0685b0 msgr2=0x7f393c0cbd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:39:37.938 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.934+0000 7f3941d14700 1 -- 172.21.15.112:0/4180144249 shutdown_connections 2023-01-24T03:39:37.938 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:37.934+0000 7f3941d14700 1 -- 172.21.15.112:0/4180144249 wait complete. 2023-01-24T03:39:37.939 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 38 2023-01-24T03:39:37.948 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:39:39.949 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:39:40.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.075+0000 7f2c7f9d7700 1 Processor -- start 2023-01-24T03:39:40.078 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.075+0000 7f2c7f9d7700 1 -- start start 2023-01-24T03:39:40.080 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.076+0000 7f2c7f9d7700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f2c780c7e40 0x7f2c780b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:40.081 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.076+0000 7f2c7f9d7700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f2c780ca3e0 0x7f2c780ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:40.081 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.076+0000 7f2c7f9d7700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c78156240 con 0x7f2c78155da0 2023-01-24T03:39:40.081 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.076+0000 7f2c7f9d7700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c780b1170 con 0x7f2c780ca3e0 2023-01-24T03:39:40.081 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.076+0000 7f2c7f9d7700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c780b1450 con 0x7f2c780c7e40 2023-01-24T03:39:40.081 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.076+0000 7f2c7df74700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f2c780ca3e0 0x7f2c780ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:40.082 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.076+0000 7f2c7df74700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f2c780ca3e0 0x7f2c780ca7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55356/0 (socket says 172.21.15.112:55356) 2023-01-24T03:39:40.082 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.076+0000 7f2c7df74700 1 -- 172.21.15.112:0/3114476828 learned_addr learned my addr 172.21.15.112:0/3114476828 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:39:40.082 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.076+0000 7f2c7d773700 1 --2- 172.21.15.112:0/3114476828 >> v2:172.21.15.142:3300/0 conn(0x7f2c780c7e40 0x7f2c780b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:40.082 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.077+0000 7f2c7df74700 1 -- 172.21.15.112:0/3114476828 >> v2:172.21.15.142:3300/0 conn(0x7f2c780c7e40 msgr2=0x7f2c780b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:39:40.083 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.077+0000 7f2c7df74700 1 --2- 172.21.15.112:0/3114476828 >> v2:172.21.15.142:3300/0 conn(0x7f2c780c7e40 0x7f2c780b0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:40.083 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.077+0000 7f2c7df74700 1 -- 172.21.15.112:0/3114476828 >> v1:172.21.15.142:6789/0 conn(0x7f2c78155da0 legacy=0x7f2c780ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:39:40.083 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.077+0000 7f2c7df74700 1 -- 172.21.15.112:0/3114476828 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c780b1730 con 0x7f2c780ca3e0 2023-01-24T03:39:40.084 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.077+0000 7f2c7df74700 1 --2- 172.21.15.112:0/3114476828 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2c780ca3e0 0x7f2c780ca7e0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f2c78089cf0 tx=0x7f2c700090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd0ea09bc26d6246 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:40.084 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.077+0000 7f2c77fff700 1 -- 172.21.15.112:0/3114476828 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c70011630 con 0x7f2c780ca3e0 2023-01-24T03:39:40.084 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.077+0000 7f2c77fff700 1 -- 172.21.15.112:0/3114476828 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c700117d0 con 0x7f2c780ca3e0 2023-01-24T03:39:40.084 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.077+0000 7f2c77fff700 1 -- 172.21.15.112:0/3114476828 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c70011b90 con 0x7f2c780ca3e0 2023-01-24T03:39:40.084 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.077+0000 7f2c7f9d7700 1 -- 172.21.15.112:0/3114476828 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2c780ca3e0 msgr2=0x7f2c780ca7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:40.085 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.077+0000 7f2c7f9d7700 1 --2- 172.21.15.112:0/3114476828 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2c780ca3e0 0x7f2c780ca7e0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f2c78089cf0 tx=0x7f2c700090f0 comp rx=0 tx=0).stop 2023-01-24T03:39:40.085 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.077+0000 7f2c7f9d7700 1 -- 172.21.15.112:0/3114476828 shutdown_connections 2023-01-24T03:39:40.085 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.077+0000 7f2c7f9d7700 1 --2- 172.21.15.112:0/3114476828 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2c780ca3e0 0x7f2c780ca7e0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:40.085 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.077+0000 7f2c7f9d7700 1 --2- 172.21.15.112:0/3114476828 >> v2:172.21.15.142:3300/0 conn(0x7f2c780c7e40 0x7f2c780b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:40.086 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.077+0000 7f2c7f9d7700 1 -- 172.21.15.112:0/3114476828 >> 172.21.15.112:0/3114476828 conn(0x7f2c7805e2c0 msgr2=0x7f2c780cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:39:40.086 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.077+0000 7f2c7f9d7700 1 -- 172.21.15.112:0/3114476828 shutdown_connections 2023-01-24T03:39:40.086 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.077+0000 7f2c7f9d7700 1 -- 172.21.15.112:0/3114476828 wait complete. 2023-01-24T03:39:40.086 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.079+0000 7f2c7f9d7700 1 Processor -- start 2023-01-24T03:39:40.086 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.079+0000 7f2c7f9d7700 1 -- start start 2023-01-24T03:39:40.089 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.079+0000 7f2c7f9d7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2c780c7e40 0x7f2c78165650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:40.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.079+0000 7f2c7f9d7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2c780ca3e0 0x7f2c78165b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:40.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.079+0000 7f2c7f9d7700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2c78155da0 0x7f2c780bfa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:40.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.079+0000 7f2c7f9d7700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c780c0870 con 0x7f2c78155da0 2023-01-24T03:39:40.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.079+0000 7f2c7f9d7700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c78169760 con 0x7f2c780c7e40 2023-01-24T03:39:40.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.079+0000 7f2c7f9d7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c78168890 con 0x7f2c780ca3e0 2023-01-24T03:39:40.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.080+0000 7f2c7df74700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2c78155da0 0x7f2c780bfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:40.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.080+0000 7f2c7df74700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2c78155da0 0x7f2c780bfa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55362/0 (socket says 172.21.15.112:55362) 2023-01-24T03:39:40.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.080+0000 7f2c7df74700 1 -- 172.21.15.112:0/1669316598 learned_addr learned my addr 172.21.15.112:0/1669316598 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:39:40.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.080+0000 7f2c7d773700 1 --2- 172.21.15.112:0/1669316598 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2c780c7e40 0x7f2c78165650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:40.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.080+0000 7f2c7cf72700 1 --2- 172.21.15.112:0/1669316598 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2c780ca3e0 0x7f2c78165b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:40.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.080+0000 7f2c7df74700 1 -- 172.21.15.112:0/1669316598 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2c780ca3e0 msgr2=0x7f2c78165b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:40.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.080+0000 7f2c7df74700 1 --2- 172.21.15.112:0/1669316598 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2c780ca3e0 0x7f2c78165b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:40.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.080+0000 7f2c7df74700 1 -- 172.21.15.112:0/1669316598 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2c780c7e40 msgr2=0x7f2c78165650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:40.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.080+0000 7f2c7df74700 1 --2- 172.21.15.112:0/1669316598 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2c780c7e40 0x7f2c78165650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:40.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.080+0000 7f2c7df74700 1 -- 172.21.15.112:0/1669316598 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c70009370 con 0x7f2c78155da0 2023-01-24T03:39:40.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.080+0000 7f2c7df74700 1 --2- 172.21.15.112:0/1669316598 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2c78155da0 0x7f2c780bfa00 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f2c78089340 tx=0x7f2c70016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:40.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.080+0000 7f2c757fa700 1 -- 172.21.15.112:0/1669316598 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c70011630 con 0x7f2c78155da0 2023-01-24T03:39:40.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.080+0000 7f2c757fa700 1 -- 172.21.15.112:0/1669316598 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c70020070 con 0x7f2c78155da0 2023-01-24T03:39:40.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.080+0000 7f2c7f9d7700 1 -- 172.21.15.112:0/1669316598 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c78168b50 con 0x7f2c78155da0 2023-01-24T03:39:40.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.080+0000 7f2c757fa700 1 -- 172.21.15.112:0/1669316598 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c700276b0 con 0x7f2c78155da0 2023-01-24T03:39:40.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.080+0000 7f2c7f9d7700 1 -- 172.21.15.112:0/1669316598 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c78169110 con 0x7f2c78155da0 2023-01-24T03:39:40.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.081+0000 7f2c7f9d7700 1 -- 172.21.15.112:0/1669316598 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c7804bb80 con 0x7f2c78155da0 2023-01-24T03:39:40.095 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.081+0000 7f2c757fa700 1 -- 172.21.15.112:0/1669316598 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f2c70017020 con 0x7f2c78155da0 2023-01-24T03:39:40.095 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.082+0000 7f2c757fa700 1 --2- 172.21.15.112:0/1669316598 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2c580611c0 0x7f2c58063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:40.095 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.082+0000 7f2c757fa700 1 -- 172.21.15.112:0/1669316598 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 7383+0+0 (secure 0 0 0) 0x7f2c70022070 con 0x7f2c78155da0 2023-01-24T03:39:40.095 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.082+0000 7f2c7d773700 1 --2- 172.21.15.112:0/1669316598 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2c580611c0 0x7f2c58063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:40.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.082+0000 7f2c7d773700 1 --2- 172.21.15.112:0/1669316598 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2c580611c0 0x7f2c58063670 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f2c680075a0 tx=0x7f2c680097e0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:40.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.086+0000 7f2c757fa700 1 -- 172.21.15.112:0/1669316598 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f2c7004f600 con 0x7f2c78155da0 2023-01-24T03:39:40.309 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.307+0000 7f2c7f9d7700 1 -- 172.21.15.112:0/1669316598 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c780c5df0 con 0x7f2c78155da0 2023-01-24T03:39:40.310 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.308+0000 7f2c757fa700 1 -- 172.21.15.112:0/1669316598 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+5364 (secure 0 0 0) 0x7f2c70011b50 con 0x7f2c78155da0 2023-01-24T03:39:40.310 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:39:40.311 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:39:37.870779+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":37,"state":"up:reconnect","state_seq":25,"addr":"172.21.15.142:6835/2136880103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2136880103},{"type":"v1","addr":"172.21.15.142:6835","nonce":2136880103}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:39:40.312 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.310+0000 7f2c7f9d7700 1 -- 172.21.15.112:0/1669316598 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2c580611c0 msgr2=0x7f2c58063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:40.312 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.310+0000 7f2c7f9d7700 1 --2- 172.21.15.112:0/1669316598 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2c580611c0 0x7f2c58063670 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f2c680075a0 tx=0x7f2c680097e0 comp rx=0 tx=0).stop 2023-01-24T03:39:40.312 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.310+0000 7f2c7f9d7700 1 -- 172.21.15.112:0/1669316598 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2c78155da0 msgr2=0x7f2c780bfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:40.313 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.310+0000 7f2c7f9d7700 1 --2- 172.21.15.112:0/1669316598 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2c78155da0 0x7f2c780bfa00 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f2c78089340 tx=0x7f2c70016000 comp rx=0 tx=0).stop 2023-01-24T03:39:40.313 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.310+0000 7f2c7f9d7700 1 -- 172.21.15.112:0/1669316598 shutdown_connections 2023-01-24T03:39:40.313 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.310+0000 7f2c7f9d7700 1 --2- 172.21.15.112:0/1669316598 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f2c580611c0 0x7f2c58063670 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:40.313 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.310+0000 7f2c7f9d7700 1 --2- 172.21.15.112:0/1669316598 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2c78155da0 0x7f2c780bfa00 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:40.314 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.310+0000 7f2c7f9d7700 1 --2- 172.21.15.112:0/1669316598 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2c780c7e40 0x7f2c78165650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:40.314 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.310+0000 7f2c7f9d7700 1 --2- 172.21.15.112:0/1669316598 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f2c780ca3e0 0x7f2c78165b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:40.314 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.310+0000 7f2c7f9d7700 1 -- 172.21.15.112:0/1669316598 >> 172.21.15.112:0/1669316598 conn(0x7f2c7805e2c0 msgr2=0x7f2c780680c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:39:40.314 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.310+0000 7f2c7f9d7700 1 -- 172.21.15.112:0/1669316598 shutdown_connections 2023-01-24T03:39:40.314 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:40.310+0000 7f2c7f9d7700 1 -- 172.21.15.112:0/1669316598 wait complete. 2023-01-24T03:39:40.315 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 38 2023-01-24T03:39:40.324 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:39:42.324 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:39:42.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.447+0000 7f27e8eb3700 1 Processor -- start 2023-01-24T03:39:42.450 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.447+0000 7f27e8eb3700 1 -- start start 2023-01-24T03:39:42.452 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.448+0000 7f27e8eb3700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f27e40caaf0 0x7f27e4155da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:42.453 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.448+0000 7f27e8eb3700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27e40b0f40 con 0x7f27e40b0aa0 2023-01-24T03:39:42.453 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.448+0000 7f27e8eb3700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27e40b1170 con 0x7f27e40c7e40 2023-01-24T03:39:42.453 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.448+0000 7f27e8eb3700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27e40b1450 con 0x7f27e40caaf0 2023-01-24T03:39:42.453 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.448+0000 7f27e2d9d700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f27e40caaf0 0x7f27e4155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:42.454 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.448+0000 7f27e1d9b700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f27e40b0aa0 0x7f27e40ca3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:36356/0 (socket says 172.21.15.112:36356) 2023-01-24T03:39:42.454 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.448+0000 7f27e1d9b700 1 -- 172.21.15.112:0/966773667 learned_addr learned my addr 172.21.15.112:0/966773667 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:39:42.454 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.449+0000 7f27e2d9d700 1 -- 172.21.15.112:0/966773667 >> v1:172.21.15.142:6789/0 conn(0x7f27e40c7e40 legacy=0x7f27e40ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:39:42.454 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.449+0000 7f27e2d9d700 1 -- 172.21.15.112:0/966773667 >> v1:172.21.15.112:6789/0 conn(0x7f27e40b0aa0 legacy=0x7f27e40ca3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:42.456 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.449+0000 7f27e159a700 1 -- 172.21.15.112:0/966773667 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1465109765 0 0) 0x7f27e40b0f40 con 0x7f27e40b0aa0 2023-01-24T03:39:42.456 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.449+0000 7f27e2d9d700 1 -- 172.21.15.112:0/966773667 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e40b1730 con 0x7f27e40caaf0 2023-01-24T03:39:42.456 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.449+0000 7f27e2d9d700 1 --2- 172.21.15.112:0/966773667 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27e40caaf0 0x7f27e4155da0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f27dc009760 tx=0x7f27dc009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55cf579b67c54411 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:42.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.449+0000 7f27e159a700 1 -- 172.21.15.112:0/966773667 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27dc011630 con 0x7f27e40caaf0 2023-01-24T03:39:42.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.449+0000 7f27e159a700 1 -- 172.21.15.112:0/966773667 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27dc011790 con 0x7f27e40caaf0 2023-01-24T03:39:42.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.449+0000 7f27e159a700 1 -- 172.21.15.112:0/966773667 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27dc011b30 con 0x7f27e40caaf0 2023-01-24T03:39:42.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.450+0000 7f27e8eb3700 1 -- 172.21.15.112:0/966773667 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27e40caaf0 msgr2=0x7f27e4155da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:42.457 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.450+0000 7f27e8eb3700 1 --2- 172.21.15.112:0/966773667 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27e40caaf0 0x7f27e4155da0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f27dc009760 tx=0x7f27dc009060 comp rx=0 tx=0).stop 2023-01-24T03:39:42.458 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.450+0000 7f27e8eb3700 1 -- 172.21.15.112:0/966773667 shutdown_connections 2023-01-24T03:39:42.458 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.450+0000 7f27e8eb3700 1 --2- 172.21.15.112:0/966773667 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27e40caaf0 0x7f27e4155da0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:42.458 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.450+0000 7f27e8eb3700 1 -- 172.21.15.112:0/966773667 >> 172.21.15.112:0/966773667 conn(0x7f27e405e2c0 msgr2=0x7f27e40cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:39:42.458 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.450+0000 7f27e8eb3700 1 -- 172.21.15.112:0/966773667 shutdown_connections 2023-01-24T03:39:42.458 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.450+0000 7f27e8eb3700 1 -- 172.21.15.112:0/966773667 wait complete. 2023-01-24T03:39:42.459 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.451+0000 7f27e8eb3700 1 Processor -- start 2023-01-24T03:39:42.459 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.451+0000 7f27e8eb3700 1 -- start start 2023-01-24T03:39:42.462 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.451+0000 7f27e8eb3700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27e40b0aa0 0x7f27e414f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:42.462 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.451+0000 7f27e8eb3700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f27e40c7e40 0x7f27e40c6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:42.462 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.451+0000 7f27e8eb3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f27e40caaf0 0x7f27e40c6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:42.463 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.451+0000 7f27e8eb3700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e40c06a0 con 0x7f27e40b0aa0 2023-01-24T03:39:42.463 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.451+0000 7f27e8eb3700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e40c0210 con 0x7f27e40c7e40 2023-01-24T03:39:42.463 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.451+0000 7f27e8eb3700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27e40c4200 con 0x7f27e40caaf0 2023-01-24T03:39:42.463 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.452+0000 7f27e259c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27e40b0aa0 0x7f27e414f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:42.464 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.452+0000 7f27e259c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27e40b0aa0 0x7f27e414f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55372/0 (socket says 172.21.15.112:55372) 2023-01-24T03:39:42.464 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.452+0000 7f27e259c700 1 -- 172.21.15.112:0/3360241227 learned_addr learned my addr 172.21.15.112:0/3360241227 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:39:42.464 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.452+0000 7f27e1d9b700 1 --2- 172.21.15.112:0/3360241227 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f27e40c7e40 0x7f27e40c6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:42.464 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.452+0000 7f27e259c700 1 -- 172.21.15.112:0/3360241227 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f27e40caaf0 msgr2=0x7f27e40c6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:42.464 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.452+0000 7f27e2d9d700 1 --2- 172.21.15.112:0/3360241227 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f27e40caaf0 0x7f27e40c6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:42.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.452+0000 7f27e259c700 1 --2- 172.21.15.112:0/3360241227 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f27e40caaf0 0x7f27e40c6eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:42.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.452+0000 7f27e259c700 1 -- 172.21.15.112:0/3360241227 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f27e40c7e40 msgr2=0x7f27e40c6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:42.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.452+0000 7f27e259c700 1 --2- 172.21.15.112:0/3360241227 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f27e40c7e40 0x7f27e40c6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:42.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.452+0000 7f27e259c700 1 -- 172.21.15.112:0/3360241227 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27dc0092a0 con 0x7f27e40b0aa0 2023-01-24T03:39:42.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.452+0000 7f27e1d9b700 1 --2- 172.21.15.112:0/3360241227 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f27e40c7e40 0x7f27e40c6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-01-24T03:39:42.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.452+0000 7f27e259c700 1 --2- 172.21.15.112:0/3360241227 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27e40b0aa0 0x7f27e414f0e0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f27d400b7e0 tx=0x7f27d4009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:42.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.452+0000 7f27da7fc700 1 -- 172.21.15.112:0/3360241227 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d400f690 con 0x7f27e40b0aa0 2023-01-24T03:39:42.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.452+0000 7f27da7fc700 1 -- 172.21.15.112:0/3360241227 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27d400f810 con 0x7f27e40b0aa0 2023-01-24T03:39:42.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.452+0000 7f27da7fc700 1 -- 172.21.15.112:0/3360241227 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d400fbd0 con 0x7f27e40b0aa0 2023-01-24T03:39:42.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.452+0000 7f27e8eb3700 1 -- 172.21.15.112:0/3360241227 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27e40c4780 con 0x7f27e40b0aa0 2023-01-24T03:39:42.467 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.452+0000 7f27e8eb3700 1 -- 172.21.15.112:0/3360241227 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27e40bf8e0 con 0x7f27e40b0aa0 2023-01-24T03:39:42.467 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.453+0000 7f27e8eb3700 1 -- 172.21.15.112:0/3360241227 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27e404bb80 con 0x7f27e40b0aa0 2023-01-24T03:39:42.467 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.453+0000 7f27da7fc700 1 -- 172.21.15.112:0/3360241227 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f27d401f070 con 0x7f27e40b0aa0 2023-01-24T03:39:42.467 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.454+0000 7f27da7fc700 1 --2- 172.21.15.112:0/3360241227 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f27c80611c0 0x7f27c8063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:42.467 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.454+0000 7f27da7fc700 1 -- 172.21.15.112:0/3360241227 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 7383+0+0 (secure 0 0 0) 0x7f27d407dd00 con 0x7f27e40b0aa0 2023-01-24T03:39:42.467 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.454+0000 7f27e1d9b700 1 --2- 172.21.15.112:0/3360241227 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f27c80611c0 0x7f27c8063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:42.468 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.454+0000 7f27e1d9b700 1 --2- 172.21.15.112:0/3360241227 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f27c80611c0 0x7f27c8063670 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f27e40ca0f0 tx=0x7f27cc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:42.468 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.458+0000 7f27da7fc700 1 -- 172.21.15.112:0/3360241227 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f27d4081030 con 0x7f27e40b0aa0 2023-01-24T03:39:42.681 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.679+0000 7f27e8eb3700 1 -- 172.21.15.112:0/3360241227 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27e41529e0 con 0x7f27e40b0aa0 2023-01-24T03:39:42.681 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.679+0000 7f27da7fc700 1 -- 172.21.15.112:0/3360241227 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+5364 (secure 0 0 0) 0x7f27d404d900 con 0x7f27e40b0aa0 2023-01-24T03:39:42.682 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:39:42.682 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:39:37.870779+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":37,"state":"up:reconnect","state_seq":25,"addr":"172.21.15.142:6835/2136880103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2136880103},{"type":"v1","addr":"172.21.15.142:6835","nonce":2136880103}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:39:42.684 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.681+0000 7f27e8eb3700 1 -- 172.21.15.112:0/3360241227 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f27c80611c0 msgr2=0x7f27c8063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:42.684 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.682+0000 7f27e8eb3700 1 --2- 172.21.15.112:0/3360241227 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f27c80611c0 0x7f27c8063670 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f27e40ca0f0 tx=0x7f27cc009040 comp rx=0 tx=0).stop 2023-01-24T03:39:42.685 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.682+0000 7f27e8eb3700 1 -- 172.21.15.112:0/3360241227 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27e40b0aa0 msgr2=0x7f27e414f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:42.685 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.682+0000 7f27e8eb3700 1 --2- 172.21.15.112:0/3360241227 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27e40b0aa0 0x7f27e414f0e0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f27d400b7e0 tx=0x7f27d4009310 comp rx=0 tx=0).stop 2023-01-24T03:39:42.685 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.682+0000 7f27e8eb3700 1 -- 172.21.15.112:0/3360241227 shutdown_connections 2023-01-24T03:39:42.685 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.682+0000 7f27e8eb3700 1 --2- 172.21.15.112:0/3360241227 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f27c80611c0 0x7f27c8063670 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:42.685 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.682+0000 7f27e8eb3700 1 --2- 172.21.15.112:0/3360241227 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f27e40b0aa0 0x7f27e414f0e0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:42.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.682+0000 7f27e8eb3700 1 --2- 172.21.15.112:0/3360241227 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f27e40caaf0 0x7f27e40c6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:42.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.682+0000 7f27e8eb3700 1 --2- 172.21.15.112:0/3360241227 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f27e40c7e40 0x7f27e40c6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:42.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.682+0000 7f27e8eb3700 1 -- 172.21.15.112:0/3360241227 >> 172.21.15.112:0/3360241227 conn(0x7f27e405e2c0 msgr2=0x7f27e40cbc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:39:42.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.682+0000 7f27e8eb3700 1 -- 172.21.15.112:0/3360241227 shutdown_connections 2023-01-24T03:39:42.687 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:42.682+0000 7f27e8eb3700 1 -- 172.21.15.112:0/3360241227 wait complete. 2023-01-24T03:39:42.687 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 38 2023-01-24T03:39:42.696 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 38, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-01-24T03:30:20.971710+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 38, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5078": { "addr": "172.21.15.142:6835/2136880103", "addrs": { "addrvec": [ { "addr": "172.21.15.142:6833", "nonce": 2136880103, "type": "v2" }, { "addr": "172.21.15.142:6835", "nonce": 2136880103, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5078, "incarnation": 37, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:reconnect", "state_seq": 25 } }, "last_failure": 0, "last_failure_osd_epoch": 48, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-01-24T03:39:37.870779+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5078 } } } ], "standbys": [ { "addr": "172.21.15.142:6837/2181345869", "addrs": { "addrvec": [ { "addr": "172.21.15.142:6836", "nonce": 2181345869, "type": "v2" }, { "addr": "172.21.15.142:6837", "nonce": 2181345869, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6835/330343927", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6834", "nonce": 330343927, "type": "v2" }, { "addr": "172.21.15.112:6835", "nonce": 330343927, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6837/12176817", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6836", "nonce": 12176817, "type": "v2" }, { "addr": "172.21.15.112:6837", "nonce": 12176817, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4276, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6839/1110983841", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6838", "nonce": 1110983841, "type": "v2" }, { "addr": "172.21.15.112:6839", "nonce": 1110983841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4281, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-01-24T03:39:44.697 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:39:44.822 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.820+0000 7f9b4b53e700 1 Processor -- start 2023-01-24T03:39:44.822 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.820+0000 7f9b4b53e700 1 -- start start 2023-01-24T03:39:44.825 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.821+0000 7f9b4b53e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9b440c7e40 0x7f9b440b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:44.825 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.821+0000 7f9b4b53e700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f9b440ca3e0 0x7f9b440ca820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:44.825 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.821+0000 7f9b4b53e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b44156210 con 0x7f9b44155da0 2023-01-24T03:39:44.826 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.821+0000 7f9b4b53e700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b440cad60 con 0x7f9b440ca3e0 2023-01-24T03:39:44.826 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.821+0000 7f9b4b53e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b440b1170 con 0x7f9b440c7e40 2023-01-24T03:39:44.826 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.821+0000 7f9b49adb700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f9b440ca3e0 0x7f9b440ca820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:44.826 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.821+0000 7f9b49adb700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f9b440ca3e0 0x7f9b440ca820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55382/0 (socket says 172.21.15.112:55382) 2023-01-24T03:39:44.827 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.821+0000 7f9b49adb700 1 -- 172.21.15.112:0/3107350873 learned_addr learned my addr 172.21.15.112:0/3107350873 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:39:44.827 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.821+0000 7f9b492da700 1 --2- 172.21.15.112:0/3107350873 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9b440c7e40 0x7f9b440b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:44.827 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.821+0000 7f9b49adb700 1 -- 172.21.15.112:0/3107350873 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9b440c7e40 msgr2=0x7f9b440b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:39:44.827 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.821+0000 7f9b49adb700 1 --2- 172.21.15.112:0/3107350873 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9b440c7e40 0x7f9b440b0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:44.827 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.821+0000 7f9b49adb700 1 -- 172.21.15.112:0/3107350873 >> v1:172.21.15.142:6789/0 conn(0x7f9b44155da0 legacy=0x7f9b440ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:44.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.821+0000 7f9b49adb700 1 -- 172.21.15.112:0/3107350873 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b440b1450 con 0x7f9b440ca3e0 2023-01-24T03:39:44.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.821+0000 7f9b49adb700 1 --2- 172.21.15.112:0/3107350873 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b440ca3e0 0x7f9b440ca820 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c009f20 tx=0x7f9b3c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70b3af6c86b77a1 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:44.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.821+0000 7f9b33fff700 1 -- 172.21.15.112:0/3107350873 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c00e070 con 0x7f9b440ca3e0 2023-01-24T03:39:44.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.821+0000 7f9b33fff700 1 -- 172.21.15.112:0/3107350873 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b3c00ba80 con 0x7f9b440ca3e0 2023-01-24T03:39:44.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.821+0000 7f9b33fff700 1 -- 172.21.15.112:0/3107350873 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c0092e0 con 0x7f9b440ca3e0 2023-01-24T03:39:44.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.822+0000 7f9b4b53e700 1 -- 172.21.15.112:0/3107350873 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b440ca3e0 msgr2=0x7f9b440ca820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:44.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.822+0000 7f9b4b53e700 1 --2- 172.21.15.112:0/3107350873 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b440ca3e0 0x7f9b440ca820 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c009f20 tx=0x7f9b3c009840 comp rx=0 tx=0).stop 2023-01-24T03:39:44.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.822+0000 7f9b4b53e700 1 -- 172.21.15.112:0/3107350873 shutdown_connections 2023-01-24T03:39:44.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.822+0000 7f9b4b53e700 1 --2- 172.21.15.112:0/3107350873 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b440ca3e0 0x7f9b440ca820 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:44.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.822+0000 7f9b4b53e700 1 --2- 172.21.15.112:0/3107350873 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9b440c7e40 0x7f9b440b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:44.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.822+0000 7f9b4b53e700 1 -- 172.21.15.112:0/3107350873 >> 172.21.15.112:0/3107350873 conn(0x7f9b4405e2c0 msgr2=0x7f9b440cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:39:44.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.822+0000 7f9b4b53e700 1 -- 172.21.15.112:0/3107350873 shutdown_connections 2023-01-24T03:39:44.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.822+0000 7f9b4b53e700 1 -- 172.21.15.112:0/3107350873 wait complete. 2023-01-24T03:39:44.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.823+0000 7f9b4b53e700 1 Processor -- start 2023-01-24T03:39:44.830 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.823+0000 7f9b4b53e700 1 -- start start 2023-01-24T03:39:44.833 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.824+0000 7f9b4b53e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9b440c7e40 0x7f9b44155000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:44.833 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.824+0000 7f9b4b53e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b440ca3e0 0x7f9b44154590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:44.834 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.824+0000 7f9b4b53e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9b44155da0 0x7f9b440c2c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:44.834 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.824+0000 7f9b4b53e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b441653c0 con 0x7f9b440ca3e0 2023-01-24T03:39:44.834 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.824+0000 7f9b4b53e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b441661c0 con 0x7f9b44155da0 2023-01-24T03:39:44.834 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.824+0000 7f9b4b53e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b44165ee0 con 0x7f9b440c7e40 2023-01-24T03:39:44.834 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.824+0000 7f9b48ad9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b440ca3e0 0x7f9b44154590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:44.835 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.824+0000 7f9b48ad9700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b440ca3e0 0x7f9b44154590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55386/0 (socket says 172.21.15.112:55386) 2023-01-24T03:39:44.835 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.824+0000 7f9b48ad9700 1 -- 172.21.15.112:0/642885327 learned_addr learned my addr 172.21.15.112:0/642885327 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:39:44.835 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.824+0000 7f9b492da700 1 --2- 172.21.15.112:0/642885327 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9b440c7e40 0x7f9b44155000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:44.835 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.824+0000 7f9b49adb700 1 --2- 172.21.15.112:0/642885327 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9b44155da0 0x7f9b440c2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:44.835 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.824+0000 7f9b48ad9700 1 -- 172.21.15.112:0/642885327 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9b440c7e40 msgr2=0x7f9b44155000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:44.836 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.824+0000 7f9b48ad9700 1 --2- 172.21.15.112:0/642885327 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9b440c7e40 0x7f9b44155000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:44.836 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.824+0000 7f9b48ad9700 1 -- 172.21.15.112:0/642885327 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9b44155da0 msgr2=0x7f9b440c2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:44.836 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.824+0000 7f9b48ad9700 1 --2- 172.21.15.112:0/642885327 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9b44155da0 0x7f9b440c2c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:44.836 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.824+0000 7f9b48ad9700 1 -- 172.21.15.112:0/642885327 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b3c009ae0 con 0x7f9b440ca3e0 2023-01-24T03:39:44.837 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.824+0000 7f9b48ad9700 1 --2- 172.21.15.112:0/642885327 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b440ca3e0 0x7f9b44154590 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f9b3400c4f0 tx=0x7f9b34007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:44.837 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.825+0000 7f9b317fa700 1 -- 172.21.15.112:0/642885327 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3400ce60 con 0x7f9b440ca3e0 2023-01-24T03:39:44.837 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.825+0000 7f9b317fa700 1 -- 172.21.15.112:0/642885327 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b34011590 con 0x7f9b440ca3e0 2023-01-24T03:39:44.837 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.825+0000 7f9b4b53e700 1 -- 172.21.15.112:0/642885327 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b44155540 con 0x7f9b440ca3e0 2023-01-24T03:39:44.837 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.825+0000 7f9b317fa700 1 -- 172.21.15.112:0/642885327 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b34011930 con 0x7f9b440ca3e0 2023-01-24T03:39:44.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.825+0000 7f9b4b53e700 1 -- 172.21.15.112:0/642885327 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b44165890 con 0x7f9b440ca3e0 2023-01-24T03:39:44.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.825+0000 7f9b4b53e700 1 -- 172.21.15.112:0/642885327 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b4404bb80 con 0x7f9b440ca3e0 2023-01-24T03:39:44.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.826+0000 7f9b317fa700 1 -- 172.21.15.112:0/642885327 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f9b34019030 con 0x7f9b440ca3e0 2023-01-24T03:39:44.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.827+0000 7f9b317fa700 1 --2- 172.21.15.112:0/642885327 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9b240611c0 0x7f9b24063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:44.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.827+0000 7f9b317fa700 1 -- 172.21.15.112:0/642885327 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 7383+0+0 (secure 0 0 0) 0x7f9b3407d000 con 0x7f9b440ca3e0 2023-01-24T03:39:44.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.827+0000 7f9b492da700 1 --2- 172.21.15.112:0/642885327 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9b240611c0 0x7f9b24063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:44.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.827+0000 7f9b492da700 1 --2- 172.21.15.112:0/642885327 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9b240611c0 0x7f9b24063670 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9b4000bb10 tx=0x7f9b40008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:44.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:44.831+0000 7f9b317fa700 1 -- 172.21.15.112:0/642885327 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f9b3404cc80 con 0x7f9b440ca3e0 2023-01-24T03:39:45.055 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:45.053+0000 7f9b4b53e700 1 -- 172.21.15.112:0/642885327 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b440c5df0 con 0x7f9b440ca3e0 2023-01-24T03:39:45.056 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:45.054+0000 7f9b317fa700 1 -- 172.21.15.112:0/642885327 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+5361 (secure 0 0 0) 0x7f9b34017020 con 0x7f9b440ca3e0 2023-01-24T03:39:45.056 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:39:45.056 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:39:43.882693+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":37,"state":"up:rejoin","state_seq":27,"addr":"172.21.15.142:6835/2136880103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2136880103},{"type":"v1","addr":"172.21.15.142:6835","nonce":2136880103}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:39:45.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:45.056+0000 7f9b4b53e700 1 -- 172.21.15.112:0/642885327 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9b240611c0 msgr2=0x7f9b24063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:45.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:45.056+0000 7f9b4b53e700 1 --2- 172.21.15.112:0/642885327 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9b240611c0 0x7f9b24063670 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9b4000bb10 tx=0x7f9b40008040 comp rx=0 tx=0).stop 2023-01-24T03:39:45.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:45.056+0000 7f9b4b53e700 1 -- 172.21.15.112:0/642885327 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b440ca3e0 msgr2=0x7f9b44154590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:45.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:45.056+0000 7f9b4b53e700 1 --2- 172.21.15.112:0/642885327 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b440ca3e0 0x7f9b44154590 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f9b3400c4f0 tx=0x7f9b34007e00 comp rx=0 tx=0).stop 2023-01-24T03:39:45.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:45.056+0000 7f9b4b53e700 1 -- 172.21.15.112:0/642885327 shutdown_connections 2023-01-24T03:39:45.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:45.056+0000 7f9b4b53e700 1 --2- 172.21.15.112:0/642885327 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f9b240611c0 0x7f9b24063670 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:45.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:45.056+0000 7f9b4b53e700 1 --2- 172.21.15.112:0/642885327 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9b44155da0 0x7f9b440c2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:45.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:45.056+0000 7f9b4b53e700 1 --2- 172.21.15.112:0/642885327 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f9b440c7e40 0x7f9b44155000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:45.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:45.056+0000 7f9b4b53e700 1 --2- 172.21.15.112:0/642885327 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9b440ca3e0 0x7f9b44154590 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:45.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:45.056+0000 7f9b4b53e700 1 -- 172.21.15.112:0/642885327 >> 172.21.15.112:0/642885327 conn(0x7f9b4405e2c0 msgr2=0x7f9b440cb8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:39:45.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:45.056+0000 7f9b4b53e700 1 -- 172.21.15.112:0/642885327 shutdown_connections 2023-01-24T03:39:45.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:45.056+0000 7f9b4b53e700 1 -- 172.21.15.112:0/642885327 wait complete. 2023-01-24T03:39:45.061 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 39 2023-01-24T03:39:45.070 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:39:47.071 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:39:47.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.194+0000 7f50410f2700 1 Processor -- start 2023-01-24T03:39:47.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.194+0000 7f50410f2700 1 -- start start 2023-01-24T03:39:47.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.194+0000 7f50410f2700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f503c0b0aa0 0x7f503c155da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:47.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.194+0000 7f50410f2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f503c0ca3e0 0x7f503c0ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:47.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.194+0000 7f50410f2700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f503c0b0f40 con 0x7f503c0c7e40 2023-01-24T03:39:47.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.194+0000 7f50410f2700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f503c0b1170 con 0x7f503c0b0aa0 2023-01-24T03:39:47.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.194+0000 7f50410f2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f503c0b1450 con 0x7f503c0ca3e0 2023-01-24T03:39:47.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.195+0000 7f5033fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f503c0b0aa0 0x7f503c155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:47.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.195+0000 7f503b59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f503c0ca3e0 0x7f503c0ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:47.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.195+0000 7f503b59e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f503c0ca3e0 0x7f503c0ca7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3301/0 says I am v2:172.21.15.112:42698/0 (socket says 172.21.15.112:42698) 2023-01-24T03:39:47.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.195+0000 7f5033fff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f503c0b0aa0 0x7f503c155da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.112:47892/0 (socket says 172.21.15.112:47892) 2023-01-24T03:39:47.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.195+0000 7f503b59e700 1 -- 172.21.15.112:0/3398048245 learned_addr learned my addr 172.21.15.112:0/3398048245 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:39:47.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.195+0000 7f5033fff700 1 -- 172.21.15.112:0/3398048245 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f503c0ca3e0 msgr2=0x7f503c0ca7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:39:47.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.195+0000 7f5033fff700 1 --2- 172.21.15.112:0/3398048245 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f503c0ca3e0 0x7f503c0ca7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:47.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.195+0000 7f5033fff700 1 -- 172.21.15.112:0/3398048245 >> v1:172.21.15.142:6789/0 conn(0x7f503c0c7e40 legacy=0x7f503c0ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:47.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.195+0000 7f5033fff700 1 -- 172.21.15.112:0/3398048245 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f503c0b1730 con 0x7f503c0b0aa0 2023-01-24T03:39:47.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.195+0000 7f5033fff700 1 --2- 172.21.15.112:0/3398048245 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f503c0b0aa0 0x7f503c155da0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5024009760 tx=0x7f5024009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9b0775c5a1a9815 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:47.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.196+0000 7f503a59c700 1 -- 172.21.15.112:0/3398048245 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5024011630 con 0x7f503c0b0aa0 2023-01-24T03:39:47.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.196+0000 7f503a59c700 1 -- 172.21.15.112:0/3398048245 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5024011790 con 0x7f503c0b0aa0 2023-01-24T03:39:47.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.196+0000 7f503a59c700 1 -- 172.21.15.112:0/3398048245 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5024011b50 con 0x7f503c0b0aa0 2023-01-24T03:39:47.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.196+0000 7f50410f2700 1 -- 172.21.15.112:0/3398048245 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f503c0b0aa0 msgr2=0x7f503c155da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:47.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.196+0000 7f50410f2700 1 --2- 172.21.15.112:0/3398048245 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f503c0b0aa0 0x7f503c155da0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5024009760 tx=0x7f5024009060 comp rx=0 tx=0).stop 2023-01-24T03:39:47.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.196+0000 7f50410f2700 1 -- 172.21.15.112:0/3398048245 shutdown_connections 2023-01-24T03:39:47.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.196+0000 7f50410f2700 1 --2- 172.21.15.112:0/3398048245 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f503c0ca3e0 0x7f503c0ca7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:47.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.196+0000 7f50410f2700 1 --2- 172.21.15.112:0/3398048245 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f503c0b0aa0 0x7f503c155da0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:47.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.196+0000 7f50410f2700 1 -- 172.21.15.112:0/3398048245 >> 172.21.15.112:0/3398048245 conn(0x7f503c05e2c0 msgr2=0x7f503c0cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:39:47.204 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.196+0000 7f50410f2700 1 -- 172.21.15.112:0/3398048245 shutdown_connections 2023-01-24T03:39:47.204 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.196+0000 7f50410f2700 1 -- 172.21.15.112:0/3398048245 wait complete. 2023-01-24T03:39:47.207 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.197+0000 7f50410f2700 1 Processor -- start 2023-01-24T03:39:47.207 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.197+0000 7f50410f2700 1 -- start start 2023-01-24T03:39:47.207 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.198+0000 7f50410f2700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f503c0b0aa0 0x7f503c0c1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:47.207 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.198+0000 7f50410f2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f503c0c7e40 0x7f503c14ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:47.207 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.198+0000 7f50410f2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f503c0ca3e0 0x7f503c14f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:47.208 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.198+0000 7f50410f2700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f503c166210 con 0x7f503c0b0aa0 2023-01-24T03:39:47.208 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.198+0000 7f50410f2700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f503c0c3c50 con 0x7f503c0c7e40 2023-01-24T03:39:47.208 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.198+0000 7f50410f2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f503c153b10 con 0x7f503c0ca3e0 2023-01-24T03:39:47.208 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.198+0000 7f503ad9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f503c0b0aa0 0x7f503c0c1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:47.208 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.198+0000 7f503ad9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f503c0b0aa0 0x7f503c0c1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55402/0 (socket says 172.21.15.112:55402) 2023-01-24T03:39:47.209 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.198+0000 7f503ad9d700 1 -- 172.21.15.112:0/3572150879 learned_addr learned my addr 172.21.15.112:0/3572150879 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:39:47.209 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.198+0000 7f5033fff700 1 --2- 172.21.15.112:0/3572150879 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f503c0c7e40 0x7f503c14ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:47.209 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.198+0000 7f503b59e700 1 --2- 172.21.15.112:0/3572150879 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f503c0ca3e0 0x7f503c14f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:47.209 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.198+0000 7f503ad9d700 1 -- 172.21.15.112:0/3572150879 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f503c0ca3e0 msgr2=0x7f503c14f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:47.209 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.198+0000 7f503ad9d700 1 --2- 172.21.15.112:0/3572150879 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f503c0ca3e0 0x7f503c14f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:47.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.198+0000 7f503ad9d700 1 -- 172.21.15.112:0/3572150879 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f503c0c7e40 msgr2=0x7f503c14ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:47.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.198+0000 7f503ad9d700 1 --2- 172.21.15.112:0/3572150879 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f503c0c7e40 0x7f503c14ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:47.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.198+0000 7f503ad9d700 1 -- 172.21.15.112:0/3572150879 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50240092a0 con 0x7f503c0b0aa0 2023-01-24T03:39:47.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.198+0000 7f5033fff700 1 --2- 172.21.15.112:0/3572150879 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f503c0c7e40 0x7f503c14ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:39:47.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.198+0000 7f503ad9d700 1 --2- 172.21.15.112:0/3572150879 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f503c0b0aa0 0x7f503c0c1a10 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f502c00c4f0 tx=0x7f502c007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:47.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.199+0000 7f5032ffd700 1 -- 172.21.15.112:0/3572150879 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f502c00ce60 con 0x7f503c0b0aa0 2023-01-24T03:39:47.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.199+0000 7f5032ffd700 1 -- 172.21.15.112:0/3572150879 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f502c011590 con 0x7f503c0b0aa0 2023-01-24T03:39:47.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.199+0000 7f50410f2700 1 -- 172.21.15.112:0/3572150879 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f503c153df0 con 0x7f503c0b0aa0 2023-01-24T03:39:47.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.199+0000 7f5032ffd700 1 -- 172.21.15.112:0/3572150879 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f502c011930 con 0x7f503c0b0aa0 2023-01-24T03:39:47.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.199+0000 7f50410f2700 1 -- 172.21.15.112:0/3572150879 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f503c1543b0 con 0x7f503c0b0aa0 2023-01-24T03:39:47.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.200+0000 7f50410f2700 1 -- 172.21.15.112:0/3572150879 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f503c04bb80 con 0x7f503c0b0aa0 2023-01-24T03:39:47.212 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.200+0000 7f5032ffd700 1 -- 172.21.15.112:0/3572150879 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f502c019030 con 0x7f503c0b0aa0 2023-01-24T03:39:47.212 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.200+0000 7f5032ffd700 1 --2- 172.21.15.112:0/3572150879 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f50200611f0 0x7f50200636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:47.212 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.200+0000 7f5032ffd700 1 -- 172.21.15.112:0/3572150879 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 7383+0+0 (secure 0 0 0) 0x7f502c07cd30 con 0x7f503c0b0aa0 2023-01-24T03:39:47.212 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.200+0000 7f5033fff700 1 --2- 172.21.15.112:0/3572150879 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f50200611f0 0x7f50200636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:47.212 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.201+0000 7f5033fff700 1 --2- 172.21.15.112:0/3572150879 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f50200611f0 0x7f50200636a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f503c0c9dc0 tx=0x7f5024017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:47.213 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.204+0000 7f5032ffd700 1 -- 172.21.15.112:0/3572150879 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f502c04c840 con 0x7f503c0b0aa0 2023-01-24T03:39:47.426 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.424+0000 7f50410f2700 1 -- 172.21.15.112:0/3572150879 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f503c0c57e0 con 0x7f503c0b0aa0 2023-01-24T03:39:47.426 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.424+0000 7f5032ffd700 1 -- 172.21.15.112:0/3572150879 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+5361 (secure 0 0 0) 0x7f502c017090 con 0x7f503c0b0aa0 2023-01-24T03:39:47.427 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:39:47.427 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:39:43.882693+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":37,"state":"up:rejoin","state_seq":27,"addr":"172.21.15.142:6835/2136880103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2136880103},{"type":"v1","addr":"172.21.15.142:6835","nonce":2136880103}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:39:47.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.426+0000 7f50410f2700 1 -- 172.21.15.112:0/3572150879 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f50200611f0 msgr2=0x7f50200636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:47.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.426+0000 7f50410f2700 1 --2- 172.21.15.112:0/3572150879 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f50200611f0 0x7f50200636a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f503c0c9dc0 tx=0x7f5024017000 comp rx=0 tx=0).stop 2023-01-24T03:39:47.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.426+0000 7f50410f2700 1 -- 172.21.15.112:0/3572150879 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f503c0b0aa0 msgr2=0x7f503c0c1a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:47.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.426+0000 7f50410f2700 1 --2- 172.21.15.112:0/3572150879 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f503c0b0aa0 0x7f503c0c1a10 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f502c00c4f0 tx=0x7f502c007e00 comp rx=0 tx=0).stop 2023-01-24T03:39:47.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.426+0000 7f50410f2700 1 -- 172.21.15.112:0/3572150879 shutdown_connections 2023-01-24T03:39:47.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.426+0000 7f50410f2700 1 --2- 172.21.15.112:0/3572150879 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f50200611f0 0x7f50200636a0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:47.430 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.426+0000 7f50410f2700 1 --2- 172.21.15.112:0/3572150879 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f503c0b0aa0 0x7f503c0c1a10 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:47.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.426+0000 7f50410f2700 1 --2- 172.21.15.112:0/3572150879 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f503c0ca3e0 0x7f503c14f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:47.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.426+0000 7f50410f2700 1 --2- 172.21.15.112:0/3572150879 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f503c0c7e40 0x7f503c14ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:47.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.426+0000 7f50410f2700 1 -- 172.21.15.112:0/3572150879 >> 172.21.15.112:0/3572150879 conn(0x7f503c05e2c0 msgr2=0x7f503c0c8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:39:47.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.426+0000 7f50410f2700 1 -- 172.21.15.112:0/3572150879 shutdown_connections 2023-01-24T03:39:47.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:47.426+0000 7f50410f2700 1 -- 172.21.15.112:0/3572150879 wait complete. 2023-01-24T03:39:47.432 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 39 2023-01-24T03:39:47.440 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:39:49.441 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:39:49.565 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.563+0000 7fdcb7613700 1 Processor -- start 2023-01-24T03:39:49.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.564+0000 7fdcb7613700 1 -- start start 2023-01-24T03:39:49.568 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.564+0000 7fdcb7613700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fdcb00c7e40 0x7fdcb00b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:49.568 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.564+0000 7fdcb7613700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcb0155da0 0x7fdcb00ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:49.568 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.564+0000 7fdcb7613700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcb0156240 con 0x7fdcb00aef20 2023-01-24T03:39:49.569 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.564+0000 7fdcb7613700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcb00b1170 con 0x7fdcb00c7e40 2023-01-24T03:39:49.569 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.564+0000 7fdcb7613700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb00b1450 con 0x7fdcb0155da0 2023-01-24T03:39:49.569 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.565+0000 7fdcb4bae700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcb0155da0 0x7fdcb00ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:49.569 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.565+0000 7fdcb53af700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fdcb00c7e40 0x7fdcb00b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:49.569 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.565+0000 7fdcb5bb0700 1 --1- >> v1:172.21.15.142:6789/0 conn(0x7fdcb00aef20 0x7fdcb00ca3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.112:50194/0 (socket says 172.21.15.112:50194) 2023-01-24T03:39:49.570 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.565+0000 7fdcb5bb0700 1 -- 172.21.15.112:0/3715544972 learned_addr learned my addr 172.21.15.112:0/3715544972 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:39:49.570 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.565+0000 7fdcaffff700 1 -- 172.21.15.112:0/3715544972 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 135202540 0 0) 0x7fdcb0156240 con 0x7fdcb00aef20 2023-01-24T03:39:49.570 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.565+0000 7fdcaffff700 1 -- 172.21.15.112:0/3715544972 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc9c0033f0 con 0x7fdcb00aef20 2023-01-24T03:39:49.570 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.565+0000 7fdcb4bae700 1 -- 172.21.15.112:0/3715544972 >> v2:172.21.15.142:3300/0 conn(0x7fdcb00c7e40 msgr2=0x7fdcb00b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:39:49.570 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.565+0000 7fdcb4bae700 1 --2- 172.21.15.112:0/3715544972 >> v2:172.21.15.142:3300/0 conn(0x7fdcb00c7e40 0x7fdcb00b0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:49.571 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.565+0000 7fdcb4bae700 1 -- 172.21.15.112:0/3715544972 >> v1:172.21.15.142:6789/0 conn(0x7fdcb00aef20 legacy=0x7fdcb00ca3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:49.571 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.565+0000 7fdcb4bae700 1 -- 172.21.15.112:0/3715544972 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb00b1730 con 0x7fdcb0155da0 2023-01-24T03:39:49.571 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.565+0000 7fdcb53af700 1 --2- 172.21.15.112:0/3715544972 >> v2:172.21.15.142:3300/0 conn(0x7fdcb00c7e40 0x7fdcb00b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:39:49.571 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.565+0000 7fdcb4bae700 1 --2- 172.21.15.112:0/3715544972 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcb0155da0 0x7fdcb00ae9e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdca4009760 tx=0x7fdca4009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23d250fb7a3cc33e server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:49.571 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.565+0000 7fdcaffff700 1 -- 172.21.15.112:0/3715544972 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca40105b0 con 0x7fdcb0155da0 2023-01-24T03:39:49.572 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.565+0000 7fdcaffff700 1 -- 172.21.15.112:0/3715544972 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdca4010710 con 0x7fdcb0155da0 2023-01-24T03:39:49.572 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.565+0000 7fdcaffff700 1 -- 172.21.15.112:0/3715544972 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca4010a90 con 0x7fdcb0155da0 2023-01-24T03:39:49.572 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.566+0000 7fdcb7613700 1 -- 172.21.15.112:0/3715544972 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcb0155da0 msgr2=0x7fdcb00ae9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:49.572 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.566+0000 7fdcb7613700 1 --2- 172.21.15.112:0/3715544972 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcb0155da0 0x7fdcb00ae9e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdca4009760 tx=0x7fdca4009060 comp rx=0 tx=0).stop 2023-01-24T03:39:49.573 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.566+0000 7fdcb7613700 1 -- 172.21.15.112:0/3715544972 shutdown_connections 2023-01-24T03:39:49.573 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.566+0000 7fdcb7613700 1 --2- 172.21.15.112:0/3715544972 >> v2:172.21.15.142:3300/0 conn(0x7fdcb00c7e40 0x7fdcb00b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:49.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.566+0000 7fdcb7613700 1 --2- 172.21.15.112:0/3715544972 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcb0155da0 0x7fdcb00ae9e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:49.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.566+0000 7fdcb7613700 1 -- 172.21.15.112:0/3715544972 >> 172.21.15.112:0/3715544972 conn(0x7fdcb005e2c0 msgr2=0x7fdcb00cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:39:49.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.566+0000 7fdcb7613700 1 -- 172.21.15.112:0/3715544972 shutdown_connections 2023-01-24T03:39:49.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.566+0000 7fdcb7613700 1 -- 172.21.15.112:0/3715544972 wait complete. 2023-01-24T03:39:49.579 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.567+0000 7fdcb7613700 1 Processor -- start 2023-01-24T03:39:49.580 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.567+0000 7fdcb7613700 1 -- start start 2023-01-24T03:39:49.580 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.568+0000 7fdcb7613700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdcb00aef20 0x7fdcb0168420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:49.581 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.568+0000 7fdcb7613700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcb00c7e40 0x7fdcb0068280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:49.581 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.568+0000 7fdcb7613700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcb0155da0 0x7fdcb014fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:49.581 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.568+0000 7fdcb7613700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb0152330 con 0x7fdcb00aef20 2023-01-24T03:39:49.581 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.568+0000 7fdcb7613700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb00c4a80 con 0x7fdcb0155da0 2023-01-24T03:39:49.582 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.568+0000 7fdcb7613700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb00cb470 con 0x7fdcb00c7e40 2023-01-24T03:39:49.582 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.568+0000 7fdcb53af700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdcb00aef20 0x7fdcb0168420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:49.582 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.568+0000 7fdcb53af700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdcb00aef20 0x7fdcb0168420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55418/0 (socket says 172.21.15.112:55418) 2023-01-24T03:39:49.583 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.568+0000 7fdcb53af700 1 -- 172.21.15.112:0/3186318083 learned_addr learned my addr 172.21.15.112:0/3186318083 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:39:49.583 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.568+0000 7fdcb5bb0700 1 --2- 172.21.15.112:0/3186318083 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcb0155da0 0x7fdcb014fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:49.583 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.568+0000 7fdcb4bae700 1 --2- 172.21.15.112:0/3186318083 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcb00c7e40 0x7fdcb0068280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:49.583 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.568+0000 7fdcb53af700 1 -- 172.21.15.112:0/3186318083 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcb00c7e40 msgr2=0x7fdcb0068280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:49.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.568+0000 7fdcb53af700 1 --2- 172.21.15.112:0/3186318083 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcb00c7e40 0x7fdcb0068280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:49.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.568+0000 7fdcb53af700 1 -- 172.21.15.112:0/3186318083 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcb0155da0 msgr2=0x7fdcb014fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:49.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.568+0000 7fdcb53af700 1 --2- 172.21.15.112:0/3186318083 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcb0155da0 0x7fdcb014fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:49.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.568+0000 7fdcb53af700 1 -- 172.21.15.112:0/3186318083 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca0009a20 con 0x7fdcb00aef20 2023-01-24T03:39:49.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.568+0000 7fdcb53af700 1 --2- 172.21.15.112:0/3186318083 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdcb00aef20 0x7fdcb0168420 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fdca000c170 tx=0x7fdca000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:49.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.568+0000 7fdcad7fa700 1 -- 172.21.15.112:0/3186318083 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca000cd40 con 0x7fdcb00aef20 2023-01-24T03:39:49.585 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.568+0000 7fdcad7fa700 1 -- 172.21.15.112:0/3186318083 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdca000f040 con 0x7fdcb00aef20 2023-01-24T03:39:49.586 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.568+0000 7fdcb7613700 1 -- 172.21.15.112:0/3186318083 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdca40092a0 con 0x7fdcb00aef20 2023-01-24T03:39:49.586 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.569+0000 7fdcad7fa700 1 -- 172.21.15.112:0/3186318083 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca0014690 con 0x7fdcb00aef20 2023-01-24T03:39:49.586 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.569+0000 7fdcb7613700 1 -- 172.21.15.112:0/3186318083 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcb0151d70 con 0x7fdcb00aef20 2023-01-24T03:39:49.586 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.570+0000 7fdcb7613700 1 -- 172.21.15.112:0/3186318083 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcb004bb80 con 0x7fdcb00aef20 2023-01-24T03:39:49.586 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.570+0000 7fdcad7fa700 1 -- 172.21.15.112:0/3186318083 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fdca0014a00 con 0x7fdcb00aef20 2023-01-24T03:39:49.587 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.570+0000 7fdcad7fa700 1 --2- 172.21.15.112:0/3186318083 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fdc900611c0 0x7fdc90063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:49.587 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.570+0000 7fdcad7fa700 1 -- 172.21.15.112:0/3186318083 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 7383+0+0 (secure 0 0 0) 0x7fdca007ebf0 con 0x7fdcb00aef20 2023-01-24T03:39:49.587 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.571+0000 7fdcb4bae700 1 --2- 172.21.15.112:0/3186318083 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fdc900611c0 0x7fdc90063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:49.587 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.571+0000 7fdcb4bae700 1 --2- 172.21.15.112:0/3186318083 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fdc900611c0 0x7fdc90063670 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fdca4007fd0 tx=0x7fdca4016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:49.588 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.575+0000 7fdcad7fa700 1 -- 172.21.15.112:0/3186318083 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fdca004e7f0 con 0x7fdcb00aef20 2023-01-24T03:39:49.795 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.793+0000 7fdcb7613700 1 -- 172.21.15.112:0/3186318083 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdcb00c5df0 con 0x7fdcb00aef20 2023-01-24T03:39:49.796 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.793+0000 7fdcad7fa700 1 -- 172.21.15.112:0/3186318083 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+5361 (secure 0 0 0) 0x7fdca000cea0 con 0x7fdcb00aef20 2023-01-24T03:39:49.796 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:39:49.796 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:39:43.882693+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":37,"state":"up:rejoin","state_seq":27,"addr":"172.21.15.142:6835/2136880103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2136880103},{"type":"v1","addr":"172.21.15.142:6835","nonce":2136880103}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:39:49.798 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.795+0000 7fdcb7613700 1 -- 172.21.15.112:0/3186318083 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fdc900611c0 msgr2=0x7fdc90063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:49.798 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.795+0000 7fdcb7613700 1 --2- 172.21.15.112:0/3186318083 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fdc900611c0 0x7fdc90063670 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fdca4007fd0 tx=0x7fdca4016000 comp rx=0 tx=0).stop 2023-01-24T03:39:49.798 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.795+0000 7fdcb7613700 1 -- 172.21.15.112:0/3186318083 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdcb00aef20 msgr2=0x7fdcb0168420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:49.798 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.795+0000 7fdcb7613700 1 --2- 172.21.15.112:0/3186318083 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdcb00aef20 0x7fdcb0168420 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fdca000c170 tx=0x7fdca000b040 comp rx=0 tx=0).stop 2023-01-24T03:39:49.798 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.795+0000 7fdcb7613700 1 -- 172.21.15.112:0/3186318083 shutdown_connections 2023-01-24T03:39:49.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.795+0000 7fdcb7613700 1 --2- 172.21.15.112:0/3186318083 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fdc900611c0 0x7fdc90063670 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:49.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.795+0000 7fdcb7613700 1 --2- 172.21.15.112:0/3186318083 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdcb00aef20 0x7fdcb0168420 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:49.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.795+0000 7fdcb7613700 1 --2- 172.21.15.112:0/3186318083 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdcb0155da0 0x7fdcb014fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:49.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.795+0000 7fdcb7613700 1 --2- 172.21.15.112:0/3186318083 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdcb00c7e40 0x7fdcb0068280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:49.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.795+0000 7fdcb7613700 1 -- 172.21.15.112:0/3186318083 >> 172.21.15.112:0/3186318083 conn(0x7fdcb005e2c0 msgr2=0x7fdcb00caba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:39:49.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.795+0000 7fdcb7613700 1 -- 172.21.15.112:0/3186318083 shutdown_connections 2023-01-24T03:39:49.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:49.795+0000 7fdcb7613700 1 -- 172.21.15.112:0/3186318083 wait complete. 2023-01-24T03:39:49.800 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 39 2023-01-24T03:39:49.809 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:39:51.810 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:39:51.936 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.934+0000 7f4cf1754700 1 Processor -- start 2023-01-24T03:39:51.937 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.934+0000 7f4cf1754700 1 -- start start 2023-01-24T03:39:51.939 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.935+0000 7f4cf1754700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4cec0c7e40 0x7f4cec0b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:51.939 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.935+0000 7f4cf1754700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f4cec0ca3e0 0x7f4cec0ca820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:51.939 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.935+0000 7f4cf1754700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cec156210 con 0x7f4cec155da0 2023-01-24T03:39:51.940 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.935+0000 7f4cf1754700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cec0cad60 con 0x7f4cec0ca3e0 2023-01-24T03:39:51.940 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.935+0000 7f4cf1754700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cec0b1170 con 0x7f4cec0c7e40 2023-01-24T03:39:51.940 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.935+0000 7f4ceb7fe700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f4cec0ca3e0 0x7f4cec0ca820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:51.940 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.935+0000 7f4ceb7fe700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f4cec0ca3e0 0x7f4cec0ca820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55426/0 (socket says 172.21.15.112:55426) 2023-01-24T03:39:51.940 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.935+0000 7f4ceb7fe700 1 -- 172.21.15.112:0/2574983084 learned_addr learned my addr 172.21.15.112:0/2574983084 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:39:51.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.935+0000 7f4ceaffd700 1 --2- 172.21.15.112:0/2574983084 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4cec0c7e40 0x7f4cec0b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:51.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.936+0000 7f4ceb7fe700 1 -- 172.21.15.112:0/2574983084 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4cec0c7e40 msgr2=0x7f4cec0b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:39:51.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.936+0000 7f4ceb7fe700 1 --2- 172.21.15.112:0/2574983084 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4cec0c7e40 0x7f4cec0b0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:51.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.936+0000 7f4ceb7fe700 1 -- 172.21.15.112:0/2574983084 >> v1:172.21.15.142:6789/0 conn(0x7f4cec155da0 legacy=0x7f4cec0ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:39:51.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.936+0000 7f4ceb7fe700 1 -- 172.21.15.112:0/2574983084 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cec0b1450 con 0x7f4cec0ca3e0 2023-01-24T03:39:51.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.936+0000 7f4ceb7fe700 1 --2- 172.21.15.112:0/2574983084 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4cec0ca3e0 0x7f4cec0ca820 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4009f20 tx=0x7f4ce4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87fe72f584633281 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:51.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.936+0000 7f4cea7fc700 1 -- 172.21.15.112:0/2574983084 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce400e070 con 0x7f4cec0ca3e0 2023-01-24T03:39:51.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.936+0000 7f4cea7fc700 1 -- 172.21.15.112:0/2574983084 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ce400ba40 con 0x7f4cec0ca3e0 2023-01-24T03:39:51.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.936+0000 7f4cea7fc700 1 -- 172.21.15.112:0/2574983084 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce40092e0 con 0x7f4cec0ca3e0 2023-01-24T03:39:51.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.936+0000 7f4cf1754700 1 -- 172.21.15.112:0/2574983084 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4cec0ca3e0 msgr2=0x7f4cec0ca820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:51.943 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.936+0000 7f4cf1754700 1 --2- 172.21.15.112:0/2574983084 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4cec0ca3e0 0x7f4cec0ca820 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f4ce4009f20 tx=0x7f4ce4009840 comp rx=0 tx=0).stop 2023-01-24T03:39:51.943 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.936+0000 7f4cf1754700 1 -- 172.21.15.112:0/2574983084 shutdown_connections 2023-01-24T03:39:51.943 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.936+0000 7f4cf1754700 1 --2- 172.21.15.112:0/2574983084 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4cec0ca3e0 0x7f4cec0ca820 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:51.943 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.936+0000 7f4cf1754700 1 --2- 172.21.15.112:0/2574983084 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4cec0c7e40 0x7f4cec0b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:51.943 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.936+0000 7f4cf1754700 1 -- 172.21.15.112:0/2574983084 >> 172.21.15.112:0/2574983084 conn(0x7f4cec05e2c0 msgr2=0x7f4cec0cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:39:51.944 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.936+0000 7f4cf1754700 1 -- 172.21.15.112:0/2574983084 shutdown_connections 2023-01-24T03:39:51.944 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.936+0000 7f4cf1754700 1 -- 172.21.15.112:0/2574983084 wait complete. 2023-01-24T03:39:51.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.938+0000 7f4cf1754700 1 Processor -- start 2023-01-24T03:39:51.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.938+0000 7f4cf1754700 1 -- start start 2023-01-24T03:39:51.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.938+0000 7f4cf1754700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4cec0c7e40 0x7f4cec0c2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:51.948 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.938+0000 7f4cf1754700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4cec0ca3e0 0x7f4cec1656f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:51.948 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.938+0000 7f4cf1754700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4cec155da0 0x7f4cec165c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:51.948 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.938+0000 7f4cf1754700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cec1652e0 con 0x7f4cec0ca3e0 2023-01-24T03:39:51.948 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.938+0000 7f4cf1754700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cec068490 con 0x7f4cec155da0 2023-01-24T03:39:51.948 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.938+0000 7f4cf1754700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cec0cbf20 con 0x7f4cec0c7e40 2023-01-24T03:39:51.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.938+0000 7f4ce27fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4cec0ca3e0 0x7f4cec1656f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:51.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.938+0000 7f4ce27fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4cec0ca3e0 0x7f4cec1656f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55432/0 (socket says 172.21.15.112:55432) 2023-01-24T03:39:51.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.938+0000 7f4ce27fc700 1 -- 172.21.15.112:0/641986753 learned_addr learned my addr 172.21.15.112:0/641986753 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:39:51.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.939+0000 7f4ceb7fe700 1 --2- 172.21.15.112:0/641986753 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4cec155da0 0x7f4cec165c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:51.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.939+0000 7f4ceaffd700 1 --2- 172.21.15.112:0/641986753 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4cec0c7e40 0x7f4cec0c2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:51.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.939+0000 7f4ce27fc700 1 -- 172.21.15.112:0/641986753 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4cec0c7e40 msgr2=0x7f4cec0c2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:51.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.939+0000 7f4ce27fc700 1 --2- 172.21.15.112:0/641986753 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4cec0c7e40 0x7f4cec0c2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:51.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.939+0000 7f4ce27fc700 1 -- 172.21.15.112:0/641986753 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4cec155da0 msgr2=0x7f4cec165c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:51.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.939+0000 7f4ce27fc700 1 --2- 172.21.15.112:0/641986753 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4cec155da0 0x7f4cec165c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:51.950 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.939+0000 7f4ce27fc700 1 -- 172.21.15.112:0/641986753 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce4009ae0 con 0x7f4cec0ca3e0 2023-01-24T03:39:51.951 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.939+0000 7f4ce27fc700 1 --2- 172.21.15.112:0/641986753 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4cec0ca3e0 0x7f4cec1656f0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f4cd400c440 tx=0x7f4cd4007de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:51.951 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.939+0000 7f4ce37fe700 1 -- 172.21.15.112:0/641986753 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd4012420 con 0x7f4cec0ca3e0 2023-01-24T03:39:51.951 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.939+0000 7f4ce37fe700 1 -- 172.21.15.112:0/641986753 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cd40125a0 con 0x7f4cec0ca3e0 2023-01-24T03:39:51.951 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.939+0000 7f4cf1754700 1 -- 172.21.15.112:0/641986753 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cec0c4b10 con 0x7f4cec0ca3e0 2023-01-24T03:39:51.951 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.939+0000 7f4ce37fe700 1 -- 172.21.15.112:0/641986753 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd4012960 con 0x7f4cec0ca3e0 2023-01-24T03:39:51.951 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.939+0000 7f4cf1754700 1 -- 172.21.15.112:0/641986753 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cec0c50d0 con 0x7f4cec0ca3e0 2023-01-24T03:39:51.952 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.940+0000 7f4cf1754700 1 -- 172.21.15.112:0/641986753 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cec04bb80 con 0x7f4cec0ca3e0 2023-01-24T03:39:51.952 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.940+0000 7f4ce37fe700 1 -- 172.21.15.112:0/641986753 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f4cd4010050 con 0x7f4cec0ca3e0 2023-01-24T03:39:51.952 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.941+0000 7f4ce37fe700 1 --2- 172.21.15.112:0/641986753 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f4cd00611f0 0x7f4cd00636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:39:51.952 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.941+0000 7f4ce37fe700 1 -- 172.21.15.112:0/641986753 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 7383+0+0 (secure 0 0 0) 0x7f4cd407d410 con 0x7f4cec0ca3e0 2023-01-24T03:39:51.952 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.941+0000 7f4ceaffd700 1 --2- 172.21.15.112:0/641986753 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f4cd00611f0 0x7f4cd00636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:39:51.953 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.941+0000 7f4ceaffd700 1 --2- 172.21.15.112:0/641986753 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f4cd00611f0 0x7f4cd00636a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc00ab00 tx=0x7f4cdc006f90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:39:51.953 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:51.945+0000 7f4ce37fe700 1 -- 172.21.15.112:0/641986753 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f4cd404d010 con 0x7f4cec0ca3e0 2023-01-24T03:39:52.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:52.164+0000 7f4cf1754700 1 -- 172.21.15.112:0/641986753 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4cec06efe0 con 0x7f4cec0ca3e0 2023-01-24T03:39:52.167 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:52.165+0000 7f4ce37fe700 1 -- 172.21.15.112:0/641986753 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+5361 (secure 0 0 0) 0x7f4cd4050630 con 0x7f4cec0ca3e0 2023-01-24T03:39:52.167 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:39:52.168 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:39:49.907279+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":29,"addr":"172.21.15.142:6835/2136880103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2136880103},{"type":"v1","addr":"172.21.15.142:6835","nonce":2136880103}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:39:52.169 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:52.167+0000 7f4cf1754700 1 -- 172.21.15.112:0/641986753 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f4cd00611f0 msgr2=0x7f4cd00636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:52.169 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:52.167+0000 7f4cf1754700 1 --2- 172.21.15.112:0/641986753 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f4cd00611f0 0x7f4cd00636a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc00ab00 tx=0x7f4cdc006f90 comp rx=0 tx=0).stop 2023-01-24T03:39:52.169 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:52.167+0000 7f4cf1754700 1 -- 172.21.15.112:0/641986753 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4cec0ca3e0 msgr2=0x7f4cec1656f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:39:52.170 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:52.167+0000 7f4cf1754700 1 --2- 172.21.15.112:0/641986753 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4cec0ca3e0 0x7f4cec1656f0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f4cd400c440 tx=0x7f4cd4007de0 comp rx=0 tx=0).stop 2023-01-24T03:39:52.170 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:52.167+0000 7f4cf1754700 1 -- 172.21.15.112:0/641986753 shutdown_connections 2023-01-24T03:39:52.170 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:52.167+0000 7f4cf1754700 1 --2- 172.21.15.112:0/641986753 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f4cd00611f0 0x7f4cd00636a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:52.170 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:52.167+0000 7f4cf1754700 1 --2- 172.21.15.112:0/641986753 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4cec155da0 0x7f4cec165c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:52.170 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:52.167+0000 7f4cf1754700 1 --2- 172.21.15.112:0/641986753 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f4cec0c7e40 0x7f4cec0c2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:52.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:52.167+0000 7f4cf1754700 1 --2- 172.21.15.112:0/641986753 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4cec0ca3e0 0x7f4cec1656f0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:39:52.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:52.167+0000 7f4cf1754700 1 -- 172.21.15.112:0/641986753 >> 172.21.15.112:0/641986753 conn(0x7f4cec05e2c0 msgr2=0x7f4cec0c6e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:39:52.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:52.167+0000 7f4cf1754700 1 -- 172.21.15.112:0/641986753 shutdown_connections 2023-01-24T03:39:52.171 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:39:52.167+0000 7f4cf1754700 1 -- 172.21.15.112:0/641986753 wait complete. 2023-01-24T03:39:52.172 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 40 2023-01-24T03:39:52.181 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=0, replacing gid=5296 2023-01-24T03:39:52.181 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before reviving mds.b 2023-01-24T03:39:58.783 INFO:tasks.ceph.mds.d.smithi142.stderr:2023-01-24T03:39:58.781+0000 7ff9b64ca700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-01-24T03:40:00.182 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2023-01-24T03:40:00.183 INFO:tasks.ceph.mds.b:Restarting daemon 2023-01-24T03:40:00.183 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-01-24T03:40:00.186 INFO:tasks.ceph.mds.b:Started 2023-01-24T03:40:00.186 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:40:00.258 INFO:tasks.ceph.mds.b.smithi142.stdout:starting mds.b at 2023-01-24T03:40:00.300 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:40:00.302 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:40:00.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.312+0000 7f969467e700 1 Processor -- start 2023-01-24T03:40:00.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.312+0000 7f969467e700 1 -- start start 2023-01-24T03:40:00.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.313+0000 7f969467e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f968c0c6f60 0x7f968c0c24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:00.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.313+0000 7f969467e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f968c0c2010 con 0x7f968c06a890 2023-01-24T03:40:00.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.313+0000 7f969467e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f968c0c4a60 con 0x7f968c0c29e0 2023-01-24T03:40:00.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.313+0000 7f969467e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f968c0c5e10 con 0x7f968c0c6f60 2023-01-24T03:40:00.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.313+0000 7f969241a700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f968c06a890 0x7f968c06ac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:36540/0 (socket says 172.21.15.112:36540) 2023-01-24T03:40:00.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.313+0000 7f969241a700 1 -- 172.21.15.112:0/2567963714 learned_addr learned my addr 172.21.15.112:0/2567963714 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:40:00.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.313+0000 7f9691c19700 1 --2- 172.21.15.112:0/2567963714 >> v2:172.21.15.142:3300/0 conn(0x7f968c0c6f60 0x7f968c0c24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:00.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.314+0000 7f9691418700 1 -- 172.21.15.112:0/2567963714 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2079331720 0 0) 0x7f968c0c2010 con 0x7f968c06a890 2023-01-24T03:40:00.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.314+0000 7f9691418700 1 -- 172.21.15.112:0/2567963714 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96740033f0 con 0x7f968c06a890 2023-01-24T03:40:00.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.314+0000 7f9691c19700 1 -- 172.21.15.112:0/2567963714 >> v1:172.21.15.142:6789/0 conn(0x7f968c0c29e0 legacy=0x7f968c151f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:00.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.314+0000 7f9691c19700 1 -- 172.21.15.112:0/2567963714 >> v1:172.21.15.112:6789/0 conn(0x7f968c06a890 legacy=0x7f968c06ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:00.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.314+0000 7f9691c19700 1 -- 172.21.15.112:0/2567963714 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f968c0c2e40 con 0x7f968c0c6f60 2023-01-24T03:40:00.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.314+0000 7f9691c19700 1 --2- 172.21.15.112:0/2567963714 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f968c0c6f60 0x7f968c0c24a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f96880096e0 tx=0x7f9688009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1e79db852ab7273 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:00.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.314+0000 7f9691418700 1 -- 172.21.15.112:0/2567963714 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96880105b0 con 0x7f968c0c6f60 2023-01-24T03:40:00.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.314+0000 7f9691418700 1 -- 172.21.15.112:0/2567963714 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9688010710 con 0x7f968c0c6f60 2023-01-24T03:40:00.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.314+0000 7f9691418700 1 -- 172.21.15.112:0/2567963714 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9688010a90 con 0x7f968c0c6f60 2023-01-24T03:40:00.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.315+0000 7f969467e700 1 -- 172.21.15.112:0/2567963714 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f968c0c6f60 msgr2=0x7f968c0c24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:00.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.315+0000 7f969467e700 1 --2- 172.21.15.112:0/2567963714 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f968c0c6f60 0x7f968c0c24a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f96880096e0 tx=0x7f9688009060 comp rx=0 tx=0).stop 2023-01-24T03:40:00.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.315+0000 7f969467e700 1 -- 172.21.15.112:0/2567963714 shutdown_connections 2023-01-24T03:40:00.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.315+0000 7f969467e700 1 --2- 172.21.15.112:0/2567963714 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f968c0c6f60 0x7f968c0c24a0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:00.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.315+0000 7f969467e700 1 -- 172.21.15.112:0/2567963714 >> 172.21.15.112:0/2567963714 conn(0x7f968c066e10 msgr2=0x7f968c067210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:00.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.315+0000 7f969467e700 1 -- 172.21.15.112:0/2567963714 shutdown_connections 2023-01-24T03:40:00.323 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.315+0000 7f969467e700 1 -- 172.21.15.112:0/2567963714 wait complete. 2023-01-24T03:40:00.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.316+0000 7f969467e700 1 Processor -- start 2023-01-24T03:40:00.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.316+0000 7f969467e700 1 -- start start 2023-01-24T03:40:00.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.317+0000 7f969467e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f968c06a890 0x7f968c160b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:00.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.317+0000 7f969467e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f968c0c29e0 0x7f968c163870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:00.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.317+0000 7f969467e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f968c0c6f60 0x7f968c06dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:00.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.317+0000 7f969467e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968c0c4fe0 con 0x7f968c0c6f60 2023-01-24T03:40:00.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.317+0000 7f969467e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968c06da10 con 0x7f968c0c29e0 2023-01-24T03:40:00.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.317+0000 7f969467e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f968c1611f0 con 0x7f968c06a890 2023-01-24T03:40:00.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.317+0000 7f9692c1b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f968c0c6f60 0x7f968c06dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:00.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.317+0000 7f9692c1b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f968c0c6f60 0x7f968c06dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55562/0 (socket says 172.21.15.112:55562) 2023-01-24T03:40:00.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.317+0000 7f9692c1b700 1 -- 172.21.15.112:0/2031671868 learned_addr learned my addr 172.21.15.112:0/2031671868 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:40:00.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.317+0000 7f9691c19700 1 --2- 172.21.15.112:0/2031671868 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f968c0c29e0 0x7f968c163870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:00.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.317+0000 7f969241a700 1 --2- 172.21.15.112:0/2031671868 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f968c06a890 0x7f968c160b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:00.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.317+0000 7f9692c1b700 1 -- 172.21.15.112:0/2031671868 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f968c06a890 msgr2=0x7f968c160b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:00.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.317+0000 7f9692c1b700 1 --2- 172.21.15.112:0/2031671868 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f968c06a890 0x7f968c160b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:00.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.317+0000 7f9692c1b700 1 -- 172.21.15.112:0/2031671868 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f968c0c29e0 msgr2=0x7f968c163870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:00.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.317+0000 7f9692c1b700 1 --2- 172.21.15.112:0/2031671868 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f968c0c29e0 0x7f968c163870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:00.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.317+0000 7f9692c1b700 1 -- 172.21.15.112:0/2031671868 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96880092a0 con 0x7f968c0c6f60 2023-01-24T03:40:00.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.317+0000 7f9692c1b700 1 --2- 172.21.15.112:0/2031671868 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f968c0c6f60 0x7f968c06dea0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f968400c8b0 tx=0x7f96840016c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:00.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.317+0000 7f967a7fc700 1 -- 172.21.15.112:0/2031671868 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9684012630 con 0x7f968c0c6f60 2023-01-24T03:40:00.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.318+0000 7f967a7fc700 1 -- 172.21.15.112:0/2031671868 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9684012790 con 0x7f968c0c6f60 2023-01-24T03:40:00.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.318+0000 7f969467e700 1 -- 172.21.15.112:0/2031671868 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f968c161490 con 0x7f968c0c6f60 2023-01-24T03:40:00.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.318+0000 7f967a7fc700 1 -- 172.21.15.112:0/2031671868 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9684012b50 con 0x7f968c0c6f60 2023-01-24T03:40:00.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.318+0000 7f969467e700 1 -- 172.21.15.112:0/2031671868 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f968c161a50 con 0x7f968c0c6f60 2023-01-24T03:40:00.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.318+0000 7f969467e700 1 -- 172.21.15.112:0/2031671868 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f968c0c5420 con 0x7f968c0c6f60 2023-01-24T03:40:00.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.319+0000 7f967a7fc700 1 -- 172.21.15.112:0/2031671868 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f9684018020 con 0x7f968c0c6f60 2023-01-24T03:40:00.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.319+0000 7f967a7fc700 1 --2- 172.21.15.112:0/2031671868 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f966c0611c0 0x7f966c063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:00.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.320+0000 7f967a7fc700 1 -- 172.21.15.112:0/2031671868 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 7383+0+0 (secure 0 0 0) 0x7f968407db30 con 0x7f968c0c6f60 2023-01-24T03:40:00.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.320+0000 7f969241a700 1 --2- 172.21.15.112:0/2031671868 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f966c0611c0 0x7f966c063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:00.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.320+0000 7f969241a700 1 --2- 172.21.15.112:0/2031671868 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f966c0611c0 0x7f966c063670 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f967c007fd0 tx=0x7f967c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:00.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.324+0000 7f967a7fc700 1 -- 172.21.15.112:0/2031671868 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f968404d730 con 0x7f968c0c6f60 2023-01-24T03:40:00.547 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.544+0000 7f969467e700 1 -- 172.21.15.112:0/2031671868 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f968c06db50 con 0x7f968c0c6f60 2023-01-24T03:40:00.548 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.545+0000 7f967a7fc700 1 -- 172.21.15.112:0/2031671868 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+5361 (secure 0 0 0) 0x7f9684050d50 con 0x7f968c0c6f60 2023-01-24T03:40:00.548 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:40:00.548 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:39:49.907279+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":29,"addr":"172.21.15.142:6835/2136880103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2136880103},{"type":"v1","addr":"172.21.15.142:6835","nonce":2136880103}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:40:00.550 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.546+0000 7f969467e700 1 -- 172.21.15.112:0/2031671868 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f966c0611c0 msgr2=0x7f966c063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:00.550 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.546+0000 7f969467e700 1 --2- 172.21.15.112:0/2031671868 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f966c0611c0 0x7f966c063670 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f967c007fd0 tx=0x7f967c009040 comp rx=0 tx=0).stop 2023-01-24T03:40:00.551 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.547+0000 7f969467e700 1 -- 172.21.15.112:0/2031671868 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f968c0c6f60 msgr2=0x7f968c06dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:00.551 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.547+0000 7f969467e700 1 --2- 172.21.15.112:0/2031671868 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f968c0c6f60 0x7f968c06dea0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f968400c8b0 tx=0x7f96840016c0 comp rx=0 tx=0).stop 2023-01-24T03:40:00.551 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.547+0000 7f969467e700 1 -- 172.21.15.112:0/2031671868 shutdown_connections 2023-01-24T03:40:00.551 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.547+0000 7f969467e700 1 --2- 172.21.15.112:0/2031671868 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f966c0611c0 0x7f966c063670 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:00.551 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.547+0000 7f969467e700 1 --2- 172.21.15.112:0/2031671868 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f968c0c6f60 0x7f968c06dea0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:00.552 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.547+0000 7f969467e700 1 --2- 172.21.15.112:0/2031671868 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f968c06a890 0x7f968c160b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:00.552 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.547+0000 7f969467e700 1 --2- 172.21.15.112:0/2031671868 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f968c0c29e0 0x7f968c163870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:00.552 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.547+0000 7f969467e700 1 -- 172.21.15.112:0/2031671868 >> 172.21.15.112:0/2031671868 conn(0x7f968c066e10 msgr2=0x7f968c06ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:00.552 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.547+0000 7f969467e700 1 -- 172.21.15.112:0/2031671868 shutdown_connections 2023-01-24T03:40:00.553 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:00.547+0000 7f969467e700 1 -- 172.21.15.112:0/2031671868 wait complete. 2023-01-24T03:40:00.553 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 40 2023-01-24T03:40:00.562 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2023-01-24T03:40:02.563 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:40:02.677 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.674+0000 7f5e9759e700 1 Processor -- start 2023-01-24T03:40:02.677 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.674+0000 7f5e9759e700 1 -- start start 2023-01-24T03:40:02.680 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.675+0000 7f5e9759e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f5e98156e40 0x7f5e980cb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:02.680 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.675+0000 7f5e9759e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e980c3d10 con 0x7f5e980c5680 2023-01-24T03:40:02.680 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.675+0000 7f5e9759e700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e980c2e20 con 0x7f5e98155e20 2023-01-24T03:40:02.680 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.675+0000 7f5e9759e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e980c4eb0 con 0x7f5e98156e40 2023-01-24T03:40:02.681 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.675+0000 7f5e9659c700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f5e980c5680 0x7f5e98156730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:36566/0 (socket says 172.21.15.112:36566) 2023-01-24T03:40:02.681 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.675+0000 7f5e9659c700 1 -- 172.21.15.112:0/3312727190 learned_addr learned my addr 172.21.15.112:0/3312727190 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:40:02.681 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.675+0000 7f5e95d9b700 1 --2- 172.21.15.112:0/3312727190 >> v2:172.21.15.142:3300/0 conn(0x7f5e98156e40 0x7f5e980cb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:02.681 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.675+0000 7f5e9559a700 1 -- 172.21.15.112:0/3312727190 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1549422577 0 0) 0x7f5e980c3d10 con 0x7f5e980c5680 2023-01-24T03:40:02.681 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.675+0000 7f5e9559a700 1 -- 172.21.15.112:0/3312727190 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e780033f0 con 0x7f5e980c5680 2023-01-24T03:40:02.682 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.675+0000 7f5e9559a700 1 -- 172.21.15.112:0/3312727190 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 134495031 0 0) 0x7f5e980c2e20 con 0x7f5e98155e20 2023-01-24T03:40:02.682 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.675+0000 7f5e9559a700 1 -- 172.21.15.112:0/3312727190 --> v1:172.21.15.142:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e980c3d10 con 0x7f5e98155e20 2023-01-24T03:40:02.682 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.675+0000 7f5e9559a700 1 -- 172.21.15.112:0/3312727190 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2526516817 0 0) 0x7f5e780033f0 con 0x7f5e980c5680 2023-01-24T03:40:02.682 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.675+0000 7f5e95d9b700 1 -- 172.21.15.112:0/3312727190 >> v1:172.21.15.142:6789/0 conn(0x7f5e98155e20 legacy=0x7f5e9814ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:02.682 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.675+0000 7f5e95d9b700 1 -- 172.21.15.112:0/3312727190 >> v1:172.21.15.112:6789/0 conn(0x7f5e980c5680 legacy=0x7f5e98156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:02.682 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.675+0000 7f5e95d9b700 1 -- 172.21.15.112:0/3312727190 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e980c3100 con 0x7f5e98156e40 2023-01-24T03:40:02.683 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.676+0000 7f5e95d9b700 1 --2- 172.21.15.112:0/3312727190 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e98156e40 0x7f5e980cb870 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5e80009760 tx=0x7f5e80009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=122e47626fe85e46 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:02.683 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.676+0000 7f5e9559a700 1 -- 172.21.15.112:0/3312727190 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e880017f0 con 0x7f5e98156e40 2023-01-24T03:40:02.683 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.676+0000 7f5e9559a700 1 -- 172.21.15.112:0/3312727190 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e80011440 con 0x7f5e98156e40 2023-01-24T03:40:02.683 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.676+0000 7f5e9559a700 1 -- 172.21.15.112:0/3312727190 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e800117c0 con 0x7f5e98156e40 2023-01-24T03:40:02.683 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.676+0000 7f5e9759e700 1 -- 172.21.15.112:0/3312727190 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e98156e40 msgr2=0x7f5e980cb870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:02.684 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.676+0000 7f5e9759e700 1 --2- 172.21.15.112:0/3312727190 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e98156e40 0x7f5e980cb870 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5e80009760 tx=0x7f5e80009060 comp rx=0 tx=0).stop 2023-01-24T03:40:02.684 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.676+0000 7f5e9759e700 1 -- 172.21.15.112:0/3312727190 shutdown_connections 2023-01-24T03:40:02.684 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.676+0000 7f5e9759e700 1 --2- 172.21.15.112:0/3312727190 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e98156e40 0x7f5e980cb870 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:02.684 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.676+0000 7f5e9759e700 1 -- 172.21.15.112:0/3312727190 >> 172.21.15.112:0/3312727190 conn(0x7f5e980685b0 msgr2=0x7f5e980689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:02.684 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.676+0000 7f5e9759e700 1 -- 172.21.15.112:0/3312727190 shutdown_connections 2023-01-24T03:40:02.685 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.676+0000 7f5e9759e700 1 -- 172.21.15.112:0/3312727190 wait complete. 2023-01-24T03:40:02.688 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.677+0000 7f5e9759e700 1 Processor -- start 2023-01-24T03:40:02.688 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.677+0000 7f5e9759e700 1 -- start start 2023-01-24T03:40:02.688 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.678+0000 7f5e9759e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e980c5680 0x7f5e980caac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:02.688 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.678+0000 7f5e9759e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5e98155e20 0x7f5e9814fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:02.689 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.678+0000 7f5e9759e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e98156e40 0x7f5e980b0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:02.689 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.678+0000 7f5e9759e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e98152710 con 0x7f5e98155e20 2023-01-24T03:40:02.689 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.678+0000 7f5e9759e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e98150f00 con 0x7f5e98156e40 2023-01-24T03:40:02.689 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.678+0000 7f5e9759e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e98151390 con 0x7f5e980c5680 2023-01-24T03:40:02.689 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.678+0000 7f5e95d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5e98155e20 0x7f5e9814fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:02.690 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.678+0000 7f5e95d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5e98155e20 0x7f5e9814fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55586/0 (socket says 172.21.15.112:55586) 2023-01-24T03:40:02.690 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.678+0000 7f5e95d9b700 1 -- 172.21.15.112:0/3649335965 learned_addr learned my addr 172.21.15.112:0/3649335965 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:40:02.690 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.678+0000 7f5e9659c700 1 --2- 172.21.15.112:0/3649335965 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e980c5680 0x7f5e980caac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:02.690 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.678+0000 7f5e96d9d700 1 --2- 172.21.15.112:0/3649335965 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e98156e40 0x7f5e980b0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:02.690 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.679+0000 7f5e95d9b700 1 -- 172.21.15.112:0/3649335965 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e980c5680 msgr2=0x7f5e980caac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:02.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.679+0000 7f5e95d9b700 1 --2- 172.21.15.112:0/3649335965 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e980c5680 0x7f5e980caac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:02.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.679+0000 7f5e95d9b700 1 -- 172.21.15.112:0/3649335965 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e98156e40 msgr2=0x7f5e980b0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:02.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.679+0000 7f5e95d9b700 1 --2- 172.21.15.112:0/3649335965 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e98156e40 0x7f5e980b0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:02.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.679+0000 7f5e95d9b700 1 -- 172.21.15.112:0/3649335965 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e800092a0 con 0x7f5e98155e20 2023-01-24T03:40:02.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.679+0000 7f5e95d9b700 1 --2- 172.21.15.112:0/3649335965 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5e98155e20 0x7f5e9814fba0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f5e80003080 tx=0x7f5e80015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:02.692 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.679+0000 7f5e8e7fc700 1 -- 172.21.15.112:0/3649335965 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e80010ac0 con 0x7f5e98155e20 2023-01-24T03:40:02.692 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.679+0000 7f5e8e7fc700 1 -- 172.21.15.112:0/3649335965 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e80010c20 con 0x7f5e98155e20 2023-01-24T03:40:02.692 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.679+0000 7f5e9759e700 1 -- 172.21.15.112:0/3649335965 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e980c5ed0 con 0x7f5e98155e20 2023-01-24T03:40:02.692 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.679+0000 7f5e8e7fc700 1 -- 172.21.15.112:0/3649335965 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e8001c420 con 0x7f5e98155e20 2023-01-24T03:40:02.692 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.679+0000 7f5e9759e700 1 -- 172.21.15.112:0/3649335965 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e981657d0 con 0x7f5e98155e20 2023-01-24T03:40:02.693 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.680+0000 7f5e9759e700 1 -- 172.21.15.112:0/3649335965 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e98150cd0 con 0x7f5e98155e20 2023-01-24T03:40:02.693 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.680+0000 7f5e8e7fc700 1 -- 172.21.15.112:0/3649335965 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f5e8000f070 con 0x7f5e98155e20 2023-01-24T03:40:02.693 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.680+0000 7f5e8e7fc700 1 --2- 172.21.15.112:0/3649335965 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5e7c0611c0 0x7f5e7c063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:02.693 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.681+0000 7f5e8e7fc700 1 -- 172.21.15.112:0/3649335965 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 7383+0+0 (secure 0 0 0) 0x7f5e8007eeb0 con 0x7f5e98155e20 2023-01-24T03:40:02.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.681+0000 7f5e9659c700 1 --2- 172.21.15.112:0/3649335965 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5e7c0611c0 0x7f5e7c063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:02.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.681+0000 7f5e9659c700 1 --2- 172.21.15.112:0/3649335965 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5e7c0611c0 0x7f5e7c063670 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f5e88001920 tx=0x7f5e88000f40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:02.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.685+0000 7f5e8e7fc700 1 -- 172.21.15.112:0/3649335965 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f5e98150cd0 con 0x7f5e98155e20 2023-01-24T03:40:02.906 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.903+0000 7f5e9759e700 1 -- 172.21.15.112:0/3649335965 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e98150cd0 con 0x7f5e98155e20 2023-01-24T03:40:02.907 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.904+0000 7f5e8e7fc700 1 -- 172.21.15.112:0/3649335965 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+6121 (secure 0 0 0) 0x7f5e98150cd0 con 0x7f5e98155e20 2023-01-24T03:40:02.907 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:40:02.908 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5186,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/2804746591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2804746591},{"type":"v1","addr":"172.21.15.142:6834","nonce":2804746591}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:39:49.907279+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":29,"addr":"172.21.15.142:6835/2136880103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2136880103},{"type":"v1","addr":"172.21.15.142:6835","nonce":2136880103}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:40:02.909 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.906+0000 7f5e9759e700 1 -- 172.21.15.112:0/3649335965 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5e7c0611c0 msgr2=0x7f5e7c063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:02.909 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.906+0000 7f5e9759e700 1 --2- 172.21.15.112:0/3649335965 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5e7c0611c0 0x7f5e7c063670 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f5e88001920 tx=0x7f5e88000f40 comp rx=0 tx=0).stop 2023-01-24T03:40:02.909 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.906+0000 7f5e9759e700 1 -- 172.21.15.112:0/3649335965 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5e98155e20 msgr2=0x7f5e9814fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:02.910 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.906+0000 7f5e9759e700 1 --2- 172.21.15.112:0/3649335965 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5e98155e20 0x7f5e9814fba0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f5e80003080 tx=0x7f5e80015000 comp rx=0 tx=0).stop 2023-01-24T03:40:02.910 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.906+0000 7f5e9759e700 1 -- 172.21.15.112:0/3649335965 shutdown_connections 2023-01-24T03:40:02.910 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.906+0000 7f5e9759e700 1 --2- 172.21.15.112:0/3649335965 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5e7c0611c0 0x7f5e7c063670 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:02.911 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.906+0000 7f5e9759e700 1 --2- 172.21.15.112:0/3649335965 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5e98156e40 0x7f5e980b0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:02.911 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.906+0000 7f5e9759e700 1 --2- 172.21.15.112:0/3649335965 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5e980c5680 0x7f5e980caac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:02.911 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.906+0000 7f5e9759e700 1 --2- 172.21.15.112:0/3649335965 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5e98155e20 0x7f5e9814fba0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:02.912 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.906+0000 7f5e9759e700 1 -- 172.21.15.112:0/3649335965 >> 172.21.15.112:0/3649335965 conn(0x7f5e980685b0 msgr2=0x7f5e9814def0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:02.912 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.906+0000 7f5e9759e700 1 -- 172.21.15.112:0/3649335965 shutdown_connections 2023-01-24T03:40:02.913 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:02.906+0000 7f5e9759e700 1 -- 172.21.15.112:0/3649335965 wait complete. 2023-01-24T03:40:02.913 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 41 2023-01-24T03:40:02.921 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2023-01-24T03:40:02.921 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 13 secs before thrashing 2023-01-24T03:40:15.922 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:40:16.047 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.045+0000 7fcdf0ddc700 1 Processor -- start 2023-01-24T03:40:16.048 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.045+0000 7fcdf0ddc700 1 -- start start 2023-01-24T03:40:16.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.046+0000 7fcdf0ddc700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fcdec0c7e40 0x7fcdec0b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:16.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.046+0000 7fcdf0ddc700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fcdec155da0 0x7fcdec0ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:16.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.046+0000 7fcdf0ddc700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdec156240 con 0x7fcdec0aef20 2023-01-24T03:40:16.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.046+0000 7fcdf0ddc700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdec0af320 con 0x7fcdec0c7e40 2023-01-24T03:40:16.051 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.046+0000 7fcdf0ddc700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdec0b1170 con 0x7fcdec155da0 2023-01-24T03:40:16.051 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.046+0000 7fcdea59c700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fcdec0c7e40 0x7fcdec0b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:16.051 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.047+0000 7fcdea59c700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fcdec0c7e40 0x7fcdec0b0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55638/0 (socket says 172.21.15.112:55638) 2023-01-24T03:40:16.051 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.047+0000 7fcdea59c700 1 -- 172.21.15.112:0/1696053222 learned_addr learned my addr 172.21.15.112:0/1696053222 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:40:16.051 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.047+0000 7fcde9d9b700 1 --2- 172.21.15.112:0/1696053222 >> v2:172.21.15.142:3300/0 conn(0x7fcdec155da0 0x7fcdec0ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:16.052 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.047+0000 7fcdea59c700 1 -- 172.21.15.112:0/1696053222 >> v2:172.21.15.142:3300/0 conn(0x7fcdec155da0 msgr2=0x7fcdec0ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:40:16.052 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.047+0000 7fcdea59c700 1 --2- 172.21.15.112:0/1696053222 >> v2:172.21.15.142:3300/0 conn(0x7fcdec155da0 0x7fcdec0ae9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:16.052 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.047+0000 7fcdea59c700 1 -- 172.21.15.112:0/1696053222 >> v1:172.21.15.142:6789/0 conn(0x7fcdec0aef20 legacy=0x7fcdec0ca3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:16.052 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.047+0000 7fcdea59c700 1 -- 172.21.15.112:0/1696053222 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdec0b1450 con 0x7fcdec0c7e40 2023-01-24T03:40:16.052 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.047+0000 7fcdea59c700 1 --2- 172.21.15.112:0/1696053222 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcdec0c7e40 0x7fcdec0b0aa0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fcdec152ea0 tx=0x7fcddc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efa7cf93a5be4319 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:16.053 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.047+0000 7fcde959a700 1 -- 172.21.15.112:0/1696053222 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcddc00e070 con 0x7fcdec0c7e40 2023-01-24T03:40:16.053 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.047+0000 7fcde959a700 1 -- 172.21.15.112:0/1696053222 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcddc00ba20 con 0x7fcdec0c7e40 2023-01-24T03:40:16.053 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.047+0000 7fcde959a700 1 -- 172.21.15.112:0/1696053222 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcddc0092e0 con 0x7fcdec0c7e40 2023-01-24T03:40:16.053 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.048+0000 7fcdf0ddc700 1 -- 172.21.15.112:0/1696053222 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcdec0c7e40 msgr2=0x7fcdec0b0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:16.053 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.048+0000 7fcdf0ddc700 1 --2- 172.21.15.112:0/1696053222 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcdec0c7e40 0x7fcdec0b0aa0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fcdec152ea0 tx=0x7fcddc009840 comp rx=0 tx=0).stop 2023-01-24T03:40:16.054 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.048+0000 7fcdf0ddc700 1 -- 172.21.15.112:0/1696053222 shutdown_connections 2023-01-24T03:40:16.054 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.048+0000 7fcdf0ddc700 1 --2- 172.21.15.112:0/1696053222 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcdec0c7e40 0x7fcdec0b0aa0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:16.054 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.048+0000 7fcdf0ddc700 1 --2- 172.21.15.112:0/1696053222 >> v2:172.21.15.142:3300/0 conn(0x7fcdec155da0 0x7fcdec0ae9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:16.055 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.048+0000 7fcdf0ddc700 1 -- 172.21.15.112:0/1696053222 >> 172.21.15.112:0/1696053222 conn(0x7fcdec05e2c0 msgr2=0x7fcdec0cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:16.055 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.048+0000 7fcdf0ddc700 1 -- 172.21.15.112:0/1696053222 shutdown_connections 2023-01-24T03:40:16.055 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.048+0000 7fcdf0ddc700 1 -- 172.21.15.112:0/1696053222 wait complete. 2023-01-24T03:40:16.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.049+0000 7fcdf0ddc700 1 Processor -- start 2023-01-24T03:40:16.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.049+0000 7fcdf0ddc700 1 -- start start 2023-01-24T03:40:16.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.049+0000 7fcdf0ddc700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcdec0aef20 0x7fcdec0cb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:16.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.049+0000 7fcdf0ddc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcdec0c7e40 0x7fcdec0c34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:16.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.049+0000 7fcdf0ddc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcdec155da0 0x7fcdec0c39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:16.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.050+0000 7fcdf0ddc700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdec151150 con 0x7fcdec0c7e40 2023-01-24T03:40:16.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.050+0000 7fcdf0ddc700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdec157170 con 0x7fcdec155da0 2023-01-24T03:40:16.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.050+0000 7fcdf0ddc700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdec169690 con 0x7fcdec0aef20 2023-01-24T03:40:16.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.050+0000 7fcde9d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcdec0c7e40 0x7fcdec0c34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:16.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.050+0000 7fcde9d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcdec0c7e40 0x7fcdec0c34b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55642/0 (socket says 172.21.15.112:55642) 2023-01-24T03:40:16.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.050+0000 7fcde9d9b700 1 -- 172.21.15.112:0/2152676807 learned_addr learned my addr 172.21.15.112:0/2152676807 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:40:16.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.050+0000 7fcdead9d700 1 --2- 172.21.15.112:0/2152676807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcdec155da0 0x7fcdec0c39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:16.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.050+0000 7fcdea59c700 1 --2- 172.21.15.112:0/2152676807 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcdec0aef20 0x7fcdec0cb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:16.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.050+0000 7fcde9d9b700 1 -- 172.21.15.112:0/2152676807 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcdec0aef20 msgr2=0x7fcdec0cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:16.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.050+0000 7fcde9d9b700 1 --2- 172.21.15.112:0/2152676807 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcdec0aef20 0x7fcdec0cb4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:16.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.050+0000 7fcde9d9b700 1 -- 172.21.15.112:0/2152676807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcdec155da0 msgr2=0x7fcdec0c39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:16.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.050+0000 7fcde9d9b700 1 --2- 172.21.15.112:0/2152676807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcdec155da0 0x7fcdec0c39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:16.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.050+0000 7fcde9d9b700 1 -- 172.21.15.112:0/2152676807 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcddc009ae0 con 0x7fcdec0c7e40 2023-01-24T03:40:16.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.050+0000 7fcde9d9b700 1 --2- 172.21.15.112:0/2152676807 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcdec0c7e40 0x7fcdec0c34b0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fcdd400e960 tx=0x7fcdd4009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:16.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.050+0000 7fcde27fc700 1 -- 172.21.15.112:0/2152676807 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4013680 con 0x7fcdec0c7e40 2023-01-24T03:40:16.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.050+0000 7fcde27fc700 1 -- 172.21.15.112:0/2152676807 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdd40137e0 con 0x7fcdec0c7e40 2023-01-24T03:40:16.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.050+0000 7fcdf0ddc700 1 -- 172.21.15.112:0/2152676807 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdec0c4040 con 0x7fcdec0c7e40 2023-01-24T03:40:16.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.050+0000 7fcde27fc700 1 -- 172.21.15.112:0/2152676807 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4013b60 con 0x7fcdec0c7e40 2023-01-24T03:40:16.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.050+0000 7fcdf0ddc700 1 -- 172.21.15.112:0/2152676807 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdec168d60 con 0x7fcdec0c7e40 2023-01-24T03:40:16.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.051+0000 7fcdf0ddc700 1 -- 172.21.15.112:0/2152676807 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdec169820 con 0x7fcdec0c7e40 2023-01-24T03:40:16.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.052+0000 7fcde27fc700 1 -- 172.21.15.112:0/2152676807 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fcdd4019030 con 0x7fcdec0c7e40 2023-01-24T03:40:16.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.052+0000 7fcde27fc700 1 --2- 172.21.15.112:0/2152676807 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fcdd00611f0 0x7fcdd00636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:16.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.052+0000 7fcde27fc700 1 -- 172.21.15.112:0/2152676807 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 7383+0+0 (secure 0 0 0) 0x7fcdd407da30 con 0x7fcdec0c7e40 2023-01-24T03:40:16.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.052+0000 7fcdea59c700 1 --2- 172.21.15.112:0/2152676807 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fcdd00611f0 0x7fcdd00636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:16.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.052+0000 7fcdea59c700 1 --2- 172.21.15.112:0/2152676807 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fcdd00611f0 0x7fcdd00636a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fcddc009880 tx=0x7fcddc018000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:16.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.056+0000 7fcde27fc700 1 -- 172.21.15.112:0/2152676807 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fcdd404d5d0 con 0x7fcdec0c7e40 2023-01-24T03:40:16.280 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.278+0000 7fcdf0ddc700 1 -- 172.21.15.112:0/2152676807 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcdec04bb80 con 0x7fcdec0c7e40 2023-01-24T03:40:16.281 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.279+0000 7fcde27fc700 1 -- 172.21.15.112:0/2152676807 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+6121 (secure 0 0 0) 0x7fcdd4050bf0 con 0x7fcdec0c7e40 2023-01-24T03:40:16.282 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:40:16.282 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5186,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/2804746591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2804746591},{"type":"v1","addr":"172.21.15.142:6834","nonce":2804746591}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:39:49.907279+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":29,"addr":"172.21.15.142:6835/2136880103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2136880103},{"type":"v1","addr":"172.21.15.142:6835","nonce":2136880103}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:40:16.285 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.281+0000 7fcdf0ddc700 1 -- 172.21.15.112:0/2152676807 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fcdd00611f0 msgr2=0x7fcdd00636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:16.285 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.281+0000 7fcdf0ddc700 1 --2- 172.21.15.112:0/2152676807 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fcdd00611f0 0x7fcdd00636a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fcddc009880 tx=0x7fcddc018000 comp rx=0 tx=0).stop 2023-01-24T03:40:16.285 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.281+0000 7fcdf0ddc700 1 -- 172.21.15.112:0/2152676807 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcdec0c7e40 msgr2=0x7fcdec0c34b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:16.286 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.281+0000 7fcdf0ddc700 1 --2- 172.21.15.112:0/2152676807 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcdec0c7e40 0x7fcdec0c34b0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fcdd400e960 tx=0x7fcdd4009060 comp rx=0 tx=0).stop 2023-01-24T03:40:16.286 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.281+0000 7fcdf0ddc700 1 -- 172.21.15.112:0/2152676807 shutdown_connections 2023-01-24T03:40:16.286 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.281+0000 7fcdf0ddc700 1 --2- 172.21.15.112:0/2152676807 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fcdd00611f0 0x7fcdd00636a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:16.287 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.281+0000 7fcdf0ddc700 1 --2- 172.21.15.112:0/2152676807 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcdec155da0 0x7fcdec0c39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:16.287 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.281+0000 7fcdf0ddc700 1 --2- 172.21.15.112:0/2152676807 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fcdec0aef20 0x7fcdec0cb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:16.287 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.281+0000 7fcdf0ddc700 1 --2- 172.21.15.112:0/2152676807 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcdec0c7e40 0x7fcdec0c34b0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:16.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.281+0000 7fcdf0ddc700 1 -- 172.21.15.112:0/2152676807 >> 172.21.15.112:0/2152676807 conn(0x7fcdec05e2c0 msgr2=0x7fcdec1551d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:16.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.281+0000 7fcdf0ddc700 1 -- 172.21.15.112:0/2152676807 shutdown_connections 2023-01-24T03:40:16.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.281+0000 7fcdf0ddc700 1 -- 172.21.15.112:0/2152676807 wait complete. 2023-01-24T03:40:16.289 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 41 2023-01-24T03:40:16.295 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=0) 2023-01-24T03:40:16.295 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-01-24T03:40:16.296 INFO:teuthology.orchestra.run:waiting for 300 2023-01-24T03:40:16.312 INFO:tasks.ceph.mds.d:Stopped 2023-01-24T03:40:16.312 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:40:16.423 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.422+0000 7f1c9cfd4700 1 Processor -- start 2023-01-24T03:40:16.424 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.422+0000 7f1c9cfd4700 1 -- start start 2023-01-24T03:40:16.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.422+0000 7f1c9cfd4700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f1c980c7e40 0x7f1c980b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:16.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.422+0000 7f1c9cfd4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c98155da0 0x7f1c980ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:16.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.422+0000 7f1c9cfd4700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f1c980aef20 0x7f1c980ca3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:16.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.422+0000 7f1c9cfd4700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c98156240 con 0x7f1c980aef20 2023-01-24T03:40:16.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.422+0000 7f1c9cfd4700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c980b1170 con 0x7f1c980c7e40 2023-01-24T03:40:16.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.422+0000 7f1c9cfd4700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c980b1450 con 0x7f1c98155da0 2023-01-24T03:40:16.434 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.423+0000 7f1c96d9d700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f1c980aef20 0x7f1c980ca3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:16.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.423+0000 7f1c96d9d700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f1c980aef20 0x7f1c980ca3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55654/0 (socket says 172.21.15.112:55654) 2023-01-24T03:40:16.435 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.423+0000 7f1c96d9d700 1 -- 172.21.15.112:0/261604960 learned_addr learned my addr 172.21.15.112:0/261604960 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:40:16.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.423+0000 7f1c95d9b700 1 --2- 172.21.15.112:0/261604960 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c98155da0 0x7f1c980ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:16.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.423+0000 7f1c9659c700 1 --2- 172.21.15.112:0/261604960 >> v2:172.21.15.142:3300/0 conn(0x7f1c980c7e40 0x7f1c980b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:16.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.423+0000 7f1c96d9d700 1 -- 172.21.15.112:0/261604960 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c98155da0 msgr2=0x7f1c980ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:40:16.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.423+0000 7f1c96d9d700 1 --2- 172.21.15.112:0/261604960 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c98155da0 0x7f1c980ae9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:16.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.423+0000 7f1c96d9d700 1 -- 172.21.15.112:0/261604960 >> v2:172.21.15.142:3300/0 conn(0x7f1c980c7e40 msgr2=0x7f1c980b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:40:16.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.423+0000 7f1c96d9d700 1 --2- 172.21.15.112:0/261604960 >> v2:172.21.15.142:3300/0 conn(0x7f1c980c7e40 0x7f1c980b0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:16.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.423+0000 7f1c96d9d700 1 -- 172.21.15.112:0/261604960 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c980b1730 con 0x7f1c980aef20 2023-01-24T03:40:16.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.423+0000 7f1c96d9d700 1 --2- 172.21.15.112:0/261604960 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1c980aef20 0x7f1c980ca3e0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f1c98089cf0 tx=0x7f1c900090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ea2e0f59433feb9 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:16.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.423+0000 7f1c9559a700 1 -- 172.21.15.112:0/261604960 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c90011630 con 0x7f1c980aef20 2023-01-24T03:40:16.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.423+0000 7f1c9559a700 1 -- 172.21.15.112:0/261604960 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c900117d0 con 0x7f1c980aef20 2023-01-24T03:40:16.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.423+0000 7f1c9559a700 1 -- 172.21.15.112:0/261604960 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c90011b90 con 0x7f1c980aef20 2023-01-24T03:40:16.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.424+0000 7f1c9cfd4700 1 -- 172.21.15.112:0/261604960 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1c980aef20 msgr2=0x7f1c980ca3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:16.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.424+0000 7f1c9cfd4700 1 --2- 172.21.15.112:0/261604960 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1c980aef20 0x7f1c980ca3e0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f1c98089cf0 tx=0x7f1c900090f0 comp rx=0 tx=0).stop 2023-01-24T03:40:16.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.424+0000 7f1c9cfd4700 1 -- 172.21.15.112:0/261604960 shutdown_connections 2023-01-24T03:40:16.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.424+0000 7f1c9cfd4700 1 --2- 172.21.15.112:0/261604960 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1c980aef20 0x7f1c980ca3e0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:16.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.424+0000 7f1c9cfd4700 1 --2- 172.21.15.112:0/261604960 >> v2:172.21.15.142:3300/0 conn(0x7f1c980c7e40 0x7f1c980b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:16.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.424+0000 7f1c9cfd4700 1 --2- 172.21.15.112:0/261604960 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c98155da0 0x7f1c980ae9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:16.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.424+0000 7f1c9cfd4700 1 -- 172.21.15.112:0/261604960 >> 172.21.15.112:0/261604960 conn(0x7f1c9805e2c0 msgr2=0x7f1c980cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:16.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.424+0000 7f1c9cfd4700 1 -- 172.21.15.112:0/261604960 shutdown_connections 2023-01-24T03:40:16.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.424+0000 7f1c9cfd4700 1 -- 172.21.15.112:0/261604960 wait complete. 2023-01-24T03:40:16.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.425+0000 7f1c9cfd4700 1 Processor -- start 2023-01-24T03:40:16.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.425+0000 7f1c9cfd4700 1 -- start start 2023-01-24T03:40:16.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.425+0000 7f1c9cfd4700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1c980aef20 0x7f1c980c3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:16.441 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.425+0000 7f1c9cfd4700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c980c7e40 0x7f1c980c5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:16.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.425+0000 7f1c9cfd4700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1c98155da0 0x7f1c980cb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:16.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.425+0000 7f1c9cfd4700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c98151620 con 0x7f1c980aef20 2023-01-24T03:40:16.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.425+0000 7f1c9cfd4700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c980c3ee0 con 0x7f1c98155da0 2023-01-24T03:40:16.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.425+0000 7f1c9cfd4700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c98169ca0 con 0x7f1c980c7e40 2023-01-24T03:40:16.442 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.426+0000 7f1c9659c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1c980aef20 0x7f1c980c3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:16.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.426+0000 7f1c9659c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1c980aef20 0x7f1c980c3850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55656/0 (socket says 172.21.15.112:55656) 2023-01-24T03:40:16.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.426+0000 7f1c9659c700 1 -- 172.21.15.112:0/1143675828 learned_addr learned my addr 172.21.15.112:0/1143675828 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:40:16.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.426+0000 7f1c95d9b700 1 --2- 172.21.15.112:0/1143675828 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c980c7e40 0x7f1c980c5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:16.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.426+0000 7f1c96d9d700 1 --2- 172.21.15.112:0/1143675828 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1c98155da0 0x7f1c980cb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:16.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.426+0000 7f1c9659c700 1 -- 172.21.15.112:0/1143675828 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c980c7e40 msgr2=0x7f1c980c5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:16.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.426+0000 7f1c9659c700 1 --2- 172.21.15.112:0/1143675828 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c980c7e40 0x7f1c980c5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:16.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.426+0000 7f1c9659c700 1 -- 172.21.15.112:0/1143675828 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1c98155da0 msgr2=0x7f1c980cb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:16.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.426+0000 7f1c9659c700 1 --2- 172.21.15.112:0/1143675828 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1c98155da0 0x7f1c980cb9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:16.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.426+0000 7f1c9659c700 1 -- 172.21.15.112:0/1143675828 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c90009370 con 0x7f1c980aef20 2023-01-24T03:40:16.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.426+0000 7f1c9659c700 1 --2- 172.21.15.112:0/1143675828 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1c980aef20 0x7f1c980c3850 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f1c88009fd0 tx=0x7f1c88009880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:16.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.426+0000 7f1c8e7fc700 1 -- 172.21.15.112:0/1143675828 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c8800ebf0 con 0x7f1c980aef20 2023-01-24T03:40:16.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.426+0000 7f1c8e7fc700 1 -- 172.21.15.112:0/1143675828 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c8800ed50 con 0x7f1c980aef20 2023-01-24T03:40:16.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.426+0000 7f1c8e7fc700 1 -- 172.21.15.112:0/1143675828 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c88014670 con 0x7f1c980aef20 2023-01-24T03:40:16.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.426+0000 7f1c9cfd4700 1 -- 172.21.15.112:0/1143675828 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c980c5b60 con 0x7f1c980aef20 2023-01-24T03:40:16.445 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.426+0000 7f1c9cfd4700 1 -- 172.21.15.112:0/1143675828 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c98169370 con 0x7f1c980aef20 2023-01-24T03:40:16.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.427+0000 7f1c9cfd4700 1 -- 172.21.15.112:0/1143675828 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c9804bb80 con 0x7f1c980aef20 2023-01-24T03:40:16.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.427+0000 7f1c8e7fc700 1 -- 172.21.15.112:0/1143675828 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f1c88012050 con 0x7f1c980aef20 2023-01-24T03:40:16.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.428+0000 7f1c8e7fc700 1 --2- 172.21.15.112:0/1143675828 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f1c7c0611c0 0x7f1c7c063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:16.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.428+0000 7f1c8e7fc700 1 -- 172.21.15.112:0/1143675828 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 7383+0+0 (secure 0 0 0) 0x7f1c8807def0 con 0x7f1c980aef20 2023-01-24T03:40:16.446 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.428+0000 7f1c95d9b700 1 --2- 172.21.15.112:0/1143675828 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f1c7c0611c0 0x7f1c7c063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:16.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.428+0000 7f1c95d9b700 1 --2- 172.21.15.112:0/1143675828 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f1c7c0611c0 0x7f1c7c063670 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f1c980c8cd0 tx=0x7f1c80008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:16.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.432+0000 7f1c8e7fc700 1 -- 172.21.15.112:0/1143675828 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f1c8804da90 con 0x7f1c980aef20 2023-01-24T03:40:16.658 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.655+0000 7f1c9cfd4700 1 -- 172.21.15.112:0/1143675828 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c980aecc0 con 0x7f1c980aef20 2023-01-24T03:40:16.658 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.655+0000 7f1c8e7fc700 1 -- 172.21.15.112:0/1143675828 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+6121 (secure 0 0 0) 0x7f1c880510b0 con 0x7f1c980aef20 2023-01-24T03:40:16.659 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:40:16.659 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5186,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/2804746591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2804746591},{"type":"v1","addr":"172.21.15.142:6834","nonce":2804746591}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:39:49.907279+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":29,"addr":"172.21.15.142:6835/2136880103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2136880103},{"type":"v1","addr":"172.21.15.142:6835","nonce":2136880103}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:40:16.661 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.657+0000 7f1c9cfd4700 1 -- 172.21.15.112:0/1143675828 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f1c7c0611c0 msgr2=0x7f1c7c063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:16.661 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.657+0000 7f1c9cfd4700 1 --2- 172.21.15.112:0/1143675828 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f1c7c0611c0 0x7f1c7c063670 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f1c980c8cd0 tx=0x7f1c80008040 comp rx=0 tx=0).stop 2023-01-24T03:40:16.662 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.657+0000 7f1c9cfd4700 1 -- 172.21.15.112:0/1143675828 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1c980aef20 msgr2=0x7f1c980c3850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:16.662 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.657+0000 7f1c9cfd4700 1 --2- 172.21.15.112:0/1143675828 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1c980aef20 0x7f1c980c3850 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f1c88009fd0 tx=0x7f1c88009880 comp rx=0 tx=0).stop 2023-01-24T03:40:16.662 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.657+0000 7f1c9cfd4700 1 -- 172.21.15.112:0/1143675828 shutdown_connections 2023-01-24T03:40:16.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.657+0000 7f1c9cfd4700 1 --2- 172.21.15.112:0/1143675828 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f1c7c0611c0 0x7f1c7c063670 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:16.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.657+0000 7f1c9cfd4700 1 --2- 172.21.15.112:0/1143675828 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1c980aef20 0x7f1c980c3850 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:16.663 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.657+0000 7f1c9cfd4700 1 --2- 172.21.15.112:0/1143675828 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1c98155da0 0x7f1c980cb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:16.664 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.657+0000 7f1c9cfd4700 1 --2- 172.21.15.112:0/1143675828 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f1c980c7e40 0x7f1c980c5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:16.664 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.657+0000 7f1c9cfd4700 1 -- 172.21.15.112:0/1143675828 >> 172.21.15.112:0/1143675828 conn(0x7f1c9805e2c0 msgr2=0x7f1c98068b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:16.664 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.657+0000 7f1c9cfd4700 1 -- 172.21.15.112:0/1143675828 shutdown_connections 2023-01-24T03:40:16.665 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:16.657+0000 7f1c9cfd4700 1 -- 172.21.15.112:0/1143675828 wait complete. 2023-01-24T03:40:16.665 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 41 2023-01-24T03:40:16.671 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-01-24T03:40:18.673 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:40:18.798 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.796+0000 7f53a8a3d700 1 Processor -- start 2023-01-24T03:40:18.798 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.796+0000 7f53a8a3d700 1 -- start start 2023-01-24T03:40:18.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.797+0000 7f53a8a3d700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f53a40c5680 0x7f53a4156730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:18.805 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.797+0000 7f53a8a3d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f53a40c5a50 0x7f53a4156c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:18.805 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.797+0000 7f53a8a3d700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53a40c3d10 con 0x7f53a4155e20 2023-01-24T03:40:18.805 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.797+0000 7f53a8a3d700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53a40c4eb0 con 0x7f53a40c5680 2023-01-24T03:40:18.805 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.797+0000 7f53a8a3d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53a40c2e20 con 0x7f53a40c5a50 2023-01-24T03:40:18.806 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.797+0000 7f539a59c700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f53a4155e20 0x7f53a40cb870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:36660/0 (socket says 172.21.15.112:36660) 2023-01-24T03:40:18.806 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.797+0000 7f539a59c700 1 -- 172.21.15.112:0/3021291393 learned_addr learned my addr 172.21.15.112:0/3021291393 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:40:18.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.797+0000 7f53a2d9d700 1 --2- 172.21.15.112:0/3021291393 >> v2:172.21.15.142:3300/0 conn(0x7f53a40c5680 0x7f53a4156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:18.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.797+0000 7f53a359e700 1 --2- 172.21.15.112:0/3021291393 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f53a40c5a50 0x7f53a4156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:18.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.797+0000 7f53a259c700 1 -- 172.21.15.112:0/3021291393 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2394095539 0 0) 0x7f53a40c3d10 con 0x7f53a4155e20 2023-01-24T03:40:18.808 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.797+0000 7f53a259c700 1 -- 172.21.15.112:0/3021291393 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53840033f0 con 0x7f53a4155e20 2023-01-24T03:40:18.808 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.797+0000 7f53a259c700 1 -- 172.21.15.112:0/3021291393 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 610659929 0 0) 0x7f53840033f0 con 0x7f53a4155e20 2023-01-24T03:40:18.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.797+0000 7f53a259c700 1 -- 172.21.15.112:0/3021291393 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f53a40c5a50 msgr2=0x7f53a4156c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:40:18.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.797+0000 7f53a259c700 1 --2- 172.21.15.112:0/3021291393 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f53a40c5a50 0x7f53a4156c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:18.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.797+0000 7f53a259c700 1 -- 172.21.15.112:0/3021291393 >> v2:172.21.15.142:3300/0 conn(0x7f53a40c5680 msgr2=0x7f53a4156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:40:18.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.797+0000 7f53a259c700 1 --2- 172.21.15.112:0/3021291393 >> v2:172.21.15.142:3300/0 conn(0x7f53a40c5680 0x7f53a4156730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:18.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.797+0000 7f53a259c700 1 -- 172.21.15.112:0/3021291393 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a40c3100 con 0x7f53a4155e20 2023-01-24T03:40:18.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.797+0000 7f53a259c700 1 -- 172.21.15.112:0/3021291393 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7f538c001470 con 0x7f53a4155e20 2023-01-24T03:40:18.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.797+0000 7f53a259c700 1 -- 172.21.15.112:0/3021291393 >> v1:172.21.15.112:6789/0 conn(0x7f53a4155e20 legacy=0x7f53a40cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:18.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.798+0000 7f53a259c700 1 --2- 172.21.15.112:0/3021291393 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f53a40c5a50 0x7f53840038e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:18.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.798+0000 7f53a259c700 1 --2- 172.21.15.112:0/3021291393 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5384003e60 0x7f5384004690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:18.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.798+0000 7f53a259c700 1 --2- 172.21.15.112:0/3021291393 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5384004bd0 0x7f5384005040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:18.812 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.798+0000 7f53a259c700 1 -- 172.21.15.112:0/3021291393 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a40c4eb0 con 0x7f5384003e60 2023-01-24T03:40:18.812 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.798+0000 7f53a259c700 1 -- 172.21.15.112:0/3021291393 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a40c2e20 con 0x7f5384004bd0 2023-01-24T03:40:18.812 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.798+0000 7f53a259c700 1 -- 172.21.15.112:0/3021291393 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f538400b910 con 0x7f53a40c5a50 2023-01-24T03:40:18.812 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.798+0000 7f539a59c700 1 --2- 172.21.15.112:0/3021291393 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5384003e60 0x7f5384004690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:18.813 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.798+0000 7f53a2d9d700 1 --2- 172.21.15.112:0/3021291393 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f53a40c5a50 0x7f53840038e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:18.813 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.798+0000 7f53a359e700 1 --2- 172.21.15.112:0/3021291393 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5384004bd0 0x7f5384005040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:18.813 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.798+0000 7f539a59c700 1 -- 172.21.15.112:0/3021291393 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f53a40c5a50 msgr2=0x7f53840038e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:40:18.813 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.798+0000 7f539a59c700 1 --2- 172.21.15.112:0/3021291393 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f53a40c5a50 0x7f53840038e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:18.813 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.798+0000 7f539a59c700 1 -- 172.21.15.112:0/3021291393 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5384004bd0 msgr2=0x7f5384005040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:40:18.814 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.798+0000 7f539a59c700 1 --2- 172.21.15.112:0/3021291393 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5384004bd0 0x7f5384005040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:18.814 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.798+0000 7f539a59c700 1 -- 172.21.15.112:0/3021291393 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53840033f0 con 0x7f5384003e60 2023-01-24T03:40:18.814 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.798+0000 7f539a59c700 1 --2- 172.21.15.112:0/3021291393 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5384003e60 0x7f5384004690 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f538c00c210 tx=0x7f538c001e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c5f3b572ee5af05 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:18.814 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.798+0000 7f53a259c700 1 -- 172.21.15.112:0/3021291393 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538c00ca80 con 0x7f5384003e60 2023-01-24T03:40:18.814 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.798+0000 7f53a259c700 1 -- 172.21.15.112:0/3021291393 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f538c00cc00 con 0x7f5384003e60 2023-01-24T03:40:18.815 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.798+0000 7f53a259c700 1 -- 172.21.15.112:0/3021291393 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538c0070d0 con 0x7f5384003e60 2023-01-24T03:40:18.815 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.799+0000 7f53a8a3d700 1 -- 172.21.15.112:0/3021291393 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5384003e60 msgr2=0x7f5384004690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:18.815 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.799+0000 7f53a8a3d700 1 --2- 172.21.15.112:0/3021291393 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5384003e60 0x7f5384004690 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f538c00c210 tx=0x7f538c001e90 comp rx=0 tx=0).stop 2023-01-24T03:40:18.815 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.799+0000 7f53a359e700 1 -- 172.21.15.112:0/3021291393 reap_dead start 2023-01-24T03:40:18.815 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.799+0000 7f53a8a3d700 1 -- 172.21.15.112:0/3021291393 shutdown_connections 2023-01-24T03:40:18.816 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.799+0000 7f53a8a3d700 1 -- 172.21.15.112:0/3021291393 >> 172.21.15.112:0/3021291393 conn(0x7f53a40685b0 msgr2=0x7f53a40689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:18.816 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.799+0000 7f53a8a3d700 1 -- 172.21.15.112:0/3021291393 shutdown_connections 2023-01-24T03:40:18.816 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.799+0000 7f53a8a3d700 1 -- 172.21.15.112:0/3021291393 wait complete. 2023-01-24T03:40:18.816 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.800+0000 7f53a8a3d700 1 Processor -- start 2023-01-24T03:40:18.816 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.800+0000 7f53a8a3d700 1 -- start start 2023-01-24T03:40:18.817 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.801+0000 7f53a8a3d700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5384003e60 0x7f53a414afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:18.817 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.801+0000 7f53a8a3d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53a40c1d60 0x7f53a40c0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:18.817 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.801+0000 7f53a8a3d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f53a4152a40 0x7f53a40c1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:18.817 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.801+0000 7f53a8a3d700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a40c0390 con 0x7f53a4152a40 2023-01-24T03:40:18.817 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.801+0000 7f53a8a3d700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a4165ea0 con 0x7f53a40c1d60 2023-01-24T03:40:18.818 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.801+0000 7f53a8a3d700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53a40bf780 con 0x7f5384003e60 2023-01-24T03:40:18.818 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.801+0000 7f53a359e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f53a4152a40 0x7f53a40c1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:18.818 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.801+0000 7f53a359e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f53a4152a40 0x7f53a40c1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55686/0 (socket says 172.21.15.112:55686) 2023-01-24T03:40:18.818 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.801+0000 7f53a359e700 1 -- 172.21.15.112:0/4169629667 learned_addr learned my addr 172.21.15.112:0/4169629667 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:40:18.818 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.801+0000 7f53a2d9d700 1 --2- 172.21.15.112:0/4169629667 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5384003e60 0x7f53a414afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:18.819 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.801+0000 7f539a59c700 1 --2- 172.21.15.112:0/4169629667 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53a40c1d60 0x7f53a40c0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:18.819 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.801+0000 7f53a359e700 1 -- 172.21.15.112:0/4169629667 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5384003e60 msgr2=0x7f53a414afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:18.819 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.801+0000 7f53a359e700 1 --2- 172.21.15.112:0/4169629667 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5384003e60 0x7f53a414afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:18.819 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.801+0000 7f53a359e700 1 -- 172.21.15.112:0/4169629667 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53a40c1d60 msgr2=0x7f53a40c0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:18.819 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.801+0000 7f53a359e700 1 --2- 172.21.15.112:0/4169629667 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53a40c1d60 0x7f53a40c0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:18.820 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.801+0000 7f53a359e700 1 -- 172.21.15.112:0/4169629667 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a40bfa60 con 0x7f53a4152a40 2023-01-24T03:40:18.820 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.801+0000 7f53a359e700 1 --2- 172.21.15.112:0/4169629667 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f53a4152a40 0x7f53a40c1070 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f53a40899d0 tx=0x7f539c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:18.820 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.801+0000 7f539b7fe700 1 -- 172.21.15.112:0/4169629667 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539c00e040 con 0x7f53a4152a40 2023-01-24T03:40:18.820 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.801+0000 7f539b7fe700 1 -- 172.21.15.112:0/4169629667 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f539c00f040 con 0x7f53a4152a40 2023-01-24T03:40:18.820 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.801+0000 7f539b7fe700 1 -- 172.21.15.112:0/4169629667 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539c00fd70 con 0x7f53a4152a40 2023-01-24T03:40:18.821 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.801+0000 7f53a8a3d700 1 -- 172.21.15.112:0/4169629667 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53a40bfd40 con 0x7f53a4152a40 2023-01-24T03:40:18.821 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.801+0000 7f53a8a3d700 1 -- 172.21.15.112:0/4169629667 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53a40c2920 con 0x7f53a4152a40 2023-01-24T03:40:18.821 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.802+0000 7f53a8a3d700 1 -- 172.21.15.112:0/4169629667 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53a40c01e0 con 0x7f53a4152a40 2023-01-24T03:40:18.821 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.803+0000 7f539b7fe700 1 -- 172.21.15.112:0/4169629667 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f539c015640 con 0x7f53a4152a40 2023-01-24T03:40:18.822 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.803+0000 7f539b7fe700 1 --2- 172.21.15.112:0/4169629667 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f53880611f0 0x7f53880636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:18.822 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.803+0000 7f539b7fe700 1 -- 172.21.15.112:0/4169629667 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 7383+0+0 (secure 0 0 0) 0x7f539c07e200 con 0x7f53a4152a40 2023-01-24T03:40:18.823 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.803+0000 7f53a2d9d700 1 --2- 172.21.15.112:0/4169629667 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f53880611f0 0x7f53880636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:18.823 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.803+0000 7f53a2d9d700 1 --2- 172.21.15.112:0/4169629667 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f53880611f0 0x7f53880636a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5394006d80 tx=0x7f5394009060 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:18.823 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:18.807+0000 7f539b7fe700 1 -- 172.21.15.112:0/4169629667 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f539c04de00 con 0x7f53a4152a40 2023-01-24T03:40:19.030 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:19.028+0000 7f53a8a3d700 1 -- 172.21.15.112:0/4169629667 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53a405ea20 con 0x7f53a4152a40 2023-01-24T03:40:19.031 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:19.029+0000 7f539b7fe700 1 -- 172.21.15.112:0/4169629667 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+6121 (secure 0 0 0) 0x7f539c051420 con 0x7f53a4152a40 2023-01-24T03:40:19.031 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:40:19.031 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5186,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/2804746591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2804746591},{"type":"v1","addr":"172.21.15.142:6834","nonce":2804746591}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:39:49.907279+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":29,"addr":"172.21.15.142:6835/2136880103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2136880103},{"type":"v1","addr":"172.21.15.142:6835","nonce":2136880103}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:40:19.033 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:19.031+0000 7f53a8a3d700 1 -- 172.21.15.112:0/4169629667 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f53880611f0 msgr2=0x7f53880636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:19.033 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:19.031+0000 7f53a8a3d700 1 --2- 172.21.15.112:0/4169629667 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f53880611f0 0x7f53880636a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5394006d80 tx=0x7f5394009060 comp rx=0 tx=0).stop 2023-01-24T03:40:19.033 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:19.031+0000 7f53a8a3d700 1 -- 172.21.15.112:0/4169629667 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f53a4152a40 msgr2=0x7f53a40c1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:19.033 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:19.031+0000 7f53a8a3d700 1 --2- 172.21.15.112:0/4169629667 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f53a4152a40 0x7f53a40c1070 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f53a40899d0 tx=0x7f539c00b040 comp rx=0 tx=0).stop 2023-01-24T03:40:19.034 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:19.031+0000 7f53a8a3d700 1 -- 172.21.15.112:0/4169629667 shutdown_connections 2023-01-24T03:40:19.034 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:19.031+0000 7f53a8a3d700 1 --2- 172.21.15.112:0/4169629667 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f53880611f0 0x7f53880636a0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:19.034 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:19.031+0000 7f53a8a3d700 1 --2- 172.21.15.112:0/4169629667 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f53a4152a40 0x7f53a40c1070 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:19.034 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:19.031+0000 7f53a8a3d700 1 --2- 172.21.15.112:0/4169629667 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5384003e60 0x7f53a414afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:19.034 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:19.031+0000 7f53a8a3d700 1 --2- 172.21.15.112:0/4169629667 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f53a40c1d60 0x7f53a40c0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:19.035 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:19.031+0000 7f53a8a3d700 1 -- 172.21.15.112:0/4169629667 >> 172.21.15.112:0/4169629667 conn(0x7f53a40685b0 msgr2=0x7f53a414bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:19.035 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:19.031+0000 7f53a8a3d700 1 -- 172.21.15.112:0/4169629667 shutdown_connections 2023-01-24T03:40:19.035 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:19.031+0000 7f53a8a3d700 1 -- 172.21.15.112:0/4169629667 wait complete. 2023-01-24T03:40:19.036 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 41 2023-01-24T03:40:19.044 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-01-24T03:40:21.045 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:40:21.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.377+0000 7f8f8ae0f700 1 Processor -- start 2023-01-24T03:40:21.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.378+0000 7f8f8ae0f700 1 -- start start 2023-01-24T03:40:21.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.379+0000 7f8f8ae0f700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f8f8406b2e0 0x7f8f841556a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:21.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.379+0000 7f8f8ae0f700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f8f840c6820 0x7f8f840c6c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:21.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.379+0000 7f8f8ae0f700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f840c71a0 con 0x7f8f8406b6b0 2023-01-24T03:40:21.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.379+0000 7f8f8ae0f700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f840c4070 con 0x7f8f840c6820 2023-01-24T03:40:21.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.379+0000 7f8f8ae0f700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f840c2b20 con 0x7f8f8406b2e0 2023-01-24T03:40:21.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.379+0000 7f8f893ac700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f8f840c6820 0x7f8f840c6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:21.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.379+0000 7f8f893ac700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f8f840c6820 0x7f8f840c6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55730/0 (socket says 172.21.15.112:55730) 2023-01-24T03:40:21.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.379+0000 7f8f893ac700 1 -- 172.21.15.112:0/343591037 learned_addr learned my addr 172.21.15.112:0/343591037 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:40:21.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.380+0000 7f8f893ac700 1 -- 172.21.15.112:0/343591037 >> v2:172.21.15.142:3300/0 conn(0x7f8f8406b2e0 msgr2=0x7f8f841556a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-01-24T03:40:21.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.380+0000 7f8f893ac700 1 --2- 172.21.15.112:0/343591037 >> v2:172.21.15.142:3300/0 conn(0x7f8f8406b2e0 0x7f8f841556a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:21.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.380+0000 7f8f893ac700 1 -- 172.21.15.112:0/343591037 >> v1:172.21.15.112:6789/0 conn(0x7f8f8406b6b0 legacy=0x7f8f840c58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:21.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.380+0000 7f8f893ac700 1 -- 172.21.15.112:0/343591037 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f840c2e00 con 0x7f8f840c6820 2023-01-24T03:40:21.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.380+0000 7f8f893ac700 1 --2- 172.21.15.112:0/343591037 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8f840c6820 0x7f8f840c6c60 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f8f840c5030 tx=0x7f8f7c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51ea2ab4018bcea7 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:21.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.380+0000 7f8f837fe700 1 -- 172.21.15.112:0/343591037 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c00e070 con 0x7f8f840c6820 2023-01-24T03:40:21.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.381+0000 7f8f837fe700 1 -- 172.21.15.112:0/343591037 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f7c00ba50 con 0x7f8f840c6820 2023-01-24T03:40:21.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.381+0000 7f8f837fe700 1 -- 172.21.15.112:0/343591037 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c0092e0 con 0x7f8f840c6820 2023-01-24T03:40:21.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.381+0000 7f8f8ae0f700 1 -- 172.21.15.112:0/343591037 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8f840c6820 msgr2=0x7f8f840c6c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:21.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.381+0000 7f8f8ae0f700 1 --2- 172.21.15.112:0/343591037 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8f840c6820 0x7f8f840c6c60 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f8f840c5030 tx=0x7f8f7c009840 comp rx=0 tx=0).stop 2023-01-24T03:40:21.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.383+0000 7f8f8ae0f700 1 -- 172.21.15.112:0/343591037 shutdown_connections 2023-01-24T03:40:21.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.383+0000 7f8f8ae0f700 1 --2- 172.21.15.112:0/343591037 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8f840c6820 0x7f8f840c6c60 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:21.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.383+0000 7f8f8ae0f700 1 --2- 172.21.15.112:0/343591037 >> v2:172.21.15.142:3300/0 conn(0x7f8f8406b2e0 0x7f8f841556a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:21.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.383+0000 7f8f8ae0f700 1 -- 172.21.15.112:0/343591037 >> 172.21.15.112:0/343591037 conn(0x7f8f840678e0 msgr2=0x7f8f840c4590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:21.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.385+0000 7f8f8ae0f700 1 -- 172.21.15.112:0/343591037 shutdown_connections 2023-01-24T03:40:21.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.385+0000 7f8f8ae0f700 1 -- 172.21.15.112:0/343591037 wait complete. 2023-01-24T03:40:21.394 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.387+0000 7f8f8ae0f700 1 Processor -- start 2023-01-24T03:40:21.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.387+0000 7f8f8ae0f700 1 -- start start 2023-01-24T03:40:21.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.388+0000 7f8f8ae0f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8f8406b2e0 0x7f8f8414a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:21.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.388+0000 7f8f8ae0f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8f8406b6b0 0x7f8f84165860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:21.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.388+0000 7f8f8ae0f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8f840c6820 0x7f8f84165da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:21.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.388+0000 7f8f8ae0f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8414fa70 con 0x7f8f8406b6b0 2023-01-24T03:40:21.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.388+0000 7f8f8ae0f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f84168820 con 0x7f8f840c6820 2023-01-24T03:40:21.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.388+0000 7f8f8ae0f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f841653d0 con 0x7f8f8406b2e0 2023-01-24T03:40:21.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.388+0000 7f8f893ac700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8f840c6820 0x7f8f84165da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:21.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.389+0000 7f8f893ac700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8f840c6820 0x7f8f84165da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.112:48228/0 (socket says 172.21.15.112:48228) 2023-01-24T03:40:21.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.389+0000 7f8f893ac700 1 -- 172.21.15.112:0/3039253480 learned_addr learned my addr 172.21.15.112:0/3039253480 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:40:21.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.389+0000 7f8f83fff700 1 --2- 172.21.15.112:0/3039253480 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8f8406b6b0 0x7f8f84165860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:21.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.389+0000 7f8f88bab700 1 --2- 172.21.15.112:0/3039253480 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8f8406b2e0 0x7f8f8414a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:21.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.389+0000 7f8f893ac700 1 -- 172.21.15.112:0/3039253480 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8f8406b2e0 msgr2=0x7f8f8414a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:21.397 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.389+0000 7f8f893ac700 1 --2- 172.21.15.112:0/3039253480 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8f8406b2e0 0x7f8f8414a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:21.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.389+0000 7f8f893ac700 1 -- 172.21.15.112:0/3039253480 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8f8406b6b0 msgr2=0x7f8f84165860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:21.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.389+0000 7f8f893ac700 1 --2- 172.21.15.112:0/3039253480 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8f8406b6b0 0x7f8f84165860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:21.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.389+0000 7f8f893ac700 1 -- 172.21.15.112:0/3039253480 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f7c009ae0 con 0x7f8f840c6820 2023-01-24T03:40:21.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.389+0000 7f8f83fff700 1 --2- 172.21.15.112:0/3039253480 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8f8406b6b0 0x7f8f84165860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:40:21.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.389+0000 7f8f88bab700 1 --2- 172.21.15.112:0/3039253480 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8f8406b2e0 0x7f8f8414a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:40:21.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.389+0000 7f8f893ac700 1 --2- 172.21.15.112:0/3039253480 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8f840c6820 0x7f8f84165da0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8f8405e440 tx=0x7f8f7c018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:21.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.389+0000 7f8f80ff9700 1 -- 172.21.15.112:0/3039253480 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c008ce0 con 0x7f8f840c6820 2023-01-24T03:40:21.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.389+0000 7f8f80ff9700 1 -- 172.21.15.112:0/3039253480 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f7c008e60 con 0x7f8f840c6820 2023-01-24T03:40:21.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.389+0000 7f8f80ff9700 1 -- 172.21.15.112:0/3039253480 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c01c660 con 0x7f8f840c6820 2023-01-24T03:40:21.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.389+0000 7f8f8ae0f700 1 -- 172.21.15.112:0/3039253480 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f84168a50 con 0x7f8f840c6820 2023-01-24T03:40:21.400 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.390+0000 7f8f8ae0f700 1 -- 172.21.15.112:0/3039253480 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f84168ff0 con 0x7f8f840c6820 2023-01-24T03:40:21.400 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.391+0000 7f8f8ae0f700 1 -- 172.21.15.112:0/3039253480 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f8404bb80 con 0x7f8f840c6820 2023-01-24T03:40:21.400 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.391+0000 7f8f80ff9700 1 -- 172.21.15.112:0/3039253480 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f8f7c01ca50 con 0x7f8f840c6820 2023-01-24T03:40:21.400 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.392+0000 7f8f80ff9700 1 --2- 172.21.15.112:0/3039253480 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f8f6c061270 0x7f8f6c063720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:21.401 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.392+0000 7f8f80ff9700 1 -- 172.21.15.112:0/3039253480 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 7383+0+0 (secure 0 0 0) 0x7f8f7c07f7b0 con 0x7f8f840c6820 2023-01-24T03:40:21.401 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.392+0000 7f8f88bab700 1 --2- 172.21.15.112:0/3039253480 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f8f6c061270 0x7f8f6c063720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:21.401 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.392+0000 7f8f88bab700 1 --2- 172.21.15.112:0/3039253480 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f8f6c061270 0x7f8f6c063720 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8f70006fd0 tx=0x7f8f70008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:21.402 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.396+0000 7f8f80ff9700 1 -- 172.21.15.112:0/3039253480 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f8f7c04f2c0 con 0x7f8f840c6820 2023-01-24T03:40:21.628 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.626+0000 7f8f8ae0f700 1 -- 172.21.15.112:0/3039253480 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f840c6220 con 0x7f8f840c6820 2023-01-24T03:40:21.629 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.627+0000 7f8f80ff9700 1 -- 172.21.15.112:0/3039253480 <== mon.1 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+6121 (secure 0 0 0) 0x7f8f7c0528e0 con 0x7f8f840c6820 2023-01-24T03:40:21.629 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:40:21.629 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5186,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/2804746591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2804746591},{"type":"v1","addr":"172.21.15.142:6834","nonce":2804746591}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:39:49.907279+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":29,"addr":"172.21.15.142:6835/2136880103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2136880103},{"type":"v1","addr":"172.21.15.142:6835","nonce":2136880103}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:40:21.631 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.628+0000 7f8f8ae0f700 1 -- 172.21.15.112:0/3039253480 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f8f6c061270 msgr2=0x7f8f6c063720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:21.631 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.628+0000 7f8f8ae0f700 1 --2- 172.21.15.112:0/3039253480 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f8f6c061270 0x7f8f6c063720 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8f70006fd0 tx=0x7f8f70008040 comp rx=0 tx=0).stop 2023-01-24T03:40:21.631 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.628+0000 7f8f8ae0f700 1 -- 172.21.15.112:0/3039253480 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8f840c6820 msgr2=0x7f8f84165da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:21.632 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.628+0000 7f8f8ae0f700 1 --2- 172.21.15.112:0/3039253480 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8f840c6820 0x7f8f84165da0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8f8405e440 tx=0x7f8f7c018000 comp rx=0 tx=0).stop 2023-01-24T03:40:21.632 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.629+0000 7f8f8ae0f700 1 -- 172.21.15.112:0/3039253480 shutdown_connections 2023-01-24T03:40:21.632 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.629+0000 7f8f8ae0f700 1 --2- 172.21.15.112:0/3039253480 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f8f6c061270 0x7f8f6c063720 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:21.632 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.629+0000 7f8f8ae0f700 1 --2- 172.21.15.112:0/3039253480 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8f840c6820 0x7f8f84165da0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:21.632 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.629+0000 7f8f8ae0f700 1 --2- 172.21.15.112:0/3039253480 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f8f8406b2e0 0x7f8f8414a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:21.633 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.629+0000 7f8f8ae0f700 1 --2- 172.21.15.112:0/3039253480 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8f8406b6b0 0x7f8f84165860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:21.633 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.629+0000 7f8f8ae0f700 1 -- 172.21.15.112:0/3039253480 >> 172.21.15.112:0/3039253480 conn(0x7f8f840678e0 msgr2=0x7f8f8406eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:21.633 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.629+0000 7f8f8ae0f700 1 -- 172.21.15.112:0/3039253480 shutdown_connections 2023-01-24T03:40:21.633 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:21.629+0000 7f8f8ae0f700 1 -- 172.21.15.112:0/3039253480 wait complete. 2023-01-24T03:40:21.634 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 41 2023-01-24T03:40:21.642 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-01-24T03:40:23.643 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:40:23.768 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.766+0000 7fb2609b1700 1 Processor -- start 2023-01-24T03:40:23.768 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.766+0000 7fb2609b1700 1 -- start start 2023-01-24T03:40:23.771 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.767+0000 7fb2609b1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb25c0c7e40 0x7fb25c0b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:23.771 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.767+0000 7fb2609b1700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fb25c0ca3e0 0x7fb25c0ca820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:23.771 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.767+0000 7fb2609b1700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb25c156210 con 0x7fb25c155da0 2023-01-24T03:40:23.771 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.767+0000 7fb2609b1700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb25c0cad60 con 0x7fb25c0ca3e0 2023-01-24T03:40:23.772 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.767+0000 7fb2609b1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb25c0b1170 con 0x7fb25c0c7e40 2023-01-24T03:40:23.772 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.767+0000 7fb25ad9d700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fb25c0ca3e0 0x7fb25c0ca820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:23.772 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.767+0000 7fb25ad9d700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fb25c0ca3e0 0x7fb25c0ca820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55746/0 (socket says 172.21.15.112:55746) 2023-01-24T03:40:23.772 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.767+0000 7fb25ad9d700 1 -- 172.21.15.112:0/3879714913 learned_addr learned my addr 172.21.15.112:0/3879714913 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:40:23.772 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.767+0000 7fb25a59c700 1 --2- 172.21.15.112:0/3879714913 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb25c0c7e40 0x7fb25c0b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:23.773 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.767+0000 7fb25ad9d700 1 -- 172.21.15.112:0/3879714913 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb25c0c7e40 msgr2=0x7fb25c0b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:40:23.773 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.767+0000 7fb25ad9d700 1 --2- 172.21.15.112:0/3879714913 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb25c0c7e40 0x7fb25c0b0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:23.773 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.767+0000 7fb25ad9d700 1 -- 172.21.15.112:0/3879714913 >> v1:172.21.15.142:6789/0 conn(0x7fb25c155da0 legacy=0x7fb25c0ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:23.773 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.767+0000 7fb25ad9d700 1 -- 172.21.15.112:0/3879714913 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb25c0b1450 con 0x7fb25c0ca3e0 2023-01-24T03:40:23.773 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.768+0000 7fb25ad9d700 1 --2- 172.21.15.112:0/3879714913 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb25c0ca3e0 0x7fb25c0ca820 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fb254006fd0 tx=0x7fb2540090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a24d1cf86028302 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:23.774 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.768+0000 7fb25959a700 1 -- 172.21.15.112:0/3879714913 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb254011630 con 0x7fb25c0ca3e0 2023-01-24T03:40:23.774 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.768+0000 7fb25959a700 1 -- 172.21.15.112:0/3879714913 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2540117d0 con 0x7fb25c0ca3e0 2023-01-24T03:40:23.774 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.768+0000 7fb25959a700 1 -- 172.21.15.112:0/3879714913 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb254011b90 con 0x7fb25c0ca3e0 2023-01-24T03:40:23.774 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.768+0000 7fb2609b1700 1 -- 172.21.15.112:0/3879714913 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb25c0ca3e0 msgr2=0x7fb25c0ca820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:23.774 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.768+0000 7fb2609b1700 1 --2- 172.21.15.112:0/3879714913 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb25c0ca3e0 0x7fb25c0ca820 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fb254006fd0 tx=0x7fb2540090f0 comp rx=0 tx=0).stop 2023-01-24T03:40:23.775 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.768+0000 7fb2609b1700 1 -- 172.21.15.112:0/3879714913 shutdown_connections 2023-01-24T03:40:23.775 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.768+0000 7fb2609b1700 1 --2- 172.21.15.112:0/3879714913 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb25c0ca3e0 0x7fb25c0ca820 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:23.776 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.768+0000 7fb2609b1700 1 --2- 172.21.15.112:0/3879714913 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb25c0c7e40 0x7fb25c0b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:23.777 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.768+0000 7fb2609b1700 1 -- 172.21.15.112:0/3879714913 >> 172.21.15.112:0/3879714913 conn(0x7fb25c05e2c0 msgr2=0x7fb25c0cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:23.777 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.768+0000 7fb2609b1700 1 -- 172.21.15.112:0/3879714913 shutdown_connections 2023-01-24T03:40:23.777 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.768+0000 7fb2609b1700 1 -- 172.21.15.112:0/3879714913 wait complete. 2023-01-24T03:40:23.780 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.769+0000 7fb2609b1700 1 Processor -- start 2023-01-24T03:40:23.780 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.769+0000 7fb2609b1700 1 -- start start 2023-01-24T03:40:23.780 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.770+0000 7fb2609b1700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb25c0c7e40 0x7fb25c169530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:23.781 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.770+0000 7fb2609b1700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb25c0ca3e0 0x7fb25c14f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:23.781 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.770+0000 7fb2609b1700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb25c155da0 0x7fb25c0c1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:23.781 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.770+0000 7fb2609b1700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25c152d10 con 0x7fb25c155da0 2023-01-24T03:40:23.781 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.770+0000 7fb2609b1700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25c14fe30 con 0x7fb25c0ca3e0 2023-01-24T03:40:23.781 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.770+0000 7fb2609b1700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb25c14f6b0 con 0x7fb25c0c7e40 2023-01-24T03:40:23.782 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.770+0000 7fb25ad9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb25c155da0 0x7fb25c0c1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:23.782 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.770+0000 7fb25ad9d700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb25c155da0 0x7fb25c0c1990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55754/0 (socket says 172.21.15.112:55754) 2023-01-24T03:40:23.782 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.770+0000 7fb25ad9d700 1 -- 172.21.15.112:0/3806533203 learned_addr learned my addr 172.21.15.112:0/3806533203 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:40:23.782 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.770+0000 7fb259d9b700 1 --2- 172.21.15.112:0/3806533203 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb25c0ca3e0 0x7fb25c14f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:23.782 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.770+0000 7fb25a59c700 1 --2- 172.21.15.112:0/3806533203 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb25c0c7e40 0x7fb25c169530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:23.783 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.771+0000 7fb25ad9d700 1 -- 172.21.15.112:0/3806533203 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb25c0c7e40 msgr2=0x7fb25c169530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:23.783 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.771+0000 7fb25ad9d700 1 --2- 172.21.15.112:0/3806533203 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb25c0c7e40 0x7fb25c169530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:23.783 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.771+0000 7fb25ad9d700 1 -- 172.21.15.112:0/3806533203 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb25c0ca3e0 msgr2=0x7fb25c14f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:23.783 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.771+0000 7fb25ad9d700 1 --2- 172.21.15.112:0/3806533203 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb25c0ca3e0 0x7fb25c14f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:23.784 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.771+0000 7fb25ad9d700 1 -- 172.21.15.112:0/3806533203 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb254009370 con 0x7fb25c155da0 2023-01-24T03:40:23.784 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.771+0000 7fb25ad9d700 1 --2- 172.21.15.112:0/3806533203 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb25c155da0 0x7fb25c0c1990 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fb254008000 tx=0x7fb254016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:23.784 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.771+0000 7fb2527fc700 1 -- 172.21.15.112:0/3806533203 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb254011630 con 0x7fb25c155da0 2023-01-24T03:40:23.784 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.771+0000 7fb2527fc700 1 -- 172.21.15.112:0/3806533203 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb254020070 con 0x7fb25c155da0 2023-01-24T03:40:23.784 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.771+0000 7fb2527fc700 1 -- 172.21.15.112:0/3806533203 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2540276d0 con 0x7fb25c155da0 2023-01-24T03:40:23.785 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.771+0000 7fb2609b1700 1 -- 172.21.15.112:0/3806533203 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb25c152470 con 0x7fb25c155da0 2023-01-24T03:40:23.785 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.771+0000 7fb2609b1700 1 -- 172.21.15.112:0/3806533203 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb25c152a30 con 0x7fb25c155da0 2023-01-24T03:40:23.785 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.772+0000 7fb2609b1700 1 -- 172.21.15.112:0/3806533203 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb25c0c0800 con 0x7fb25c155da0 2023-01-24T03:40:23.785 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.772+0000 7fb2527fc700 1 -- 172.21.15.112:0/3806533203 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fb254017020 con 0x7fb25c155da0 2023-01-24T03:40:23.785 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.772+0000 7fb2527fc700 1 --2- 172.21.15.112:0/3806533203 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb2400611c0 0x7fb240063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:23.786 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.772+0000 7fb2527fc700 1 -- 172.21.15.112:0/3806533203 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 7383+0+0 (secure 0 0 0) 0x7fb254022070 con 0x7fb25c155da0 2023-01-24T03:40:23.786 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.773+0000 7fb25a59c700 1 --2- 172.21.15.112:0/3806533203 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb2400611c0 0x7fb240063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:23.786 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.773+0000 7fb25a59c700 1 --2- 172.21.15.112:0/3806533203 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb2400611c0 0x7fb240063670 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb24c009fd0 tx=0x7fb24c00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:23.786 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.777+0000 7fb2527fc700 1 -- 172.21.15.112:0/3806533203 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fb25404f550 con 0x7fb25c155da0 2023-01-24T03:40:23.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.995+0000 7fb2609b1700 1 -- 172.21.15.112:0/3806533203 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb25c04bb80 con 0x7fb25c155da0 2023-01-24T03:40:23.998 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.996+0000 7fb2527fc700 1 -- 172.21.15.112:0/3806533203 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+6121 (secure 0 0 0) 0x7fb254011b50 con 0x7fb25c155da0 2023-01-24T03:40:23.998 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:40:23.998 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5186,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/2804746591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2804746591},{"type":"v1","addr":"172.21.15.142:6834","nonce":2804746591}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:39:49.907279+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":29,"addr":"172.21.15.142:6835/2136880103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2136880103},{"type":"v1","addr":"172.21.15.142:6835","nonce":2136880103}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:40:24.000 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.998+0000 7fb2609b1700 1 -- 172.21.15.112:0/3806533203 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb2400611c0 msgr2=0x7fb240063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:24.000 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.998+0000 7fb2609b1700 1 --2- 172.21.15.112:0/3806533203 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb2400611c0 0x7fb240063670 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb24c009fd0 tx=0x7fb24c00b040 comp rx=0 tx=0).stop 2023-01-24T03:40:24.000 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.998+0000 7fb2609b1700 1 -- 172.21.15.112:0/3806533203 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb25c155da0 msgr2=0x7fb25c0c1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:24.001 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.998+0000 7fb2609b1700 1 --2- 172.21.15.112:0/3806533203 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb25c155da0 0x7fb25c0c1990 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fb254008000 tx=0x7fb254016000 comp rx=0 tx=0).stop 2023-01-24T03:40:24.001 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.998+0000 7fb2609b1700 1 -- 172.21.15.112:0/3806533203 shutdown_connections 2023-01-24T03:40:24.001 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.998+0000 7fb2609b1700 1 --2- 172.21.15.112:0/3806533203 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fb2400611c0 0x7fb240063670 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:24.001 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.998+0000 7fb2609b1700 1 --2- 172.21.15.112:0/3806533203 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb25c155da0 0x7fb25c0c1990 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:24.001 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.998+0000 7fb2609b1700 1 --2- 172.21.15.112:0/3806533203 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fb25c0c7e40 0x7fb25c169530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:24.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.998+0000 7fb2609b1700 1 --2- 172.21.15.112:0/3806533203 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb25c0ca3e0 0x7fb25c14f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:24.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.998+0000 7fb2609b1700 1 -- 172.21.15.112:0/3806533203 >> 172.21.15.112:0/3806533203 conn(0x7fb25c05e2c0 msgr2=0x7fb25c154f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:24.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.998+0000 7fb2609b1700 1 -- 172.21.15.112:0/3806533203 shutdown_connections 2023-01-24T03:40:24.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:23.998+0000 7fb2609b1700 1 -- 172.21.15.112:0/3806533203 wait complete. 2023-01-24T03:40:24.003 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 41 2023-01-24T03:40:24.012 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-01-24T03:40:26.013 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:40:26.137 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.135+0000 7fc7ff088700 1 Processor -- start 2023-01-24T03:40:26.137 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.135+0000 7fc7ff088700 1 -- start start 2023-01-24T03:40:26.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.135+0000 7fc7ff088700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc7f80c7e40 0x7fc7f80b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:26.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.135+0000 7fc7ff088700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fc7f8155da0 0x7fc7f80ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:26.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.135+0000 7fc7ff088700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fc7f80aef20 0x7fc7f80ca3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:26.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.135+0000 7fc7ff088700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7f8156210 con 0x7fc7f80aef20 2023-01-24T03:40:26.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.135+0000 7fc7ff088700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7f80b1170 con 0x7fc7f8155da0 2023-01-24T03:40:26.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.135+0000 7fc7ff088700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f80b1450 con 0x7fc7f80c7e40 2023-01-24T03:40:26.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.136+0000 7fc7fd625700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fc7f80aef20 0x7fc7f80ca3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:26.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.136+0000 7fc7fd625700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fc7f80aef20 0x7fc7f80ca3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55764/0 (socket says 172.21.15.112:55764) 2023-01-24T03:40:26.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.136+0000 7fc7fd625700 1 -- 172.21.15.112:0/1331580788 learned_addr learned my addr 172.21.15.112:0/1331580788 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:40:26.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.136+0000 7fc7fce24700 1 --2- 172.21.15.112:0/1331580788 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc7f80c7e40 0x7fc7f80b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:26.148 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.136+0000 7fc7f7fff700 1 --2- 172.21.15.112:0/1331580788 >> v2:172.21.15.142:3300/0 conn(0x7fc7f8155da0 0x7fc7f80ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:26.148 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.136+0000 7fc7fd625700 1 -- 172.21.15.112:0/1331580788 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc7f80c7e40 msgr2=0x7fc7f80b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:40:26.148 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.136+0000 7fc7fd625700 1 --2- 172.21.15.112:0/1331580788 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc7f80c7e40 0x7fc7f80b0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:26.148 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.136+0000 7fc7fd625700 1 -- 172.21.15.112:0/1331580788 >> v2:172.21.15.142:3300/0 conn(0x7fc7f8155da0 msgr2=0x7fc7f80ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:40:26.148 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.136+0000 7fc7fd625700 1 --2- 172.21.15.112:0/1331580788 >> v2:172.21.15.142:3300/0 conn(0x7fc7f8155da0 0x7fc7f80ae9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:26.149 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.136+0000 7fc7fd625700 1 -- 172.21.15.112:0/1331580788 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f80b1730 con 0x7fc7f80aef20 2023-01-24T03:40:26.149 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.136+0000 7fc7fce24700 1 --2- 172.21.15.112:0/1331580788 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc7f80c7e40 0x7fc7f80b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-01-24T03:40:26.149 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.136+0000 7fc7f7fff700 1 --2- 172.21.15.112:0/1331580788 >> v2:172.21.15.142:3300/0 conn(0x7fc7f8155da0 0x7fc7f80ae9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-01-24T03:40:26.149 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.136+0000 7fc7fd625700 1 --2- 172.21.15.112:0/1331580788 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7f80aef20 0x7fc7f80ca3e0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8067fd0 tx=0x7fc7f00090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aee47948781de90b server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:26.150 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.136+0000 7fc7f77fe700 1 -- 172.21.15.112:0/1331580788 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f0011630 con 0x7fc7f80aef20 2023-01-24T03:40:26.150 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.136+0000 7fc7f77fe700 1 -- 172.21.15.112:0/1331580788 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7f00117d0 con 0x7fc7f80aef20 2023-01-24T03:40:26.150 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.136+0000 7fc7f77fe700 1 -- 172.21.15.112:0/1331580788 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f0011b90 con 0x7fc7f80aef20 2023-01-24T03:40:26.150 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.137+0000 7fc7ff088700 1 -- 172.21.15.112:0/1331580788 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7f80aef20 msgr2=0x7fc7f80ca3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:26.150 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.137+0000 7fc7ff088700 1 --2- 172.21.15.112:0/1331580788 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7f80aef20 0x7fc7f80ca3e0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8067fd0 tx=0x7fc7f00090f0 comp rx=0 tx=0).stop 2023-01-24T03:40:26.151 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.137+0000 7fc7ff088700 1 -- 172.21.15.112:0/1331580788 shutdown_connections 2023-01-24T03:40:26.151 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.137+0000 7fc7ff088700 1 --2- 172.21.15.112:0/1331580788 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7f80aef20 0x7fc7f80ca3e0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:26.151 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.137+0000 7fc7ff088700 1 --2- 172.21.15.112:0/1331580788 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc7f80c7e40 0x7fc7f80b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:26.151 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.137+0000 7fc7ff088700 1 --2- 172.21.15.112:0/1331580788 >> v2:172.21.15.142:3300/0 conn(0x7fc7f8155da0 0x7fc7f80ae9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:26.151 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.137+0000 7fc7ff088700 1 -- 172.21.15.112:0/1331580788 >> 172.21.15.112:0/1331580788 conn(0x7fc7f805e2c0 msgr2=0x7fc7f80cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:26.152 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.137+0000 7fc7ff088700 1 -- 172.21.15.112:0/1331580788 shutdown_connections 2023-01-24T03:40:26.152 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.137+0000 7fc7ff088700 1 -- 172.21.15.112:0/1331580788 wait complete. 2023-01-24T03:40:26.152 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.138+0000 7fc7ff088700 1 Processor -- start 2023-01-24T03:40:26.152 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.138+0000 7fc7ff088700 1 -- start start 2023-01-24T03:40:26.152 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.138+0000 7fc7ff088700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc7f80aef20 0x7fc7f80cb410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:26.153 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.139+0000 7fc7ff088700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7f80c7e40 0x7fc7f80c32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:26.153 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.139+0000 7fc7ff088700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc7f8155da0 0x7fc7f80c37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:26.153 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.139+0000 7fc7ff088700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f8165990 con 0x7fc7f80c7e40 2023-01-24T03:40:26.153 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.139+0000 7fc7ff088700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f80c2e20 con 0x7fc7f80aef20 2023-01-24T03:40:26.153 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.139+0000 7fc7ff088700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f8164e30 con 0x7fc7f8155da0 2023-01-24T03:40:26.154 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.139+0000 7fc7f7fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7f80c7e40 0x7fc7f80c32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:26.154 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.139+0000 7fc7f7fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7f80c7e40 0x7fc7f80c32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55768/0 (socket says 172.21.15.112:55768) 2023-01-24T03:40:26.154 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.139+0000 7fc7f7fff700 1 -- 172.21.15.112:0/168878695 learned_addr learned my addr 172.21.15.112:0/168878695 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:40:26.154 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.139+0000 7fc7fd625700 1 --2- 172.21.15.112:0/168878695 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc7f8155da0 0x7fc7f80c37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:26.155 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.139+0000 7fc7fce24700 1 --2- 172.21.15.112:0/168878695 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc7f80aef20 0x7fc7f80cb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:26.155 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.139+0000 7fc7f7fff700 1 -- 172.21.15.112:0/168878695 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc7f8155da0 msgr2=0x7fc7f80c37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:26.155 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.139+0000 7fc7f7fff700 1 --2- 172.21.15.112:0/168878695 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc7f8155da0 0x7fc7f80c37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:26.155 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.139+0000 7fc7f7fff700 1 -- 172.21.15.112:0/168878695 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc7f80aef20 msgr2=0x7fc7f80cb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:26.155 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.139+0000 7fc7f7fff700 1 --2- 172.21.15.112:0/168878695 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc7f80aef20 0x7fc7f80cb410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:26.156 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.139+0000 7fc7f7fff700 1 -- 172.21.15.112:0/168878695 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f0009370 con 0x7fc7f80c7e40 2023-01-24T03:40:26.156 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.139+0000 7fc7f7fff700 1 --2- 172.21.15.112:0/168878695 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7f80c7e40 0x7fc7f80c32b0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fc7e000cfb0 tx=0x7fc7e0007870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:26.156 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.139+0000 7fc7f4ff9700 1 -- 172.21.15.112:0/168878695 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e000adb0 con 0x7fc7f80c7e40 2023-01-24T03:40:26.156 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.139+0000 7fc7f4ff9700 1 -- 172.21.15.112:0/168878695 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7e0016420 con 0x7fc7f80c7e40 2023-01-24T03:40:26.156 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.139+0000 7fc7ff088700 1 -- 172.21.15.112:0/168878695 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7f81650f0 con 0x7fc7f80c7e40 2023-01-24T03:40:26.157 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.139+0000 7fc7f4ff9700 1 -- 172.21.15.112:0/168878695 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e00167e0 con 0x7fc7f80c7e40 2023-01-24T03:40:26.157 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.139+0000 7fc7ff088700 1 -- 172.21.15.112:0/168878695 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7f81656b0 con 0x7fc7f80c7e40 2023-01-24T03:40:26.157 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.140+0000 7fc7ff088700 1 -- 172.21.15.112:0/168878695 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7f804bb80 con 0x7fc7f80c7e40 2023-01-24T03:40:26.157 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.141+0000 7fc7f4ff9700 1 -- 172.21.15.112:0/168878695 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fc7e0016bd0 con 0x7fc7f80c7e40 2023-01-24T03:40:26.157 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.141+0000 7fc7f4ff9700 1 --2- 172.21.15.112:0/168878695 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc7dc0611c0 0x7fc7dc063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:26.158 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.141+0000 7fc7f4ff9700 1 -- 172.21.15.112:0/168878695 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 7383+0+0 (secure 0 0 0) 0x7fc7e0012070 con 0x7fc7f80c7e40 2023-01-24T03:40:26.158 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.141+0000 7fc7fce24700 1 --2- 172.21.15.112:0/168878695 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc7dc0611c0 0x7fc7dc063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:26.158 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.141+0000 7fc7fce24700 1 --2- 172.21.15.112:0/168878695 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc7dc0611c0 0x7fc7dc063670 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc7e800abd0 tx=0x7fc7e8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:26.158 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.145+0000 7fc7f4ff9700 1 -- 172.21.15.112:0/168878695 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fc7e004d1d0 con 0x7fc7f80c7e40 2023-01-24T03:40:26.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.363+0000 7fc7ff088700 1 -- 172.21.15.112:0/168878695 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7f80c5df0 con 0x7fc7f80c7e40 2023-01-24T03:40:26.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.364+0000 7fc7f4ff9700 1 -- 172.21.15.112:0/168878695 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+6121 (secure 0 0 0) 0x7fc7e0016e60 con 0x7fc7f80c7e40 2023-01-24T03:40:26.366 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:40:26.366 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5186,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6834/2804746591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2804746591},{"type":"v1","addr":"172.21.15.142:6834","nonce":2804746591}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:39:49.907279+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5078},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5078":{"gid":5078,"name":"d","rank":0,"incarnation":37,"state":"up:active","state_seq":29,"addr":"172.21.15.142:6835/2136880103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":2136880103},{"type":"v1","addr":"172.21.15.142:6835","nonce":2136880103}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:40:26.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.365+0000 7fc7ff088700 1 -- 172.21.15.112:0/168878695 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc7dc0611c0 msgr2=0x7fc7dc063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:26.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.366+0000 7fc7ff088700 1 --2- 172.21.15.112:0/168878695 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc7dc0611c0 0x7fc7dc063670 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc7e800abd0 tx=0x7fc7e8008040 comp rx=0 tx=0).stop 2023-01-24T03:40:26.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.366+0000 7fc7ff088700 1 -- 172.21.15.112:0/168878695 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7f80c7e40 msgr2=0x7fc7f80c32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:26.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.366+0000 7fc7ff088700 1 --2- 172.21.15.112:0/168878695 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7f80c7e40 0x7fc7f80c32b0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fc7e000cfb0 tx=0x7fc7e0007870 comp rx=0 tx=0).stop 2023-01-24T03:40:26.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.366+0000 7fc7ff088700 1 -- 172.21.15.112:0/168878695 shutdown_connections 2023-01-24T03:40:26.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.366+0000 7fc7ff088700 1 --2- 172.21.15.112:0/168878695 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fc7dc0611c0 0x7fc7dc063670 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:26.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.366+0000 7fc7ff088700 1 --2- 172.21.15.112:0/168878695 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fc7f8155da0 0x7fc7f80c37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:26.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.366+0000 7fc7ff088700 1 --2- 172.21.15.112:0/168878695 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc7f80aef20 0x7fc7f80cb410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:26.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.366+0000 7fc7ff088700 1 --2- 172.21.15.112:0/168878695 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc7f80c7e40 0x7fc7f80c32b0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:26.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.366+0000 7fc7ff088700 1 -- 172.21.15.112:0/168878695 >> 172.21.15.112:0/168878695 conn(0x7fc7f805e2c0 msgr2=0x7fc7f80c8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:26.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.366+0000 7fc7ff088700 1 -- 172.21.15.112:0/168878695 shutdown_connections 2023-01-24T03:40:26.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:26.366+0000 7fc7ff088700 1 -- 172.21.15.112:0/168878695 wait complete. 2023-01-24T03:40:26.371 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 41 2023-01-24T03:40:26.379 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-01-24T03:40:28.380 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:40:28.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.503+0000 7f37fc4f2700 1 Processor -- start 2023-01-24T03:40:28.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.503+0000 7f37fc4f2700 1 -- start start 2023-01-24T03:40:28.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.504+0000 7f37fc4f2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37f40c2490 0x7f37f4155710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:28.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.504+0000 7f37fc4f2700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f40c18f0 con 0x7f37f414c290 2023-01-24T03:40:28.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.504+0000 7f37fc4f2700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f40c1030 con 0x7f37f414c660 2023-01-24T03:40:28.512 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.504+0000 7f37fc4f2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37f406bad0 con 0x7f37f40c2490 2023-01-24T03:40:28.512 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.504+0000 7f37fa28e700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f37f414c290 0x7f37f406ec80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:36762/0 (socket says 172.21.15.112:36762) 2023-01-24T03:40:28.512 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.504+0000 7f37fa28e700 1 -- 172.21.15.112:0/1004591260 learned_addr learned my addr 172.21.15.112:0/1004591260 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:40:28.512 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.504+0000 7f37f928c700 1 -- 172.21.15.112:0/1004591260 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3808574028 0 0) 0x7f37f40c18f0 con 0x7f37f414c290 2023-01-24T03:40:28.512 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.504+0000 7f37f928c700 1 -- 172.21.15.112:0/1004591260 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37e00033f0 con 0x7f37f414c290 2023-01-24T03:40:28.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.505+0000 7f37faa8f700 1 --2- 172.21.15.112:0/1004591260 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37f40c2490 0x7f37f4155710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:28.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.505+0000 7f37f928c700 1 -- 172.21.15.112:0/1004591260 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4087611336 0 0) 0x7f37e00033f0 con 0x7f37f414c290 2023-01-24T03:40:28.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.505+0000 7f37f928c700 1 -- 172.21.15.112:0/1004591260 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37f40c2490 msgr2=0x7f37f4155710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:40:28.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.505+0000 7f37f928c700 1 --2- 172.21.15.112:0/1004591260 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37f40c2490 0x7f37f4155710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:28.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.505+0000 7f37f928c700 1 -- 172.21.15.112:0/1004591260 >> v1:172.21.15.142:6789/0 conn(0x7f37f414c660 legacy=0x7f37f40c1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:28.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.505+0000 7f37f928c700 1 -- 172.21.15.112:0/1004591260 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f406bdb0 con 0x7f37f414c290 2023-01-24T03:40:28.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.505+0000 7f37f928c700 1 -- 172.21.15.112:0/1004591260 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7f37e40017f0 con 0x7f37f414c290 2023-01-24T03:40:28.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.505+0000 7f37faa8f700 1 --2- 172.21.15.112:0/1004591260 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37f40c2490 0x7f37f4155710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:40:28.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.505+0000 7f37f928c700 1 -- 172.21.15.112:0/1004591260 >> v1:172.21.15.112:6789/0 conn(0x7f37f414c290 legacy=0x7f37f406ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:28.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.505+0000 7f37f928c700 1 --2- 172.21.15.112:0/1004591260 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37f40c2490 0x7f37e0003910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:28.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.505+0000 7f37f928c700 1 --2- 172.21.15.112:0/1004591260 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37e0003f00 0x7f37e0004690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:28.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.505+0000 7f37f928c700 1 --2- 172.21.15.112:0/1004591260 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37e0004bd0 0x7f37e0005040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:28.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.505+0000 7f37f928c700 1 -- 172.21.15.112:0/1004591260 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f406bad0 con 0x7f37e0004bd0 2023-01-24T03:40:28.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.505+0000 7f37f928c700 1 -- 172.21.15.112:0/1004591260 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e000bb20 con 0x7f37e0003f00 2023-01-24T03:40:28.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.505+0000 7f37f928c700 1 -- 172.21.15.112:0/1004591260 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37e000bce0 con 0x7f37f40c2490 2023-01-24T03:40:28.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.505+0000 7f37faa8f700 1 --2- 172.21.15.112:0/1004591260 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37e0004bd0 0x7f37e0005040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:28.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.505+0000 7f37fa28e700 1 --2- 172.21.15.112:0/1004591260 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37f40c2490 0x7f37e0003910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:28.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.505+0000 7f37f9a8d700 1 --2- 172.21.15.112:0/1004591260 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37e0003f00 0x7f37e0004690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:28.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.505+0000 7f37faa8f700 1 -- 172.21.15.112:0/1004591260 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37f40c2490 msgr2=0x7f37e0003910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:40:28.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.505+0000 7f37faa8f700 1 --2- 172.21.15.112:0/1004591260 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37f40c2490 0x7f37e0003910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:28.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.505+0000 7f37faa8f700 1 -- 172.21.15.112:0/1004591260 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37e0003f00 msgr2=0x7f37e0004690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:40:28.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.505+0000 7f37faa8f700 1 --2- 172.21.15.112:0/1004591260 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37e0003f00 0x7f37e0004690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:28.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.505+0000 7f37faa8f700 1 -- 172.21.15.112:0/1004591260 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e00033f0 con 0x7f37e0004bd0 2023-01-24T03:40:28.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.505+0000 7f37faa8f700 1 --2- 172.21.15.112:0/1004591260 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37e0004bd0 0x7f37e0005040 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f37f40c28f0 tx=0x7f37ec008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16b2c65a8b2dfe98 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:28.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.506+0000 7f37f928c700 1 -- 172.21.15.112:0/1004591260 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ec00b400 con 0x7f37e0004bd0 2023-01-24T03:40:28.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.506+0000 7f37f928c700 1 -- 172.21.15.112:0/1004591260 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37ec00d5d0 con 0x7f37e0004bd0 2023-01-24T03:40:28.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.506+0000 7f37f928c700 1 -- 172.21.15.112:0/1004591260 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ec00d990 con 0x7f37e0004bd0 2023-01-24T03:40:28.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.506+0000 7f37fc4f2700 1 -- 172.21.15.112:0/1004591260 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37e0004bd0 msgr2=0x7f37e0005040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:28.519 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.506+0000 7f37fc4f2700 1 --2- 172.21.15.112:0/1004591260 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37e0004bd0 0x7f37e0005040 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f37f40c28f0 tx=0x7f37ec008040 comp rx=0 tx=0).stop 2023-01-24T03:40:28.519 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.506+0000 7f37faa8f700 1 -- 172.21.15.112:0/1004591260 reap_dead start 2023-01-24T03:40:28.519 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.506+0000 7f37fc4f2700 1 -- 172.21.15.112:0/1004591260 shutdown_connections 2023-01-24T03:40:28.519 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.506+0000 7f37fc4f2700 1 -- 172.21.15.112:0/1004591260 >> 172.21.15.112:0/1004591260 conn(0x7f37f4067f20 msgr2=0x7f37f4068320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:28.519 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.506+0000 7f37fc4f2700 1 -- 172.21.15.112:0/1004591260 shutdown_connections 2023-01-24T03:40:28.520 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.506+0000 7f37fc4f2700 1 -- 172.21.15.112:0/1004591260 wait complete. 2023-01-24T03:40:28.520 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.507+0000 7f37fc4f2700 1 Processor -- start 2023-01-24T03:40:28.520 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.507+0000 7f37fc4f2700 1 -- start start 2023-01-24T03:40:28.520 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.508+0000 7f37fc4f2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37e0004bd0 0x7f37f414e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:28.520 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.508+0000 7f37fc4f2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37f414b920 0x7f37f414dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:28.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.508+0000 7f37fc4f2700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37f40c8750 0x7f37f4149730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:28.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.508+0000 7f37fc4f2700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f4147b70 con 0x7f37f40c8750 2023-01-24T03:40:28.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.508+0000 7f37fc4f2700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f414d410 con 0x7f37e0004bd0 2023-01-24T03:40:28.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.508+0000 7f37fc4f2700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37f414cb20 con 0x7f37f414b920 2023-01-24T03:40:28.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.508+0000 7f37faa8f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37f40c8750 0x7f37f4149730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:28.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.508+0000 7f37faa8f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37f40c8750 0x7f37f4149730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55790/0 (socket says 172.21.15.112:55790) 2023-01-24T03:40:28.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.508+0000 7f37faa8f700 1 -- 172.21.15.112:0/4002362366 learned_addr learned my addr 172.21.15.112:0/4002362366 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:40:28.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.508+0000 7f37fa28e700 1 --2- 172.21.15.112:0/4002362366 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37e0004bd0 0x7f37f414e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:28.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.508+0000 7f37f9a8d700 1 --2- 172.21.15.112:0/4002362366 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37f414b920 0x7f37f414dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:28.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.508+0000 7f37faa8f700 1 -- 172.21.15.112:0/4002362366 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37f414b920 msgr2=0x7f37f414dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:28.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.508+0000 7f37faa8f700 1 --2- 172.21.15.112:0/4002362366 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37f414b920 0x7f37f414dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:28.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.508+0000 7f37faa8f700 1 -- 172.21.15.112:0/4002362366 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37e0004bd0 msgr2=0x7f37f414e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:28.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.508+0000 7f37faa8f700 1 --2- 172.21.15.112:0/4002362366 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37e0004bd0 0x7f37f414e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:28.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.508+0000 7f37faa8f700 1 -- 172.21.15.112:0/4002362366 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f4165ab0 con 0x7f37f40c8750 2023-01-24T03:40:28.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.508+0000 7f37faa8f700 1 --2- 172.21.15.112:0/4002362366 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37f40c8750 0x7f37f4149730 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f37ec014000 tx=0x7f37ec01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:28.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.509+0000 7f37f27fc700 1 -- 172.21.15.112:0/4002362366 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ec012630 con 0x7f37f40c8750 2023-01-24T03:40:28.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.509+0000 7f37f27fc700 1 -- 172.21.15.112:0/4002362366 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37ec012790 con 0x7f37f40c8750 2023-01-24T03:40:28.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.509+0000 7f37fc4f2700 1 -- 172.21.15.112:0/4002362366 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37f414cdc0 con 0x7f37f40c8750 2023-01-24T03:40:28.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.509+0000 7f37f27fc700 1 -- 172.21.15.112:0/4002362366 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ec012b30 con 0x7f37f40c8750 2023-01-24T03:40:28.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.509+0000 7f37fc4f2700 1 -- 172.21.15.112:0/4002362366 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37f40c47f0 con 0x7f37f40c8750 2023-01-24T03:40:28.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.509+0000 7f37fc4f2700 1 -- 172.21.15.112:0/4002362366 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37f404bb80 con 0x7f37f40c8750 2023-01-24T03:40:28.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.510+0000 7f37f27fc700 1 -- 172.21.15.112:0/4002362366 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f37ec025030 con 0x7f37f40c8750 2023-01-24T03:40:28.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.510+0000 7f37f27fc700 1 --2- 172.21.15.112:0/4002362366 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f37d40611f0 0x7f37d40636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:28.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.510+0000 7f37f27fc700 1 -- 172.21.15.112:0/4002362366 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 7383+0+0 (secure 0 0 0) 0x7f37ec000e40 con 0x7f37f40c8750 2023-01-24T03:40:28.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.510+0000 7f37fa28e700 1 --2- 172.21.15.112:0/4002362366 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f37d40611f0 0x7f37d40636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:28.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.511+0000 7f37fa28e700 1 --2- 172.21.15.112:0/4002362366 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f37d40611f0 0x7f37d40636a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f37e4001e30 tx=0x7f37e4007dd0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:28.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.514+0000 7f37f27fc700 1 -- 172.21.15.112:0/4002362366 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f37ec04df20 con 0x7f37f40c8750 2023-01-24T03:40:28.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.745+0000 7f37fc4f2700 1 -- 172.21.15.112:0/4002362366 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37f414d550 con 0x7f37f40c8750 2023-01-24T03:40:28.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.746+0000 7f37f27fc700 1 -- 172.21.15.112:0/4002362366 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+5360 (secure 0 0 0) 0x7f37ec051540 con 0x7f37f40c8750 2023-01-24T03:40:28.748 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:40:28.748 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:40:28.725494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5186},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5186":{"gid":5186,"name":"b","rank":0,"incarnation":42,"state":"up:replay","state_seq":1,"addr":"172.21.15.142:6834/2804746591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2804746591},{"type":"v1","addr":"172.21.15.142:6834","nonce":2804746591}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:40:28.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.747+0000 7f37fc4f2700 1 -- 172.21.15.112:0/4002362366 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f37d40611f0 msgr2=0x7f37d40636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:28.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.747+0000 7f37fc4f2700 1 --2- 172.21.15.112:0/4002362366 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f37d40611f0 0x7f37d40636a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f37e4001e30 tx=0x7f37e4007dd0 comp rx=0 tx=0).stop 2023-01-24T03:40:28.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.748+0000 7f37fc4f2700 1 -- 172.21.15.112:0/4002362366 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37f40c8750 msgr2=0x7f37f4149730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:28.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.748+0000 7f37fc4f2700 1 --2- 172.21.15.112:0/4002362366 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37f40c8750 0x7f37f4149730 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f37ec014000 tx=0x7f37ec01a000 comp rx=0 tx=0).stop 2023-01-24T03:40:28.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.748+0000 7f37fc4f2700 1 -- 172.21.15.112:0/4002362366 shutdown_connections 2023-01-24T03:40:28.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.748+0000 7f37fc4f2700 1 --2- 172.21.15.112:0/4002362366 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f37d40611f0 0x7f37d40636a0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:28.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.748+0000 7f37fc4f2700 1 --2- 172.21.15.112:0/4002362366 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37f40c8750 0x7f37f4149730 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:28.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.748+0000 7f37fc4f2700 1 --2- 172.21.15.112:0/4002362366 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f37e0004bd0 0x7f37f414e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:28.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.748+0000 7f37fc4f2700 1 --2- 172.21.15.112:0/4002362366 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f37f414b920 0x7f37f414dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:28.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.748+0000 7f37fc4f2700 1 -- 172.21.15.112:0/4002362366 >> 172.21.15.112:0/4002362366 conn(0x7f37f4067f20 msgr2=0x7f37f40c3070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:28.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.748+0000 7f37fc4f2700 1 -- 172.21.15.112:0/4002362366 shutdown_connections 2023-01-24T03:40:28.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.748+0000 7f37fc4f2700 1 -- 172.21.15.112:0/4002362366 wait complete. 2023-01-24T03:40:28.755 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 42 2023-01-24T03:40:28.761 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2023-01-24T03:40:28.762 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-01-24T03:40:28.762 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:40:28.874 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.872+0000 7f884337c700 1 Processor -- start 2023-01-24T03:40:28.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.872+0000 7f884337c700 1 -- start start 2023-01-24T03:40:28.879 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.873+0000 7f884337c700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f883c06cf50 0x7f883c0c1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:28.880 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.873+0000 7f884337c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f883c0c3560 0x7f883c0c4790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:28.880 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.873+0000 7f884337c700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f883c153270 con 0x7f883c06d320 2023-01-24T03:40:28.880 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.873+0000 7f884337c700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f883c153e80 con 0x7f883c06cf50 2023-01-24T03:40:28.880 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.873+0000 7f884337c700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f883c0c2870 con 0x7f883c0c3560 2023-01-24T03:40:28.880 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.873+0000 7f8841118700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f883c06cf50 0x7f883c0c1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:28.881 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.873+0000 7f8841118700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f883c06cf50 0x7f883c0c1560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55804/0 (socket says 172.21.15.112:55804) 2023-01-24T03:40:28.881 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.873+0000 7f8841118700 1 -- 172.21.15.112:0/2800455007 learned_addr learned my addr 172.21.15.112:0/2800455007 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:40:28.881 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.874+0000 7f8841919700 1 --2- 172.21.15.112:0/2800455007 >> v2:172.21.15.142:3300/0 conn(0x7f883c0c3560 0x7f883c0c4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:28.881 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.874+0000 7f8841118700 1 -- 172.21.15.112:0/2800455007 >> v2:172.21.15.142:3300/0 conn(0x7f883c0c3560 msgr2=0x7f883c0c4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:40:28.881 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.874+0000 7f8841118700 1 --2- 172.21.15.112:0/2800455007 >> v2:172.21.15.142:3300/0 conn(0x7f883c0c3560 0x7f883c0c4790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:28.882 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.874+0000 7f8841118700 1 -- 172.21.15.112:0/2800455007 >> v1:172.21.15.142:6789/0 conn(0x7f883c06d320 legacy=0x7f883c0c2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:40:28.882 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.874+0000 7f8841118700 1 -- 172.21.15.112:0/2800455007 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f883c1534e0 con 0x7f883c06cf50 2023-01-24T03:40:28.882 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.874+0000 7f8841118700 1 --2- 172.21.15.112:0/2800455007 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f883c06cf50 0x7f883c0c1560 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f883c06eeb0 tx=0x7f88380090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2074b23a98f01900 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:28.882 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.874+0000 7f882bfff700 1 -- 172.21.15.112:0/2800455007 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8838011630 con 0x7f883c06cf50 2023-01-24T03:40:28.882 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.874+0000 7f882bfff700 1 -- 172.21.15.112:0/2800455007 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88380117b0 con 0x7f883c06cf50 2023-01-24T03:40:28.883 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.874+0000 7f882bfff700 1 -- 172.21.15.112:0/2800455007 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8838011b70 con 0x7f883c06cf50 2023-01-24T03:40:28.883 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.874+0000 7f884337c700 1 -- 172.21.15.112:0/2800455007 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f883c06cf50 msgr2=0x7f883c0c1560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:28.883 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.874+0000 7f884337c700 1 --2- 172.21.15.112:0/2800455007 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f883c06cf50 0x7f883c0c1560 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f883c06eeb0 tx=0x7f88380090f0 comp rx=0 tx=0).stop 2023-01-24T03:40:28.883 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.874+0000 7f884337c700 1 -- 172.21.15.112:0/2800455007 shutdown_connections 2023-01-24T03:40:28.884 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.874+0000 7f884337c700 1 --2- 172.21.15.112:0/2800455007 >> v2:172.21.15.142:3300/0 conn(0x7f883c0c3560 0x7f883c0c4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:28.884 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.874+0000 7f884337c700 1 --2- 172.21.15.112:0/2800455007 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f883c06cf50 0x7f883c0c1560 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:28.884 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.874+0000 7f884337c700 1 -- 172.21.15.112:0/2800455007 >> 172.21.15.112:0/2800455007 conn(0x7f883c0c9820 msgr2=0x7f883c0c9c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:28.884 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.874+0000 7f884337c700 1 -- 172.21.15.112:0/2800455007 shutdown_connections 2023-01-24T03:40:28.885 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.874+0000 7f884337c700 1 -- 172.21.15.112:0/2800455007 wait complete. 2023-01-24T03:40:28.885 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.875+0000 7f884337c700 1 Processor -- start 2023-01-24T03:40:28.885 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.875+0000 7f884337c700 1 -- start start 2023-01-24T03:40:28.885 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.876+0000 7f884337c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f883c06cf50 0x7f883c14af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:28.885 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.876+0000 7f884337c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f883c06d320 0x7f883c158b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:28.886 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.876+0000 7f884337c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f883c0c3560 0x7f883c1499e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:28.886 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.876+0000 7f884337c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f883c069fa0 con 0x7f883c06cf50 2023-01-24T03:40:28.886 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.876+0000 7f884337c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f883c14aac0 con 0x7f883c06d320 2023-01-24T03:40:28.886 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.876+0000 7f884337c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f883c069cc0 con 0x7f883c0c3560 2023-01-24T03:40:28.886 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.876+0000 7f8841118700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f883c06cf50 0x7f883c14af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:28.887 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.876+0000 7f8841118700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f883c06cf50 0x7f883c14af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55810/0 (socket says 172.21.15.112:55810) 2023-01-24T03:40:28.887 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.876+0000 7f8841118700 1 -- 172.21.15.112:0/2889908238 learned_addr learned my addr 172.21.15.112:0/2889908238 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:40:28.887 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.877+0000 7f8841118700 1 -- 172.21.15.112:0/2889908238 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f883c0c3560 msgr2=0x7f883c1499e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:28.887 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.877+0000 7f8841118700 1 --2- 172.21.15.112:0/2889908238 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f883c0c3560 0x7f883c1499e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:28.887 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.877+0000 7f8841118700 1 -- 172.21.15.112:0/2889908238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f883c06d320 msgr2=0x7f883c158b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:28.888 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.877+0000 7f8840917700 1 --2- 172.21.15.112:0/2889908238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f883c06d320 0x7f883c158b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:28.888 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.877+0000 7f8841118700 1 --2- 172.21.15.112:0/2889908238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f883c06d320 0x7f883c158b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:28.888 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.877+0000 7f8841118700 1 -- 172.21.15.112:0/2889908238 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8838009370 con 0x7f883c06cf50 2023-01-24T03:40:28.888 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.877+0000 7f8841118700 1 --2- 172.21.15.112:0/2889908238 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f883c06cf50 0x7f883c14af50 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f8838013000 tx=0x7f8838014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:28.889 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.877+0000 7f88297fa700 1 -- 172.21.15.112:0/2889908238 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8838011b30 con 0x7f883c06cf50 2023-01-24T03:40:28.889 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.877+0000 7f88297fa700 1 -- 172.21.15.112:0/2889908238 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8838026450 con 0x7f883c06cf50 2023-01-24T03:40:28.889 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.877+0000 7f884337c700 1 -- 172.21.15.112:0/2889908238 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f883c14dc40 con 0x7f883c06cf50 2023-01-24T03:40:28.889 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.877+0000 7f88297fa700 1 -- 172.21.15.112:0/2889908238 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8838026810 con 0x7f883c06cf50 2023-01-24T03:40:28.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.877+0000 7f884337c700 1 -- 172.21.15.112:0/2889908238 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f883c14b490 con 0x7f883c06cf50 2023-01-24T03:40:28.891 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.878+0000 7f884337c700 1 -- 172.21.15.112:0/2889908238 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f883c04bb80 con 0x7f883c06cf50 2023-01-24T03:40:28.891 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.878+0000 7f88297fa700 1 -- 172.21.15.112:0/2889908238 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f8838023070 con 0x7f883c06cf50 2023-01-24T03:40:28.891 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.879+0000 7f88297fa700 1 --2- 172.21.15.112:0/2889908238 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f881c0611c0 0x7f881c063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:28.892 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.879+0000 7f88297fa700 1 -- 172.21.15.112:0/2889908238 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 7469+0+0 (secure 0 0 0) 0x7f883807f440 con 0x7f883c06cf50 2023-01-24T03:40:28.892 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.879+0000 7f8840917700 1 --2- 172.21.15.112:0/2889908238 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f881c0611c0 0x7f881c063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:28.892 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.879+0000 7f8840917700 1 --2- 172.21.15.112:0/2889908238 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f881c0611c0 0x7f881c063670 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f882c007fd0 tx=0x7f882c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:28.892 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:28.883+0000 7f88297fa700 1 -- 172.21.15.112:0/2889908238 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f883804eff0 con 0x7f883c06cf50 2023-01-24T03:40:29.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:29.111+0000 7f884337c700 1 -- 172.21.15.112:0/2889908238 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f883c1537c0 con 0x7f883c06cf50 2023-01-24T03:40:29.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:29.112+0000 7f88297fa700 1 -- 172.21.15.112:0/2889908238 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+5360 (secure 0 0 0) 0x7f8838015090 con 0x7f883c06cf50 2023-01-24T03:40:29.114 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:40:29.115 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:40:28.725494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5186},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5186":{"gid":5186,"name":"b","rank":0,"incarnation":42,"state":"up:replay","state_seq":1,"addr":"172.21.15.142:6834/2804746591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2804746591},{"type":"v1","addr":"172.21.15.142:6834","nonce":2804746591}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:40:29.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:29.114+0000 7f884337c700 1 -- 172.21.15.112:0/2889908238 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f881c0611c0 msgr2=0x7f881c063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:29.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:29.114+0000 7f884337c700 1 --2- 172.21.15.112:0/2889908238 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f881c0611c0 0x7f881c063670 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f882c007fd0 tx=0x7f882c009040 comp rx=0 tx=0).stop 2023-01-24T03:40:29.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:29.114+0000 7f884337c700 1 -- 172.21.15.112:0/2889908238 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f883c06cf50 msgr2=0x7f883c14af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:29.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:29.114+0000 7f884337c700 1 --2- 172.21.15.112:0/2889908238 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f883c06cf50 0x7f883c14af50 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f8838013000 tx=0x7f8838014000 comp rx=0 tx=0).stop 2023-01-24T03:40:29.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:29.114+0000 7f884337c700 1 -- 172.21.15.112:0/2889908238 shutdown_connections 2023-01-24T03:40:29.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:29.114+0000 7f884337c700 1 --2- 172.21.15.112:0/2889908238 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f881c0611c0 0x7f881c063670 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:29.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:29.114+0000 7f884337c700 1 --2- 172.21.15.112:0/2889908238 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f883c06cf50 0x7f883c14af50 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:29.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:29.114+0000 7f884337c700 1 --2- 172.21.15.112:0/2889908238 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f883c0c3560 0x7f883c1499e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:29.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:29.114+0000 7f884337c700 1 --2- 172.21.15.112:0/2889908238 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f883c06d320 0x7f883c158b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:29.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:29.114+0000 7f884337c700 1 -- 172.21.15.112:0/2889908238 >> 172.21.15.112:0/2889908238 conn(0x7f883c0c9820 msgr2=0x7f883c0c0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:29.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:29.114+0000 7f884337c700 1 -- 172.21.15.112:0/2889908238 shutdown_connections 2023-01-24T03:40:29.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:29.114+0000 7f884337c700 1 -- 172.21.15.112:0/2889908238 wait complete. 2023-01-24T03:40:29.119 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 42 2023-01-24T03:40:29.128 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 42, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-01-24T03:30:20.971710+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 42, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5186": { "addr": "172.21.15.142:6834/2804746591", "addrs": { "addrvec": [ { "addr": "172.21.15.142:6832", "nonce": 2804746591, "type": "v2" }, { "addr": "172.21.15.142:6834", "nonce": 2804746591, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5186, "incarnation": 42, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 49, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-01-24T03:40:28.725494+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5186 } } } ], "standbys": [ { "addr": "172.21.15.142:6837/2181345869", "addrs": { "addrvec": [ { "addr": "172.21.15.142:6836", "nonce": 2181345869, "type": "v2" }, { "addr": "172.21.15.142:6837", "nonce": 2181345869, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6835/330343927", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6834", "nonce": 330343927, "type": "v2" }, { "addr": "172.21.15.112:6835", "nonce": 330343927, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6837/12176817", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6836", "nonce": 12176817, "type": "v2" }, { "addr": "172.21.15.112:6837", "nonce": 12176817, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4276, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6839/1110983841", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6838", "nonce": 1110983841, "type": "v2" }, { "addr": "172.21.15.112:6839", "nonce": 1110983841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4281, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-01-24T03:40:30.365 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:40:30.368 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:40:31.130 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:40:31.254 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.251+0000 7f22edfe8700 1 Processor -- start 2023-01-24T03:40:31.254 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.251+0000 7f22edfe8700 1 -- start start 2023-01-24T03:40:31.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.252+0000 7f22edfe8700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f22e80c5680 0x7f22e8156730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:31.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.252+0000 7f22edfe8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22e80c5a50 0x7f22e8156c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:31.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.252+0000 7f22edfe8700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22e80c3d10 con 0x7f22e8155e20 2023-01-24T03:40:31.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.252+0000 7f22edfe8700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22e80c4eb0 con 0x7f22e80c5680 2023-01-24T03:40:31.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.252+0000 7f22edfe8700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22e80c2e20 con 0x7f22e80c5a50 2023-01-24T03:40:31.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.252+0000 7f22e7fff700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f22e8155e20 0x7f22e80cb870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:36806/0 (socket says 172.21.15.112:36806) 2023-01-24T03:40:31.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.252+0000 7f22e7fff700 1 -- 172.21.15.112:0/591021104 learned_addr learned my addr 172.21.15.112:0/591021104 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:40:31.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.252+0000 7f22ecfe6700 1 --2- 172.21.15.112:0/591021104 >> v2:172.21.15.142:3300/0 conn(0x7f22e80c5680 0x7f22e8156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:31.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.252+0000 7f22ed7e7700 1 --2- 172.21.15.112:0/591021104 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22e80c5a50 0x7f22e8156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:31.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.252+0000 7f22e77fe700 1 -- 172.21.15.112:0/591021104 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4019294149 0 0) 0x7f22e80c3d10 con 0x7f22e8155e20 2023-01-24T03:40:31.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.252+0000 7f22e77fe700 1 -- 172.21.15.112:0/591021104 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22c80033f0 con 0x7f22e8155e20 2023-01-24T03:40:31.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.253+0000 7f22ecfe6700 1 -- 172.21.15.112:0/591021104 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22e80c5a50 msgr2=0x7f22e8156c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:40:31.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.253+0000 7f22ecfe6700 1 --2- 172.21.15.112:0/591021104 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22e80c5a50 0x7f22e8156c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:31.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.253+0000 7f22ecfe6700 1 -- 172.21.15.112:0/591021104 >> v1:172.21.15.112:6789/0 conn(0x7f22e8155e20 legacy=0x7f22e80cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:31.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.253+0000 7f22ecfe6700 1 -- 172.21.15.112:0/591021104 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e80c3100 con 0x7f22e80c5680 2023-01-24T03:40:31.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.253+0000 7f22ed7e7700 1 --2- 172.21.15.112:0/591021104 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22e80c5a50 0x7f22e8156c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:40:31.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.253+0000 7f22ecfe6700 1 --2- 172.21.15.112:0/591021104 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f22e80c5680 0x7f22e8156730 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f22e0009760 tx=0x7f22e0009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2927f1595638113 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:31.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.253+0000 7f22e77fe700 1 -- 172.21.15.112:0/591021104 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e00105b0 con 0x7f22e80c5680 2023-01-24T03:40:31.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.253+0000 7f22e77fe700 1 -- 172.21.15.112:0/591021104 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22e0010710 con 0x7f22e80c5680 2023-01-24T03:40:31.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.253+0000 7f22e77fe700 1 -- 172.21.15.112:0/591021104 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e0010ab0 con 0x7f22e80c5680 2023-01-24T03:40:31.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.253+0000 7f22edfe8700 1 -- 172.21.15.112:0/591021104 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f22e80c5680 msgr2=0x7f22e8156730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:31.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.253+0000 7f22edfe8700 1 --2- 172.21.15.112:0/591021104 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f22e80c5680 0x7f22e8156730 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f22e0009760 tx=0x7f22e0009060 comp rx=0 tx=0).stop 2023-01-24T03:40:31.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.253+0000 7f22edfe8700 1 -- 172.21.15.112:0/591021104 shutdown_connections 2023-01-24T03:40:31.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.253+0000 7f22edfe8700 1 --2- 172.21.15.112:0/591021104 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22e80c5a50 0x7f22e8156c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:31.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.253+0000 7f22edfe8700 1 --2- 172.21.15.112:0/591021104 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f22e80c5680 0x7f22e8156730 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:31.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.254+0000 7f22edfe8700 1 -- 172.21.15.112:0/591021104 >> 172.21.15.112:0/591021104 conn(0x7f22e80685b0 msgr2=0x7f22e80689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:31.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.254+0000 7f22edfe8700 1 -- 172.21.15.112:0/591021104 shutdown_connections 2023-01-24T03:40:31.263 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.254+0000 7f22edfe8700 1 -- 172.21.15.112:0/591021104 wait complete. 2023-01-24T03:40:31.263 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.255+0000 7f22edfe8700 1 Processor -- start 2023-01-24T03:40:31.265 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.255+0000 7f22edfe8700 1 -- start start 2023-01-24T03:40:31.266 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.255+0000 7f22edfe8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f22e80c5680 0x7f22e815a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:31.266 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.255+0000 7f22edfe8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22e80c5a50 0x7f22e80c6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:31.266 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.255+0000 7f22edfe8700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f22e8155e20 0x7f22e80c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:31.266 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.255+0000 7f22edfe8700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e8165e40 con 0x7f22e8155e20 2023-01-24T03:40:31.266 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.255+0000 7f22edfe8700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e81694a0 con 0x7f22e80c5680 2023-01-24T03:40:31.267 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.255+0000 7f22edfe8700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22e8165230 con 0x7f22e80c5a50 2023-01-24T03:40:31.267 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.256+0000 7f22ed7e7700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f22e8155e20 0x7f22e80c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:31.267 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.256+0000 7f22ed7e7700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f22e8155e20 0x7f22e80c6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55826/0 (socket says 172.21.15.112:55826) 2023-01-24T03:40:31.267 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.256+0000 7f22ed7e7700 1 -- 172.21.15.112:0/2364378853 learned_addr learned my addr 172.21.15.112:0/2364378853 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:40:31.267 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.256+0000 7f22ecfe6700 1 --2- 172.21.15.112:0/2364378853 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f22e80c5680 0x7f22e815a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:31.268 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.256+0000 7f22e7fff700 1 --2- 172.21.15.112:0/2364378853 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22e80c5a50 0x7f22e80c6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:31.268 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.256+0000 7f22ed7e7700 1 -- 172.21.15.112:0/2364378853 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22e80c5a50 msgr2=0x7f22e80c6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:31.268 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.256+0000 7f22ed7e7700 1 --2- 172.21.15.112:0/2364378853 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22e80c5a50 0x7f22e80c6890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:31.268 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.256+0000 7f22ed7e7700 1 -- 172.21.15.112:0/2364378853 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f22e80c5680 msgr2=0x7f22e815a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:31.268 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.256+0000 7f22ed7e7700 1 --2- 172.21.15.112:0/2364378853 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f22e80c5680 0x7f22e815a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:31.269 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.256+0000 7f22ed7e7700 1 -- 172.21.15.112:0/2364378853 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22d00092a0 con 0x7f22e8155e20 2023-01-24T03:40:31.269 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.256+0000 7f22ed7e7700 1 --2- 172.21.15.112:0/2364378853 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f22e8155e20 0x7f22e80c6dd0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f22d000e980 tx=0x7f22d0009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:31.269 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.256+0000 7f22e4ff9700 1 -- 172.21.15.112:0/2364378853 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d000c850 con 0x7f22e8155e20 2023-01-24T03:40:31.269 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.256+0000 7f22e4ff9700 1 -- 172.21.15.112:0/2364378853 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22d000c9b0 con 0x7f22e8155e20 2023-01-24T03:40:31.269 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.256+0000 7f22edfe8700 1 -- 172.21.15.112:0/2364378853 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22e00092a0 con 0x7f22e8155e20 2023-01-24T03:40:31.270 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.256+0000 7f22e4ff9700 1 -- 172.21.15.112:0/2364378853 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d000cd70 con 0x7f22e8155e20 2023-01-24T03:40:31.270 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.256+0000 7f22edfe8700 1 -- 172.21.15.112:0/2364378853 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22e81657f0 con 0x7f22e8155e20 2023-01-24T03:40:31.270 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.257+0000 7f22edfe8700 1 -- 172.21.15.112:0/2364378853 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22e8165fb0 con 0x7f22e8155e20 2023-01-24T03:40:31.270 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.257+0000 7f22e4ff9700 1 -- 172.21.15.112:0/2364378853 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f22d0013680 con 0x7f22e8155e20 2023-01-24T03:40:31.270 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.258+0000 7f22e4ff9700 1 --2- 172.21.15.112:0/2364378853 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f22cc0611c0 0x7f22cc063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:31.270 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.258+0000 7f22e4ff9700 1 -- 172.21.15.112:0/2364378853 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 7469+0+0 (secure 0 0 0) 0x7f22d007dbf0 con 0x7f22e8155e20 2023-01-24T03:40:31.271 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.258+0000 7f22ecfe6700 1 --2- 172.21.15.112:0/2364378853 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f22cc0611c0 0x7f22cc063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:31.271 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.258+0000 7f22ecfe6700 1 --2- 172.21.15.112:0/2364378853 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f22cc0611c0 0x7f22cc063670 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f22e0009760 tx=0x7f22e0016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:31.271 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.262+0000 7f22e4ff9700 1 -- 172.21.15.112:0/2364378853 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f22d004d7a0 con 0x7f22e8155e20 2023-01-24T03:40:31.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.482+0000 7f22edfe8700 1 -- 172.21.15.112:0/2364378853 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22e805ea20 con 0x7f22e8155e20 2023-01-24T03:40:31.485 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.482+0000 7f22e4ff9700 1 -- 172.21.15.112:0/2364378853 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+5360 (secure 0 0 0) 0x7f22d0050dc0 con 0x7f22e8155e20 2023-01-24T03:40:31.486 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:40:31.486 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:40:28.725494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5186},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5186":{"gid":5186,"name":"b","rank":0,"incarnation":42,"state":"up:replay","state_seq":1,"addr":"172.21.15.142:6834/2804746591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2804746591},{"type":"v1","addr":"172.21.15.142:6834","nonce":2804746591}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:40:31.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.484+0000 7f22edfe8700 1 -- 172.21.15.112:0/2364378853 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f22cc0611c0 msgr2=0x7f22cc063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:31.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.484+0000 7f22edfe8700 1 --2- 172.21.15.112:0/2364378853 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f22cc0611c0 0x7f22cc063670 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f22e0009760 tx=0x7f22e0016000 comp rx=0 tx=0).stop 2023-01-24T03:40:31.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.484+0000 7f22edfe8700 1 -- 172.21.15.112:0/2364378853 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f22e8155e20 msgr2=0x7f22e80c6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:31.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.484+0000 7f22edfe8700 1 --2- 172.21.15.112:0/2364378853 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f22e8155e20 0x7f22e80c6dd0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f22d000e980 tx=0x7f22d0009060 comp rx=0 tx=0).stop 2023-01-24T03:40:31.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.484+0000 7f22edfe8700 1 -- 172.21.15.112:0/2364378853 shutdown_connections 2023-01-24T03:40:31.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.484+0000 7f22edfe8700 1 --2- 172.21.15.112:0/2364378853 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f22cc0611c0 0x7f22cc063670 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:31.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.484+0000 7f22edfe8700 1 --2- 172.21.15.112:0/2364378853 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f22e8155e20 0x7f22e80c6dd0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:31.489 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.484+0000 7f22edfe8700 1 --2- 172.21.15.112:0/2364378853 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f22e80c5680 0x7f22e815a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:31.489 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.484+0000 7f22edfe8700 1 --2- 172.21.15.112:0/2364378853 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f22e80c5a50 0x7f22e80c6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:31.489 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.484+0000 7f22edfe8700 1 -- 172.21.15.112:0/2364378853 >> 172.21.15.112:0/2364378853 conn(0x7f22e80685b0 msgr2=0x7f22e805d830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:31.489 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.484+0000 7f22edfe8700 1 -- 172.21.15.112:0/2364378853 shutdown_connections 2023-01-24T03:40:31.489 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:31.484+0000 7f22edfe8700 1 -- 172.21.15.112:0/2364378853 wait complete. 2023-01-24T03:40:31.490 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 42 2023-01-24T03:40:31.499 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:40:33.500 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:40:33.625 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.621+0000 7f9655ce7700 1 Processor -- start 2023-01-24T03:40:33.625 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.622+0000 7f9655ce7700 1 -- start start 2023-01-24T03:40:33.627 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.622+0000 7f9655ce7700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f9650068b10 0x7f96500cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:33.627 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.622+0000 7f9655ce7700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f96500aef60 0x7f96500b14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:33.627 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.622+0000 7f9655ce7700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96501560e0 con 0x7f96500aeb90 2023-01-24T03:40:33.628 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.622+0000 7f9655ce7700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9650155530 con 0x7f9650068b10 2023-01-24T03:40:33.628 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.622+0000 7f9655ce7700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9650069ac0 con 0x7f96500aef60 2023-01-24T03:40:33.628 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.622+0000 7f964effd700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f9650068b10 0x7f96500cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:33.628 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.623+0000 7f964effd700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f9650068b10 0x7f96500cc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55834/0 (socket says 172.21.15.112:55834) 2023-01-24T03:40:33.628 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.623+0000 7f964effd700 1 -- 172.21.15.112:0/744243016 learned_addr learned my addr 172.21.15.112:0/744243016 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:40:33.629 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.623+0000 7f964ffff700 1 --2- 172.21.15.112:0/744243016 >> v2:172.21.15.142:3300/0 conn(0x7f96500aef60 0x7f96500b14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:33.629 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.623+0000 7f964effd700 1 -- 172.21.15.112:0/744243016 >> v2:172.21.15.142:3300/0 conn(0x7f96500aef60 msgr2=0x7f96500b14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:40:33.629 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.623+0000 7f964effd700 1 --2- 172.21.15.112:0/744243016 >> v2:172.21.15.142:3300/0 conn(0x7f96500aef60 0x7f96500b14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:33.629 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.623+0000 7f964effd700 1 -- 172.21.15.112:0/744243016 >> v1:172.21.15.142:6789/0 conn(0x7f96500aeb90 legacy=0x7f9650068400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:40:33.629 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.623+0000 7f964effd700 1 -- 172.21.15.112:0/744243016 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9650069d80 con 0x7f9650068b10 2023-01-24T03:40:33.630 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.623+0000 7f964effd700 1 --2- 172.21.15.112:0/744243016 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9650068b10 0x7f96500cc3e0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f96500c2a50 tx=0x7f96440090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9e82192264ac9cc server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:33.630 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.623+0000 7f964e7fc700 1 -- 172.21.15.112:0/744243016 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9644011630 con 0x7f9650068b10 2023-01-24T03:40:33.630 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.623+0000 7f964e7fc700 1 -- 172.21.15.112:0/744243016 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9644011790 con 0x7f9650068b10 2023-01-24T03:40:33.630 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.623+0000 7f964e7fc700 1 -- 172.21.15.112:0/744243016 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9644011b50 con 0x7f9650068b10 2023-01-24T03:40:33.630 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.624+0000 7f9655ce7700 1 -- 172.21.15.112:0/744243016 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9650068b10 msgr2=0x7f96500cc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:33.631 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.624+0000 7f9655ce7700 1 --2- 172.21.15.112:0/744243016 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9650068b10 0x7f96500cc3e0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f96500c2a50 tx=0x7f96440090f0 comp rx=0 tx=0).stop 2023-01-24T03:40:33.631 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.624+0000 7f9655ce7700 1 -- 172.21.15.112:0/744243016 shutdown_connections 2023-01-24T03:40:33.631 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.624+0000 7f9655ce7700 1 --2- 172.21.15.112:0/744243016 >> v2:172.21.15.142:3300/0 conn(0x7f96500aef60 0x7f96500b14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:33.631 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.624+0000 7f9655ce7700 1 --2- 172.21.15.112:0/744243016 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9650068b10 0x7f96500cc3e0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:33.631 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.624+0000 7f9655ce7700 1 -- 172.21.15.112:0/744243016 >> 172.21.15.112:0/744243016 conn(0x7f965005e300 msgr2=0x7f96500cba20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:33.632 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.624+0000 7f9655ce7700 1 -- 172.21.15.112:0/744243016 shutdown_connections 2023-01-24T03:40:33.632 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.624+0000 7f9655ce7700 1 -- 172.21.15.112:0/744243016 wait complete. 2023-01-24T03:40:33.634 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.625+0000 7f9655ce7700 1 Processor -- start 2023-01-24T03:40:33.635 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.625+0000 7f9655ce7700 1 -- start start 2023-01-24T03:40:33.635 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.626+0000 7f9655ce7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9650068b10 0x7f965014fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:33.635 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.626+0000 7f9655ce7700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f96500aeb90 0x7f9650148eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:33.635 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.626+0000 7f9655ce7700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f96500aef60 0x7f965015aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:33.635 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.626+0000 7f9655ce7700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9650166200 con 0x7f96500aeb90 2023-01-24T03:40:33.636 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.626+0000 7f9655ce7700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96500c0be0 con 0x7f9650068b10 2023-01-24T03:40:33.636 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.626+0000 7f9655ce7700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9650165c00 con 0x7f96500aef60 2023-01-24T03:40:33.636 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.626+0000 7f964effd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f96500aeb90 0x7f9650148eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:33.636 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.626+0000 7f964effd700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f96500aeb90 0x7f9650148eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55840/0 (socket says 172.21.15.112:55840) 2023-01-24T03:40:33.637 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.626+0000 7f964effd700 1 -- 172.21.15.112:0/3354376839 learned_addr learned my addr 172.21.15.112:0/3354376839 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:40:33.637 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.626+0000 7f964f7fe700 1 --2- 172.21.15.112:0/3354376839 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9650068b10 0x7f965014fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:33.637 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.626+0000 7f964ffff700 1 --2- 172.21.15.112:0/3354376839 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f96500aef60 0x7f965015aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:33.637 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.626+0000 7f964effd700 1 -- 172.21.15.112:0/3354376839 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f96500aef60 msgr2=0x7f965015aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:33.637 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.626+0000 7f964effd700 1 --2- 172.21.15.112:0/3354376839 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f96500aef60 0x7f965015aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:33.638 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.626+0000 7f964effd700 1 -- 172.21.15.112:0/3354376839 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9650068b10 msgr2=0x7f965014fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:33.638 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.626+0000 7f964effd700 1 --2- 172.21.15.112:0/3354376839 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9650068b10 0x7f965014fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:33.638 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.626+0000 7f964effd700 1 -- 172.21.15.112:0/3354376839 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9644009370 con 0x7f96500aeb90 2023-01-24T03:40:33.638 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.626+0000 7f964effd700 1 --2- 172.21.15.112:0/3354376839 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f96500aeb90 0x7f9650148eb0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f9650067190 tx=0x7f9644014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:33.638 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.626+0000 7f96377fe700 1 -- 172.21.15.112:0/3354376839 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9644011900 con 0x7f96500aeb90 2023-01-24T03:40:33.638 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.626+0000 7f96377fe700 1 -- 172.21.15.112:0/3354376839 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9644011a60 con 0x7f96500aeb90 2023-01-24T03:40:33.639 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.626+0000 7f9655ce7700 1 -- 172.21.15.112:0/3354376839 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96500c0e50 con 0x7f96500aeb90 2023-01-24T03:40:33.639 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.626+0000 7f96377fe700 1 -- 172.21.15.112:0/3354376839 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9644025640 con 0x7f96500aeb90 2023-01-24T03:40:33.639 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.627+0000 7f9655ce7700 1 -- 172.21.15.112:0/3354376839 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96500c1e80 con 0x7f96500aeb90 2023-01-24T03:40:33.639 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.628+0000 7f9655ce7700 1 -- 172.21.15.112:0/3354376839 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9650166050 con 0x7f96500aeb90 2023-01-24T03:40:33.640 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.628+0000 7f96377fe700 1 -- 172.21.15.112:0/3354376839 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f964400f050 con 0x7f96500aeb90 2023-01-24T03:40:33.640 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.628+0000 7f96377fe700 1 --2- 172.21.15.112:0/3354376839 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f96300611f0 0x7f96300636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:33.640 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.628+0000 7f96377fe700 1 -- 172.21.15.112:0/3354376839 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 7469+0+0 (secure 0 0 0) 0x7f964407ef60 con 0x7f96500aeb90 2023-01-24T03:40:33.640 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.628+0000 7f964f7fe700 1 --2- 172.21.15.112:0/3354376839 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f96300611f0 0x7f96300636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:33.640 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.628+0000 7f964f7fe700 1 --2- 172.21.15.112:0/3354376839 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f96300611f0 0x7f96300636a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9640003fb0 tx=0x7f9640002530 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:33.641 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.632+0000 7f96377fe700 1 -- 172.21.15.112:0/3354376839 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f96440af1d0 con 0x7f96500aeb90 2023-01-24T03:40:33.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.854+0000 7f9655ce7700 1 -- 172.21.15.112:0/3354376839 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f96500c8cd0 con 0x7f96500aeb90 2023-01-24T03:40:33.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.855+0000 7f96377fe700 1 -- 172.21.15.112:0/3354376839 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+5360 (secure 0 0 0) 0x7f96500c8cd0 con 0x7f96500aeb90 2023-01-24T03:40:33.858 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:40:33.858 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:40:28.725494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5186},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5186":{"gid":5186,"name":"b","rank":0,"incarnation":42,"state":"up:replay","state_seq":1,"addr":"172.21.15.142:6834/2804746591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2804746591},{"type":"v1","addr":"172.21.15.142:6834","nonce":2804746591}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:40:33.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.857+0000 7f9655ce7700 1 -- 172.21.15.112:0/3354376839 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f96300611f0 msgr2=0x7f96300636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:33.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.857+0000 7f9655ce7700 1 --2- 172.21.15.112:0/3354376839 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f96300611f0 0x7f96300636a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9640003fb0 tx=0x7f9640002530 comp rx=0 tx=0).stop 2023-01-24T03:40:33.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.857+0000 7f9655ce7700 1 -- 172.21.15.112:0/3354376839 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f96500aeb90 msgr2=0x7f9650148eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:33.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.857+0000 7f9655ce7700 1 --2- 172.21.15.112:0/3354376839 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f96500aeb90 0x7f9650148eb0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f9650067190 tx=0x7f9644014000 comp rx=0 tx=0).stop 2023-01-24T03:40:33.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.857+0000 7f9655ce7700 1 -- 172.21.15.112:0/3354376839 shutdown_connections 2023-01-24T03:40:33.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.857+0000 7f9655ce7700 1 --2- 172.21.15.112:0/3354376839 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f96300611f0 0x7f96300636a0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:33.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.857+0000 7f9655ce7700 1 --2- 172.21.15.112:0/3354376839 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f96500aef60 0x7f965015aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:33.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.857+0000 7f9655ce7700 1 --2- 172.21.15.112:0/3354376839 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9650068b10 0x7f965014fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:33.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.857+0000 7f9655ce7700 1 --2- 172.21.15.112:0/3354376839 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f96500aeb90 0x7f9650148eb0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:33.862 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.857+0000 7f9655ce7700 1 -- 172.21.15.112:0/3354376839 >> 172.21.15.112:0/3354376839 conn(0x7f965005e300 msgr2=0x7f965014b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:33.862 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.857+0000 7f9655ce7700 1 -- 172.21.15.112:0/3354376839 shutdown_connections 2023-01-24T03:40:33.862 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:33.857+0000 7f9655ce7700 1 -- 172.21.15.112:0/3354376839 wait complete. 2023-01-24T03:40:33.862 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 42 2023-01-24T03:40:33.871 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:40:35.872 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:40:35.998 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.995+0000 7f510c9fa700 1 Processor -- start 2023-01-24T03:40:35.998 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.995+0000 7f510c9fa700 1 -- start start 2023-01-24T03:40:36.001 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.996+0000 7f510c9fa700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f51080c5680 0x7f5108156730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:36.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.996+0000 7f510c9fa700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f51080c5a50 0x7f5108156c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:36.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.996+0000 7f510c9fa700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51080c3d10 con 0x7f5108155e20 2023-01-24T03:40:36.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.996+0000 7f510c9fa700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51080c4eb0 con 0x7f51080c5680 2023-01-24T03:40:36.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.996+0000 7f510c9fa700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51080c2e20 con 0x7f51080c5a50 2023-01-24T03:40:36.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.996+0000 7f510659c700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f5108155e20 0x7f51080cb870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:36838/0 (socket says 172.21.15.112:36838) 2023-01-24T03:40:36.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.996+0000 7f510659c700 1 -- 172.21.15.112:0/2439899357 learned_addr learned my addr 172.21.15.112:0/2439899357 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:40:36.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.996+0000 7f5106d9d700 1 --2- 172.21.15.112:0/2439899357 >> v2:172.21.15.142:3300/0 conn(0x7f51080c5680 0x7f5108156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:36.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.996+0000 7f510759e700 1 --2- 172.21.15.112:0/2439899357 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f51080c5a50 0x7f5108156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:36.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.996+0000 7f5105d9b700 1 -- 172.21.15.112:0/2439899357 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 773099868 0 0) 0x7f51080c3d10 con 0x7f5108155e20 2023-01-24T03:40:36.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.996+0000 7f5105d9b700 1 -- 172.21.15.112:0/2439899357 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50e80033f0 con 0x7f5108155e20 2023-01-24T03:40:36.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.996+0000 7f5106d9d700 1 -- 172.21.15.112:0/2439899357 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f51080c5a50 msgr2=0x7f5108156c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:40:36.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.996+0000 7f5105d9b700 1 -- 172.21.15.112:0/2439899357 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2917955432 0 0) 0x7f50e80033f0 con 0x7f5108155e20 2023-01-24T03:40:36.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.996+0000 7f5106d9d700 1 --2- 172.21.15.112:0/2439899357 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f51080c5a50 0x7f5108156c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:36.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.996+0000 7f5106d9d700 1 -- 172.21.15.112:0/2439899357 >> v1:172.21.15.112:6789/0 conn(0x7f5108155e20 legacy=0x7f51080cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:36.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.996+0000 7f5106d9d700 1 -- 172.21.15.112:0/2439899357 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51080c3100 con 0x7f51080c5680 2023-01-24T03:40:36.006 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.996+0000 7f510759e700 1 --2- 172.21.15.112:0/2439899357 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f51080c5a50 0x7f5108156c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:40:36.006 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.997+0000 7f5106d9d700 1 --2- 172.21.15.112:0/2439899357 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f51080c5680 0x7f5108156730 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f50f8009e60 tx=0x7f50f8009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5019ab164a2f6ca server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:36.006 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.997+0000 7f5105d9b700 1 -- 172.21.15.112:0/2439899357 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f800e070 con 0x7f51080c5680 2023-01-24T03:40:36.006 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.997+0000 7f5105d9b700 1 -- 172.21.15.112:0/2439899357 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50f800b7a0 con 0x7f51080c5680 2023-01-24T03:40:36.006 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.997+0000 7f5105d9b700 1 -- 172.21.15.112:0/2439899357 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f0001470 con 0x7f51080c5680 2023-01-24T03:40:36.007 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.997+0000 7f510c9fa700 1 -- 172.21.15.112:0/2439899357 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f51080c5680 msgr2=0x7f5108156730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:36.007 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.997+0000 7f510c9fa700 1 --2- 172.21.15.112:0/2439899357 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f51080c5680 0x7f5108156730 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f50f8009e60 tx=0x7f50f8009780 comp rx=0 tx=0).stop 2023-01-24T03:40:36.007 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.997+0000 7f510c9fa700 1 -- 172.21.15.112:0/2439899357 shutdown_connections 2023-01-24T03:40:36.007 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.997+0000 7f510c9fa700 1 --2- 172.21.15.112:0/2439899357 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f51080c5a50 0x7f5108156c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:36.007 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.997+0000 7f510c9fa700 1 --2- 172.21.15.112:0/2439899357 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f51080c5680 0x7f5108156730 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:36.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.997+0000 7f510c9fa700 1 -- 172.21.15.112:0/2439899357 >> 172.21.15.112:0/2439899357 conn(0x7f51080685b0 msgr2=0x7f51080689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:36.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.997+0000 7f510c9fa700 1 -- 172.21.15.112:0/2439899357 shutdown_connections 2023-01-24T03:40:36.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.997+0000 7f510c9fa700 1 -- 172.21.15.112:0/2439899357 wait complete. 2023-01-24T03:40:36.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.998+0000 7f510c9fa700 1 Processor -- start 2023-01-24T03:40:36.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.999+0000 7f510c9fa700 1 -- start start 2023-01-24T03:40:36.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.999+0000 7f510c9fa700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f51080c5680 0x7f51080c8b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:36.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.999+0000 7f510c9fa700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51080c5a50 0x7f5108153e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:36.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.999+0000 7f510c9fa700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5108155e20 0x7f51080c6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:36.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.999+0000 7f510c9fa700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5108165430 con 0x7f51080c5a50 2023-01-24T03:40:36.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.999+0000 7f510c9fa700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5108165f90 con 0x7f5108155e20 2023-01-24T03:40:36.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.999+0000 7f510c9fa700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51081656f0 con 0x7f51080c5680 2023-01-24T03:40:36.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.999+0000 7f510659c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51080c5a50 0x7f5108153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:36.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.999+0000 7f510659c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51080c5a50 0x7f5108153e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55852/0 (socket says 172.21.15.112:55852) 2023-01-24T03:40:36.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.999+0000 7f510659c700 1 -- 172.21.15.112:0/934927551 learned_addr learned my addr 172.21.15.112:0/934927551 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:40:36.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.999+0000 7f510759e700 1 --2- 172.21.15.112:0/934927551 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5108155e20 0x7f51080c6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:36.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:35.999+0000 7f5106d9d700 1 --2- 172.21.15.112:0/934927551 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f51080c5680 0x7f51080c8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:36.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:36.000+0000 7f510659c700 1 -- 172.21.15.112:0/934927551 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f51080c5680 msgr2=0x7f51080c8b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:36.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:36.000+0000 7f510659c700 1 --2- 172.21.15.112:0/934927551 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f51080c5680 0x7f51080c8b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:36.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:36.000+0000 7f510659c700 1 -- 172.21.15.112:0/934927551 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5108155e20 msgr2=0x7f51080c6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:36.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:36.000+0000 7f510659c700 1 --2- 172.21.15.112:0/934927551 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5108155e20 0x7f51080c6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:36.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:36.000+0000 7f510659c700 1 -- 172.21.15.112:0/934927551 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51000092a0 con 0x7f51080c5a50 2023-01-24T03:40:36.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:36.000+0000 7f510659c700 1 --2- 172.21.15.112:0/934927551 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51080c5a50 0x7f5108153e20 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f50f000c810 tx=0x7f50f0001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:36.015 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:36.000+0000 7f50feffd700 1 -- 172.21.15.112:0/934927551 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f0007360 con 0x7f51080c5a50 2023-01-24T03:40:36.015 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:36.000+0000 7f510c9fa700 1 -- 172.21.15.112:0/934927551 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50f8009a20 con 0x7f51080c5a50 2023-01-24T03:40:36.015 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:36.000+0000 7f50feffd700 1 -- 172.21.15.112:0/934927551 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50f00074c0 con 0x7f51080c5a50 2023-01-24T03:40:36.015 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:36.000+0000 7f50feffd700 1 -- 172.21.15.112:0/934927551 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f00048c0 con 0x7f51080c5a50 2023-01-24T03:40:36.015 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:36.000+0000 7f510c9fa700 1 -- 172.21.15.112:0/934927551 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5108165cb0 con 0x7f51080c5a50 2023-01-24T03:40:36.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:36.001+0000 7f510c9fa700 1 -- 172.21.15.112:0/934927551 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f510805ea20 con 0x7f51080c5a50 2023-01-24T03:40:36.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:36.001+0000 7f50feffd700 1 -- 172.21.15.112:0/934927551 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f50f001a030 con 0x7f51080c5a50 2023-01-24T03:40:36.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:36.001+0000 7f50feffd700 1 --2- 172.21.15.112:0/934927551 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f50ec0611f0 0x7f50ec0636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:36.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:36.002+0000 7f50feffd700 1 -- 172.21.15.112:0/934927551 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 7469+0+0 (secure 0 0 0) 0x7f50f007ce30 con 0x7f51080c5a50 2023-01-24T03:40:36.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:36.002+0000 7f5106d9d700 1 --2- 172.21.15.112:0/934927551 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f50ec0611f0 0x7f50ec0636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:36.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:36.002+0000 7f5106d9d700 1 --2- 172.21.15.112:0/934927551 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f50ec0611f0 0x7f50ec0636a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f50f80097c0 tx=0x7f50f8018000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:36.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:36.006+0000 7f50feffd700 1 -- 172.21.15.112:0/934927551 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f50f004b1d0 con 0x7f51080c5a50 2023-01-24T03:40:36.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:36.229+0000 7f510c9fa700 1 -- 172.21.15.112:0/934927551 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f510804b340 con 0x7f51080c5a50 2023-01-24T03:40:36.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:36.230+0000 7f50feffd700 1 -- 172.21.15.112:0/934927551 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+5360 (secure 0 0 0) 0x7f50f0018020 con 0x7f51080c5a50 2023-01-24T03:40:36.233 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:40:36.233 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:40:28.725494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5186},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5186":{"gid":5186,"name":"b","rank":0,"incarnation":42,"state":"up:replay","state_seq":1,"addr":"172.21.15.142:6834/2804746591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2804746591},{"type":"v1","addr":"172.21.15.142:6834","nonce":2804746591}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:40:36.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:36.232+0000 7f510c9fa700 1 -- 172.21.15.112:0/934927551 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f50ec0611f0 msgr2=0x7f50ec0636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:36.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:36.232+0000 7f510c9fa700 1 --2- 172.21.15.112:0/934927551 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f50ec0611f0 0x7f50ec0636a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f50f80097c0 tx=0x7f50f8018000 comp rx=0 tx=0).stop 2023-01-24T03:40:36.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:36.232+0000 7f510c9fa700 1 -- 172.21.15.112:0/934927551 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51080c5a50 msgr2=0x7f5108153e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:36.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:36.232+0000 7f510c9fa700 1 --2- 172.21.15.112:0/934927551 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51080c5a50 0x7f5108153e20 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f50f000c810 tx=0x7f50f0001a80 comp rx=0 tx=0).stop 2023-01-24T03:40:36.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:36.232+0000 7f510c9fa700 1 -- 172.21.15.112:0/934927551 shutdown_connections 2023-01-24T03:40:36.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:36.232+0000 7f510c9fa700 1 --2- 172.21.15.112:0/934927551 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f50ec0611f0 0x7f50ec0636a0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:36.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:36.232+0000 7f510c9fa700 1 --2- 172.21.15.112:0/934927551 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5108155e20 0x7f51080c6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:36.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:36.232+0000 7f510c9fa700 1 --2- 172.21.15.112:0/934927551 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f51080c5680 0x7f51080c8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:36.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:36.232+0000 7f510c9fa700 1 --2- 172.21.15.112:0/934927551 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51080c5a50 0x7f5108153e20 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:36.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:36.232+0000 7f510c9fa700 1 -- 172.21.15.112:0/934927551 >> 172.21.15.112:0/934927551 conn(0x7f51080685b0 msgr2=0x7f5108154f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:36.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:36.232+0000 7f510c9fa700 1 -- 172.21.15.112:0/934927551 shutdown_connections 2023-01-24T03:40:36.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:36.232+0000 7f510c9fa700 1 -- 172.21.15.112:0/934927551 wait complete. 2023-01-24T03:40:36.238 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 42 2023-01-24T03:40:36.246 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:40:38.247 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:40:38.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.368+0000 7f011bed8700 1 Processor -- start 2023-01-24T03:40:38.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.368+0000 7f011bed8700 1 -- start start 2023-01-24T03:40:38.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.369+0000 7f011bed8700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f0114068b10 0x7f01140cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:38.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.369+0000 7f011bed8700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01141560e0 con 0x7f01140aef60 2023-01-24T03:40:38.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.369+0000 7f011bed8700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0114155530 con 0x7f01140aeb90 2023-01-24T03:40:38.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.369+0000 7f011bed8700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0114069ac0 con 0x7f0114068b10 2023-01-24T03:40:38.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.369+0000 7f0119473700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f0114068b10 0x7f01140cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:38.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.369+0000 7f011a475700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f01140aef60 0x7f011414f370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:36850/0 (socket says 172.21.15.112:36850) 2023-01-24T03:40:38.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.369+0000 7f011a475700 1 -- 172.21.15.112:0/2960782462 learned_addr learned my addr 172.21.15.112:0/2960782462 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:40:38.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.369+0000 7f0119473700 1 -- 172.21.15.112:0/2960782462 >> v1:172.21.15.142:6789/0 conn(0x7f01140aeb90 legacy=0x7f0114068400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:38.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.369+0000 7f0119473700 1 -- 172.21.15.112:0/2960782462 >> v1:172.21.15.112:6789/0 conn(0x7f01140aef60 legacy=0x7f011414f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:38.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.370+0000 7f0118c72700 1 -- 172.21.15.112:0/2960782462 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3470258349 0 0) 0x7f01141560e0 con 0x7f01140aef60 2023-01-24T03:40:38.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.370+0000 7f0119473700 1 -- 172.21.15.112:0/2960782462 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0114069d80 con 0x7f0114068b10 2023-01-24T03:40:38.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.370+0000 7f0119473700 1 --2- 172.21.15.112:0/2960782462 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0114068b10 0x7f01140cc3e0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f0108009760 tx=0x7f0108009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff22d07d90c49644 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:38.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.370+0000 7f0118c72700 1 -- 172.21.15.112:0/2960782462 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0108011630 con 0x7f0114068b10 2023-01-24T03:40:38.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.370+0000 7f0118c72700 1 -- 172.21.15.112:0/2960782462 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0108011790 con 0x7f0114068b10 2023-01-24T03:40:38.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.370+0000 7f0118c72700 1 -- 172.21.15.112:0/2960782462 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0108011b30 con 0x7f0114068b10 2023-01-24T03:40:38.381 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.370+0000 7f011bed8700 1 -- 172.21.15.112:0/2960782462 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0114068b10 msgr2=0x7f01140cc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:38.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.370+0000 7f011bed8700 1 --2- 172.21.15.112:0/2960782462 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0114068b10 0x7f01140cc3e0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f0108009760 tx=0x7f0108009060 comp rx=0 tx=0).stop 2023-01-24T03:40:38.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.370+0000 7f011bed8700 1 -- 172.21.15.112:0/2960782462 shutdown_connections 2023-01-24T03:40:38.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.370+0000 7f011bed8700 1 --2- 172.21.15.112:0/2960782462 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0114068b10 0x7f01140cc3e0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:38.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.370+0000 7f011bed8700 1 -- 172.21.15.112:0/2960782462 >> 172.21.15.112:0/2960782462 conn(0x7f011405e300 msgr2=0x7f01140cba20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:38.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.370+0000 7f011bed8700 1 -- 172.21.15.112:0/2960782462 shutdown_connections 2023-01-24T03:40:38.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.371+0000 7f011bed8700 1 -- 172.21.15.112:0/2960782462 wait complete. 2023-01-24T03:40:38.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.372+0000 7f011bed8700 1 Processor -- start 2023-01-24T03:40:38.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.372+0000 7f011bed8700 1 -- start start 2023-01-24T03:40:38.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.372+0000 7f011bed8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0114068b10 0x7f0114149d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:38.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.372+0000 7f011bed8700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01140aeb90 0x7f011414e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:38.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.372+0000 7f011bed8700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f01140aef60 0x7f0114168fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:38.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.372+0000 7f011bed8700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01140c60c0 con 0x7f01140aeb90 2023-01-24T03:40:38.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.372+0000 7f011bed8700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01140c6710 con 0x7f0114068b10 2023-01-24T03:40:38.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.372+0000 7f011bed8700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01140c0a70 con 0x7f01140aef60 2023-01-24T03:40:38.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.372+0000 7f0119473700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01140aeb90 0x7f011414e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:38.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.372+0000 7f0119473700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01140aeb90 0x7f011414e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55868/0 (socket says 172.21.15.112:55868) 2023-01-24T03:40:38.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.372+0000 7f0119473700 1 -- 172.21.15.112:0/1797991810 learned_addr learned my addr 172.21.15.112:0/1797991810 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:40:38.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.372+0000 7f0119c74700 1 --2- 172.21.15.112:0/1797991810 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0114068b10 0x7f0114149d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:38.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.373+0000 7f011a475700 1 --2- 172.21.15.112:0/1797991810 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f01140aef60 0x7f0114168fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:38.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.373+0000 7f0119473700 1 -- 172.21.15.112:0/1797991810 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f01140aef60 msgr2=0x7f0114168fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:38.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.373+0000 7f0119473700 1 --2- 172.21.15.112:0/1797991810 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f01140aef60 0x7f0114168fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:38.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.373+0000 7f0119473700 1 -- 172.21.15.112:0/1797991810 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0114068b10 msgr2=0x7f0114149d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:38.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.373+0000 7f0119473700 1 --2- 172.21.15.112:0/1797991810 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0114068b10 0x7f0114149d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:38.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.373+0000 7f0119473700 1 -- 172.21.15.112:0/1797991810 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01080092a0 con 0x7f01140aeb90 2023-01-24T03:40:38.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.373+0000 7f0119473700 1 --2- 172.21.15.112:0/1797991810 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01140aeb90 0x7f011414e210 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f0108009eb0 tx=0x7f0108017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:38.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.373+0000 7f0111ffb700 1 -- 172.21.15.112:0/1797991810 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0108000bc0 con 0x7f01140aeb90 2023-01-24T03:40:38.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.373+0000 7f0111ffb700 1 -- 172.21.15.112:0/1797991810 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0108011900 con 0x7f01140aeb90 2023-01-24T03:40:38.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.373+0000 7f011bed8700 1 -- 172.21.15.112:0/1797991810 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01140bfc90 con 0x7f01140aeb90 2023-01-24T03:40:38.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.373+0000 7f0111ffb700 1 -- 172.21.15.112:0/1797991810 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0108025420 con 0x7f01140aeb90 2023-01-24T03:40:38.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.373+0000 7f011bed8700 1 -- 172.21.15.112:0/1797991810 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01140bff30 con 0x7f01140aeb90 2023-01-24T03:40:38.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.374+0000 7f011bed8700 1 -- 172.21.15.112:0/1797991810 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f011404bb80 con 0x7f01140aeb90 2023-01-24T03:40:38.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.375+0000 7f0111ffb700 1 -- 172.21.15.112:0/1797991810 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f010802c030 con 0x7f01140aeb90 2023-01-24T03:40:38.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.375+0000 7f0111ffb700 1 --2- 172.21.15.112:0/1797991810 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f00f40611c0 0x7f00f4063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:38.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.375+0000 7f0111ffb700 1 -- 172.21.15.112:0/1797991810 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 7469+0+0 (secure 0 0 0) 0x7f010807efb0 con 0x7f01140aeb90 2023-01-24T03:40:38.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.375+0000 7f0119c74700 1 --2- 172.21.15.112:0/1797991810 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f00f40611c0 0x7f00f4063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:38.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.375+0000 7f0119c74700 1 --2- 172.21.15.112:0/1797991810 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f00f40611c0 0x7f00f4063670 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0104004290 tx=0x7f0104001560 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:38.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.379+0000 7f0111ffb700 1 -- 172.21.15.112:0/1797991810 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f010804eb60 con 0x7f01140aeb90 2023-01-24T03:40:38.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.599+0000 7f011bed8700 1 -- 172.21.15.112:0/1797991810 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01140af470 con 0x7f01140aeb90 2023-01-24T03:40:38.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.599+0000 7f0111ffb700 1 -- 172.21.15.112:0/1797991810 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+5360 (secure 0 0 0) 0x7f0108015090 con 0x7f01140aeb90 2023-01-24T03:40:38.603 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:40:38.603 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:40:28.725494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5186},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5186":{"gid":5186,"name":"b","rank":0,"incarnation":42,"state":"up:replay","state_seq":1,"addr":"172.21.15.142:6834/2804746591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2804746591},{"type":"v1","addr":"172.21.15.142:6834","nonce":2804746591}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:40:38.605 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.602+0000 7f011bed8700 1 -- 172.21.15.112:0/1797991810 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f00f40611c0 msgr2=0x7f00f4063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:38.605 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.602+0000 7f011bed8700 1 --2- 172.21.15.112:0/1797991810 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f00f40611c0 0x7f00f4063670 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0104004290 tx=0x7f0104001560 comp rx=0 tx=0).stop 2023-01-24T03:40:38.606 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.602+0000 7f011bed8700 1 -- 172.21.15.112:0/1797991810 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01140aeb90 msgr2=0x7f011414e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:38.606 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.602+0000 7f011bed8700 1 --2- 172.21.15.112:0/1797991810 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01140aeb90 0x7f011414e210 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f0108009eb0 tx=0x7f0108017000 comp rx=0 tx=0).stop 2023-01-24T03:40:38.606 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.602+0000 7f011bed8700 1 -- 172.21.15.112:0/1797991810 shutdown_connections 2023-01-24T03:40:38.606 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.602+0000 7f011bed8700 1 --2- 172.21.15.112:0/1797991810 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f00f40611c0 0x7f00f4063670 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:38.606 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.602+0000 7f011bed8700 1 --2- 172.21.15.112:0/1797991810 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f01140aef60 0x7f0114168fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:38.607 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.602+0000 7f011bed8700 1 --2- 172.21.15.112:0/1797991810 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0114068b10 0x7f0114149d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:38.607 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.602+0000 7f011bed8700 1 --2- 172.21.15.112:0/1797991810 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01140aeb90 0x7f011414e210 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:38.607 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.602+0000 7f011bed8700 1 -- 172.21.15.112:0/1797991810 >> 172.21.15.112:0/1797991810 conn(0x7f011405e300 msgr2=0x7f0114154cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:38.607 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.602+0000 7f011bed8700 1 -- 172.21.15.112:0/1797991810 shutdown_connections 2023-01-24T03:40:38.607 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:38.602+0000 7f011bed8700 1 -- 172.21.15.112:0/1797991810 wait complete. 2023-01-24T03:40:38.608 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 42 2023-01-24T03:40:38.617 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:40:40.617 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:40:40.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.738+0000 7f98c88c5700 1 Processor -- start 2023-01-24T03:40:40.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.738+0000 7f98c88c5700 1 -- start start 2023-01-24T03:40:40.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.739+0000 7f98c88c5700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f98c40b0aa0 0x7f98c4155da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:40.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.739+0000 7f98c88c5700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f98c40ca3e0 0x7f98c40ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:40.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.739+0000 7f98c88c5700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98c40b0f40 con 0x7f98c40c7e40 2023-01-24T03:40:40.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.739+0000 7f98c88c5700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98c40b1170 con 0x7f98c40ca3e0 2023-01-24T03:40:40.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.739+0000 7f98c88c5700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98c40b1450 con 0x7f98c40b0aa0 2023-01-24T03:40:40.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.740+0000 7f98c2d9d700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f98c40ca3e0 0x7f98c40ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:40.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.740+0000 7f98c259c700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f98c40c7e40 0x7f98c40ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:36870/0 (socket says 172.21.15.112:36870) 2023-01-24T03:40:40.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.740+0000 7f98c259c700 1 -- 172.21.15.112:0/1132916977 learned_addr learned my addr 172.21.15.112:0/1132916977 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:40:40.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.740+0000 7f98c1d9b700 1 --2- 172.21.15.112:0/1132916977 >> v2:172.21.15.142:3300/0 conn(0x7f98c40b0aa0 0x7f98c4155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:40.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.740+0000 7f98c2d9d700 1 -- 172.21.15.112:0/1132916977 >> v2:172.21.15.142:3300/0 conn(0x7f98c40b0aa0 msgr2=0x7f98c4155da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:40:40.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.740+0000 7f98c2d9d700 1 --2- 172.21.15.112:0/1132916977 >> v2:172.21.15.142:3300/0 conn(0x7f98c40b0aa0 0x7f98c4155da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:40.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.740+0000 7f98c2d9d700 1 -- 172.21.15.112:0/1132916977 >> v1:172.21.15.112:6789/0 conn(0x7f98c40c7e40 legacy=0x7f98c40ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:40.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.740+0000 7f98c2d9d700 1 -- 172.21.15.112:0/1132916977 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c40b1730 con 0x7f98c40ca3e0 2023-01-24T03:40:40.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.740+0000 7f98c159a700 1 -- 172.21.15.112:0/1132916977 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3161708990 0 0) 0x7f98c40b0f40 con 0x7f98c40c7e40 2023-01-24T03:40:40.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.740+0000 7f98c2d9d700 1 --2- 172.21.15.112:0/1132916977 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f98c40ca3e0 0x7f98c40ca7e0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f98c4152ea0 tx=0x7f98bc009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92297532f6f75b36 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:40.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.740+0000 7f98c159a700 1 -- 172.21.15.112:0/1132916977 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98bc011630 con 0x7f98c40ca3e0 2023-01-24T03:40:40.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.740+0000 7f98c159a700 1 -- 172.21.15.112:0/1132916977 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98bc011790 con 0x7f98c40ca3e0 2023-01-24T03:40:40.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.740+0000 7f98c159a700 1 -- 172.21.15.112:0/1132916977 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98bc011b50 con 0x7f98c40ca3e0 2023-01-24T03:40:40.748 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.741+0000 7f98c88c5700 1 -- 172.21.15.112:0/1132916977 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f98c40ca3e0 msgr2=0x7f98c40ca7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:40.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.741+0000 7f98c88c5700 1 --2- 172.21.15.112:0/1132916977 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f98c40ca3e0 0x7f98c40ca7e0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f98c4152ea0 tx=0x7f98bc009060 comp rx=0 tx=0).stop 2023-01-24T03:40:40.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.741+0000 7f98c88c5700 1 -- 172.21.15.112:0/1132916977 shutdown_connections 2023-01-24T03:40:40.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.741+0000 7f98c88c5700 1 --2- 172.21.15.112:0/1132916977 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f98c40ca3e0 0x7f98c40ca7e0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:40.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.741+0000 7f98c88c5700 1 --2- 172.21.15.112:0/1132916977 >> v2:172.21.15.142:3300/0 conn(0x7f98c40b0aa0 0x7f98c4155da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:40.749 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.741+0000 7f98c88c5700 1 -- 172.21.15.112:0/1132916977 >> 172.21.15.112:0/1132916977 conn(0x7f98c405e2c0 msgr2=0x7f98c40cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:40.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.741+0000 7f98c88c5700 1 -- 172.21.15.112:0/1132916977 shutdown_connections 2023-01-24T03:40:40.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.741+0000 7f98c88c5700 1 -- 172.21.15.112:0/1132916977 wait complete. 2023-01-24T03:40:40.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.742+0000 7f98c88c5700 1 Processor -- start 2023-01-24T03:40:40.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.742+0000 7f98c88c5700 1 -- start start 2023-01-24T03:40:40.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.742+0000 7f98c88c5700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f98c40b0aa0 0x7f98c4155050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:40.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.742+0000 7f98c88c5700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f98c40c7e40 0x7f98c4165a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:40.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.742+0000 7f98c88c5700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f98c40ca3e0 0x7f98c414dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:40.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.742+0000 7f98c88c5700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c4165050 con 0x7f98c40c7e40 2023-01-24T03:40:40.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.742+0000 7f98c88c5700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c41655b0 con 0x7f98c40b0aa0 2023-01-24T03:40:40.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.742+0000 7f98c88c5700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98c4165f80 con 0x7f98c40ca3e0 2023-01-24T03:40:40.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.743+0000 7f98c1d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f98c40c7e40 0x7f98c4165a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:40.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.743+0000 7f98c1d9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f98c40c7e40 0x7f98c4165a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55888/0 (socket says 172.21.15.112:55888) 2023-01-24T03:40:40.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.743+0000 7f98c1d9b700 1 -- 172.21.15.112:0/1440323143 learned_addr learned my addr 172.21.15.112:0/1440323143 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:40:40.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.743+0000 7f98c259c700 1 --2- 172.21.15.112:0/1440323143 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f98c40b0aa0 0x7f98c4155050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:40.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.743+0000 7f98c2d9d700 1 --2- 172.21.15.112:0/1440323143 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f98c40ca3e0 0x7f98c414dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:40.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.743+0000 7f98c1d9b700 1 -- 172.21.15.112:0/1440323143 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f98c40ca3e0 msgr2=0x7f98c414dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:40.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.743+0000 7f98c1d9b700 1 --2- 172.21.15.112:0/1440323143 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f98c40ca3e0 0x7f98c414dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:40.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.743+0000 7f98c1d9b700 1 -- 172.21.15.112:0/1440323143 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f98c40b0aa0 msgr2=0x7f98c4155050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:40.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.743+0000 7f98c1d9b700 1 --2- 172.21.15.112:0/1440323143 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f98c40b0aa0 0x7f98c4155050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:40.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.743+0000 7f98c1d9b700 1 -- 172.21.15.112:0/1440323143 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98bc0092a0 con 0x7f98c40c7e40 2023-01-24T03:40:40.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.743+0000 7f98c259c700 1 --2- 172.21.15.112:0/1440323143 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f98c40b0aa0 0x7f98c4155050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:40:40.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.743+0000 7f98c1d9b700 1 --2- 172.21.15.112:0/1440323143 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f98c40c7e40 0x7f98c4165a40 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f98ac00d8c0 tx=0x7f98ac008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:40.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.743+0000 7f98ba7fc700 1 -- 172.21.15.112:0/1440323143 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ac0095b0 con 0x7f98c40c7e40 2023-01-24T03:40:40.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.743+0000 7f98ba7fc700 1 -- 172.21.15.112:0/1440323143 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98ac009710 con 0x7f98c40c7e40 2023-01-24T03:40:40.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.743+0000 7f98c88c5700 1 -- 172.21.15.112:0/1440323143 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98c40c2e10 con 0x7f98c40c7e40 2023-01-24T03:40:40.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.743+0000 7f98ba7fc700 1 -- 172.21.15.112:0/1440323143 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ac009ab0 con 0x7f98c40c7e40 2023-01-24T03:40:40.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.743+0000 7f98c88c5700 1 -- 172.21.15.112:0/1440323143 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98c40c3370 con 0x7f98c40c7e40 2023-01-24T03:40:40.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.744+0000 7f98c88c5700 1 -- 172.21.15.112:0/1440323143 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98c4165740 con 0x7f98c40c7e40 2023-01-24T03:40:40.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.744+0000 7f98ba7fc700 1 -- 172.21.15.112:0/1440323143 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f98ac024070 con 0x7f98c40c7e40 2023-01-24T03:40:40.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.745+0000 7f98ba7fc700 1 --2- 172.21.15.112:0/1440323143 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f98a80611c0 0x7f98a8063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:40.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.745+0000 7f98c259c700 1 --2- 172.21.15.112:0/1440323143 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f98a80611c0 0x7f98a8063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:40.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.745+0000 7f98ba7fc700 1 -- 172.21.15.112:0/1440323143 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 7469+0+0 (secure 0 0 0) 0x7f98ac07e030 con 0x7f98c40c7e40 2023-01-24T03:40:40.759 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.745+0000 7f98c259c700 1 --2- 172.21.15.112:0/1440323143 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f98a80611c0 0x7f98a8063670 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f98b4001e10 tx=0x7f98b4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:40.759 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.749+0000 7f98ba7fc700 1 -- 172.21.15.112:0/1440323143 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f98ac04dbe0 con 0x7f98c40c7e40 2023-01-24T03:40:40.975 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.972+0000 7f98c88c5700 1 -- 172.21.15.112:0/1440323143 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98c40c5df0 con 0x7f98c40c7e40 2023-01-24T03:40:40.975 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.972+0000 7f98ba7fc700 1 -- 172.21.15.112:0/1440323143 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+5364 (secure 0 0 0) 0x7f98ac051200 con 0x7f98c40c7e40 2023-01-24T03:40:40.976 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:40:40.976 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:40:38.808944+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5186},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5186":{"gid":5186,"name":"b","rank":0,"incarnation":42,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.142:6834/2804746591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2804746591},{"type":"v1","addr":"172.21.15.142:6834","nonce":2804746591}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:40:40.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.974+0000 7f98c88c5700 1 -- 172.21.15.112:0/1440323143 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f98a80611c0 msgr2=0x7f98a8063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:40.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.974+0000 7f98c88c5700 1 --2- 172.21.15.112:0/1440323143 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f98a80611c0 0x7f98a8063670 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f98b4001e10 tx=0x7f98b4009040 comp rx=0 tx=0).stop 2023-01-24T03:40:40.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.974+0000 7f98c88c5700 1 -- 172.21.15.112:0/1440323143 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f98c40c7e40 msgr2=0x7f98c4165a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:40.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.974+0000 7f98c88c5700 1 --2- 172.21.15.112:0/1440323143 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f98c40c7e40 0x7f98c4165a40 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f98ac00d8c0 tx=0x7f98ac008040 comp rx=0 tx=0).stop 2023-01-24T03:40:40.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.974+0000 7f98c88c5700 1 -- 172.21.15.112:0/1440323143 shutdown_connections 2023-01-24T03:40:40.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.974+0000 7f98c88c5700 1 --2- 172.21.15.112:0/1440323143 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f98a80611c0 0x7f98a8063670 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:40.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.974+0000 7f98c88c5700 1 --2- 172.21.15.112:0/1440323143 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f98c40ca3e0 0x7f98c414dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:40.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.974+0000 7f98c88c5700 1 --2- 172.21.15.112:0/1440323143 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f98c40b0aa0 0x7f98c4155050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:40.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.974+0000 7f98c88c5700 1 --2- 172.21.15.112:0/1440323143 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f98c40c7e40 0x7f98c4165a40 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:40.979 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.974+0000 7f98c88c5700 1 -- 172.21.15.112:0/1440323143 >> 172.21.15.112:0/1440323143 conn(0x7f98c405e2c0 msgr2=0x7f98c414eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:40.980 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.974+0000 7f98c88c5700 1 -- 172.21.15.112:0/1440323143 shutdown_connections 2023-01-24T03:40:40.980 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:40.974+0000 7f98c88c5700 1 -- 172.21.15.112:0/1440323143 wait complete. 2023-01-24T03:40:40.980 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 43 2023-01-24T03:40:40.989 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 43, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-01-24T03:30:20.971710+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 43, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5186": { "addr": "172.21.15.142:6834/2804746591", "addrs": { "addrvec": [ { "addr": "172.21.15.142:6832", "nonce": 2804746591, "type": "v2" }, { "addr": "172.21.15.142:6834", "nonce": 2804746591, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5186, "incarnation": 42, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:reconnect", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 49, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-01-24T03:40:38.808944+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5186 } } } ], "standbys": [ { "addr": "172.21.15.142:6837/2181345869", "addrs": { "addrvec": [ { "addr": "172.21.15.142:6836", "nonce": 2181345869, "type": "v2" }, { "addr": "172.21.15.142:6837", "nonce": 2181345869, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6835/330343927", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6834", "nonce": 330343927, "type": "v2" }, { "addr": "172.21.15.112:6835", "nonce": 330343927, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6837/12176817", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6836", "nonce": 12176817, "type": "v2" }, { "addr": "172.21.15.112:6837", "nonce": 12176817, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4276, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6839/1110983841", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6838", "nonce": 1110983841, "type": "v2" }, { "addr": "172.21.15.112:6839", "nonce": 1110983841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4281, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-01-24T03:40:42.990 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:40:43.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.113+0000 7f7197f2f700 1 Processor -- start 2023-01-24T03:40:43.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.113+0000 7f7197f2f700 1 -- start start 2023-01-24T03:40:43.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.114+0000 7f7197f2f700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f71900caaf0 0x7f7190155da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:43.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.114+0000 7f7197f2f700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71900b0f40 con 0x7f71900b0aa0 2023-01-24T03:40:43.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.114+0000 7f7197f2f700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71900b1170 con 0x7f71900c7e40 2023-01-24T03:40:43.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.114+0000 7f7197f2f700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71900b1450 con 0x7f71900caaf0 2023-01-24T03:40:43.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.114+0000 7f71954ca700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f71900b0aa0 0x7f71900ca3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:36888/0 (socket says 172.21.15.112:36888) 2023-01-24T03:40:43.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.114+0000 7f71954ca700 1 -- 172.21.15.112:0/3509797754 learned_addr learned my addr 172.21.15.112:0/3509797754 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:40:43.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.114+0000 7f71964cc700 1 --2- 172.21.15.112:0/3509797754 >> v2:172.21.15.142:3300/0 conn(0x7f71900caaf0 0x7f7190155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:43.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.114+0000 7f7194cc9700 1 -- 172.21.15.112:0/3509797754 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 363456557 0 0) 0x7f71900b0f40 con 0x7f71900b0aa0 2023-01-24T03:40:43.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.114+0000 7f7194cc9700 1 -- 172.21.15.112:0/3509797754 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71780033f0 con 0x7f71900b0aa0 2023-01-24T03:40:43.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.114+0000 7f71964cc700 1 -- 172.21.15.112:0/3509797754 >> v1:172.21.15.142:6789/0 conn(0x7f71900c7e40 legacy=0x7f71900ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:43.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.114+0000 7f7194cc9700 1 -- 172.21.15.112:0/3509797754 <== mon.1 v1:172.21.15.142:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3810194619 0 0) 0x7f71900b1170 con 0x7f71900c7e40 2023-01-24T03:40:43.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.114+0000 7f71964cc700 1 -- 172.21.15.112:0/3509797754 >> v1:172.21.15.112:6789/0 conn(0x7f71900b0aa0 legacy=0x7f71900ca3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:43.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.114+0000 7f71964cc700 1 -- 172.21.15.112:0/3509797754 --> v2:172.21.15.142:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71900b1730 con 0x7f71900caaf0 2023-01-24T03:40:43.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.115+0000 7f71964cc700 1 --2- 172.21.15.112:0/3509797754 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71900caaf0 0x7f7190155da0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7188009e60 tx=0x7f7188009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cc9fec84fc5f74b server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:43.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.115+0000 7f7194cc9700 1 -- 172.21.15.112:0/3509797754 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f718800e070 con 0x7f71900caaf0 2023-01-24T03:40:43.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.115+0000 7f7194cc9700 1 -- 172.21.15.112:0/3509797754 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f718800bb60 con 0x7f71900caaf0 2023-01-24T03:40:43.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.115+0000 7f7194cc9700 1 -- 172.21.15.112:0/3509797754 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7188012430 con 0x7f71900caaf0 2023-01-24T03:40:43.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.115+0000 7f7197f2f700 1 -- 172.21.15.112:0/3509797754 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71900caaf0 msgr2=0x7f7190155da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:43.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.115+0000 7f7197f2f700 1 --2- 172.21.15.112:0/3509797754 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71900caaf0 0x7f7190155da0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7188009e60 tx=0x7f7188009780 comp rx=0 tx=0).stop 2023-01-24T03:40:43.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.115+0000 7f7197f2f700 1 -- 172.21.15.112:0/3509797754 shutdown_connections 2023-01-24T03:40:43.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.115+0000 7f7197f2f700 1 --2- 172.21.15.112:0/3509797754 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71900caaf0 0x7f7190155da0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:43.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.115+0000 7f7197f2f700 1 -- 172.21.15.112:0/3509797754 >> 172.21.15.112:0/3509797754 conn(0x7f719005e2c0 msgr2=0x7f71900cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:43.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.115+0000 7f7197f2f700 1 -- 172.21.15.112:0/3509797754 shutdown_connections 2023-01-24T03:40:43.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.115+0000 7f7197f2f700 1 -- 172.21.15.112:0/3509797754 wait complete. 2023-01-24T03:40:43.126 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.116+0000 7f7197f2f700 1 Processor -- start 2023-01-24T03:40:43.126 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.116+0000 7f7197f2f700 1 -- start start 2023-01-24T03:40:43.126 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.117+0000 7f7197f2f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f71900b0aa0 0x7f719014ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:43.127 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.117+0000 7f7197f2f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71900c7e40 0x7f719014d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:43.127 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.117+0000 7f7197f2f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71900caaf0 0x7f7190151a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:43.127 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.117+0000 7f7197f2f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7190165940 con 0x7f71900b0aa0 2023-01-24T03:40:43.127 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.117+0000 7f7197f2f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71901515c0 con 0x7f71900c7e40 2023-01-24T03:40:43.127 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.117+0000 7f7197f2f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7190165e00 con 0x7f71900caaf0 2023-01-24T03:40:43.128 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.117+0000 7f7195ccb700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f71900b0aa0 0x7f719014ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:43.128 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.117+0000 7f7195ccb700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f71900b0aa0 0x7f719014ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55904/0 (socket says 172.21.15.112:55904) 2023-01-24T03:40:43.128 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.117+0000 7f7195ccb700 1 -- 172.21.15.112:0/1451086036 learned_addr learned my addr 172.21.15.112:0/1451086036 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:40:43.128 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.117+0000 7f71964cc700 1 --2- 172.21.15.112:0/1451086036 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71900caaf0 0x7f7190151a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:43.128 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.117+0000 7f71954ca700 1 --2- 172.21.15.112:0/1451086036 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71900c7e40 0x7f719014d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:43.129 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.117+0000 7f7195ccb700 1 -- 172.21.15.112:0/1451086036 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71900caaf0 msgr2=0x7f7190151a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:43.129 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.117+0000 7f7195ccb700 1 --2- 172.21.15.112:0/1451086036 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71900caaf0 0x7f7190151a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:43.129 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.117+0000 7f7195ccb700 1 -- 172.21.15.112:0/1451086036 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71900c7e40 msgr2=0x7f719014d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:43.129 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.117+0000 7f7195ccb700 1 --2- 172.21.15.112:0/1451086036 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71900c7e40 0x7f719014d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:43.129 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.117+0000 7f7195ccb700 1 -- 172.21.15.112:0/1451086036 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7188009a20 con 0x7f71900b0aa0 2023-01-24T03:40:43.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.117+0000 7f71964cc700 1 --2- 172.21.15.112:0/1451086036 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71900caaf0 0x7f7190151a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:40:43.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.117+0000 7f71954ca700 1 --2- 172.21.15.112:0/1451086036 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71900c7e40 0x7f719014d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:40:43.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.118+0000 7f7195ccb700 1 --2- 172.21.15.112:0/1451086036 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f71900b0aa0 0x7f719014ff10 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f718c00c410 tx=0x7f718c007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:43.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.118+0000 7f717dffb700 1 -- 172.21.15.112:0/1451086036 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f718c011590 con 0x7f71900b0aa0 2023-01-24T03:40:43.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.118+0000 7f717dffb700 1 -- 172.21.15.112:0/1451086036 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f718c011710 con 0x7f71900b0aa0 2023-01-24T03:40:43.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.118+0000 7f7197f2f700 1 -- 172.21.15.112:0/1451086036 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7190157e70 con 0x7f71900b0aa0 2023-01-24T03:40:43.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.118+0000 7f717dffb700 1 -- 172.21.15.112:0/1451086036 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f718c011ad0 con 0x7f71900b0aa0 2023-01-24T03:40:43.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.118+0000 7f7197f2f700 1 -- 172.21.15.112:0/1451086036 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7190151250 con 0x7f71900b0aa0 2023-01-24T03:40:43.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.119+0000 7f7197f2f700 1 -- 172.21.15.112:0/1451086036 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71901518a0 con 0x7f71900b0aa0 2023-01-24T03:40:43.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.119+0000 7f717dffb700 1 -- 172.21.15.112:0/1451086036 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f718c019030 con 0x7f71900b0aa0 2023-01-24T03:40:43.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.120+0000 7f717dffb700 1 --2- 172.21.15.112:0/1451086036 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f71700611c0 0x7f7170063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:43.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.120+0000 7f717dffb700 1 -- 172.21.15.112:0/1451086036 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 7469+0+0 (secure 0 0 0) 0x7f718c07cd80 con 0x7f71900b0aa0 2023-01-24T03:40:43.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.120+0000 7f71954ca700 1 --2- 172.21.15.112:0/1451086036 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f71700611c0 0x7f7170063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:43.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.120+0000 7f71954ca700 1 --2- 172.21.15.112:0/1451086036 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f71700611c0 0x7f7170063670 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7190067480 tx=0x7f7180009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:43.133 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.124+0000 7f717dffb700 1 -- 172.21.15.112:0/1451086036 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f718c04c8d0 con 0x7f71900b0aa0 2023-01-24T03:40:43.346 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.343+0000 7f7197f2f700 1 -- 172.21.15.112:0/1451086036 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71900af5d0 con 0x7f71900b0aa0 2023-01-24T03:40:43.347 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.344+0000 7f717dffb700 1 -- 172.21.15.112:0/1451086036 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+5364 (secure 0 0 0) 0x7f718c017090 con 0x7f71900b0aa0 2023-01-24T03:40:43.347 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:40:43.348 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:40:38.808944+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5186},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5186":{"gid":5186,"name":"b","rank":0,"incarnation":42,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.142:6834/2804746591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2804746591},{"type":"v1","addr":"172.21.15.142:6834","nonce":2804746591}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:40:43.349 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.346+0000 7f7197f2f700 1 -- 172.21.15.112:0/1451086036 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f71700611c0 msgr2=0x7f7170063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:43.349 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.346+0000 7f7197f2f700 1 --2- 172.21.15.112:0/1451086036 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f71700611c0 0x7f7170063670 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7190067480 tx=0x7f7180009040 comp rx=0 tx=0).stop 2023-01-24T03:40:43.350 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.346+0000 7f7197f2f700 1 -- 172.21.15.112:0/1451086036 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f71900b0aa0 msgr2=0x7f719014ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:43.350 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.346+0000 7f7197f2f700 1 --2- 172.21.15.112:0/1451086036 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f71900b0aa0 0x7f719014ff10 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f718c00c410 tx=0x7f718c007e00 comp rx=0 tx=0).stop 2023-01-24T03:40:43.350 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.346+0000 7f7197f2f700 1 -- 172.21.15.112:0/1451086036 shutdown_connections 2023-01-24T03:40:43.350 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.346+0000 7f7197f2f700 1 --2- 172.21.15.112:0/1451086036 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f71700611c0 0x7f7170063670 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:43.350 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.346+0000 7f7197f2f700 1 --2- 172.21.15.112:0/1451086036 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f71900b0aa0 0x7f719014ff10 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:43.351 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.346+0000 7f7197f2f700 1 --2- 172.21.15.112:0/1451086036 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f71900caaf0 0x7f7190151a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:43.351 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.346+0000 7f7197f2f700 1 --2- 172.21.15.112:0/1451086036 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f71900c7e40 0x7f719014d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:43.351 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.346+0000 7f7197f2f700 1 -- 172.21.15.112:0/1451086036 >> 172.21.15.112:0/1451086036 conn(0x7f719005e2c0 msgr2=0x7f71900680c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:43.351 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.346+0000 7f7197f2f700 1 -- 172.21.15.112:0/1451086036 shutdown_connections 2023-01-24T03:40:43.351 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:43.346+0000 7f7197f2f700 1 -- 172.21.15.112:0/1451086036 wait complete. 2023-01-24T03:40:43.352 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 43 2023-01-24T03:40:43.361 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:40:45.362 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:40:45.489 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.486+0000 7fd5b336f700 1 Processor -- start 2023-01-24T03:40:45.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.486+0000 7fd5b336f700 1 -- start start 2023-01-24T03:40:45.504 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.487+0000 7fd5b336f700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fd5ac0c7e40 0x7fd5ac0b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:45.504 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.487+0000 7fd5b336f700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fd5ac155da0 0x7fd5ac0ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:45.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.487+0000 7fd5b336f700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5ac156240 con 0x7fd5ac0aef20 2023-01-24T03:40:45.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.487+0000 7fd5b336f700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5ac0af320 con 0x7fd5ac155da0 2023-01-24T03:40:45.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.487+0000 7fd5b336f700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5ac0b1170 con 0x7fd5ac0c7e40 2023-01-24T03:40:45.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.487+0000 7fd5b090a700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fd5ac155da0 0x7fd5ac0ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:45.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.487+0000 7fd5b190c700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fd5ac0aef20 0x7fd5ac0ca3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:36902/0 (socket says 172.21.15.112:36902) 2023-01-24T03:40:45.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.487+0000 7fd5b190c700 1 -- 172.21.15.112:0/1841594922 learned_addr learned my addr 172.21.15.112:0/1841594922 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:40:45.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.488+0000 7fd5b110b700 1 --2- 172.21.15.112:0/1841594922 >> v2:172.21.15.142:3300/0 conn(0x7fd5ac0c7e40 0x7fd5ac0b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:45.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.488+0000 7fd5b090a700 1 -- 172.21.15.112:0/1841594922 >> v2:172.21.15.142:3300/0 conn(0x7fd5ac0c7e40 msgr2=0x7fd5ac0b0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:40:45.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.488+0000 7fd5b090a700 1 --2- 172.21.15.112:0/1841594922 >> v2:172.21.15.142:3300/0 conn(0x7fd5ac0c7e40 0x7fd5ac0b0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:45.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.488+0000 7fd5b090a700 1 -- 172.21.15.112:0/1841594922 >> v1:172.21.15.112:6789/0 conn(0x7fd5ac0aef20 legacy=0x7fd5ac0ca3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:45.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.488+0000 7fd5b090a700 1 -- 172.21.15.112:0/1841594922 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5ac0b1450 con 0x7fd5ac155da0 2023-01-24T03:40:45.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.488+0000 7fd5abfff700 1 -- 172.21.15.112:0/1841594922 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1449002534 0 0) 0x7fd5ac156240 con 0x7fd5ac0aef20 2023-01-24T03:40:45.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.488+0000 7fd5b090a700 1 --2- 172.21.15.112:0/1841594922 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd5ac155da0 0x7fd5ac0ae9e0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac089cf0 tx=0x7fd5a0009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5529505a4210e79d server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:45.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.488+0000 7fd5abfff700 1 -- 172.21.15.112:0/1841594922 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a0011630 con 0x7fd5ac155da0 2023-01-24T03:40:45.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.488+0000 7fd5abfff700 1 -- 172.21.15.112:0/1841594922 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5a0011790 con 0x7fd5ac155da0 2023-01-24T03:40:45.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.488+0000 7fd5abfff700 1 -- 172.21.15.112:0/1841594922 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a0011b50 con 0x7fd5ac155da0 2023-01-24T03:40:45.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.488+0000 7fd5b336f700 1 -- 172.21.15.112:0/1841594922 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd5ac155da0 msgr2=0x7fd5ac0ae9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:45.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.488+0000 7fd5b336f700 1 --2- 172.21.15.112:0/1841594922 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd5ac155da0 0x7fd5ac0ae9e0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac089cf0 tx=0x7fd5a0009060 comp rx=0 tx=0).stop 2023-01-24T03:40:45.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.488+0000 7fd5b336f700 1 -- 172.21.15.112:0/1841594922 shutdown_connections 2023-01-24T03:40:45.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.488+0000 7fd5b336f700 1 --2- 172.21.15.112:0/1841594922 >> v2:172.21.15.142:3300/0 conn(0x7fd5ac0c7e40 0x7fd5ac0b0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:45.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.488+0000 7fd5b336f700 1 --2- 172.21.15.112:0/1841594922 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd5ac155da0 0x7fd5ac0ae9e0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:45.510 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.488+0000 7fd5b336f700 1 -- 172.21.15.112:0/1841594922 >> 172.21.15.112:0/1841594922 conn(0x7fd5ac05e2c0 msgr2=0x7fd5ac0cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:45.510 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.488+0000 7fd5b336f700 1 -- 172.21.15.112:0/1841594922 shutdown_connections 2023-01-24T03:40:45.510 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.488+0000 7fd5b336f700 1 -- 172.21.15.112:0/1841594922 wait complete. 2023-01-24T03:40:45.510 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.489+0000 7fd5b336f700 1 Processor -- start 2023-01-24T03:40:45.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.490+0000 7fd5b336f700 1 -- start start 2023-01-24T03:40:45.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.490+0000 7fd5b336f700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd5ac0aef20 0x7fd5ac0c1b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:45.512 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.490+0000 7fd5b336f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5ac0c7e40 0x7fd5ac0c2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:45.512 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.490+0000 7fd5b336f700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5ac155da0 0x7fd5ac0c3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:45.512 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.490+0000 7fd5b336f700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ac1656f0 con 0x7fd5ac0aef20 2023-01-24T03:40:45.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.490+0000 7fd5b336f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ac1659d0 con 0x7fd5ac0c7e40 2023-01-24T03:40:45.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.490+0000 7fd5b336f700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ac165cb0 con 0x7fd5ac155da0 2023-01-24T03:40:45.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.490+0000 7fd5b110b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd5ac0aef20 0x7fd5ac0c1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:45.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.490+0000 7fd5b110b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd5ac0aef20 0x7fd5ac0c1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55918/0 (socket says 172.21.15.112:55918) 2023-01-24T03:40:45.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.490+0000 7fd5b110b700 1 -- 172.21.15.112:0/3987694937 learned_addr learned my addr 172.21.15.112:0/3987694937 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:40:45.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.491+0000 7fd5b090a700 1 --2- 172.21.15.112:0/3987694937 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5ac0c7e40 0x7fd5ac0c2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:45.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.491+0000 7fd5b110b700 1 -- 172.21.15.112:0/3987694937 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5ac155da0 msgr2=0x7fd5ac0c3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:45.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.491+0000 7fd5b110b700 1 --2- 172.21.15.112:0/3987694937 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5ac155da0 0x7fd5ac0c3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:45.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.491+0000 7fd5b110b700 1 -- 172.21.15.112:0/3987694937 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5ac0c7e40 msgr2=0x7fd5ac0c2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:45.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.491+0000 7fd5b110b700 1 --2- 172.21.15.112:0/3987694937 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5ac0c7e40 0x7fd5ac0c2e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:45.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.491+0000 7fd5b110b700 1 -- 172.21.15.112:0/3987694937 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a00092a0 con 0x7fd5ac0aef20 2023-01-24T03:40:45.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.491+0000 7fd5b110b700 1 --2- 172.21.15.112:0/3987694937 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd5ac0aef20 0x7fd5ac0c1b50 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fd59c00ce10 tx=0x7fd59c007810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:45.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.491+0000 7fd5a97fa700 1 -- 172.21.15.112:0/3987694937 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd59c00ab90 con 0x7fd5ac0aef20 2023-01-24T03:40:45.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.491+0000 7fd5a97fa700 1 -- 172.21.15.112:0/3987694937 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd59c00acf0 con 0x7fd5ac0aef20 2023-01-24T03:40:45.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.491+0000 7fd5b336f700 1 -- 172.21.15.112:0/3987694937 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5ac165f90 con 0x7fd5ac0aef20 2023-01-24T03:40:45.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.491+0000 7fd5a97fa700 1 -- 172.21.15.112:0/3987694937 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd59c017660 con 0x7fd5ac0aef20 2023-01-24T03:40:45.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.491+0000 7fd5b336f700 1 -- 172.21.15.112:0/3987694937 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5ac157230 con 0x7fd5ac0aef20 2023-01-24T03:40:45.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.492+0000 7fd5b336f700 1 -- 172.21.15.112:0/3987694937 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5ac0c0830 con 0x7fd5ac0aef20 2023-01-24T03:40:45.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.492+0000 7fd5a97fa700 1 -- 172.21.15.112:0/3987694937 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fd59c01e060 con 0x7fd5ac0aef20 2023-01-24T03:40:45.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.493+0000 7fd5a97fa700 1 --2- 172.21.15.112:0/3987694937 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd58c0611c0 0x7fd58c063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:45.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.493+0000 7fd5a97fa700 1 -- 172.21.15.112:0/3987694937 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 7469+0+0 (secure 0 0 0) 0x7fd59c013070 con 0x7fd5ac0aef20 2023-01-24T03:40:45.519 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.493+0000 7fd5b090a700 1 --2- 172.21.15.112:0/3987694937 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd58c0611c0 0x7fd58c063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:45.519 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.493+0000 7fd5b090a700 1 --2- 172.21.15.112:0/3987694937 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd58c0611c0 0x7fd58c063670 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd5a0009fd0 tx=0x7fd5a0017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:45.519 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.497+0000 7fd5a97fa700 1 -- 172.21.15.112:0/3987694937 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fd59c04de10 con 0x7fd5ac0aef20 2023-01-24T03:40:45.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.715+0000 7fd5b336f700 1 -- 172.21.15.112:0/3987694937 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5ac0c2c90 con 0x7fd5ac0aef20 2023-01-24T03:40:45.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.715+0000 7fd5a97fa700 1 -- 172.21.15.112:0/3987694937 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+5361 (secure 0 0 0) 0x7fd59c00ae60 con 0x7fd5ac0aef20 2023-01-24T03:40:45.719 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:40:45.719 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:40:44.844101+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5186},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5186":{"gid":5186,"name":"b","rank":0,"incarnation":42,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.142:6834/2804746591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2804746591},{"type":"v1","addr":"172.21.15.142:6834","nonce":2804746591}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:40:45.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.717+0000 7fd5b336f700 1 -- 172.21.15.112:0/3987694937 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd58c0611c0 msgr2=0x7fd58c063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:45.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.717+0000 7fd5b336f700 1 --2- 172.21.15.112:0/3987694937 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd58c0611c0 0x7fd58c063670 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd5a0009fd0 tx=0x7fd5a0017000 comp rx=0 tx=0).stop 2023-01-24T03:40:45.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.717+0000 7fd5b336f700 1 -- 172.21.15.112:0/3987694937 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd5ac0aef20 msgr2=0x7fd5ac0c1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:45.721 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.717+0000 7fd5b336f700 1 --2- 172.21.15.112:0/3987694937 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd5ac0aef20 0x7fd5ac0c1b50 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fd59c00ce10 tx=0x7fd59c007810 comp rx=0 tx=0).stop 2023-01-24T03:40:45.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.717+0000 7fd5b336f700 1 -- 172.21.15.112:0/3987694937 shutdown_connections 2023-01-24T03:40:45.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.717+0000 7fd5b336f700 1 --2- 172.21.15.112:0/3987694937 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fd58c0611c0 0x7fd58c063670 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:45.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.717+0000 7fd5b336f700 1 --2- 172.21.15.112:0/3987694937 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd5ac0aef20 0x7fd5ac0c1b50 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:45.722 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.717+0000 7fd5b336f700 1 --2- 172.21.15.112:0/3987694937 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fd5ac155da0 0x7fd5ac0c3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:45.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.717+0000 7fd5b336f700 1 --2- 172.21.15.112:0/3987694937 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd5ac0c7e40 0x7fd5ac0c2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:45.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.717+0000 7fd5b336f700 1 -- 172.21.15.112:0/3987694937 >> 172.21.15.112:0/3987694937 conn(0x7fd5ac05e2c0 msgr2=0x7fd5ac1591f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:45.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.717+0000 7fd5b336f700 1 -- 172.21.15.112:0/3987694937 shutdown_connections 2023-01-24T03:40:45.723 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:45.717+0000 7fd5b336f700 1 -- 172.21.15.112:0/3987694937 wait complete. 2023-01-24T03:40:45.724 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 44 2023-01-24T03:40:45.732 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:40:47.733 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:40:47.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.856+0000 7f55fbfff700 1 Processor -- start 2023-01-24T03:40:47.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.856+0000 7f55fbfff700 1 -- start start 2023-01-24T03:40:47.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.857+0000 7f55fbfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55fc0c29e0 0x7f55fc0c2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:47.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.857+0000 7f55fbfff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f55fc152520 0x7f55fc06d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:47.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.857+0000 7f55fbfff700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55fc06a5c0 con 0x7f55fc06ebe0 2023-01-24T03:40:47.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.857+0000 7f55fbfff700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55fc06d100 con 0x7f55fc152520 2023-01-24T03:40:47.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.857+0000 7f55fbfff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55fc06a810 con 0x7f55fc0c29e0 2023-01-24T03:40:47.862 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.857+0000 7f55fb7fe700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f55fc152520 0x7f55fc06d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:47.862 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.857+0000 7f55fa7fc700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f55fc06ebe0 0x7f55fc151e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:36918/0 (socket says 172.21.15.112:36918) 2023-01-24T03:40:47.862 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.857+0000 7f55fa7fc700 1 -- 172.21.15.112:0/2888069080 learned_addr learned my addr 172.21.15.112:0/2888069080 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:40:47.862 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.857+0000 7f55faffd700 1 --2- 172.21.15.112:0/2888069080 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55fc0c29e0 0x7f55fc0c2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:47.862 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.857+0000 7f55fb7fe700 1 -- 172.21.15.112:0/2888069080 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55fc0c29e0 msgr2=0x7f55fc0c2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:40:47.863 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.857+0000 7f55fb7fe700 1 --2- 172.21.15.112:0/2888069080 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55fc0c29e0 0x7f55fc0c2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:47.863 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.857+0000 7f55fb7fe700 1 -- 172.21.15.112:0/2888069080 >> v1:172.21.15.112:6789/0 conn(0x7f55fc06ebe0 legacy=0x7f55fc151e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:47.863 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.857+0000 7f55fb7fe700 1 -- 172.21.15.112:0/2888069080 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55fc06aaf0 con 0x7f55fc152520 2023-01-24T03:40:47.863 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.857+0000 7f55f9ffb700 1 -- 172.21.15.112:0/2888069080 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2301980137 0 0) 0x7f55fc06a5c0 con 0x7f55fc06ebe0 2023-01-24T03:40:47.863 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.857+0000 7f55faffd700 1 --2- 172.21.15.112:0/2888069080 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55fc0c29e0 0x7f55fc0c2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:40:47.864 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.857+0000 7f55fb7fe700 1 --2- 172.21.15.112:0/2888069080 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55fc152520 0x7f55fc06d550 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f55f4009760 tx=0x7f55f4009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59aaa6e83ebe9b48 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:47.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.859+0000 7f55f9ffb700 1 -- 172.21.15.112:0/2888069080 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f4011630 con 0x7f55fc152520 2023-01-24T03:40:47.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.859+0000 7f55f9ffb700 1 -- 172.21.15.112:0/2888069080 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55f4011790 con 0x7f55fc152520 2023-01-24T03:40:47.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.859+0000 7f55f9ffb700 1 -- 172.21.15.112:0/2888069080 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f4011b30 con 0x7f55fc152520 2023-01-24T03:40:47.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.860+0000 7f55fbfff700 1 -- 172.21.15.112:0/2888069080 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55fc152520 msgr2=0x7f55fc06d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:47.869 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.860+0000 7f55fbfff700 1 --2- 172.21.15.112:0/2888069080 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55fc152520 0x7f55fc06d550 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f55f4009760 tx=0x7f55f4009060 comp rx=0 tx=0).stop 2023-01-24T03:40:47.869 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.860+0000 7f55fbfff700 1 -- 172.21.15.112:0/2888069080 shutdown_connections 2023-01-24T03:40:47.869 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.860+0000 7f55fbfff700 1 --2- 172.21.15.112:0/2888069080 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55fc152520 0x7f55fc06d550 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:47.869 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.860+0000 7f55fbfff700 1 --2- 172.21.15.112:0/2888069080 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55fc0c29e0 0x7f55fc0c2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:47.869 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.860+0000 7f55fbfff700 1 -- 172.21.15.112:0/2888069080 >> 172.21.15.112:0/2888069080 conn(0x7f55fc066e10 msgr2=0x7f55fc067210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:47.870 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.860+0000 7f55fbfff700 1 -- 172.21.15.112:0/2888069080 shutdown_connections 2023-01-24T03:40:47.870 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.860+0000 7f55fbfff700 1 -- 172.21.15.112:0/2888069080 wait complete. 2023-01-24T03:40:47.870 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.861+0000 7f55fbfff700 1 Processor -- start 2023-01-24T03:40:47.870 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.861+0000 7f55fbfff700 1 -- start start 2023-01-24T03:40:47.870 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.862+0000 7f55fbfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55fc06ebe0 0x7f55fc0c7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:47.871 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.862+0000 7f55fbfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55fc0c29e0 0x7f55fc0b5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:47.871 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.862+0000 7f55fbfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55fc152520 0x7f55fc0b5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:47.871 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.862+0000 7f55fbfff700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55fc16da00 con 0x7f55fc152520 2023-01-24T03:40:47.871 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.862+0000 7f55fbfff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55fc0ba810 con 0x7f55fc06ebe0 2023-01-24T03:40:47.871 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.862+0000 7f55fbfff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55fc0b6aa0 con 0x7f55fc0c29e0 2023-01-24T03:40:47.872 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.862+0000 7f55fb7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55fc152520 0x7f55fc0b5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:47.872 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.862+0000 7f55fb7fe700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55fc152520 0x7f55fc0b5680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55936/0 (socket says 172.21.15.112:55936) 2023-01-24T03:40:47.872 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.862+0000 7f55fb7fe700 1 -- 172.21.15.112:0/744458570 learned_addr learned my addr 172.21.15.112:0/744458570 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:40:47.872 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.862+0000 7f55fa7fc700 1 --2- 172.21.15.112:0/744458570 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55fc0c29e0 0x7f55fc0b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:47.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.862+0000 7f55faffd700 1 --2- 172.21.15.112:0/744458570 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55fc06ebe0 0x7f55fc0c7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:47.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.862+0000 7f55fb7fe700 1 -- 172.21.15.112:0/744458570 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55fc0c29e0 msgr2=0x7f55fc0b5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:47.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.862+0000 7f55fb7fe700 1 --2- 172.21.15.112:0/744458570 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55fc0c29e0 0x7f55fc0b5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:47.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.862+0000 7f55fb7fe700 1 -- 172.21.15.112:0/744458570 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55fc06ebe0 msgr2=0x7f55fc0c7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:47.874 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.862+0000 7f55fb7fe700 1 --2- 172.21.15.112:0/744458570 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55fc06ebe0 0x7f55fc0c7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:47.874 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.862+0000 7f55fb7fe700 1 -- 172.21.15.112:0/744458570 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55f40092a0 con 0x7f55fc152520 2023-01-24T03:40:47.874 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.862+0000 7f55fb7fe700 1 --2- 172.21.15.112:0/744458570 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55fc152520 0x7f55fc0b5680 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f55f40091e0 tx=0x7f55f4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:47.874 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.863+0000 7f55f2ffd700 1 -- 172.21.15.112:0/744458570 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f4011900 con 0x7f55fc152520 2023-01-24T03:40:47.874 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.863+0000 7f55f2ffd700 1 -- 172.21.15.112:0/744458570 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55f4011a60 con 0x7f55fc152520 2023-01-24T03:40:47.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.863+0000 7f55fbfff700 1 -- 172.21.15.112:0/744458570 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55fc0b5e90 con 0x7f55fc152520 2023-01-24T03:40:47.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.863+0000 7f55f2ffd700 1 -- 172.21.15.112:0/744458570 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f4026640 con 0x7f55fc152520 2023-01-24T03:40:47.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.863+0000 7f55fbfff700 1 -- 172.21.15.112:0/744458570 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55fc0b6450 con 0x7f55fc152520 2023-01-24T03:40:47.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.864+0000 7f55fbfff700 1 -- 172.21.15.112:0/744458570 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55fc05ea20 con 0x7f55fc152520 2023-01-24T03:40:47.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.864+0000 7f55f2ffd700 1 -- 172.21.15.112:0/744458570 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f55f400f050 con 0x7f55fc152520 2023-01-24T03:40:47.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.864+0000 7f55f2ffd700 1 --2- 172.21.15.112:0/744458570 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f55e00611c0 0x7f55e0063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:47.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.864+0000 7f55f2ffd700 1 -- 172.21.15.112:0/744458570 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 7469+0+0 (secure 0 0 0) 0x7f55f407f780 con 0x7f55fc152520 2023-01-24T03:40:47.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.864+0000 7f55faffd700 1 --2- 172.21.15.112:0/744458570 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f55e00611c0 0x7f55e0063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:47.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.865+0000 7f55faffd700 1 --2- 172.21.15.112:0/744458570 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f55e00611c0 0x7f55e0063670 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f55ec007740 tx=0x7f55ec0099d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:47.877 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:47.869+0000 7f55f2ffd700 1 -- 172.21.15.112:0/744458570 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f55f404f350 con 0x7f55fc152520 2023-01-24T03:40:48.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:48.088+0000 7f55fbfff700 1 -- 172.21.15.112:0/744458570 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55fc0b6be0 con 0x7f55fc152520 2023-01-24T03:40:48.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:48.088+0000 7f55f2ffd700 1 -- 172.21.15.112:0/744458570 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+5361 (secure 0 0 0) 0x7f55f4052970 con 0x7f55fc152520 2023-01-24T03:40:48.091 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:40:48.091 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:40:44.844101+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5186},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5186":{"gid":5186,"name":"b","rank":0,"incarnation":42,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.142:6834/2804746591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2804746591},{"type":"v1","addr":"172.21.15.142:6834","nonce":2804746591}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:40:48.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:48.090+0000 7f55fbfff700 1 -- 172.21.15.112:0/744458570 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f55e00611c0 msgr2=0x7f55e0063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:48.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:48.090+0000 7f55fbfff700 1 --2- 172.21.15.112:0/744458570 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f55e00611c0 0x7f55e0063670 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f55ec007740 tx=0x7f55ec0099d0 comp rx=0 tx=0).stop 2023-01-24T03:40:48.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:48.090+0000 7f55fbfff700 1 -- 172.21.15.112:0/744458570 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55fc152520 msgr2=0x7f55fc0b5680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:48.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:48.090+0000 7f55fbfff700 1 --2- 172.21.15.112:0/744458570 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55fc152520 0x7f55fc0b5680 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f55f40091e0 tx=0x7f55f4017000 comp rx=0 tx=0).stop 2023-01-24T03:40:48.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:48.090+0000 7f55fbfff700 1 -- 172.21.15.112:0/744458570 shutdown_connections 2023-01-24T03:40:48.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:48.090+0000 7f55fbfff700 1 --2- 172.21.15.112:0/744458570 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f55e00611c0 0x7f55e0063670 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:48.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:48.090+0000 7f55fbfff700 1 --2- 172.21.15.112:0/744458570 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f55fc152520 0x7f55fc0b5680 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:48.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:48.090+0000 7f55fbfff700 1 --2- 172.21.15.112:0/744458570 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55fc06ebe0 0x7f55fc0c7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:48.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:48.090+0000 7f55fbfff700 1 --2- 172.21.15.112:0/744458570 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f55fc0c29e0 0x7f55fc0b5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:48.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:48.090+0000 7f55fbfff700 1 -- 172.21.15.112:0/744458570 >> 172.21.15.112:0/744458570 conn(0x7f55fc066e10 msgr2=0x7f55fc0c2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:48.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:48.090+0000 7f55fbfff700 1 -- 172.21.15.112:0/744458570 shutdown_connections 2023-01-24T03:40:48.095 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:48.090+0000 7f55fbfff700 1 -- 172.21.15.112:0/744458570 wait complete. 2023-01-24T03:40:48.095 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 44 2023-01-24T03:40:48.104 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:40:50.105 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:40:50.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.228+0000 7ff77bfff700 1 Processor -- start 2023-01-24T03:40:50.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.228+0000 7ff77bfff700 1 -- start start 2023-01-24T03:40:50.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.229+0000 7ff77bfff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7ff77c0c3360 0x7ff77c05df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:50.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.229+0000 7ff77bfff700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7ff77c068a70 0x7ff77c155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:50.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.229+0000 7ff77bfff700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff77c05e5b0 con 0x7ff77c0c3730 2023-01-24T03:40:50.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.229+0000 7ff77bfff700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff77c155ff0 con 0x7ff77c068a70 2023-01-24T03:40:50.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.229+0000 7ff77bfff700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff77c06cd70 con 0x7ff77c0c3360 2023-01-24T03:40:50.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.229+0000 7ff77b7fe700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7ff77c068a70 0x7ff77c155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:50.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.229+0000 7ff77a7fc700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7ff77c0c3730 0x7ff77c067f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:36928/0 (socket says 172.21.15.112:36928) 2023-01-24T03:40:50.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.229+0000 7ff77a7fc700 1 -- 172.21.15.112:0/333264150 learned_addr learned my addr 172.21.15.112:0/333264150 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:40:50.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.229+0000 7ff77affd700 1 --2- 172.21.15.112:0/333264150 >> v2:172.21.15.142:3300/0 conn(0x7ff77c0c3360 0x7ff77c05df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:50.238 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.229+0000 7ff77b7fe700 1 -- 172.21.15.112:0/333264150 >> v2:172.21.15.142:3300/0 conn(0x7ff77c0c3360 msgr2=0x7ff77c05df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:40:50.238 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.229+0000 7ff77b7fe700 1 --2- 172.21.15.112:0/333264150 >> v2:172.21.15.142:3300/0 conn(0x7ff77c0c3360 0x7ff77c05df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:50.238 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.229+0000 7ff77b7fe700 1 -- 172.21.15.112:0/333264150 >> v1:172.21.15.112:6789/0 conn(0x7ff77c0c3730 legacy=0x7ff77c067f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:50.238 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.229+0000 7ff77b7fe700 1 -- 172.21.15.112:0/333264150 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff77c06d050 con 0x7ff77c068a70 2023-01-24T03:40:50.238 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.229+0000 7ff779ffb700 1 -- 172.21.15.112:0/333264150 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3413137486 0 0) 0x7ff77c05e5b0 con 0x7ff77c0c3730 2023-01-24T03:40:50.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.229+0000 7ff77affd700 1 --2- 172.21.15.112:0/333264150 >> v2:172.21.15.142:3300/0 conn(0x7ff77c0c3360 0x7ff77c05df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-01-24T03:40:50.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.230+0000 7ff77b7fe700 1 --2- 172.21.15.112:0/333264150 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff77c068a70 0x7ff77c155ab0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7ff77c0ca740 tx=0x7ff774009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32bbc7cd4186a13e server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:50.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.230+0000 7ff779ffb700 1 -- 172.21.15.112:0/333264150 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff774011630 con 0x7ff77c068a70 2023-01-24T03:40:50.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.230+0000 7ff779ffb700 1 -- 172.21.15.112:0/333264150 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff774011790 con 0x7ff77c068a70 2023-01-24T03:40:50.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.230+0000 7ff779ffb700 1 -- 172.21.15.112:0/333264150 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff774011b50 con 0x7ff77c068a70 2023-01-24T03:40:50.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.230+0000 7ff77bfff700 1 -- 172.21.15.112:0/333264150 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff77c068a70 msgr2=0x7ff77c155ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:50.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.230+0000 7ff77bfff700 1 --2- 172.21.15.112:0/333264150 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff77c068a70 0x7ff77c155ab0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7ff77c0ca740 tx=0x7ff774009060 comp rx=0 tx=0).stop 2023-01-24T03:40:50.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.230+0000 7ff77bfff700 1 -- 172.21.15.112:0/333264150 shutdown_connections 2023-01-24T03:40:50.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.230+0000 7ff77bfff700 1 --2- 172.21.15.112:0/333264150 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff77c068a70 0x7ff77c155ab0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:50.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.230+0000 7ff77bfff700 1 --2- 172.21.15.112:0/333264150 >> v2:172.21.15.142:3300/0 conn(0x7ff77c0c3360 0x7ff77c05df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:50.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.230+0000 7ff77bfff700 1 -- 172.21.15.112:0/333264150 >> 172.21.15.112:0/333264150 conn(0x7ff77c06c360 msgr2=0x7ff77c05d830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:50.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.230+0000 7ff77bfff700 1 -- 172.21.15.112:0/333264150 shutdown_connections 2023-01-24T03:40:50.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.230+0000 7ff77bfff700 1 -- 172.21.15.112:0/333264150 wait complete. 2023-01-24T03:40:50.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.231+0000 7ff77bfff700 1 Processor -- start 2023-01-24T03:40:50.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.231+0000 7ff77bfff700 1 -- start start 2023-01-24T03:40:50.242 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.232+0000 7ff77bfff700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff77c068a70 0x7ff77c1500a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:50.242 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.232+0000 7ff77bfff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff77c0c3360 0x7ff77c154ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:50.242 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.232+0000 7ff77bfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff77c156200 0x7ff77c164c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:50.242 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.232+0000 7ff77bfff700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff77c152b10 con 0x7ff77c0c3360 2023-01-24T03:40:50.242 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.232+0000 7ff77bfff700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff77c14ff40 con 0x7ff77c156200 2023-01-24T03:40:50.243 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.232+0000 7ff77bfff700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff77c151500 con 0x7ff77c068a70 2023-01-24T03:40:50.243 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.232+0000 7ff77a7fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff77c0c3360 0x7ff77c154ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:50.243 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.232+0000 7ff77a7fc700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff77c0c3360 0x7ff77c154ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55948/0 (socket says 172.21.15.112:55948) 2023-01-24T03:40:50.243 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.232+0000 7ff77a7fc700 1 -- 172.21.15.112:0/4038881290 learned_addr learned my addr 172.21.15.112:0/4038881290 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:40:50.243 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.232+0000 7ff77affd700 1 --2- 172.21.15.112:0/4038881290 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff77c068a70 0x7ff77c1500a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:50.243 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.232+0000 7ff77b7fe700 1 --2- 172.21.15.112:0/4038881290 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff77c156200 0x7ff77c164c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:50.244 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.232+0000 7ff77a7fc700 1 -- 172.21.15.112:0/4038881290 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff77c068a70 msgr2=0x7ff77c1500a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:50.244 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.232+0000 7ff77a7fc700 1 --2- 172.21.15.112:0/4038881290 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff77c068a70 0x7ff77c1500a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:50.244 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.232+0000 7ff77a7fc700 1 -- 172.21.15.112:0/4038881290 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff77c156200 msgr2=0x7ff77c164c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:50.244 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.233+0000 7ff77a7fc700 1 --2- 172.21.15.112:0/4038881290 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff77c156200 0x7ff77c164c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:50.244 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.233+0000 7ff77a7fc700 1 -- 172.21.15.112:0/4038881290 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7740092a0 con 0x7ff77c0c3360 2023-01-24T03:40:50.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.233+0000 7ff77a7fc700 1 --2- 172.21.15.112:0/4038881290 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff77c0c3360 0x7ff77c154ab0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7ff76400c4e0 tx=0x7ff764007c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:50.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.233+0000 7ff772ffd700 1 -- 172.21.15.112:0/4038881290 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff76400ce90 con 0x7ff77c0c3360 2023-01-24T03:40:50.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.233+0000 7ff772ffd700 1 -- 172.21.15.112:0/4038881290 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7640072e0 con 0x7ff77c0c3360 2023-01-24T03:40:50.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.233+0000 7ff77bfff700 1 -- 172.21.15.112:0/4038881290 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff77c1505e0 con 0x7ff77c0c3360 2023-01-24T03:40:50.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.233+0000 7ff772ffd700 1 -- 172.21.15.112:0/4038881290 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff764005780 con 0x7ff77c0c3360 2023-01-24T03:40:50.246 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.233+0000 7ff77bfff700 1 -- 172.21.15.112:0/4038881290 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff77c14fc60 con 0x7ff77c0c3360 2023-01-24T03:40:50.246 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.234+0000 7ff77bfff700 1 -- 172.21.15.112:0/4038881290 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff77c069ab0 con 0x7ff77c0c3360 2023-01-24T03:40:50.246 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.234+0000 7ff772ffd700 1 -- 172.21.15.112:0/4038881290 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7ff764019030 con 0x7ff77c0c3360 2023-01-24T03:40:50.247 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.234+0000 7ff772ffd700 1 --2- 172.21.15.112:0/4038881290 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff7600611c0 0x7ff760063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:50.247 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.234+0000 7ff772ffd700 1 -- 172.21.15.112:0/4038881290 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 7469+0+0 (secure 0 0 0) 0x7ff76407cb00 con 0x7ff77c0c3360 2023-01-24T03:40:50.247 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.234+0000 7ff77affd700 1 --2- 172.21.15.112:0/4038881290 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff7600611c0 0x7ff760063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:50.248 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.235+0000 7ff77affd700 1 --2- 172.21.15.112:0/4038881290 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff7600611c0 0x7ff760063670 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff76c00d6e0 tx=0x7ff76c007850 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:50.248 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.239+0000 7ff772ffd700 1 -- 172.21.15.112:0/4038881290 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7ff76404c6b0 con 0x7ff77c0c3360 2023-01-24T03:40:50.460 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.458+0000 7ff77bfff700 1 -- 172.21.15.112:0/4038881290 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff77c1505e0 con 0x7ff77c0c3360 2023-01-24T03:40:50.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.459+0000 7ff772ffd700 1 -- 172.21.15.112:0/4038881290 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+5361 (secure 0 0 0) 0x7ff77c1505e0 con 0x7ff77c0c3360 2023-01-24T03:40:50.465 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:40:50.465 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:40:44.844101+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5186},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5186":{"gid":5186,"name":"b","rank":0,"incarnation":42,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.142:6834/2804746591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2804746591},{"type":"v1","addr":"172.21.15.142:6834","nonce":2804746591}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:40:50.467 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.461+0000 7ff77bfff700 1 -- 172.21.15.112:0/4038881290 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff7600611c0 msgr2=0x7ff760063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:50.467 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.461+0000 7ff77bfff700 1 --2- 172.21.15.112:0/4038881290 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff7600611c0 0x7ff760063670 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff76c00d6e0 tx=0x7ff76c007850 comp rx=0 tx=0).stop 2023-01-24T03:40:50.468 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.461+0000 7ff77bfff700 1 -- 172.21.15.112:0/4038881290 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff77c0c3360 msgr2=0x7ff77c154ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:50.468 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.461+0000 7ff77bfff700 1 --2- 172.21.15.112:0/4038881290 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff77c0c3360 0x7ff77c154ab0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7ff76400c4e0 tx=0x7ff764007c70 comp rx=0 tx=0).stop 2023-01-24T03:40:50.468 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.461+0000 7ff77bfff700 1 -- 172.21.15.112:0/4038881290 shutdown_connections 2023-01-24T03:40:50.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.461+0000 7ff77bfff700 1 --2- 172.21.15.112:0/4038881290 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7ff7600611c0 0x7ff760063670 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:50.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.461+0000 7ff77bfff700 1 --2- 172.21.15.112:0/4038881290 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff77c156200 0x7ff77c164c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:50.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.461+0000 7ff77bfff700 1 --2- 172.21.15.112:0/4038881290 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7ff77c068a70 0x7ff77c1500a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:50.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.461+0000 7ff77bfff700 1 --2- 172.21.15.112:0/4038881290 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff77c0c3360 0x7ff77c154ab0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:50.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.461+0000 7ff77bfff700 1 -- 172.21.15.112:0/4038881290 >> 172.21.15.112:0/4038881290 conn(0x7ff77c06c360 msgr2=0x7ff77c14eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:50.471 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.461+0000 7ff77bfff700 1 -- 172.21.15.112:0/4038881290 shutdown_connections 2023-01-24T03:40:50.471 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:50.461+0000 7ff77bfff700 1 -- 172.21.15.112:0/4038881290 wait complete. 2023-01-24T03:40:50.472 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 44 2023-01-24T03:40:50.477 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:40:52.478 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:40:52.603 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.601+0000 7fa8bd7a3700 1 Processor -- start 2023-01-24T03:40:52.603 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.601+0000 7fa8bd7a3700 1 -- start start 2023-01-24T03:40:52.606 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.602+0000 7fa8bd7a3700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa8b80c52a0 0x7fa8b80c4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:52.606 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.602+0000 7fa8bd7a3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8b81586b0 0x7fa8b814ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:52.606 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.602+0000 7fa8bd7a3700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b806ba30 con 0x7fa8b80c3ec0 2023-01-24T03:40:52.606 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.602+0000 7fa8bd7a3700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8b81595e0 con 0x7fa8b80c52a0 2023-01-24T03:40:52.607 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.602+0000 7fa8bd7a3700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b8158aa0 con 0x7fa8b81586b0 2023-01-24T03:40:52.607 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.602+0000 7fa8affff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa8b80c52a0 0x7fa8b80c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:52.607 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.602+0000 7fa8bcfa2700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8b81586b0 0x7fa8b814ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:52.607 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.602+0000 7fa8affff700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fa8b80c52a0 0x7fa8b80c4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.112:48452/0 (socket says 172.21.15.112:48452) 2023-01-24T03:40:52.608 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.602+0000 7fa8affff700 1 -- 172.21.15.112:0/3396496423 learned_addr learned my addr 172.21.15.112:0/3396496423 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:40:52.608 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.602+0000 7fa8b7fff700 1 --1- 172.21.15.112:0/3396496423 >> v1:172.21.15.142:6789/0 conn(0x7fa8b80c3ec0 0x7fa8b806af40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.142:6789/0 says I am v1:172.21.15.112:50732/0 (socket says 172.21.15.112:50732) 2023-01-24T03:40:52.608 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.603+0000 7fa8bcfa2700 1 -- 172.21.15.112:0/3396496423 >> v2:172.21.15.142:3300/0 conn(0x7fa8b80c52a0 msgr2=0x7fa8b80c4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:40:52.608 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.603+0000 7fa8bcfa2700 1 --2- 172.21.15.112:0/3396496423 >> v2:172.21.15.142:3300/0 conn(0x7fa8b80c52a0 0x7fa8b80c4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:52.608 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.603+0000 7fa8bcfa2700 1 -- 172.21.15.112:0/3396496423 >> v1:172.21.15.142:6789/0 conn(0x7fa8b80c3ec0 legacy=0x7fa8b806af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:52.609 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.603+0000 7fa8bcfa2700 1 -- 172.21.15.112:0/3396496423 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b80c2d30 con 0x7fa8b81586b0 2023-01-24T03:40:52.609 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.603+0000 7fa8affff700 1 --2- 172.21.15.112:0/3396496423 >> v2:172.21.15.142:3300/0 conn(0x7fa8b80c52a0 0x7fa8b80c4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-01-24T03:40:52.609 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.603+0000 7fa8bcfa2700 1 --2- 172.21.15.112:0/3396496423 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8b81586b0 0x7fa8b814ed10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa8b0009e60 tx=0x7fa8b0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71d65f289aaaf838 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:52.609 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.603+0000 7fa8b77fe700 1 -- 172.21.15.112:0/3396496423 <== mon.2 v2:172.21.15.142:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b000e070 con 0x7fa8b81586b0 2023-01-24T03:40:52.609 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.603+0000 7fa8b77fe700 1 -- 172.21.15.112:0/3396496423 <== mon.2 v2:172.21.15.142:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8b000bbf0 con 0x7fa8b81586b0 2023-01-24T03:40:52.610 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.603+0000 7fa8b77fe700 1 -- 172.21.15.112:0/3396496423 <== mon.2 v2:172.21.15.142:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b0012430 con 0x7fa8b81586b0 2023-01-24T03:40:52.610 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.604+0000 7fa8bd7a3700 1 -- 172.21.15.112:0/3396496423 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8b81586b0 msgr2=0x7fa8b814ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:52.610 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.604+0000 7fa8bd7a3700 1 --2- 172.21.15.112:0/3396496423 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8b81586b0 0x7fa8b814ed10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa8b0009e60 tx=0x7fa8b0009780 comp rx=0 tx=0).stop 2023-01-24T03:40:52.610 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.604+0000 7fa8bd7a3700 1 -- 172.21.15.112:0/3396496423 shutdown_connections 2023-01-24T03:40:52.610 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.604+0000 7fa8bd7a3700 1 --2- 172.21.15.112:0/3396496423 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8b81586b0 0x7fa8b814ed10 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:52.611 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.604+0000 7fa8bd7a3700 1 --2- 172.21.15.112:0/3396496423 >> v2:172.21.15.142:3300/0 conn(0x7fa8b80c52a0 0x7fa8b80c4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:52.611 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.604+0000 7fa8bd7a3700 1 -- 172.21.15.112:0/3396496423 >> 172.21.15.112:0/3396496423 conn(0x7fa8b8066e10 msgr2=0x7fa8b8067210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:52.611 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.604+0000 7fa8bd7a3700 1 -- 172.21.15.112:0/3396496423 shutdown_connections 2023-01-24T03:40:52.611 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.604+0000 7fa8bd7a3700 1 -- 172.21.15.112:0/3396496423 wait complete. 2023-01-24T03:40:52.614 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.605+0000 7fa8bd7a3700 1 Processor -- start 2023-01-24T03:40:52.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.605+0000 7fa8bd7a3700 1 -- start start 2023-01-24T03:40:52.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.605+0000 7fa8bd7a3700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa8b80c3ec0 0x7fa8b8165840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:52.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.605+0000 7fa8bd7a3700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa8b80c52a0 0x7fa8b806de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:52.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.605+0000 7fa8bd7a3700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8b81586b0 0x7fa8b8168480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:52.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.605+0000 7fa8bd7a3700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b80bd320 con 0x7fa8b80c52a0 2023-01-24T03:40:52.616 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.605+0000 7fa8bd7a3700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b8156240 con 0x7fa8b80c3ec0 2023-01-24T03:40:52.616 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.605+0000 7fa8bd7a3700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b81653b0 con 0x7fa8b81586b0 2023-01-24T03:40:52.616 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.606+0000 7fa8affff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa8b80c52a0 0x7fa8b806de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:52.616 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.606+0000 7fa8affff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa8b80c52a0 0x7fa8b806de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55962/0 (socket says 172.21.15.112:55962) 2023-01-24T03:40:52.616 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.606+0000 7fa8affff700 1 -- 172.21.15.112:0/1601927478 learned_addr learned my addr 172.21.15.112:0/1601927478 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:40:52.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.606+0000 7fa8b7fff700 1 --2- 172.21.15.112:0/1601927478 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa8b80c3ec0 0x7fa8b8165840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:52.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.606+0000 7fa8bcfa2700 1 --2- 172.21.15.112:0/1601927478 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8b81586b0 0x7fa8b8168480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:52.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.606+0000 7fa8affff700 1 -- 172.21.15.112:0/1601927478 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8b81586b0 msgr2=0x7fa8b8168480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:52.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.606+0000 7fa8affff700 1 --2- 172.21.15.112:0/1601927478 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8b81586b0 0x7fa8b8168480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:52.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.606+0000 7fa8affff700 1 -- 172.21.15.112:0/1601927478 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa8b80c3ec0 msgr2=0x7fa8b8165840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:52.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.606+0000 7fa8affff700 1 --2- 172.21.15.112:0/1601927478 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa8b80c3ec0 0x7fa8b8165840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:52.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.606+0000 7fa8affff700 1 -- 172.21.15.112:0/1601927478 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a0009370 con 0x7fa8b80c52a0 2023-01-24T03:40:52.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.606+0000 7fa8affff700 1 --2- 172.21.15.112:0/1601927478 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa8b80c52a0 0x7fa8b806de50 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fa8a000ea50 tx=0x7fa8a00090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:52.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.606+0000 7fa8b4ff9700 1 -- 172.21.15.112:0/1601927478 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a000ee90 con 0x7fa8b80c52a0 2023-01-24T03:40:52.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.606+0000 7fa8b4ff9700 1 -- 172.21.15.112:0/1601927478 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8a000c850 con 0x7fa8b80c52a0 2023-01-24T03:40:52.619 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.606+0000 7fa8bd7a3700 1 -- 172.21.15.112:0/1601927478 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8b0009a20 con 0x7fa8b80c52a0 2023-01-24T03:40:52.619 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.606+0000 7fa8b4ff9700 1 -- 172.21.15.112:0/1601927478 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a000cc10 con 0x7fa8b80c52a0 2023-01-24T03:40:52.619 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.606+0000 7fa8bd7a3700 1 -- 172.21.15.112:0/1601927478 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8b80b6c20 con 0x7fa8b80c52a0 2023-01-24T03:40:52.619 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.607+0000 7fa8bd7a3700 1 -- 172.21.15.112:0/1601927478 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8b8165520 con 0x7fa8b80c52a0 2023-01-24T03:40:52.619 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.608+0000 7fa8b4ff9700 1 -- 172.21.15.112:0/1601927478 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fa8a0019030 con 0x7fa8b80c52a0 2023-01-24T03:40:52.620 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.608+0000 7fa8b4ff9700 1 --2- 172.21.15.112:0/1601927478 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa89c0611f0 0x7fa89c0636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:52.620 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.608+0000 7fa8b4ff9700 1 -- 172.21.15.112:0/1601927478 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 7469+0+0 (secure 0 0 0) 0x7fa8a007d7f0 con 0x7fa8b80c52a0 2023-01-24T03:40:52.620 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.608+0000 7fa8b7fff700 1 --2- 172.21.15.112:0/1601927478 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa89c0611f0 0x7fa89c0636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:52.620 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.609+0000 7fa8b7fff700 1 --2- 172.21.15.112:0/1601927478 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa89c0611f0 0x7fa89c0636a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa8a8007fd0 tx=0x7fa8a8000bc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:52.620 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.612+0000 7fa8b4ff9700 1 -- 172.21.15.112:0/1601927478 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fa8a004d3a0 con 0x7fa8b80c52a0 2023-01-24T03:40:52.836 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.834+0000 7fa8bd7a3700 1 -- 172.21.15.112:0/1601927478 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8b805ea20 con 0x7fa8b80c52a0 2023-01-24T03:40:52.837 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.834+0000 7fa8b4ff9700 1 -- 172.21.15.112:0/1601927478 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+5367 (secure 0 0 0) 0x7fa8a00509c0 con 0x7fa8b80c52a0 2023-01-24T03:40:52.837 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:40:52.837 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:40:50.861448+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5186},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5186":{"gid":5186,"name":"b","rank":0,"incarnation":42,"state":"up:clientreplay","state_seq":15,"addr":"172.21.15.142:6834/2804746591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2804746591},{"type":"v1","addr":"172.21.15.142:6834","nonce":2804746591}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:40:52.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.836+0000 7fa8bd7a3700 1 -- 172.21.15.112:0/1601927478 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa89c0611f0 msgr2=0x7fa89c0636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:52.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.836+0000 7fa8bd7a3700 1 --2- 172.21.15.112:0/1601927478 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa89c0611f0 0x7fa89c0636a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa8a8007fd0 tx=0x7fa8a8000bc0 comp rx=0 tx=0).stop 2023-01-24T03:40:52.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.836+0000 7fa8bd7a3700 1 -- 172.21.15.112:0/1601927478 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa8b80c52a0 msgr2=0x7fa8b806de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:52.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.836+0000 7fa8bd7a3700 1 --2- 172.21.15.112:0/1601927478 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa8b80c52a0 0x7fa8b806de50 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fa8a000ea50 tx=0x7fa8a00090f0 comp rx=0 tx=0).stop 2023-01-24T03:40:52.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.836+0000 7fa8bd7a3700 1 -- 172.21.15.112:0/1601927478 shutdown_connections 2023-01-24T03:40:52.840 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.836+0000 7fa8bd7a3700 1 --2- 172.21.15.112:0/1601927478 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fa89c0611f0 0x7fa89c0636a0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:52.840 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.836+0000 7fa8bd7a3700 1 --2- 172.21.15.112:0/1601927478 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fa8b81586b0 0x7fa8b8168480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:52.840 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.836+0000 7fa8bd7a3700 1 --2- 172.21.15.112:0/1601927478 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa8b80c3ec0 0x7fa8b8165840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:52.840 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.836+0000 7fa8bd7a3700 1 --2- 172.21.15.112:0/1601927478 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa8b80c52a0 0x7fa8b806de50 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:52.840 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.836+0000 7fa8bd7a3700 1 -- 172.21.15.112:0/1601927478 >> 172.21.15.112:0/1601927478 conn(0x7fa8b8066e10 msgr2=0x7fa8b806d070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:52.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.836+0000 7fa8bd7a3700 1 -- 172.21.15.112:0/1601927478 shutdown_connections 2023-01-24T03:40:52.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:52.836+0000 7fa8bd7a3700 1 -- 172.21.15.112:0/1601927478 wait complete. 2023-01-24T03:40:52.841 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 45 2023-01-24T03:40:52.852 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 45, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-01-24T03:30:20.971710+0000", "damaged": [], "data_pools": [ 4, 5 ], "enabled": true, "epoch": 45, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5186": { "addr": "172.21.15.142:6834/2804746591", "addrs": { "addrvec": [ { "addr": "172.21.15.142:6832", "nonce": 2804746591, "type": "v2" }, { "addr": "172.21.15.142:6834", "nonce": 2804746591, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 5186, "incarnation": 42, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:clientreplay", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 49, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-01-24T03:40:50.861448+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5186 } } } ], "standbys": [ { "addr": "172.21.15.142:6837/2181345869", "addrs": { "addrvec": [ { "addr": "172.21.15.142:6836", "nonce": 2181345869, "type": "v2" }, { "addr": "172.21.15.142:6837", "nonce": 2181345869, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4253, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6835/330343927", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6834", "nonce": 330343927, "type": "v2" }, { "addr": "172.21.15.112:6835", "nonce": 330343927, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4266, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6837/12176817", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6836", "nonce": 12176817, "type": "v2" }, { "addr": "172.21.15.112:6837", "nonce": 12176817, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 2, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4276, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.112:6839/1110983841", "addrs": { "addrvec": [ { "addr": "172.21.15.112:6838", "nonce": 1110983841, "type": "v2" }, { "addr": "172.21.15.112:6839", "nonce": 1110983841, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138320759226367, "flags": 0, "gid": 4281, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-01-24T03:40:54.853 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:40:54.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.976+0000 7f5b29e95700 1 Processor -- start 2023-01-24T03:40:54.978 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.976+0000 7f5b29e95700 1 -- start start 2023-01-24T03:40:54.980 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.976+0000 7f5b29e95700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f5b240c7e40 0x7f5b240b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:54.980 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.976+0000 7f5b29e95700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f5b24155da0 0x7f5b240ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:54.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.976+0000 7f5b29e95700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b24156240 con 0x7f5b240aef20 2023-01-24T03:40:54.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.976+0000 7f5b29e95700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b240af320 con 0x7f5b240c7e40 2023-01-24T03:40:54.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.976+0000 7f5b29e95700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b240b1170 con 0x7f5b24155da0 2023-01-24T03:40:54.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.977+0000 7f5b237fe700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f5b240c7e40 0x7f5b240b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:54.981 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.977+0000 7f5b237fe700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f5b240c7e40 0x7f5b240b0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55968/0 (socket says 172.21.15.112:55968) 2023-01-24T03:40:54.982 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.977+0000 7f5b237fe700 1 -- 172.21.15.112:0/1873229647 learned_addr learned my addr 172.21.15.112:0/1873229647 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:40:54.982 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.977+0000 7f5b22ffd700 1 --2- 172.21.15.112:0/1873229647 >> v2:172.21.15.142:3300/0 conn(0x7f5b24155da0 0x7f5b240ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:54.982 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.977+0000 7f5b237fe700 1 -- 172.21.15.112:0/1873229647 >> v2:172.21.15.142:3300/0 conn(0x7f5b24155da0 msgr2=0x7f5b240ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:40:54.982 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.977+0000 7f5b237fe700 1 --2- 172.21.15.112:0/1873229647 >> v2:172.21.15.142:3300/0 conn(0x7f5b24155da0 0x7f5b240ae9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:54.982 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.977+0000 7f5b237fe700 1 -- 172.21.15.112:0/1873229647 >> v1:172.21.15.142:6789/0 conn(0x7f5b240aef20 legacy=0x7f5b240ca3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:40:54.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.977+0000 7f5b237fe700 1 -- 172.21.15.112:0/1873229647 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b240b1450 con 0x7f5b240c7e40 2023-01-24T03:40:54.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.977+0000 7f5b237fe700 1 --2- 172.21.15.112:0/1873229647 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5b240c7e40 0x7f5b240b0aa0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f5b24152ea0 tx=0x7f5b1c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cca04b7aacf8e34b server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:54.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.977+0000 7f5b227fc700 1 -- 172.21.15.112:0/1873229647 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c011630 con 0x7f5b240c7e40 2023-01-24T03:40:54.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.977+0000 7f5b227fc700 1 -- 172.21.15.112:0/1873229647 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1c0117d0 con 0x7f5b240c7e40 2023-01-24T03:40:54.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.977+0000 7f5b227fc700 1 -- 172.21.15.112:0/1873229647 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c011b90 con 0x7f5b240c7e40 2023-01-24T03:40:54.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.978+0000 7f5b29e95700 1 -- 172.21.15.112:0/1873229647 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5b240c7e40 msgr2=0x7f5b240b0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:54.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.978+0000 7f5b29e95700 1 --2- 172.21.15.112:0/1873229647 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5b240c7e40 0x7f5b240b0aa0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f5b24152ea0 tx=0x7f5b1c0090f0 comp rx=0 tx=0).stop 2023-01-24T03:40:54.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.978+0000 7f5b29e95700 1 -- 172.21.15.112:0/1873229647 shutdown_connections 2023-01-24T03:40:54.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.978+0000 7f5b29e95700 1 --2- 172.21.15.112:0/1873229647 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5b240c7e40 0x7f5b240b0aa0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:54.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.978+0000 7f5b29e95700 1 --2- 172.21.15.112:0/1873229647 >> v2:172.21.15.142:3300/0 conn(0x7f5b24155da0 0x7f5b240ae9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:54.985 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.978+0000 7f5b29e95700 1 -- 172.21.15.112:0/1873229647 >> 172.21.15.112:0/1873229647 conn(0x7f5b2405e2c0 msgr2=0x7f5b240cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:54.985 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.978+0000 7f5b29e95700 1 -- 172.21.15.112:0/1873229647 shutdown_connections 2023-01-24T03:40:54.985 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.978+0000 7f5b29e95700 1 -- 172.21.15.112:0/1873229647 wait complete. 2023-01-24T03:40:54.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.979+0000 7f5b29e95700 1 Processor -- start 2023-01-24T03:40:54.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.979+0000 7f5b29e95700 1 -- start start 2023-01-24T03:40:54.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.980+0000 7f5b29e95700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b240aef20 0x7f5b24165fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:54.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.980+0000 7f5b29e95700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b240c7e40 0x7f5b2414f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:54.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.980+0000 7f5b29e95700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5b24155da0 0x7f5b24150310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:54.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.980+0000 7f5b29e95700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b24166510 con 0x7f5b24155da0 2023-01-24T03:40:54.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.980+0000 7f5b29e95700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b241522a0 con 0x7f5b240c7e40 2023-01-24T03:40:54.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.980+0000 7f5b29e95700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b24152580 con 0x7f5b240aef20 2023-01-24T03:40:54.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.980+0000 7f5b23fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5b24155da0 0x7f5b24150310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:54.990 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.980+0000 7f5b23fff700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5b24155da0 0x7f5b24150310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55978/0 (socket says 172.21.15.112:55978) 2023-01-24T03:40:54.990 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.980+0000 7f5b23fff700 1 -- 172.21.15.112:0/3077540412 learned_addr learned my addr 172.21.15.112:0/3077540412 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:40:54.990 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.980+0000 7f5b22ffd700 1 --2- 172.21.15.112:0/3077540412 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b240c7e40 0x7f5b2414f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:54.990 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.980+0000 7f5b237fe700 1 --2- 172.21.15.112:0/3077540412 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b240aef20 0x7f5b24165fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:54.990 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.980+0000 7f5b23fff700 1 -- 172.21.15.112:0/3077540412 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b240aef20 msgr2=0x7f5b24165fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:54.991 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.980+0000 7f5b23fff700 1 --2- 172.21.15.112:0/3077540412 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b240aef20 0x7f5b24165fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:54.991 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.980+0000 7f5b23fff700 1 -- 172.21.15.112:0/3077540412 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b240c7e40 msgr2=0x7f5b2414f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:54.991 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.980+0000 7f5b23fff700 1 --2- 172.21.15.112:0/3077540412 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b240c7e40 0x7f5b2414f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:54.991 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.980+0000 7f5b23fff700 1 -- 172.21.15.112:0/3077540412 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b1c009370 con 0x7f5b24155da0 2023-01-24T03:40:54.991 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.980+0000 7f5b23fff700 1 --2- 172.21.15.112:0/3077540412 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5b24155da0 0x7f5b24150310 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f5b1400c440 tx=0x7f5b14007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:54.992 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.980+0000 7f5b0b7fe700 1 -- 172.21.15.112:0/3077540412 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b14011590 con 0x7f5b24155da0 2023-01-24T03:40:54.992 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.980+0000 7f5b0b7fe700 1 -- 172.21.15.112:0/3077540412 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b14011710 con 0x7f5b24155da0 2023-01-24T03:40:54.992 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.980+0000 7f5b29e95700 1 -- 172.21.15.112:0/3077540412 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b24152860 con 0x7f5b24155da0 2023-01-24T03:40:54.992 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.980+0000 7f5b0b7fe700 1 -- 172.21.15.112:0/3077540412 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b14011ad0 con 0x7f5b24155da0 2023-01-24T03:40:54.992 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.981+0000 7f5b29e95700 1 -- 172.21.15.112:0/3077540412 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b2414fdf0 con 0x7f5b24155da0 2023-01-24T03:40:54.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.981+0000 7f5b29e95700 1 -- 172.21.15.112:0/3077540412 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b24150010 con 0x7f5b24155da0 2023-01-24T03:40:54.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.982+0000 7f5b0b7fe700 1 -- 172.21.15.112:0/3077540412 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f5b14019030 con 0x7f5b24155da0 2023-01-24T03:40:54.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.982+0000 7f5b0b7fe700 1 --2- 172.21.15.112:0/3077540412 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5b040611c0 0x7f5b04063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:54.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.982+0000 7f5b0b7fe700 1 -- 172.21.15.112:0/3077540412 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 7469+0+0 (secure 0 0 0) 0x7f5b1407ccc0 con 0x7f5b24155da0 2023-01-24T03:40:54.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.982+0000 7f5b237fe700 1 --2- 172.21.15.112:0/3077540412 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5b040611c0 0x7f5b04063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:54.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.982+0000 7f5b237fe700 1 --2- 172.21.15.112:0/3077540412 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5b040611c0 0x7f5b04063670 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c009340 tx=0x7f5b1c016000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:54.994 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:54.986+0000 7f5b0b7fe700 1 -- 172.21.15.112:0/3077540412 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f5b1404c870 con 0x7f5b24155da0 2023-01-24T03:40:55.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:55.208+0000 7f5b29e95700 1 -- 172.21.15.112:0/3077540412 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b240c40c0 con 0x7f5b24155da0 2023-01-24T03:40:55.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:55.209+0000 7f5b0b7fe700 1 -- 172.21.15.112:0/3077540412 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+5367 (secure 0 0 0) 0x7f5b14017090 con 0x7f5b24155da0 2023-01-24T03:40:55.211 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:40:55.212 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:40:50.861448+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5186},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5186":{"gid":5186,"name":"b","rank":0,"incarnation":42,"state":"up:clientreplay","state_seq":15,"addr":"172.21.15.142:6834/2804746591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2804746591},{"type":"v1","addr":"172.21.15.142:6834","nonce":2804746591}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:40:55.213 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:55.211+0000 7f5b29e95700 1 -- 172.21.15.112:0/3077540412 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5b040611c0 msgr2=0x7f5b04063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:55.213 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:55.211+0000 7f5b29e95700 1 --2- 172.21.15.112:0/3077540412 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5b040611c0 0x7f5b04063670 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c009340 tx=0x7f5b1c016000 comp rx=0 tx=0).stop 2023-01-24T03:40:55.214 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:55.211+0000 7f5b29e95700 1 -- 172.21.15.112:0/3077540412 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5b24155da0 msgr2=0x7f5b24150310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:55.214 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:55.211+0000 7f5b29e95700 1 --2- 172.21.15.112:0/3077540412 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5b24155da0 0x7f5b24150310 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f5b1400c440 tx=0x7f5b14007dd0 comp rx=0 tx=0).stop 2023-01-24T03:40:55.214 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:55.211+0000 7f5b29e95700 1 -- 172.21.15.112:0/3077540412 shutdown_connections 2023-01-24T03:40:55.214 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:55.211+0000 7f5b29e95700 1 --2- 172.21.15.112:0/3077540412 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f5b040611c0 0x7f5b04063670 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:55.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:55.211+0000 7f5b29e95700 1 --2- 172.21.15.112:0/3077540412 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5b24155da0 0x7f5b24150310 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:55.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:55.211+0000 7f5b29e95700 1 --2- 172.21.15.112:0/3077540412 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f5b240aef20 0x7f5b24165fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:55.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:55.211+0000 7f5b29e95700 1 --2- 172.21.15.112:0/3077540412 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b240c7e40 0x7f5b2414f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:55.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:55.211+0000 7f5b29e95700 1 -- 172.21.15.112:0/3077540412 >> 172.21.15.112:0/3077540412 conn(0x7f5b2405e2c0 msgr2=0x7f5b24156700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:55.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:55.211+0000 7f5b29e95700 1 -- 172.21.15.112:0/3077540412 shutdown_connections 2023-01-24T03:40:55.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:55.211+0000 7f5b29e95700 1 -- 172.21.15.112:0/3077540412 wait complete. 2023-01-24T03:40:55.216 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 45 2023-01-24T03:40:55.225 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:40:57.226 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:40:57.351 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.349+0000 7fce70597700 1 Processor -- start 2023-01-24T03:40:57.352 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.349+0000 7fce70597700 1 -- start start 2023-01-24T03:40:57.354 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.350+0000 7fce70597700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fce680c7e40 0x7fce680b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:57.354 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.350+0000 7fce70597700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7fce680ca3e0 0x7fce680ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:57.354 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.350+0000 7fce70597700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce68156240 con 0x7fce68155da0 2023-01-24T03:40:57.355 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.350+0000 7fce70597700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce680b1170 con 0x7fce680c7e40 2023-01-24T03:40:57.355 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.350+0000 7fce70597700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce680b1450 con 0x7fce680ca3e0 2023-01-24T03:40:57.355 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.350+0000 7fce6e333700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fce680c7e40 0x7fce680b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:57.355 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.350+0000 7fce6db32700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fce68155da0 0x7fce680ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:36982/0 (socket says 172.21.15.112:36982) 2023-01-24T03:40:57.355 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.350+0000 7fce6db32700 1 -- 172.21.15.112:0/473223656 learned_addr learned my addr 172.21.15.112:0/473223656 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:40:57.356 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.350+0000 7fce6eb34700 1 --2- 172.21.15.112:0/473223656 >> v2:172.21.15.142:3300/0 conn(0x7fce680ca3e0 0x7fce680ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:57.356 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.351+0000 7fce6d331700 1 -- 172.21.15.112:0/473223656 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1000039844 0 0) 0x7fce68156240 con 0x7fce68155da0 2023-01-24T03:40:57.356 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.351+0000 7fce6d331700 1 -- 172.21.15.112:0/473223656 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce540033f0 con 0x7fce68155da0 2023-01-24T03:40:57.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.351+0000 7fce6e333700 1 -- 172.21.15.112:0/473223656 >> v2:172.21.15.142:3300/0 conn(0x7fce680ca3e0 msgr2=0x7fce680ca7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:40:57.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.351+0000 7fce6e333700 1 --2- 172.21.15.112:0/473223656 >> v2:172.21.15.142:3300/0 conn(0x7fce680ca3e0 0x7fce680ca7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:57.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.351+0000 7fce6e333700 1 -- 172.21.15.112:0/473223656 >> v1:172.21.15.112:6789/0 conn(0x7fce68155da0 legacy=0x7fce680ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:57.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.351+0000 7fce6e333700 1 -- 172.21.15.112:0/473223656 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce680b1730 con 0x7fce680c7e40 2023-01-24T03:40:57.357 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.351+0000 7fce6e333700 1 --2- 172.21.15.112:0/473223656 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fce680c7e40 0x7fce680b0aa0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fce6805e810 tx=0x7fce58009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82f1536d5775365d server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:57.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.351+0000 7fce6d331700 1 -- 172.21.15.112:0/473223656 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5800e070 con 0x7fce680c7e40 2023-01-24T03:40:57.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.351+0000 7fce6d331700 1 -- 172.21.15.112:0/473223656 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce5800bab0 con 0x7fce680c7e40 2023-01-24T03:40:57.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.351+0000 7fce6d331700 1 -- 172.21.15.112:0/473223656 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5800be50 con 0x7fce680c7e40 2023-01-24T03:40:57.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.351+0000 7fce70597700 1 -- 172.21.15.112:0/473223656 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fce680c7e40 msgr2=0x7fce680b0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:57.358 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.351+0000 7fce70597700 1 --2- 172.21.15.112:0/473223656 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fce680c7e40 0x7fce680b0aa0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fce6805e810 tx=0x7fce58009780 comp rx=0 tx=0).stop 2023-01-24T03:40:57.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.352+0000 7fce70597700 1 -- 172.21.15.112:0/473223656 shutdown_connections 2023-01-24T03:40:57.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.352+0000 7fce70597700 1 --2- 172.21.15.112:0/473223656 >> v2:172.21.15.142:3300/0 conn(0x7fce680ca3e0 0x7fce680ca7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:57.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.352+0000 7fce70597700 1 --2- 172.21.15.112:0/473223656 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fce680c7e40 0x7fce680b0aa0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:57.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.352+0000 7fce70597700 1 -- 172.21.15.112:0/473223656 >> 172.21.15.112:0/473223656 conn(0x7fce6805e2c0 msgr2=0x7fce680cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:57.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.352+0000 7fce70597700 1 -- 172.21.15.112:0/473223656 shutdown_connections 2023-01-24T03:40:57.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.352+0000 7fce70597700 1 -- 172.21.15.112:0/473223656 wait complete. 2023-01-24T03:40:57.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.353+0000 7fce70597700 1 Processor -- start 2023-01-24T03:40:57.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.353+0000 7fce70597700 1 -- start start 2023-01-24T03:40:57.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.353+0000 7fce70597700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fce680c7e40 0x7fce68168700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:57.363 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.353+0000 7fce70597700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce680ca3e0 0x7fce680bfa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:57.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.353+0000 7fce70597700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce68155da0 0x7fce6814bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:57.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.353+0000 7fce70597700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce680c1830 con 0x7fce680c7e40 2023-01-24T03:40:57.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.353+0000 7fce70597700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce681554a0 con 0x7fce68155da0 2023-01-24T03:40:57.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.353+0000 7fce70597700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce68155740 con 0x7fce680ca3e0 2023-01-24T03:40:57.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.353+0000 7fce6e333700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fce680c7e40 0x7fce68168700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:57.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.353+0000 7fce6e333700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fce680c7e40 0x7fce68168700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:55996/0 (socket says 172.21.15.112:55996) 2023-01-24T03:40:57.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.353+0000 7fce6e333700 1 -- 172.21.15.112:0/3711091695 learned_addr learned my addr 172.21.15.112:0/3711091695 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:40:57.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.353+0000 7fce6db32700 1 --2- 172.21.15.112:0/3711091695 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce680ca3e0 0x7fce680bfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:57.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.353+0000 7fce6eb34700 1 --2- 172.21.15.112:0/3711091695 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce68155da0 0x7fce6814bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:57.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.354+0000 7fce6e333700 1 -- 172.21.15.112:0/3711091695 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce680ca3e0 msgr2=0x7fce680bfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:57.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.354+0000 7fce6e333700 1 --2- 172.21.15.112:0/3711091695 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce680ca3e0 0x7fce680bfa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:57.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.354+0000 7fce6e333700 1 -- 172.21.15.112:0/3711091695 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce68155da0 msgr2=0x7fce6814bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:57.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.354+0000 7fce6e333700 1 --2- 172.21.15.112:0/3711091695 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce68155da0 0x7fce6814bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:57.366 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.354+0000 7fce6e333700 1 -- 172.21.15.112:0/3711091695 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce58009a20 con 0x7fce680c7e40 2023-01-24T03:40:57.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.354+0000 7fce6eb34700 1 --2- 172.21.15.112:0/3711091695 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce68155da0 0x7fce6814bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:40:57.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.354+0000 7fce6e333700 1 --2- 172.21.15.112:0/3711091695 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fce680c7e40 0x7fce68168700 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fce58009fd0 tx=0x7fce58019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:57.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.354+0000 7fce667fc700 1 -- 172.21.15.112:0/3711091695 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce58012a20 con 0x7fce680c7e40 2023-01-24T03:40:57.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.354+0000 7fce667fc700 1 -- 172.21.15.112:0/3711091695 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce58021070 con 0x7fce680c7e40 2023-01-24T03:40:57.367 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.354+0000 7fce70597700 1 -- 172.21.15.112:0/3711091695 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce680c5530 con 0x7fce680c7e40 2023-01-24T03:40:57.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.354+0000 7fce667fc700 1 -- 172.21.15.112:0/3711091695 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce58012df0 con 0x7fce680c7e40 2023-01-24T03:40:57.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.354+0000 7fce70597700 1 -- 172.21.15.112:0/3711091695 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce680c1aa0 con 0x7fce680c7e40 2023-01-24T03:40:57.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.355+0000 7fce70597700 1 -- 172.21.15.112:0/3711091695 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce6804bb80 con 0x7fce680c7e40 2023-01-24T03:40:57.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.355+0000 7fce667fc700 1 -- 172.21.15.112:0/3711091695 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fce5802c020 con 0x7fce680c7e40 2023-01-24T03:40:57.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.355+0000 7fce667fc700 1 --2- 172.21.15.112:0/3711091695 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fce480611c0 0x7fce48063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:57.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.356+0000 7fce667fc700 1 -- 172.21.15.112:0/3711091695 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 7469+0+0 (secure 0 0 0) 0x7fce5807fba0 con 0x7fce680c7e40 2023-01-24T03:40:57.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.356+0000 7fce6db32700 1 --2- 172.21.15.112:0/3711091695 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fce480611c0 0x7fce48063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:57.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.356+0000 7fce6db32700 1 --2- 172.21.15.112:0/3711091695 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fce480611c0 0x7fce48063670 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fce68068200 tx=0x7fce5c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:57.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.360+0000 7fce667fc700 1 -- 172.21.15.112:0/3711091695 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fce5804f750 con 0x7fce680c7e40 2023-01-24T03:40:57.594 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.593+0000 7fce70597700 1 -- 172.21.15.112:0/3711091695 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce6806efe0 con 0x7fce680c7e40 2023-01-24T03:40:57.595 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.593+0000 7fce667fc700 1 -- 172.21.15.112:0/3711091695 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+5367 (secure 0 0 0) 0x7fce58052d70 con 0x7fce680c7e40 2023-01-24T03:40:57.595 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:40:57.596 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:40:50.861448+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5186},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5186":{"gid":5186,"name":"b","rank":0,"incarnation":42,"state":"up:clientreplay","state_seq":15,"addr":"172.21.15.142:6834/2804746591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2804746591},{"type":"v1","addr":"172.21.15.142:6834","nonce":2804746591}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:40:57.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.595+0000 7fce70597700 1 -- 172.21.15.112:0/3711091695 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fce480611c0 msgr2=0x7fce48063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:57.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.595+0000 7fce70597700 1 --2- 172.21.15.112:0/3711091695 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fce480611c0 0x7fce48063670 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fce68068200 tx=0x7fce5c009040 comp rx=0 tx=0).stop 2023-01-24T03:40:57.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.595+0000 7fce70597700 1 -- 172.21.15.112:0/3711091695 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fce680c7e40 msgr2=0x7fce68168700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:57.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.595+0000 7fce70597700 1 --2- 172.21.15.112:0/3711091695 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fce680c7e40 0x7fce68168700 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fce58009fd0 tx=0x7fce58019000 comp rx=0 tx=0).stop 2023-01-24T03:40:57.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.595+0000 7fce70597700 1 -- 172.21.15.112:0/3711091695 shutdown_connections 2023-01-24T03:40:57.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.595+0000 7fce70597700 1 --2- 172.21.15.112:0/3711091695 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fce480611c0 0x7fce48063670 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:57.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.595+0000 7fce70597700 1 --2- 172.21.15.112:0/3711091695 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fce680c7e40 0x7fce68168700 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:57.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.595+0000 7fce70597700 1 --2- 172.21.15.112:0/3711091695 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fce68155da0 0x7fce6814bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:57.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.595+0000 7fce70597700 1 --2- 172.21.15.112:0/3711091695 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fce680ca3e0 0x7fce680bfa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:57.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.595+0000 7fce70597700 1 -- 172.21.15.112:0/3711091695 >> 172.21.15.112:0/3711091695 conn(0x7fce6805e2c0 msgr2=0x7fce680683f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:57.600 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.595+0000 7fce70597700 1 -- 172.21.15.112:0/3711091695 shutdown_connections 2023-01-24T03:40:57.600 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:57.595+0000 7fce70597700 1 -- 172.21.15.112:0/3711091695 wait complete. 2023-01-24T03:40:57.600 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 45 2023-01-24T03:40:57.609 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-01-24T03:40:59.610 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:40:59.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.724+0000 7f43f4e0e700 1 Processor -- start 2023-01-24T03:40:59.726 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.724+0000 7f43f4e0e700 1 -- start start 2023-01-24T03:40:59.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.725+0000 7f43f4e0e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f43f00b0aa0 0x7f43f0155da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:59.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.725+0000 7f43f4e0e700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f43f00ca3e0 0x7f43f00ca840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:59.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.725+0000 7f43f4e0e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43f00b0f40 con 0x7f43f00c7e40 2023-01-24T03:40:59.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.725+0000 7f43f4e0e700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43f00cad80 con 0x7f43f00ca3e0 2023-01-24T03:40:59.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.725+0000 7f43f4e0e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43f00b1170 con 0x7f43f00b0aa0 2023-01-24T03:40:59.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.725+0000 7f43ee59c700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7f43f00c7e40 0x7f43f00ae9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:36996/0 (socket says 172.21.15.112:36996) 2023-01-24T03:40:59.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.725+0000 7f43ee59c700 1 -- 172.21.15.112:0/2187661170 learned_addr learned my addr 172.21.15.112:0/2187661170 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:40:59.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.725+0000 7f43eed9d700 1 --2- 172.21.15.112:0/2187661170 >> v2:172.21.15.142:3300/0 conn(0x7f43f00ca3e0 0x7f43f00ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:59.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.725+0000 7f43edd9b700 1 --2- 172.21.15.112:0/2187661170 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f43f00b0aa0 0x7f43f0155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:59.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.725+0000 7f43ed59a700 1 -- 172.21.15.112:0/2187661170 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3840608302 0 0) 0x7f43f00b0f40 con 0x7f43f00c7e40 2023-01-24T03:40:59.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.725+0000 7f43ed59a700 1 -- 172.21.15.112:0/2187661170 --> v1:172.21.15.112:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43d00033f0 con 0x7f43f00c7e40 2023-01-24T03:40:59.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.726+0000 7f43ed59a700 1 -- 172.21.15.112:0/2187661170 <== mon.0 v1:172.21.15.112:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4090308692 0 0) 0x7f43d00033f0 con 0x7f43f00c7e40 2023-01-24T03:40:59.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.726+0000 7f43ed59a700 1 -- 172.21.15.112:0/2187661170 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f43f00b0aa0 msgr2=0x7f43f0155da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:40:59.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.726+0000 7f43ed59a700 1 --2- 172.21.15.112:0/2187661170 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f43f00b0aa0 0x7f43f0155da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:59.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.726+0000 7f43ed59a700 1 -- 172.21.15.112:0/2187661170 >> v2:172.21.15.142:3300/0 conn(0x7f43f00ca3e0 msgr2=0x7f43f00ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:40:59.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.726+0000 7f43ed59a700 1 --2- 172.21.15.112:0/2187661170 >> v2:172.21.15.142:3300/0 conn(0x7f43f00ca3e0 0x7f43f00ca840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:59.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.726+0000 7f43ed59a700 1 -- 172.21.15.112:0/2187661170 --> v1:172.21.15.112:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f00b1450 con 0x7f43f00c7e40 2023-01-24T03:40:59.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.726+0000 7f43ed59a700 1 -- 172.21.15.112:0/2187661170 <== mon.0 v1:172.21.15.112:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 315957160 0 0) 0x7f43e00017f0 con 0x7f43f00c7e40 2023-01-24T03:40:59.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.726+0000 7f43ed59a700 1 -- 172.21.15.112:0/2187661170 >> v1:172.21.15.112:6789/0 conn(0x7f43f00c7e40 legacy=0x7f43f00ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:59.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.726+0000 7f43ed59a700 1 --2- 172.21.15.112:0/2187661170 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f43f00b0aa0 0x7f43d00038e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:59.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.726+0000 7f43ed59a700 1 --2- 172.21.15.112:0/2187661170 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43d0003e60 0x7f43d0004690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:59.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.726+0000 7f43ed59a700 1 --2- 172.21.15.112:0/2187661170 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f43d0004bd0 0x7f43d0005040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:59.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.726+0000 7f43ed59a700 1 -- 172.21.15.112:0/2187661170 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f00cad80 con 0x7f43d0004bd0 2023-01-24T03:40:59.734 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.726+0000 7f43ed59a700 1 -- 172.21.15.112:0/2187661170 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f00b1170 con 0x7f43d0003e60 2023-01-24T03:40:59.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.726+0000 7f43ed59a700 1 -- 172.21.15.112:0/2187661170 --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43d000b910 con 0x7f43f00b0aa0 2023-01-24T03:40:59.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.726+0000 7f43eed9d700 1 --2- 172.21.15.112:0/2187661170 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f43d0004bd0 0x7f43d0005040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:59.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.726+0000 7f43ee59c700 1 --2- 172.21.15.112:0/2187661170 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f43f00b0aa0 0x7f43d00038e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:59.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.726+0000 7f43edd9b700 1 --2- 172.21.15.112:0/2187661170 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43d0003e60 0x7f43d0004690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:59.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.726+0000 7f43eed9d700 1 -- 172.21.15.112:0/2187661170 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f43f00b0aa0 msgr2=0x7f43d00038e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:40:59.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.726+0000 7f43eed9d700 1 --2- 172.21.15.112:0/2187661170 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f43f00b0aa0 0x7f43d00038e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:59.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.726+0000 7f43eed9d700 1 -- 172.21.15.112:0/2187661170 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43d0003e60 msgr2=0x7f43d0004690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:40:59.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.726+0000 7f43eed9d700 1 --2- 172.21.15.112:0/2187661170 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43d0003e60 0x7f43d0004690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:59.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.726+0000 7f43eed9d700 1 -- 172.21.15.112:0/2187661170 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d00033f0 con 0x7f43d0004bd0 2023-01-24T03:40:59.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.726+0000 7f43eed9d700 1 --2- 172.21.15.112:0/2187661170 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f43d0004bd0 0x7f43d0005040 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f43e8009a80 tx=0x7f43e8009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5290ac1ae5e4ec18 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:59.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.726+0000 7f43ed59a700 1 -- 172.21.15.112:0/2187661170 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e8009430 con 0x7f43d0004bd0 2023-01-24T03:40:59.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.726+0000 7f43ed59a700 1 -- 172.21.15.112:0/2187661170 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43e800e5e0 con 0x7f43d0004bd0 2023-01-24T03:40:59.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.726+0000 7f43ed59a700 1 -- 172.21.15.112:0/2187661170 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e800e9a0 con 0x7f43d0004bd0 2023-01-24T03:40:59.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.727+0000 7f43f4e0e700 1 -- 172.21.15.112:0/2187661170 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f43d0004bd0 msgr2=0x7f43d0005040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:59.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.727+0000 7f43f4e0e700 1 --2- 172.21.15.112:0/2187661170 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f43d0004bd0 0x7f43d0005040 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f43e8009a80 tx=0x7f43e8009060 comp rx=0 tx=0).stop 2023-01-24T03:40:59.737 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.727+0000 7f43eed9d700 1 -- 172.21.15.112:0/2187661170 reap_dead start 2023-01-24T03:40:59.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.727+0000 7f43f4e0e700 1 -- 172.21.15.112:0/2187661170 shutdown_connections 2023-01-24T03:40:59.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.727+0000 7f43f4e0e700 1 -- 172.21.15.112:0/2187661170 >> 172.21.15.112:0/2187661170 conn(0x7f43f005e2c0 msgr2=0x7f43f00cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:59.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.727+0000 7f43f4e0e700 1 -- 172.21.15.112:0/2187661170 shutdown_connections 2023-01-24T03:40:59.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.727+0000 7f43f4e0e700 1 -- 172.21.15.112:0/2187661170 wait complete. 2023-01-24T03:40:59.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.728+0000 7f43f4e0e700 1 Processor -- start 2023-01-24T03:40:59.738 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.728+0000 7f43f4e0e700 1 -- start start 2023-01-24T03:40:59.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.729+0000 7f43f4e0e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f43d0004bd0 0x7f43f00c4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:59.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.729+0000 7f43f4e0e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f43f0155190 0x7f43f015aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:59.741 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.729+0000 7f43f4e0e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43f00c4cb0 0x7f43f015afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:59.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.729+0000 7f43f4e0e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f015b8a0 con 0x7f43f0155190 2023-01-24T03:40:59.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.729+0000 7f43f4e0e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f0165400 con 0x7f43f00c4cb0 2023-01-24T03:40:59.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.729+0000 7f43f4e0e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43f0166180 con 0x7f43d0004bd0 2023-01-24T03:40:59.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.729+0000 7f43edd9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f43f0155190 0x7f43f015aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:59.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.729+0000 7f43edd9b700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f43f0155190 0x7f43f015aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:56022/0 (socket says 172.21.15.112:56022) 2023-01-24T03:40:59.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.729+0000 7f43edd9b700 1 -- 172.21.15.112:0/1022272163 learned_addr learned my addr 172.21.15.112:0/1022272163 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:40:59.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.729+0000 7f43ee59c700 1 --2- 172.21.15.112:0/1022272163 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f43d0004bd0 0x7f43f00c4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:59.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.729+0000 7f43eed9d700 1 --2- 172.21.15.112:0/1022272163 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43f00c4cb0 0x7f43f015afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:59.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.729+0000 7f43edd9b700 1 -- 172.21.15.112:0/1022272163 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f43d0004bd0 msgr2=0x7f43f00c4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:59.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.729+0000 7f43edd9b700 1 --2- 172.21.15.112:0/1022272163 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f43d0004bd0 0x7f43f00c4770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:59.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.729+0000 7f43edd9b700 1 -- 172.21.15.112:0/1022272163 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43f00c4cb0 msgr2=0x7f43f015afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:59.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.729+0000 7f43edd9b700 1 --2- 172.21.15.112:0/1022272163 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43f00c4cb0 0x7f43f015afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:59.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.729+0000 7f43edd9b700 1 -- 172.21.15.112:0/1022272163 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f0169360 con 0x7f43f0155190 2023-01-24T03:40:59.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.729+0000 7f43edd9b700 1 --2- 172.21.15.112:0/1022272163 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f43f0155190 0x7f43f015aaa0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f43d80119c0 tx=0x7f43d8009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:59.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.729+0000 7f43e67fc700 1 -- 172.21.15.112:0/1022272163 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d800f790 con 0x7f43f0155190 2023-01-24T03:40:59.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.729+0000 7f43e67fc700 1 -- 172.21.15.112:0/1022272163 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43d800f8f0 con 0x7f43f0155190 2023-01-24T03:40:59.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.729+0000 7f43f4e0e700 1 -- 172.21.15.112:0/1022272163 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43f01656a0 con 0x7f43f0155190 2023-01-24T03:40:59.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.729+0000 7f43e67fc700 1 -- 172.21.15.112:0/1022272163 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d800fc90 con 0x7f43f0155190 2023-01-24T03:40:59.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.729+0000 7f43f4e0e700 1 -- 172.21.15.112:0/1022272163 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43f0165c60 con 0x7f43f0155190 2023-01-24T03:40:59.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.730+0000 7f43f4e0e700 1 -- 172.21.15.112:0/1022272163 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43f004bb80 con 0x7f43f0155190 2023-01-24T03:40:59.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.731+0000 7f43e67fc700 1 -- 172.21.15.112:0/1022272163 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f43d8014650 con 0x7f43f0155190 2023-01-24T03:40:59.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.731+0000 7f43e67fc700 1 --2- 172.21.15.112:0/1022272163 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f43d40611c0 0x7f43d4063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:40:59.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.731+0000 7f43e67fc700 1 -- 172.21.15.112:0/1022272163 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 7469+0+0 (secure 0 0 0) 0x7f43d8018030 con 0x7f43f0155190 2023-01-24T03:40:59.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.731+0000 7f43ee59c700 1 --2- 172.21.15.112:0/1022272163 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f43d40611c0 0x7f43d4063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:40:59.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.731+0000 7f43ee59c700 1 --2- 172.21.15.112:0/1022272163 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f43d40611c0 0x7f43d4063670 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f43e0004a90 tx=0x7f43e0001d90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:40:59.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.735+0000 7f43e67fc700 1 -- 172.21.15.112:0/1022272163 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f43d804e400 con 0x7f43f0155190 2023-01-24T03:40:59.959 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.957+0000 7f43f4e0e700 1 -- 172.21.15.112:0/1022272163 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43f00c5df0 con 0x7f43f0155190 2023-01-24T03:40:59.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.958+0000 7f43e67fc700 1 -- 172.21.15.112:0/1022272163 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+5361 (secure 0 0 0) 0x7f43d8051a20 con 0x7f43f0155190 2023-01-24T03:40:59.960 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:40:59.960 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:40:59.897812+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5186},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5186":{"gid":5186,"name":"b","rank":0,"incarnation":42,"state":"up:active","state_seq":18,"addr":"172.21.15.142:6834/2804746591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2804746591},{"type":"v1","addr":"172.21.15.142:6834","nonce":2804746591}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:40:59.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.960+0000 7f43f4e0e700 1 -- 172.21.15.112:0/1022272163 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f43d40611c0 msgr2=0x7f43d4063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:59.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.960+0000 7f43f4e0e700 1 --2- 172.21.15.112:0/1022272163 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f43d40611c0 0x7f43d4063670 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f43e0004a90 tx=0x7f43e0001d90 comp rx=0 tx=0).stop 2023-01-24T03:40:59.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.960+0000 7f43f4e0e700 1 -- 172.21.15.112:0/1022272163 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f43f0155190 msgr2=0x7f43f015aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:40:59.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.960+0000 7f43f4e0e700 1 --2- 172.21.15.112:0/1022272163 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f43f0155190 0x7f43f015aaa0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f43d80119c0 tx=0x7f43d8009060 comp rx=0 tx=0).stop 2023-01-24T03:40:59.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.960+0000 7f43f4e0e700 1 -- 172.21.15.112:0/1022272163 shutdown_connections 2023-01-24T03:40:59.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.960+0000 7f43f4e0e700 1 --2- 172.21.15.112:0/1022272163 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f43d40611c0 0x7f43d4063670 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:59.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.960+0000 7f43f4e0e700 1 --2- 172.21.15.112:0/1022272163 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43f00c4cb0 0x7f43f015afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:59.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.960+0000 7f43f4e0e700 1 --2- 172.21.15.112:0/1022272163 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f43d0004bd0 0x7f43f00c4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:59.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.960+0000 7f43f4e0e700 1 --2- 172.21.15.112:0/1022272163 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f43f0155190 0x7f43f015aaa0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:40:59.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.960+0000 7f43f4e0e700 1 -- 172.21.15.112:0/1022272163 >> 172.21.15.112:0/1022272163 conn(0x7f43f005e2c0 msgr2=0x7f43f014eda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:40:59.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.960+0000 7f43f4e0e700 1 -- 172.21.15.112:0/1022272163 shutdown_connections 2023-01-24T03:40:59.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:40:59.960+0000 7f43f4e0e700 1 -- 172.21.15.112:0/1022272163 wait complete. 2023-01-24T03:40:59.965 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 46 2023-01-24T03:40:59.973 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=5078 2023-01-24T03:40:59.974 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.d 2023-01-24T03:41:00.431 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:41:00.434 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:41:05.939 INFO:tasks.ceph.mds.b.smithi142.stderr:2023-01-24T03:41:05.937+0000 7f97b0dd9700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-01-24T03:41:05.990 INFO:tasks.workunit.client.0.smithi112.stderr:Updating files: 2% (257/11669) Updating files: 3% (351/11669) Updating files: 4% (467/11669) Updating files: 4% (517/11669) Updating files: 4% (536/11669) Updating files: 5% (584/11669) Updating files: 5% (692/11669) Updating files: 6% (701/11669) Updating files: 6% (727/11669) Updating files: 7% (817/11669) Updating files: 8% (934/11669) Updating files: 8% (1049/11669) Updating files: 9% (1051/11669) Updating files: 9% (1089/11669) Updating files: 9% (1164/11669) Updating files: 10% (1167/11669) Updating files: 11% (1284/11669) Updating files: 11% (1327/11669) Updating files: 12% (1401/11669) Updating files: 12% (1478/11669) Updating files: 13% (1517/11669) Updating files: 14% (1634/11669) Updating files: 14% (1699/11669) Updating files: 15% (1751/11669) Updating files: 15% (1785/11669) Updating files: 16% (1868/11669) Updating files: 16% (1934/11669) Updating files: 17% (1984/11669) Updating files: 17% (2050/11669) Updating files: 17% (2099/11669) Updating files: 18% (2101/11669) Updating files: 19% (2218/11669) Updating files: 19% (2318/11669) Updating files: 19% (2323/11669) Updating files: 20% (2334/11669) Updating files: 20% (2397/11669) Updating files: 21% (2451/11669) Updating files: 22% (2568/11669) Updating files: 22% (2596/11669) Updating files: 23% (2684/11669) Updating files: 23% (2799/11669) Updating files: 24% (2801/11669) Updating files: 24% (2804/11669) Updating files: 24% (2852/11669) Updating files: 25% (2918/11669) Updating files: 25% (3006/11669) Updating files: 26% (3034/11669) Updating files: 26% (3121/11669) Updating files: 27% (3151/11669) Updating files: 27% (3170/11669) Updating files: 27% (3206/11669) Updating files: 28% (3268/11669) Updating files: 29% (3385/11669) Updating files: 29% (3415/11669) Updating files: 30% (3501/11669) Updating files: 31% (3618/11669) Updating files: 32% (3735/11669) Updating files: 32% (3800/11669) Updating files: 33% (3851/11669) Updating files: 33% (3960/11669) Updating files: 34% (3968/11669) Updating files: 34% (4016/11669) Updating files: 35% (4085/11669) Updating files: 35% (4146/11669) Updating files: 36% (4201/11669) Updating files: 37% (4318/11669) Updating files: 37% (4433/11669) Updating files: 38% (4435/11669) Updating files: 38% (4514/11669) Updating files: 39% (4551/11669) Updating files: 39% (4606/11669) Updating files: 39% (4608/11669) Updating files: 40% (4668/11669) Updating files: 41% (4785/11669) Updating files: 41% (4899/11669) Updating files: 42% (4901/11669) Updating files: 43% (5018/11669) Updating files: 43% (5026/11669) Updating files: 44% (5135/11669) Updating files: 44% (5171/11669) Updating files: 44% (5185/11669) Updating files: 44% (5188/11669) Updating files: 45% (5252/11669) Updating files: 46% (5368/11669) Updating files: 47% (5485/11669) Updating files: 47% (5515/11669) Updating files: 47% (5563/11669) Updating files: 47% (5582/11669) Updating files: 48% (5602/11669) Updating files: 49% (5718/11669) Updating files: 50% (5835/11669) Updating files: 50% (5836/11669) Updating files: 51% (5952/11669) Updating files: 51% (5971/11669) Updating files: 51% (6013/11669) Updating files: 52% (6068/11669) Updating files: 53% (6185/11669) Updating files: 54% (6302/11669) Updating files: 55% (6418/11669) Updating files: 56% (6535/11669) Updating files: 56% (6568/11669) Updating files: 57% (6652/11669) Updating files: 58% (6769/11669) Updating files: 59% (6885/11669) Updating files: 59% (6933/11669) Updating files: 60% (7002/11669) Updating files: 61% (7119/11669) Updating files: 61% (7122/11669) Updating files: 61% (7187/11669) Updating files: 62% (7235/11669) Updating files: 62% (7260/11669) Updating files: 63% (7352/11669) Updating files: 64% (7469/11669) Updating files: 64% (7475/11669) Updating files: 64% (7557/11669) Updating files: 65% (7585/11669) Updating files: 66% (7702/11669) Updating files: 66% (7806/11669) Updating files: 67% (7819/11669) Updating files: 67% (7901/11669) Updating files: 68% (7935/11669) Updating files: 68% (8027/11669) Updating files: 69% (8052/11669) Updating files: 69% (8092/11669) Updating files: 70% (8169/11669) Updating files: 70% (8219/11669) Updating files: 71% (8285/11669) Updating files: 72% (8402/11669) Updating files: 72% (8477/11669) Updating files: 73% (8519/11669) Updating files: 74% (8636/11669) Updating files: 75% (8752/11669) Updating files: 75% (8813/11669) Updating files: 76% (8869/11669) Updating files: 76% (8887/11669) Updating files: 77% (8986/11669) Updating files: 77% (9041/11669) Updating files: 78% (9102/11669) Updating files: 78% (9136/11669) Updating files: 79% (9219/11669) Updating files: 79% (9250/11669) Updating files: 79% (9318/11669) Updating files: 80% (9336/11669) Updating files: 80% (9346/11669) Updating files: 80% (9385/11669) Updating files: 80% (9408/11669) Updating files: 81% (9452/11669) Updating files: 81% (9559/11669) Updating files: 82% (9569/11669) Updating files: 83% (9686/11669) Updating files: 84% (9802/11669) Updating files: 84% (9809/11669) Updating files: 85% (9919/11669) Updating files: 85% (10004/11669) Updating files: 86% (10036/11669) Updating files: 86% (10152/11669) Updating files: 87% (10153/11669) Updating files: 87% (10168/11669) Updating files: 87% (10242/11669) Updating files: 88% (10269/11669) Updating files: 88% (10278/11669) Updating files: 89% (10386/11669) Updating files: 89% (10475/11669) Updating files: 90% (10503/11669) Updating files: 90% (10599/11669) Updating files: 91% (10619/11669) Updating files: 91% (10671/11669) Updating files: 92% (10736/11669) Updating files: 93% (10853/11669) Updating files: 93% (10896/11669) Updating files: 94% (10969/11669) Updating files: 94% (11039/11669) Updating files: 95% (11086/11669) Updating files: 96% (11203/11669) Updating files: 97% (11319/11669) Updating files: 97% (11350/11669) Updating files: 98% (11436/11669) Updating files: 99% (11553/11669) Updating files: 99% (11639/11669) Updating files: 100% (11669/11669) Updating files: 100% (11669/11669), done. 2023-01-24T03:41:06.369 DEBUG:teuthology.orchestra.run:got remote process result: 128 2023-01-24T03:41:06.370 INFO:tasks.workunit:Stopping ['fs/snaps'] on client.0... 2023-01-24T03:41:06.371 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-01-24T03:41:06.610 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/run_tasks.py", line 103, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/run_tasks.py", line 82, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_8387d427d91a301da19bd82e090b96e1348bfadb/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_8387d427d91a301da19bd82e090b96e1348bfadb/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_8387d427d91a301da19bd82e090b96e1348bfadb/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi112 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8387d427d91a301da19bd82e090b96e1348bfadb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2023-01-24T03:41:06.739 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=ad7cdddef5c34704ad3f0970b2210445 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/run_tasks.py", line 103, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/run_tasks.py", line 82, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_8387d427d91a301da19bd82e090b96e1348bfadb/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_8387d427d91a301da19bd82e090b96e1348bfadb/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_8387d427d91a301da19bd82e090b96e1348bfadb/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi112 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8387d427d91a301da19bd82e090b96e1348bfadb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2023-01-24T03:41:06.742 DEBUG:teuthology.run_tasks:Unwinding manager mds_thrash 2023-01-24T03:41:06.753 INFO:tasks.mds_thrash:joining mds_thrasher 2023-01-24T03:41:06.975 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2023-01-24T03:41:06.976 INFO:tasks.ceph.mds.d:Restarting daemon 2023-01-24T03:41:06.976 DEBUG:teuthology.orchestra.run.smithi142:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-01-24T03:41:06.979 INFO:tasks.ceph.mds.d:Started 2023-01-24T03:41:06.980 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:41:07.061 INFO:tasks.ceph.mds.d.smithi142.stdout:starting mds.d at 2023-01-24T03:41:07.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.092+0000 7f45fee7c700 1 Processor -- start 2023-01-24T03:41:07.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.092+0000 7f45fee7c700 1 -- start start 2023-01-24T03:41:07.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.093+0000 7f45fee7c700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f45f80c5680 0x7f45f8156730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:41:07.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.093+0000 7f45fee7c700 1 --2- >> v2:172.21.15.142:3300/0 conn(0x7f45f80c5a50 0x7f45f8156c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:41:07.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.093+0000 7f45fee7c700 1 -- --> v1:172.21.15.142:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45f80c3d10 con 0x7f45f8155e20 2023-01-24T03:41:07.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.093+0000 7f45fee7c700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45f80c4eb0 con 0x7f45f80c5680 2023-01-24T03:41:07.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.093+0000 7f45fee7c700 1 -- --> v2:172.21.15.142:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45f80c2e20 con 0x7f45f80c5a50 2023-01-24T03:41:07.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.093+0000 7f45fde7a700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f45f80c5680 0x7f45f8156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:41:07.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.093+0000 7f45fde7a700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7f45f80c5680 0x7f45f8156730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:56042/0 (socket says 172.21.15.112:56042) 2023-01-24T03:41:07.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.093+0000 7f45fde7a700 1 -- 172.21.15.112:0/3758092283 learned_addr learned my addr 172.21.15.112:0/3758092283 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:41:07.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.093+0000 7f45fe67b700 1 --2- 172.21.15.112:0/3758092283 >> v2:172.21.15.142:3300/0 conn(0x7f45f80c5a50 0x7f45f8156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:41:07.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.093+0000 7f45fde7a700 1 -- 172.21.15.112:0/3758092283 >> v2:172.21.15.142:3300/0 conn(0x7f45f80c5a50 msgr2=0x7f45f8156c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:41:07.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.093+0000 7f45fde7a700 1 --2- 172.21.15.112:0/3758092283 >> v2:172.21.15.142:3300/0 conn(0x7f45f80c5a50 0x7f45f8156c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:41:07.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.093+0000 7f45fde7a700 1 -- 172.21.15.112:0/3758092283 >> v1:172.21.15.142:6789/0 conn(0x7f45f8155e20 legacy=0x7f45f80cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:41:07.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.093+0000 7f45fde7a700 1 -- 172.21.15.112:0/3758092283 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f80c3100 con 0x7f45f80c5680 2023-01-24T03:41:07.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.093+0000 7f45fde7a700 1 --2- 172.21.15.112:0/3758092283 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f45f80c5680 0x7f45f8156730 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f45f80894c0 tx=0x7f45f0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f19c00e8aba476d server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:41:07.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.093+0000 7f45fce78700 1 -- 172.21.15.112:0/3758092283 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45f000e070 con 0x7f45f80c5680 2023-01-24T03:41:07.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.093+0000 7f45fce78700 1 -- 172.21.15.112:0/3758092283 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45f000bae0 con 0x7f45f80c5680 2023-01-24T03:41:07.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.093+0000 7f45fce78700 1 -- 172.21.15.112:0/3758092283 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45f00092e0 con 0x7f45f80c5680 2023-01-24T03:41:07.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.094+0000 7f45fee7c700 1 -- 172.21.15.112:0/3758092283 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f45f80c5680 msgr2=0x7f45f8156730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:41:07.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.094+0000 7f45fee7c700 1 --2- 172.21.15.112:0/3758092283 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f45f80c5680 0x7f45f8156730 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f45f80894c0 tx=0x7f45f0009840 comp rx=0 tx=0).stop 2023-01-24T03:41:07.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.094+0000 7f45fee7c700 1 -- 172.21.15.112:0/3758092283 shutdown_connections 2023-01-24T03:41:07.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.094+0000 7f45fee7c700 1 --2- 172.21.15.112:0/3758092283 >> v2:172.21.15.142:3300/0 conn(0x7f45f80c5a50 0x7f45f8156c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:41:07.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.094+0000 7f45fee7c700 1 --2- 172.21.15.112:0/3758092283 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f45f80c5680 0x7f45f8156730 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:41:07.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.094+0000 7f45fee7c700 1 -- 172.21.15.112:0/3758092283 >> 172.21.15.112:0/3758092283 conn(0x7f45f80685b0 msgr2=0x7f45f80689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:41:07.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.094+0000 7f45fee7c700 1 -- 172.21.15.112:0/3758092283 shutdown_connections 2023-01-24T03:41:07.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.094+0000 7f45fee7c700 1 -- 172.21.15.112:0/3758092283 wait complete. 2023-01-24T03:41:07.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.095+0000 7f45fee7c700 1 Processor -- start 2023-01-24T03:41:07.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.095+0000 7f45fee7c700 1 -- start start 2023-01-24T03:41:07.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.096+0000 7f45fee7c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f45f80c5680 0x7f45f8166160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:41:07.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.096+0000 7f45fee7c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f45f80c5a50 0x7f45f81516b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:41:07.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.096+0000 7f45fee7c700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f45f8155e20 0x7f45f80c8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:41:07.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.096+0000 7f45fee7c700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f80ca0e0 con 0x7f45f80c5a50 2023-01-24T03:41:07.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.096+0000 7f45fee7c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f8151bf0 con 0x7f45f80c5680 2023-01-24T03:41:07.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.096+0000 7f45fee7c700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45f80cb130 con 0x7f45f8155e20 2023-01-24T03:41:07.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.096+0000 7f45fd679700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f45f80c5a50 0x7f45f81516b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:41:07.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.096+0000 7f45fd679700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f45f80c5a50 0x7f45f81516b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:56046/0 (socket says 172.21.15.112:56046) 2023-01-24T03:41:07.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.096+0000 7f45fd679700 1 -- 172.21.15.112:0/2171585146 learned_addr learned my addr 172.21.15.112:0/2171585146 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:41:07.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.096+0000 7f45fde7a700 1 --2- 172.21.15.112:0/2171585146 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f45f80c5680 0x7f45f8166160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:41:07.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.096+0000 7f45fe67b700 1 --2- 172.21.15.112:0/2171585146 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f45f8155e20 0x7f45f80c8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:41:07.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.096+0000 7f45fd679700 1 -- 172.21.15.112:0/2171585146 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f45f8155e20 msgr2=0x7f45f80c8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:41:07.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.096+0000 7f45fd679700 1 --2- 172.21.15.112:0/2171585146 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f45f8155e20 0x7f45f80c8400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:41:07.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.096+0000 7f45fd679700 1 -- 172.21.15.112:0/2171585146 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f45f80c5680 msgr2=0x7f45f8166160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:41:07.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.096+0000 7f45fd679700 1 --2- 172.21.15.112:0/2171585146 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f45f80c5680 0x7f45f8166160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:41:07.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.096+0000 7f45fd679700 1 -- 172.21.15.112:0/2171585146 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f0009ae0 con 0x7f45f80c5a50 2023-01-24T03:41:07.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.096+0000 7f45fd679700 1 --2- 172.21.15.112:0/2171585146 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f45f80c5a50 0x7f45f81516b0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f45e800c470 tx=0x7f45e8007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:41:07.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.096+0000 7f45e5ffb700 1 -- 172.21.15.112:0/2171585146 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e8012420 con 0x7f45f80c5a50 2023-01-24T03:41:07.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.096+0000 7f45e5ffb700 1 -- 172.21.15.112:0/2171585146 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45e80125a0 con 0x7f45f80c5a50 2023-01-24T03:41:07.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.096+0000 7f45fee7c700 1 -- 172.21.15.112:0/2171585146 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45f80cb410 con 0x7f45f80c5a50 2023-01-24T03:41:07.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.096+0000 7f45e5ffb700 1 -- 172.21.15.112:0/2171585146 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e8012960 con 0x7f45f80c5a50 2023-01-24T03:41:07.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.097+0000 7f45fee7c700 1 -- 172.21.15.112:0/2171585146 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45f80c9d70 con 0x7f45f80c5a50 2023-01-24T03:41:07.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.097+0000 7f45fee7c700 1 -- 172.21.15.112:0/2171585146 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45f81666a0 con 0x7f45f80c5a50 2023-01-24T03:41:07.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.098+0000 7f45e5ffb700 1 -- 172.21.15.112:0/2171585146 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7f45e8019030 con 0x7f45f80c5a50 2023-01-24T03:41:07.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.098+0000 7f45e5ffb700 1 --2- 172.21.15.112:0/2171585146 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f45d80611c0 0x7f45d8063670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:41:07.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.098+0000 7f45e5ffb700 1 -- 172.21.15.112:0/2171585146 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 7469+0+0 (secure 0 0 0) 0x7f45e807cf00 con 0x7f45f80c5a50 2023-01-24T03:41:07.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.098+0000 7f45fde7a700 1 --2- 172.21.15.112:0/2171585146 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f45d80611c0 0x7f45d8063670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:41:07.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.099+0000 7f45fde7a700 1 --2- 172.21.15.112:0/2171585146 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f45d80611c0 0x7f45d8063670 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f45f0009880 tx=0x7f45f0018000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:41:07.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.102+0000 7f45e5ffb700 1 -- 172.21.15.112:0/2171585146 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7f45e804c960 con 0x7f45f80c5a50 2023-01-24T03:41:07.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.323+0000 7f45fee7c700 1 -- 172.21.15.112:0/2171585146 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45f80c98f0 con 0x7f45f80c5a50 2023-01-24T03:41:07.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.324+0000 7f45e5ffb700 1 -- 172.21.15.112:0/2171585146 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+5361 (secure 0 0 0) 0x7f45f80c98f0 con 0x7f45f80c5a50 2023-01-24T03:41:07.326 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:41:07.327 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:40:59.897812+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5186},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5186":{"gid":5186,"name":"b","rank":0,"incarnation":42,"state":"up:active","state_seq":18,"addr":"172.21.15.142:6834/2804746591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2804746591},{"type":"v1","addr":"172.21.15.142:6834","nonce":2804746591}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:41:07.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.326+0000 7f45fee7c700 1 -- 172.21.15.112:0/2171585146 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f45d80611c0 msgr2=0x7f45d8063670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:41:07.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.326+0000 7f45fee7c700 1 --2- 172.21.15.112:0/2171585146 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f45d80611c0 0x7f45d8063670 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f45f0009880 tx=0x7f45f0018000 comp rx=0 tx=0).stop 2023-01-24T03:41:07.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.326+0000 7f45fee7c700 1 -- 172.21.15.112:0/2171585146 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f45f80c5a50 msgr2=0x7f45f81516b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:41:07.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.326+0000 7f45fee7c700 1 --2- 172.21.15.112:0/2171585146 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f45f80c5a50 0x7f45f81516b0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f45e800c470 tx=0x7f45e8007e00 comp rx=0 tx=0).stop 2023-01-24T03:41:07.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.326+0000 7f45fee7c700 1 -- 172.21.15.112:0/2171585146 shutdown_connections 2023-01-24T03:41:07.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.326+0000 7f45fee7c700 1 --2- 172.21.15.112:0/2171585146 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7f45d80611c0 0x7f45d8063670 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:41:07.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.326+0000 7f45fee7c700 1 --2- 172.21.15.112:0/2171585146 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7f45f8155e20 0x7f45f80c8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:41:07.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.326+0000 7f45fee7c700 1 --2- 172.21.15.112:0/2171585146 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f45f80c5680 0x7f45f8166160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:41:07.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.326+0000 7f45fee7c700 1 --2- 172.21.15.112:0/2171585146 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f45f80c5a50 0x7f45f81516b0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:41:07.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.326+0000 7f45fee7c700 1 -- 172.21.15.112:0/2171585146 >> 172.21.15.112:0/2171585146 conn(0x7f45f80685b0 msgr2=0x7f45f814bd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:41:07.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.326+0000 7f45fee7c700 1 -- 172.21.15.112:0/2171585146 shutdown_connections 2023-01-24T03:41:07.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:07.326+0000 7f45fee7c700 1 -- 172.21.15.112:0/2171585146 wait complete. 2023-01-24T03:41:07.332 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 46 2023-01-24T03:41:07.340 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2023-01-24T03:41:09.342 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-01-24T03:41:09.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.463+0000 7fdb5759e700 1 Processor -- start 2023-01-24T03:41:09.467 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.463+0000 7fdb5759e700 1 -- start start 2023-01-24T03:41:09.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.464+0000 7fdb5759e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb580c5680 0x7fdb58156730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:41:09.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.464+0000 7fdb5759e700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fdb58155e20 0x7fdb58156c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:41:09.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.464+0000 7fdb5759e700 1 -- --> v1:172.21.15.112:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb580c3d10 con 0x7fdb580c5a50 2023-01-24T03:41:09.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.464+0000 7fdb5759e700 1 -- --> v2:172.21.15.112:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb580c4eb0 con 0x7fdb58155e20 2023-01-24T03:41:09.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.464+0000 7fdb5759e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb580c2e20 con 0x7fdb580c5680 2023-01-24T03:41:09.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.464+0000 7fdb55d9b700 1 --2- >> v2:172.21.15.112:3300/0 conn(0x7fdb58155e20 0x7fdb58156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:41:09.474 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.464+0000 7fdb56d9d700 1 --1- >> v1:172.21.15.112:6789/0 conn(0x7fdb580c5a50 0x7fdb580cb870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.112:6789/0 says I am v1:172.21.15.112:37044/0 (socket says 172.21.15.112:37044) 2023-01-24T03:41:09.474 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.464+0000 7fdb56d9d700 1 -- 172.21.15.112:0/2846756567 learned_addr learned my addr 172.21.15.112:0/2846756567 (peer_addr_for_me v1:172.21.15.112:0/0) 2023-01-24T03:41:09.474 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.464+0000 7fdb5659c700 1 --2- 172.21.15.112:0/2846756567 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb580c5680 0x7fdb58156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:41:09.474 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.464+0000 7fdb55d9b700 1 -- 172.21.15.112:0/2846756567 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb580c5680 msgr2=0x7fdb58156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-01-24T03:41:09.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.464+0000 7fdb55d9b700 1 --2- 172.21.15.112:0/2846756567 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb580c5680 0x7fdb58156730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:41:09.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.464+0000 7fdb55d9b700 1 -- 172.21.15.112:0/2846756567 >> v1:172.21.15.112:6789/0 conn(0x7fdb580c5a50 legacy=0x7fdb580cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:41:09.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.464+0000 7fdb5559a700 1 -- 172.21.15.112:0/2846756567 <== mon.0 v1:172.21.15.112:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2423430174 0 0) 0x7fdb580c3d10 con 0x7fdb580c5a50 2023-01-24T03:41:09.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.464+0000 7fdb55d9b700 1 -- 172.21.15.112:0/2846756567 --> v2:172.21.15.112:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb580c3100 con 0x7fdb58155e20 2023-01-24T03:41:09.476 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.464+0000 7fdb5659c700 1 --2- 172.21.15.112:0/2846756567 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb580c5680 0x7fdb58156730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-01-24T03:41:09.476 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.465+0000 7fdb55d9b700 1 --2- 172.21.15.112:0/2846756567 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdb58155e20 0x7fdb58156c70 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fdb40009790 tx=0x7fdb40009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f77a9c15477858d server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:41:09.476 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.465+0000 7fdb5559a700 1 -- 172.21.15.112:0/2846756567 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb40011630 con 0x7fdb58155e20 2023-01-24T03:41:09.477 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.465+0000 7fdb5559a700 1 -- 172.21.15.112:0/2846756567 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb40011790 con 0x7fdb58155e20 2023-01-24T03:41:09.477 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.465+0000 7fdb5559a700 1 -- 172.21.15.112:0/2846756567 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb40011b50 con 0x7fdb58155e20 2023-01-24T03:41:09.478 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.465+0000 7fdb5759e700 1 -- 172.21.15.112:0/2846756567 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdb58155e20 msgr2=0x7fdb58156c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:41:09.478 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.465+0000 7fdb5759e700 1 --2- 172.21.15.112:0/2846756567 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdb58155e20 0x7fdb58156c70 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fdb40009790 tx=0x7fdb40009060 comp rx=0 tx=0).stop 2023-01-24T03:41:09.479 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.465+0000 7fdb5759e700 1 -- 172.21.15.112:0/2846756567 shutdown_connections 2023-01-24T03:41:09.479 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.465+0000 7fdb5759e700 1 --2- 172.21.15.112:0/2846756567 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb580c5680 0x7fdb58156730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:41:09.479 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.465+0000 7fdb5759e700 1 --2- 172.21.15.112:0/2846756567 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdb58155e20 0x7fdb58156c70 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:41:09.480 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.466+0000 7fdb5759e700 1 -- 172.21.15.112:0/2846756567 >> 172.21.15.112:0/2846756567 conn(0x7fdb580685b0 msgr2=0x7fdb580689b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:41:09.480 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.466+0000 7fdb5759e700 1 -- 172.21.15.112:0/2846756567 shutdown_connections 2023-01-24T03:41:09.480 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.466+0000 7fdb5759e700 1 -- 172.21.15.112:0/2846756567 wait complete. 2023-01-24T03:41:09.481 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.467+0000 7fdb5759e700 1 Processor -- start 2023-01-24T03:41:09.481 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.467+0000 7fdb5759e700 1 -- start start 2023-01-24T03:41:09.481 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.467+0000 7fdb5759e700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdb580c5680 0x7fdb58150de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:41:09.481 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.467+0000 7fdb5759e700 1 --2- >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb580c5a50 0x7fdb58152140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:41:09.482 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.467+0000 7fdb5759e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb58155e20 0x7fdb58152680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:41:09.482 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.467+0000 7fdb5759e700 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb58150950 con 0x7fdb580c5680 2023-01-24T03:41:09.482 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.467+0000 7fdb5759e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb581663c0 con 0x7fdb58155e20 2023-01-24T03:41:09.483 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.467+0000 7fdb5759e700 1 -- --> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb58152bc0 con 0x7fdb580c5a50 2023-01-24T03:41:09.483 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.467+0000 7fdb5659c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdb580c5680 0x7fdb58150de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:41:09.483 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.467+0000 7fdb5659c700 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdb580c5680 0x7fdb58150de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:56060/0 (socket says 172.21.15.112:56060) 2023-01-24T03:41:09.483 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.467+0000 7fdb5659c700 1 -- 172.21.15.112:0/3412217072 learned_addr learned my addr 172.21.15.112:0/3412217072 (peer_addr_for_me v2:172.21.15.112:0/0) 2023-01-24T03:41:09.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.468+0000 7fdb56d9d700 1 --2- 172.21.15.112:0/3412217072 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb58155e20 0x7fdb58152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:41:09.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.468+0000 7fdb5659c700 1 -- 172.21.15.112:0/3412217072 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb580c5a50 msgr2=0x7fdb58152140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:41:09.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.468+0000 7fdb5659c700 1 --2- 172.21.15.112:0/3412217072 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb580c5a50 0x7fdb58152140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:41:09.485 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.468+0000 7fdb5659c700 1 -- 172.21.15.112:0/3412217072 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb58155e20 msgr2=0x7fdb58152680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:41:09.485 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.468+0000 7fdb5659c700 1 --2- 172.21.15.112:0/3412217072 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb58155e20 0x7fdb58152680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:41:09.485 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.468+0000 7fdb5659c700 1 -- 172.21.15.112:0/3412217072 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb400092a0 con 0x7fdb580c5680 2023-01-24T03:41:09.486 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.468+0000 7fdb5659c700 1 --2- 172.21.15.112:0/3412217072 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdb580c5680 0x7fdb58150de0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fdb4800d1b0 tx=0x7fdb480099f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:41:09.486 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.468+0000 7fdb4e7fc700 1 -- 172.21.15.112:0/3412217072 <== mon.0 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4800dee0 con 0x7fdb580c5680 2023-01-24T03:41:09.486 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.468+0000 7fdb5759e700 1 -- 172.21.15.112:0/3412217072 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb5814fd40 con 0x7fdb580c5680 2023-01-24T03:41:09.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.468+0000 7fdb5759e700 1 -- 172.21.15.112:0/3412217072 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb58150300 con 0x7fdb580c5680 2023-01-24T03:41:09.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.468+0000 7fdb4e7fc700 1 -- 172.21.15.112:0/3412217072 <== mon.0 v2:172.21.15.112:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb4800f040 con 0x7fdb580c5680 2023-01-24T03:41:09.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.468+0000 7fdb4e7fc700 1 -- 172.21.15.112:0/3412217072 <== mon.0 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb48015650 con 0x7fdb580c5680 2023-01-24T03:41:09.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.469+0000 7fdb5759e700 1 -- 172.21.15.112:0/3412217072 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb58150c30 con 0x7fdb580c5680 2023-01-24T03:41:09.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.469+0000 7fdb4e7fc700 1 -- 172.21.15.112:0/3412217072 <== mon.0 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 5) v1 ==== 79892+0+0 (secure 0 0 0) 0x7fdb4801c030 con 0x7fdb580c5680 2023-01-24T03:41:09.489 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.470+0000 7fdb4e7fc700 1 --2- 172.21.15.112:0/3412217072 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fdb3c0611f0 0x7fdb3c0636a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-01-24T03:41:09.489 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.470+0000 7fdb4e7fc700 1 -- 172.21.15.112:0/3412217072 <== mon.0 v2:172.21.15.112:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 7469+0+0 (secure 0 0 0) 0x7fdb4807de70 con 0x7fdb580c5680 2023-01-24T03:41:09.490 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.470+0000 7fdb55d9b700 1 --2- 172.21.15.112:0/3412217072 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fdb3c0611f0 0x7fdb3c0636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-01-24T03:41:09.490 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.470+0000 7fdb55d9b700 1 --2- 172.21.15.112:0/3412217072 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fdb3c0611f0 0x7fdb3c0636a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fdb40009790 tx=0x7fdb40017000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-01-24T03:41:09.490 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.474+0000 7fdb4e7fc700 1 -- 172.21.15.112:0/3412217072 <== mon.0 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+168377 (secure 0 0 0) 0x7fdb4804dad0 con 0x7fdb580c5680 2023-01-24T03:41:09.695 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.692+0000 7fdb5759e700 1 -- 172.21.15.112:0/3412217072 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb5805ea20 con 0x7fdb580c5680 2023-01-24T03:41:09.696 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.693+0000 7fdb4e7fc700 1 -- 172.21.15.112:0/3412217072 <== mon.0 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+6118 (secure 0 0 0) 0x7fdb480510f0 con 0x7fdb580c5680 2023-01-24T03:41:09.696 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-01-24T03:41:09.697 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/2181345869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":2181345869},{"type":"v1","addr":"172.21.15.142:6837","nonce":2181345869}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/330343927","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":330343927},{"type":"v1","addr":"172.21.15.112:6835","nonce":330343927}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4276,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/12176817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":12176817},{"type":"v1","addr":"172.21.15.112:6837","nonce":12176817}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4281,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/1110983841","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":1110983841},{"type":"v1","addr":"172.21.15.112:6839","nonce":1110983841}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5689,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/637421608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6833","nonce":637421608},{"type":"v1","addr":"172.21.15.142:6835","nonce":637421608}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-01-24T03:30:20.971710+0000","modified":"2023-01-24T03:40:59.897812+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5186},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5186":{"gid":5186,"name":"b","rank":0,"incarnation":42,"state":"up:active","state_seq":18,"addr":"172.21.15.142:6834/2804746591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":2804746591},{"type":"v1","addr":"172.21.15.142:6834","nonce":2804746591}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4,5],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-01-24T03:41:09.698 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.694+0000 7fdb5759e700 1 -- 172.21.15.112:0/3412217072 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fdb3c0611f0 msgr2=0x7fdb3c0636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:41:09.698 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.694+0000 7fdb5759e700 1 --2- 172.21.15.112:0/3412217072 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fdb3c0611f0 0x7fdb3c0636a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fdb40009790 tx=0x7fdb40017000 comp rx=0 tx=0).stop 2023-01-24T03:41:09.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.694+0000 7fdb5759e700 1 -- 172.21.15.112:0/3412217072 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdb580c5680 msgr2=0x7fdb58150de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-01-24T03:41:09.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.694+0000 7fdb5759e700 1 --2- 172.21.15.112:0/3412217072 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdb580c5680 0x7fdb58150de0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fdb4800d1b0 tx=0x7fdb480099f0 comp rx=0 tx=0).stop 2023-01-24T03:41:09.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.694+0000 7fdb5759e700 1 -- 172.21.15.112:0/3412217072 shutdown_connections 2023-01-24T03:41:09.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.694+0000 7fdb5759e700 1 --2- 172.21.15.112:0/3412217072 >> [v2:172.21.15.112:6832/84276,v1:172.21.15.112:6833/84276] conn(0x7fdb3c0611f0 0x7fdb3c0636a0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:41:09.700 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.694+0000 7fdb5759e700 1 --2- 172.21.15.112:0/3412217072 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdb580c5680 0x7fdb58150de0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:41:09.700 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.694+0000 7fdb5759e700 1 --2- 172.21.15.112:0/3412217072 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdb58155e20 0x7fdb58152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:41:09.700 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.694+0000 7fdb5759e700 1 --2- 172.21.15.112:0/3412217072 >> [v2:172.21.15.142:3301/0,v1:172.21.15.142:6790/0] conn(0x7fdb580c5a50 0x7fdb58152140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-01-24T03:41:09.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.694+0000 7fdb5759e700 1 -- 172.21.15.112:0/3412217072 >> 172.21.15.112:0/3412217072 conn(0x7fdb580685b0 msgr2=0x7fdb5814db90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-01-24T03:41:09.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.695+0000 7fdb5759e700 1 -- 172.21.15.112:0/3412217072 shutdown_connections 2023-01-24T03:41:09.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-01-24T03:41:09.695+0000 7fdb5759e700 1 -- 172.21.15.112:0/3412217072 wait complete. 2023-01-24T03:41:09.702 INFO:teuthology.orchestra.run.smithi112.stderr:dumped fsmap epoch 47 2023-01-24T03:41:09.709 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2023-01-24T03:41:09.710 INFO:tasks.mds_thrash.fs.[cephfs]:stat['max_mds'] = 0 2023-01-24T03:41:09.710 INFO:tasks.mds_thrash.fs.[cephfs]:stat['deactivate'] = 0 2023-01-24T03:41:09.710 INFO:tasks.mds_thrash.fs.[cephfs]:stat['kill'] = 7 2023-01-24T03:41:09.711 INFO:tasks.mds_thrash:done joining 2023-01-24T03:41:09.711 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-01-24T03:41:09.722 INFO:tasks.kclient:Unmounting kernel clients... 2023-01-24T03:41:09.722 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-01-24T03:41:09.723 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-01-24T03:41:09.723 DEBUG:teuthology.orchestra.run.smithi112:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-01-24T03:41:30.517 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:41:30.520 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:42:00.605 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:42:00.608 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:42:30.672 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:42:30.676 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:43:00.740 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:43:00.743 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:43:30.840 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:43:30.843 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:44:00.907 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:44:00.910 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:44:30.989 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:44:30.992 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:45:01.060 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:45:01.072 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:45:31.141 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:45:31.144 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:46:01.208 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:46:01.211 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:46:31.293 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:46:31.296 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:47:01.381 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:47:01.384 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:47:31.461 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:47:31.463 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:48:01.524 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:48:01.527 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:48:31.587 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:48:31.590 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:49:01.652 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:49:01.655 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:49:31.715 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:49:31.718 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:50:01.815 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:50:01.818 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:50:31.909 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:50:31.912 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:51:01.967 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:51:01.970 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:51:32.040 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:51:32.043 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:52:02.154 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:52:02.190 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:52:32.259 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:52:32.263 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:53:02.333 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:53:02.337 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:53:32.427 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:53:32.429 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:54:02.492 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:54:02.495 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:54:32.563 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:54:32.566 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:55:02.644 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:55:02.647 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:55:32.736 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:55:32.739 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:56:02.798 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:56:02.801 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:56:09.730 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2023-01-24T03:56:09.781 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 699, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1361, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() socket.timeout 2023-01-24T03:56:32.921 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:56:32.924 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:57:03.022 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:57:03.025 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:57:33.098 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:57:33.102 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:58:03.151 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:58:03.155 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:58:33.264 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:58:33.267 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:59:03.322 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:59:03.324 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:59:33.387 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T03:59:33.391 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:00:03.474 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:00:03.477 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:00:33.612 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:00:33.615 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:01:03.726 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:01:03.729 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:01:33.863 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:01:33.867 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:02:03.923 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:02:03.926 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:02:34.158 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:02:34.161 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:03:04.244 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:03:04.247 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:03:34.367 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:03:34.371 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:04:04.489 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:04:04.492 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:04:34.602 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:04:34.605 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:05:04.731 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:05:04.734 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:05:34.820 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:05:34.823 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:06:04.914 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:06:04.917 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:06:35.049 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:06:35.054 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:07:05.136 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:07:05.139 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:07:35.290 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:07:35.293 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:08:05.362 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:08:05.365 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:08:35.426 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:08:35.429 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:09:05.499 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:09:05.501 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:09:35.568 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:09:35.572 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:10:05.639 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:10:05.643 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:10:35.796 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:10:35.799 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:11:05.882 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:11:05.886 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:11:36.020 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:11:36.024 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:12:06.135 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:12:06.138 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:12:36.266 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:12:36.269 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:13:06.405 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:13:06.408 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:13:36.533 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:13:36.558 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:14:06.688 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:14:06.691 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:14:36.847 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:14:36.855 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:15:06.923 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:15:06.926 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:15:37.059 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:15:37.062 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:16:07.291 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:16:07.295 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:16:37.351 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:16:37.354 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:17:07.511 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:17:07.515 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:17:37.634 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:17:37.637 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:18:07.764 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:18:07.768 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:18:37.900 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:18:37.904 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:19:08.022 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:19:08.025 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:19:38.171 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:19:38.173 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:20:08.259 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:20:08.262 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:20:38.398 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:20:38.401 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:21:08.530 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:21:08.533 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:21:38.689 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:21:38.692 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:22:08.836 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:22:08.839 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:22:38.972 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:22:38.975 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:23:09.111 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:23:09.113 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:23:39.175 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:23:39.178 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:24:09.314 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:24:09.317 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:24:39.386 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:24:39.390 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:25:09.533 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:25:09.536 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:25:39.603 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:25:39.606 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:26:09.737 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:26:09.740 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:26:39.831 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:26:39.834 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:27:09.960 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:27:09.964 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:27:40.083 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:27:40.087 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:28:10.213 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:28:10.217 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:28:40.325 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:28:40.328 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:29:10.472 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:29:10.475 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:29:40.596 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:29:40.599 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:30:10.717 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:30:10.721 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:30:40.864 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:30:40.870 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:31:11.020 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:31:11.023 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:31:41.199 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:31:41.202 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:32:11.262 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:32:11.266 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:32:41.398 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:32:41.401 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:33:11.522 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:33:11.526 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:33:41.652 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:33:41.655 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:34:11.799 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:34:11.802 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:34:41.904 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:34:41.907 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:35:12.034 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:35:12.037 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:35:42.115 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:35:42.118 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:36:12.196 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:36:12.199 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:36:42.336 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:36:42.339 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:37:12.491 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:37:12.494 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:37:42.618 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:37:42.622 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:38:12.695 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:38:12.698 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:38:42.869 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:38:42.872 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:39:13.007 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:39:13.010 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:39:43.096 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:39:43.099 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:40:13.245 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:40:13.248 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:40:43.341 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:40:43.344 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:41:13.480 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:41:13.483 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:41:43.619 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:41:43.622 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:42:13.741 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:42:13.745 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:42:43.883 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:42:43.886 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:43:13.993 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:43:13.996 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:43:44.140 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:43:44.143 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:44:14.308 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:44:14.311 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:44:44.408 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:44:44.411 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:45:14.604 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:45:14.607 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:45:44.743 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:45:44.746 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:46:14.807 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:46:14.810 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:46:44.944 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:46:44.947 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:47:15.144 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:47:15.148 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:47:45.221 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:47:45.224 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:48:15.435 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:48:15.439 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:48:45.579 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:48:45.675 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:49:15.768 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:49:15.771 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:49:45.891 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:49:45.894 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:50:16.030 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:50:16.033 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:50:46.186 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:50:46.189 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:51:16.292 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:51:16.295 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:51:46.590 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:51:46.593 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:52:16.704 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:52:16.707 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:52:46.837 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:52:46.840 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:53:16.976 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:53:16.979 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:53:47.121 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:53:47.124 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:54:17.236 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:54:17.239 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:54:47.383 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:54:47.386 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:55:17.513 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:55:17.517 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:55:47.653 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:55:47.656 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:56:17.815 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:56:17.818 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:56:47.920 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:56:47.923 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:57:18.055 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:57:18.058 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:57:48.184 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:57:48.187 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:58:18.310 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:58:18.313 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:58:48.406 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:58:48.409 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:59:18.561 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:59:18.564 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:59:48.642 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T04:59:48.645 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:00:18.773 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:00:18.776 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:00:48.886 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:00:48.889 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:01:19.024 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:01:19.028 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:01:49.155 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:01:49.158 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:02:19.318 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:02:19.321 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:02:49.440 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:02:49.443 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:03:19.603 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:03:19.606 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:03:49.733 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:03:49.736 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:04:19.880 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:04:19.884 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:04:50.152 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:04:50.155 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:05:20.215 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:05:20.218 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:05:50.353 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:05:50.357 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:06:20.502 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:06:20.505 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:06:50.639 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:06:50.642 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:07:20.797 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:07:20.800 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:07:51.007 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:07:51.010 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:08:21.075 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:08:21.079 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:08:51.210 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:08:51.213 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:09:21.279 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:09:21.284 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:09:51.359 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:09:51.362 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:10:21.507 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:10:21.511 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:10:51.647 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:10:51.650 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:11:21.768 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:11:21.772 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:11:51.908 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:11:51.912 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:12:22.046 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:12:22.049 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:12:52.170 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:12:52.172 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:13:22.426 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:13:22.429 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:13:52.515 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:13:52.518 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:14:22.651 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:14:22.654 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:14:52.793 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:14:52.797 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:15:22.927 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:15:22.930 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:15:53.067 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:15:53.070 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:16:23.129 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:16:23.133 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:16:53.260 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:16:53.263 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:17:23.373 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:17:23.376 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:17:53.670 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:17:53.673 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:18:23.742 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:18:23.747 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:18:53.871 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:18:53.875 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:19:24.009 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:19:24.013 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:19:54.168 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:19:54.171 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:20:24.236 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:20:24.239 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:20:54.324 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:20:54.327 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:21:24.427 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:21:24.430 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:21:54.568 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:21:54.572 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:22:24.690 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:22:24.693 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:22:54.820 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:22:54.823 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:23:24.950 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:23:24.953 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:23:55.107 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:23:55.110 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:24:25.319 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:24:25.322 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:24:55.432 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:24:55.435 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:25:25.571 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:25:25.574 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:25:55.748 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:25:55.752 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:26:25.888 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:26:25.891 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:26:56.010 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:26:56.013 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:27:26.124 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:27:26.127 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:27:56.245 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:27:56.249 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:28:26.369 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:28:26.372 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:28:56.505 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:28:56.509 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:29:26.622 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:29:26.625 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:29:56.766 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:29:56.769 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:30:26.921 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:30:26.924 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:30:57.207 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:30:57.210 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:31:27.273 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:31:27.276 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:31:57.412 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:31:57.415 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:32:27.566 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:32:27.569 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:32:57.689 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:32:57.692 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:33:27.852 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:33:27.856 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:33:57.991 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:33:57.994 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:34:28.104 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:34:28.107 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:34:58.236 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:34:58.239 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:35:28.350 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:35:28.354 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:35:58.513 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:35:58.516 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:36:28.625 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:36:28.628 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:36:58.776 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:36:58.779 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:37:28.907 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:37:28.911 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:37:59.022 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:37:59.025 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:38:29.136 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:38:29.139 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:38:59.201 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:38:59.205 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:39:29.355 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:39:29.358 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:39:59.468 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:39:59.472 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:40:29.607 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:40:29.610 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:40:59.681 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:40:59.684 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:41:29.926 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:41:29.930 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:41:59.997 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:42:00.000 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:42:30.133 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:42:30.136 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:43:00.233 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:43:00.236 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:43:30.397 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:43:30.401 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:44:00.527 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:44:00.530 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:44:30.647 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:44:30.650 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:45:00.778 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:45:00.781 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:45:30.901 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:45:30.904 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:46:01.031 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:46:01.034 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:46:31.169 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:46:31.173 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:47:01.301 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:47:01.304 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:47:31.457 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:47:31.461 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:48:01.609 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:48:01.613 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:48:31.757 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:48:31.760 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:49:01.845 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:49:01.848 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:49:32.006 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:49:32.009 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:50:02.129 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:50:02.132 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:50:32.276 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:50:32.279 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:51:02.424 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:51:02.428 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:51:32.530 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:51:32.533 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:52:02.700 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:52:02.703 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:52:32.830 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:52:32.833 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:53:02.954 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:53:02.957 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:53:33.089 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:53:33.093 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:54:03.232 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:54:03.235 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:54:33.375 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:54:33.378 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:55:03.528 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:55:03.531 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:55:33.675 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:55:33.678 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:56:03.823 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:56:03.860 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:56:33.929 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:56:33.932 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:57:04.076 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:57:04.079 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:57:34.157 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:57:34.160 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:58:04.279 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:58:04.281 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:58:34.393 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:58:34.397 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:59:04.533 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:59:04.535 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:59:34.600 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T05:59:34.603 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:00:04.745 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:00:04.748 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:00:34.901 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:00:34.904 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:01:04.974 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:01:04.977 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:01:35.129 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:01:35.132 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:02:05.210 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:02:05.213 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:02:35.347 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:02:35.350 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:03:05.506 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:03:05.509 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:03:35.662 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:03:35.665 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:04:05.784 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:04:05.787 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:04:35.907 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:04:35.910 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:05:06.054 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:05:06.057 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:05:36.186 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:05:36.189 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:06:06.325 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:06:06.328 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:06:36.472 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:06:36.475 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:07:06.611 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:07:06.614 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:07:36.750 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:07:36.753 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:08:06.912 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:08:06.915 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:08:37.054 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:08:37.057 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:09:07.185 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:09:07.188 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:09:37.276 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:09:37.280 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:10:07.390 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:10:07.393 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:10:37.537 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:10:37.540 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:11:07.677 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:11:07.681 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:11:37.748 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:11:37.751 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:12:07.894 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:12:07.897 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:12:38.010 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:12:38.013 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:13:08.132 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:13:08.135 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:13:38.280 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:13:38.283 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:14:08.406 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:14:08.409 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:14:38.520 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:14:38.523 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:15:08.649 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:15:08.652 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:15:38.771 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:15:38.774 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:16:08.910 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:16:08.913 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:16:39.023 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:16:39.026 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:17:09.170 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:17:09.173 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:17:39.283 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:17:39.286 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:18:09.423 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:18:09.426 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:18:39.572 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:18:39.575 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:19:09.727 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:19:09.730 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:19:39.821 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:19:39.824 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:20:09.967 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:20:09.970 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:20:40.072 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:20:40.076 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:21:10.303 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:21:10.306 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:21:40.373 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:21:40.376 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:22:10.513 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:22:10.516 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:22:40.617 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:22:40.620 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:23:10.740 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:23:10.743 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:23:40.861 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:23:40.864 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:24:10.991 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:24:10.995 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:24:41.137 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:24:41.140 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:25:11.229 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:25:11.233 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:25:41.347 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:25:41.351 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:26:11.528 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:26:11.531 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:26:41.666 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:26:41.669 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:27:11.779 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:27:11.782 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:27:41.935 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:27:41.938 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:28:12.065 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:28:12.069 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:28:42.420 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:28:42.423 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:29:12.509 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:29:12.512 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:29:42.866 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:29:42.869 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:30:12.927 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:30:12.932 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:30:43.078 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:30:43.081 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:31:13.288 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:31:13.291 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:31:43.347 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:31:43.351 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:32:13.507 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:32:13.510 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:32:43.646 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:32:43.654 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:33:13.753 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:33:13.756 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:33:43.908 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:33:43.911 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:34:13.963 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:34:13.966 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:34:44.034 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:34:44.037 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:35:14.173 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:35:14.177 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:35:44.237 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:35:44.240 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:36:14.360 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:36:14.363 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:36:44.465 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:36:44.468 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:37:14.611 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:37:14.614 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:37:44.691 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:37:44.694 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:38:14.825 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:38:14.828 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:38:44.959 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:38:44.962 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:39:15.080 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:39:15.083 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:39:45.227 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:39:45.230 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:40:15.332 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:40:15.335 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:40:45.479 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:40:45.482 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:41:15.567 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:41:15.571 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:41:45.691 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:41:45.694 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:42:15.788 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:42:15.791 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:42:46.035 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:42:46.038 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:43:16.116 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:43:16.119 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:43:46.250 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:43:46.253 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:44:16.378 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:44:16.381 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:44:46.535 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:44:46.538 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:45:16.666 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:45:16.669 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:45:46.812 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:45:46.815 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:46:16.942 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:46:16.945 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:46:47.049 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:46:47.052 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:47:17.179 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:47:17.182 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:47:47.332 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:47:47.335 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:48:17.478 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:48:17.481 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:48:47.609 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:48:47.612 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:49:17.747 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:49:17.750 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:49:47.894 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:49:47.897 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:50:17.997 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:50:18.000 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:50:48.137 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:50:48.140 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:51:18.267 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:51:18.270 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:51:48.397 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:51:48.400 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:52:18.511 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:52:18.515 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:52:48.641 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:52:48.644 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:53:18.779 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:53:18.783 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:53:48.918 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:53:48.921 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:54:19.060 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:54:19.063 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:54:49.118 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:54:49.122 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:55:19.266 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:55:19.269 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:55:49.405 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:55:49.409 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:56:19.543 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:56:19.546 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:56:49.688 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:56:49.692 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:57:19.822 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:57:19.825 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:57:49.957 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:57:49.960 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:58:20.067 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:58:20.070 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:58:50.173 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:58:50.176 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:59:20.323 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:59:20.326 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:59:50.469 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T06:59:50.472 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:00:20.649 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:00:20.652 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:00:50.780 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:00:50.783 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:01:20.936 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:01:20.940 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:01:51.067 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:01:51.070 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:02:21.181 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:02:21.184 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:02:51.326 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:02:51.329 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:03:21.406 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:03:21.410 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:03:51.557 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:03:51.560 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:04:21.655 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:04:21.659 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:04:51.763 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:04:51.766 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:05:21.892 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:05:21.896 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:05:52.241 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:05:52.244 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:06:22.344 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:06:22.351 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:06:52.490 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:06:52.493 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:07:22.588 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:07:22.591 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:07:52.739 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:07:52.742 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:08:22.880 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:08:22.884 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:08:53.132 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:08:53.135 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:09:23.199 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:09:23.203 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:09:53.329 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:09:53.332 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:10:23.468 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:10:23.472 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:10:53.606 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:10:53.612 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:11:23.727 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:11:23.730 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:11:54.095 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:11:54.098 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:12:24.159 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:12:24.162 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:12:54.265 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:12:54.268 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:13:24.412 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:13:24.415 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:13:54.510 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:13:54.513 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:14:24.641 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:14:24.644 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:14:54.780 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:14:54.783 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:15:24.919 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:15:24.923 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:15:55.051 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:15:55.054 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:16:25.196 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:16:25.199 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:16:55.351 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:16:55.354 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:17:25.410 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:17:25.413 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:17:55.544 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:17:55.547 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:18:25.691 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:18:25.695 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:18:55.802 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:18:55.805 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:19:25.960 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:19:25.964 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:19:56.091 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:19:56.094 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:20:26.195 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:20:26.198 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:20:56.252 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:20:56.255 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:21:26.374 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:21:26.378 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:21:56.545 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:21:56.548 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:22:26.683 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:22:26.686 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:22:56.816 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:22:56.819 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:23:26.959 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:23:26.962 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:23:57.073 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:23:57.077 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:24:27.200 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:24:27.204 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:24:57.358 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:24:57.361 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:25:27.496 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:25:27.499 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:25:57.643 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:25:57.646 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:26:27.732 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:26:27.735 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:26:57.879 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:26:57.882 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:27:28.017 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:27:28.020 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:27:58.149 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:27:58.152 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:28:28.280 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:28:28.283 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:28:58.432 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:28:58.435 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:29:28.587 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:29:28.590 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:29:58.697 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:29:58.700 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:30:28.845 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:30:28.848 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:30:58.984 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:30:58.987 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:31:29.098 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:31:29.102 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:31:59.229 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:31:59.232 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:32:29.343 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:32:29.347 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:32:59.443 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:32:59.446 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:33:29.588 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:33:29.591 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:33:59.702 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:33:59.705 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:34:29.762 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:34:29.765 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:34:59.931 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:34:59.934 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:35:30.070 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:35:30.073 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:36:00.210 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:36:00.214 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:36:30.331 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:36:30.334 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:37:00.471 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:37:00.474 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:37:30.542 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:37:30.545 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:38:00.689 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:38:00.692 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:38:30.785 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:38:30.788 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:39:00.923 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:39:00.926 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:39:31.060 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:39:31.063 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:40:01.184 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:40:01.187 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:40:31.338 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:40:31.342 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:41:01.486 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:41:01.489 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:41:31.708 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:41:31.711 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:42:01.764 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:42:01.767 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:42:31.903 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:42:31.906 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:43:02.042 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:43:02.046 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:43:32.203 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:43:32.206 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:44:02.354 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:44:02.357 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:44:32.510 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:44:32.513 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:45:02.635 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:45:02.638 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:45:32.757 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:45:32.760 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:46:02.903 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:46:02.907 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:46:33.042 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:46:33.045 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:47:03.188 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:47:03.192 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:47:33.345 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:47:33.348 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:48:03.492 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:48:03.495 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:48:33.615 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:48:33.618 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:49:03.757 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:49:03.761 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:49:33.866 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:49:33.869 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:50:03.996 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:50:03.999 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:50:34.130 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:50:34.133 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:51:04.254 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:51:04.257 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:51:34.431 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:51:34.435 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:52:04.498 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:52:04.501 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:52:34.662 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:52:34.666 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:53:04.795 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:53:04.798 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:53:34.966 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:53:34.970 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:54:05.105 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:54:05.109 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:54:35.243 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:54:35.246 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:55:05.323 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:55:05.326 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:55:35.382 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:55:35.385 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:56:05.522 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:56:05.525 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:56:35.654 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:56:35.657 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:57:05.792 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:57:05.797 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:57:35.931 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:57:35.935 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:58:06.054 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:58:06.057 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:58:36.174 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:58:36.177 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:59:06.387 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:59:06.391 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:59:36.467 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T07:59:36.470 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:00:06.614 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:00:06.617 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:00:36.744 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:00:36.747 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:01:06.891 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:01:06.894 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:01:37.012 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:01:37.015 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:02:07.142 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:02:07.145 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:02:37.256 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:02:37.260 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:03:07.411 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:03:07.415 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:03:37.491 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:03:37.494 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:04:07.637 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:04:07.640 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:04:37.775 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:04:37.779 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:05:07.930 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:05:07.933 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:05:38.073 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:05:38.077 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:06:08.205 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:06:08.209 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:06:38.329 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:06:38.332 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:07:08.467 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:07:08.470 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:07:38.616 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:07:38.619 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:08:08.738 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:08:08.741 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:08:38.874 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:08:38.877 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:09:09.055 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:09:09.058 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:09:39.176 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:09:39.179 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:10:09.478 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:10:09.481 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:10:39.635 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:10:39.638 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:11:09.781 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:11:09.785 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:11:39.946 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:11:39.949 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:12:10.210 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:12:10.213 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:12:40.348 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:12:40.352 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:13:10.479 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:13:10.482 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:13:40.541 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:13:40.544 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:14:10.683 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:14:10.686 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:14:40.788 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:14:40.792 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:15:10.927 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:15:10.930 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:15:41.070 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:15:41.074 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:16:11.216 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:16:11.221 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:16:41.387 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:16:41.390 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:17:11.518 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:17:11.521 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:17:41.698 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:17:41.707 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:18:11.853 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:18:11.856 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:18:41.985 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:18:41.988 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:19:12.113 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:19:12.116 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:19:42.259 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:19:42.262 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:20:12.335 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:20:12.339 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:20:42.486 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:20:42.489 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:21:12.540 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:21:12.543 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:21:42.696 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:21:42.699 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:22:12.860 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:22:12.863 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:22:42.990 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:22:42.993 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:23:13.095 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:23:13.098 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:23:43.234 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:23:43.237 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:24:13.487 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:24:13.490 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:24:43.596 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:24:43.599 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:25:13.736 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:25:13.739 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:25:43.858 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:25:43.861 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:26:13.955 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:26:13.958 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:26:44.113 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:26:44.116 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:27:14.243 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:27:14.246 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:27:44.322 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:27:44.325 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:28:14.461 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:28:14.463 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:28:44.585 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:28:44.588 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:29:14.723 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:29:14.726 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:29:44.870 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:29:44.874 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:30:15.014 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:30:15.017 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:30:45.140 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:30:45.143 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:31:15.200 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:31:15.203 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:31:45.344 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:31:45.347 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:32:15.477 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:32:15.480 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:32:45.565 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:32:45.568 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:33:15.704 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:33:15.707 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:33:45.852 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:33:45.855 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:34:15.982 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:34:15.985 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:34:46.093 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:34:46.096 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:35:16.147 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:35:16.150 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:35:46.295 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:35:46.298 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:36:16.409 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:36:16.412 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:36:46.539 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:36:46.542 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:37:16.669 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:37:16.673 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:37:46.749 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:37:46.752 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:38:16.814 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:38:16.817 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:38:46.894 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:38:46.897 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:39:16.955 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:39:16.958 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:39:47.014 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:39:47.017 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:40:17.078 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:40:17.081 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:40:47.137 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:40:47.140 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:41:17.193 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:41:17.197 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:41:47.279 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:41:47.282 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:42:17.336 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:42:17.339 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:42:47.401 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:42:47.404 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:43:17.463 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:43:17.466 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:43:47.519 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:43:47.522 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:44:17.581 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:44:17.584 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:44:47.647 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:44:47.651 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:45:17.724 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:45:17.727 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:45:47.785 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:45:47.788 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:46:17.868 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:46:17.871 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:46:47.922 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:46:47.925 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:47:17.993 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:47:17.999 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:47:48.066 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:47:48.070 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:48:18.128 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:48:18.131 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:48:48.200 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:48:48.203 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:49:18.272 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:49:18.275 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:49:48.344 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:49:48.347 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:50:18.406 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:50:18.409 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:50:48.470 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:50:48.473 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:51:18.534 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:51:18.537 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:51:48.589 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:51:48.592 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:52:18.677 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:52:18.681 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:52:48.734 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:52:48.737 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:53:18.790 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:53:18.793 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:53:48.851 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:53:48.854 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:54:18.910 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:54:18.913 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:54:48.969 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:54:48.972 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:55:19.091 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:55:19.094 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:55:49.145 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:55:49.148 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:56:19.301 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:56:19.304 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:56:49.455 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:56:49.458 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:57:19.593 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:57:19.596 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:57:49.791 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:57:49.794 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:58:19.886 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:58:19.890 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:58:50.018 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:58:50.021 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:59:20.155 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:59:20.158 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:59:50.235 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T08:59:50.238 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:00:20.373 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:00:20.378 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:00:50.505 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:00:50.508 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:01:20.625 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:01:20.628 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:01:50.755 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:01:50.758 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:02:20.902 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:02:20.905 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:02:50.961 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:02:50.964 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:03:21.111 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:03:21.114 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:03:51.191 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:03:51.206 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:04:21.355 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:04:21.358 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:04:51.426 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:04:51.430 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:05:21.549 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:05:21.552 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:05:51.671 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:05:51.674 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:06:21.736 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:06:21.739 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:06:51.810 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:06:51.812 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:07:21.877 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:07:21.880 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:07:51.967 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:07:51.969 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:08:22.096 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:08:22.099 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:08:52.243 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:08:52.246 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:09:22.374 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:09:22.377 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:09:52.436 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:09:52.440 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:10:22.491 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:10:22.495 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:10:52.639 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:10:52.642 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:11:22.777 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:11:22.780 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:11:52.890 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:11:52.893 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:12:23.046 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:12:23.049 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:12:53.104 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:12:53.108 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:13:23.248 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:13:23.251 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:13:53.407 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:13:53.410 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:14:23.512 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:14:23.515 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:14:53.680 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:14:53.684 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:15:23.816 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:15:23.819 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:15:53.980 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:15:53.984 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:16:24.094 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:16:24.097 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:16:54.234 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:16:54.237 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:17:24.297 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:17:24.301 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:17:54.428 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:17:54.431 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:18:24.550 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:18:24.554 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:18:54.722 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:18:54.725 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:19:24.861 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:19:24.864 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:19:54.976 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:19:54.979 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:20:25.090 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:20:25.092 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:20:55.213 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:20:55.217 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:21:25.352 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:21:25.355 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:21:55.466 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:21:55.469 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:22:25.572 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:22:25.575 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:22:55.694 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:22:55.698 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:23:25.816 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:23:25.821 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:23:55.939 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:23:55.942 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:24:26.061 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:24:26.064 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:24:56.225 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:24:56.228 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:25:26.281 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:25:26.285 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:25:56.411 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:25:56.414 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:26:26.558 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:26:26.562 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:26:56.881 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:26:56.884 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:27:26.953 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:27:26.956 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:27:57.018 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:27:57.021 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:28:27.079 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:28:27.082 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:28:57.146 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:28:57.149 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:29:27.231 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:29:27.234 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:29:57.369 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:29:57.435 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:30:27.548 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:30:27.552 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:30:57.696 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:30:57.700 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:31:27.844 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:31:27.847 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:31:57.974 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:31:57.977 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:32:28.129 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:32:28.132 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:32:58.284 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:32:58.287 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:33:28.373 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:33:28.377 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:33:58.512 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:33:58.515 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:34:28.643 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:34:28.646 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:34:58.795 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:34:58.798 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:35:28.933 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:35:28.937 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:35:59.069 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:35:59.073 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:36:29.207 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:36:29.213 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:36:59.355 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:36:59.358 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:37:29.436 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:37:29.439 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:37:59.566 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:37:59.569 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:38:29.688 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:38:29.691 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:38:59.744 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:38:59.747 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:39:29.792 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:39:29.795 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:39:59.845 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:39:59.849 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:40:29.979 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:40:29.982 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:41:00.118 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:41:00.121 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:41:30.198 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:41:30.201 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:42:00.330 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:42:00.333 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:42:30.494 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:42:30.497 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:43:00.624 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:43:00.627 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:43:30.705 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:43:30.708 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:44:00.844 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:44:00.848 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:44:30.958 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:44:30.961 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:45:01.022 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:45:01.026 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:45:31.077 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:45:31.081 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:46:01.134 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:46:01.137 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:46:31.264 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:46:31.267 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:47:01.328 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:47:01.331 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:47:31.398 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:47:31.401 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:48:01.553 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:48:01.556 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:48:31.683 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:48:31.687 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:49:01.789 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:49:01.792 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:49:31.944 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:49:31.948 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:50:02.050 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:50:02.053 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:50:32.205 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:50:32.208 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:51:02.352 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:51:02.355 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:51:32.491 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:51:32.494 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:52:02.554 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:52:02.557 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:52:32.618 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:52:32.621 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:53:02.825 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:53:02.828 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:53:32.886 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:53:32.889 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:54:03.016 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:54:03.020 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:54:33.072 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:54:33.075 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:55:03.129 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:55:03.131 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:55:33.189 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:55:33.192 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:56:03.265 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:56:03.268 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:56:33.391 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:56:33.394 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:57:03.486 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:57:03.489 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:57:33.558 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:57:33.562 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:58:03.622 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:58:03.625 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:58:33.778 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:58:33.782 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:59:03.876 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:59:03.879 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:59:33.997 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T09:59:34.000 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:00:04.136 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:00:04.139 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:00:34.275 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:00:34.278 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:01:04.337 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:01:04.340 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:01:34.485 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:01:34.488 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:02:04.559 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:02:04.562 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:02:34.703 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:02:34.706 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:03:04.757 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:03:04.761 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:03:34.902 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:03:34.905 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:04:05.068 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:04:05.071 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:04:35.155 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:04:35.158 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:05:05.319 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:05:05.323 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:05:35.475 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:05:35.478 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:06:05.622 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:06:05.625 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:06:35.727 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:06:35.730 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:07:05.858 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:07:05.861 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:07:35.988 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:07:35.991 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:08:06.126 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:08:06.129 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:08:36.269 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:08:36.272 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:09:06.319 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:09:06.322 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:09:36.449 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:09:36.452 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:10:06.587 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:10:06.590 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:10:36.643 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:10:36.646 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:11:06.781 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:11:06.784 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:11:36.886 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:11:36.890 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:12:07.043 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:12:07.046 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:12:37.187 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:12:37.190 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:13:07.328 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:13:07.331 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:13:37.482 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:13:37.485 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:14:07.606 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:14:07.610 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:14:37.700 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:14:37.703 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:15:07.854 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:15:07.857 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:15:37.948 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:15:37.951 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:16:08.097 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:16:08.101 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:16:38.221 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:16:38.224 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:17:08.424 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:17:08.428 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:17:38.497 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:17:38.501 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:18:08.702 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:18:08.705 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:18:38.756 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:18:38.759 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:19:08.828 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:19:08.831 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:19:38.960 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:19:38.963 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:20:09.046 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:20:09.049 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:20:39.169 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:20:39.173 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:21:09.310 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:21:09.314 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:21:39.425 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:21:39.429 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:22:09.564 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:22:09.568 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:22:39.720 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:22:39.723 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:23:09.817 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:23:09.820 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:23:39.948 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:23:39.952 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:24:10.079 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:24:10.082 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:24:40.244 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:24:40.247 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:25:10.407 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:25:10.411 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:25:40.538 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:25:40.541 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:26:10.660 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:26:10.663 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:26:40.765 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:26:40.768 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:27:10.904 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:27:10.907 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:27:41.049 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:27:41.053 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:28:11.188 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:28:11.191 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:28:41.322 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:28:41.325 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:29:11.431 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:29:11.436 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:29:41.611 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:29:41.615 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:30:11.691 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:30:11.694 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:30:41.822 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:30:41.825 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:31:11.969 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:31:11.973 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:31:42.100 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:31:42.103 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:32:12.246 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:32:12.249 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:32:42.401 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:32:42.404 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:33:12.531 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:33:12.534 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:33:42.670 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:33:42.673 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:34:12.816 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:34:12.819 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:34:42.938 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:34:42.941 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:35:13.085 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:35:13.088 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:35:43.198 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:35:43.201 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:36:13.337 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:36:13.340 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:36:43.466 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:36:43.470 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:37:13.596 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:37:13.599 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:37:43.743 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:37:43.746 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:38:13.806 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:38:13.809 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:38:43.948 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:38:43.951 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:39:14.016 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:39:14.020 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:39:44.139 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:39:44.142 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:40:14.293 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:40:14.296 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:40:44.374 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:40:44.377 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:41:14.529 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:41:14.532 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:41:44.599 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:41:44.603 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:42:14.663 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:42:14.666 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:42:44.793 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:42:44.797 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:43:14.940 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:43:14.944 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:43:45.070 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:43:45.074 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:44:15.192 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:44:15.196 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:44:45.314 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:44:45.317 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:45:15.444 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:45:15.448 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:45:45.574 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:45:45.577 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:46:15.654 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:46:15.657 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:46:45.785 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:46:45.788 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:47:15.932 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:47:15.936 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:47:46.078 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:47:46.082 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:48:16.133 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:48:16.136 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:48:46.262 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:48:46.265 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:49:16.393 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:49:16.396 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:49:46.531 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:49:46.535 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:50:16.661 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:50:16.664 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:50:46.792 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:50:46.795 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:51:16.914 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:51:16.917 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:51:47.053 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:51:47.056 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:52:17.191 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:52:17.194 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:52:47.330 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:52:47.333 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:53:17.468 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:53:17.471 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:53:47.530 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:53:47.533 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:54:17.586 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:54:17.589 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:54:47.724 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:54:47.728 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:55:17.862 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:55:17.865 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:55:47.993 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:55:47.996 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:56:18.114 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:56:18.117 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:56:48.254 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:56:48.257 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:57:18.399 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:57:18.402 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:57:48.478 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:57:48.482 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:58:18.629 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:58:18.633 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:58:48.726 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:58:48.729 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:59:18.886 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:59:18.889 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:59:48.938 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T10:59:48.941 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:00:18.995 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:00:18.998 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:00:49.050 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:00:49.053 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:01:19.106 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:01:19.109 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:01:49.161 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:01:49.164 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:02:19.234 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:02:19.237 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:02:49.312 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:02:49.316 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:03:19.378 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:03:19.382 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:03:49.439 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:03:49.443 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:04:19.497 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:04:19.500 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:04:49.624 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:04:49.628 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:05:19.738 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:05:19.741 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:05:49.801 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:05:49.804 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:06:19.864 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:06:19.868 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:06:50.045 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:06:50.049 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:07:20.107 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:07:20.111 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:07:50.271 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:07:50.274 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:08:20.384 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:08:20.387 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:08:50.505 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:08:50.508 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:09:20.569 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:09:20.573 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:09:50.632 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:09:50.635 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:10:20.704 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:10:20.707 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:10:50.769 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:10:50.772 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:11:20.831 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:11:20.834 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:11:50.887 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:11:50.890 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:12:20.942 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:12:20.946 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:12:51.005 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:12:51.008 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:13:21.077 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:13:21.080 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:13:51.140 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:13:51.143 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:14:21.194 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:14:21.197 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:14:51.345 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:14:51.348 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:15:21.509 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:15:21.512 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:15:51.568 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:15:51.571 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:16:21.696 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:16:21.699 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:16:51.753 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:16:51.756 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:17:21.817 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:17:21.820 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:17:51.954 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:17:51.958 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:18:22.069 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:18:22.072 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:18:52.215 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:18:52.219 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:19:22.347 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:19:22.350 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:19:52.427 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:19:52.430 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:20:22.524 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:20:22.527 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:20:52.689 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:20:52.692 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:21:22.778 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:21:22.781 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:21:52.883 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:21:52.887 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:22:23.031 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:22:23.034 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:22:53.162 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:22:53.165 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:23:23.266 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:23:23.269 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:23:53.414 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:23:53.417 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:24:23.546 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:24:23.549 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:24:53.610 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:24:53.613 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:25:23.740 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:25:23.744 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:25:53.812 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:25:53.815 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:26:23.867 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:26:23.870 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:26:53.939 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:26:53.943 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:27:24.005 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:27:24.008 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:27:54.059 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:27:54.063 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:28:24.200 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:28:24.203 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:28:54.288 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:28:54.291 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:29:24.627 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:29:24.630 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:29:54.686 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:29:54.689 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:30:24.759 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:30:24.762 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:30:54.811 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:30:54.814 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:31:24.869 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:31:24.872 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:31:54.933 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:31:54.936 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:32:24.987 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:32:24.991 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:32:55.043 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:32:55.047 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:33:25.124 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:33:25.126 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:33:55.197 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:33:55.199 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:34:25.260 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:34:25.263 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:34:55.316 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:34:55.319 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:35:25.376 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:35:25.379 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:35:55.428 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:35:55.432 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:36:25.503 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:36:25.506 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:36:55.566 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:36:55.569 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:37:25.629 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:37:25.632 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:37:55.685 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:37:55.688 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:38:25.748 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:38:25.752 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:38:55.804 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:38:55.807 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:39:25.874 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:39:25.877 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:39:55.932 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:39:55.936 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:40:25.988 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:40:25.991 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:40:56.050 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:40:56.053 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:41:26.109 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:41:26.112 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:41:56.164 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:41:56.167 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:42:26.224 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:42:26.227 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:42:56.278 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:42:56.281 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:43:26.347 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:43:26.350 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:43:56.405 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:43:56.409 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:44:26.478 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:44:26.481 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:44:56.533 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:44:56.536 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:45:26.597 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:45:26.600 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:45:56.652 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:45:56.655 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:46:26.807 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:46:26.811 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:46:56.862 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:46:56.866 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:47:26.925 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:47:26.929 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:47:56.999 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:47:57.002 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:48:27.054 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:48:27.057 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:48:57.118 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:48:57.121 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:49:27.187 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:49:27.190 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:49:57.242 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:49:57.245 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:50:27.370 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:50:27.373 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:50:57.510 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:50:57.515 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:51:27.577 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:51:27.581 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:51:57.679 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:51:57.682 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:52:27.820 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:52:27.824 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:52:57.967 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:52:57.970 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:53:28.097 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:53:28.102 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:53:58.160 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:53:58.163 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:54:28.309 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:54:28.312 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:54:58.420 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:54:58.423 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:55:28.550 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:55:28.553 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:55:58.690 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:55:58.693 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:56:28.761 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:56:28.764 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:56:58.907 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:56:58.910 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:57:29.020 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:57:29.024 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:57:59.152 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:57:59.155 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:58:29.275 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:58:29.278 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:58:59.336 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:58:59.339 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:59:29.408 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:59:29.411 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:59:59.467 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T11:59:59.470 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:00:29.531 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:00:29.535 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:00:59.590 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:00:59.593 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:01:29.646 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:01:29.650 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:01:59.709 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:01:59.712 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:02:29.876 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:02:29.879 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:02:59.945 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:02:59.948 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:03:29.999 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:03:30.003 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:04:00.125 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:04:00.128 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:04:30.206 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:04:30.210 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:05:00.329 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:05:00.332 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:05:30.402 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:05:30.405 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:06:00.459 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:06:00.462 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:06:30.604 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:06:30.607 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:07:00.719 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:07:00.722 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:07:30.883 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:07:30.886 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:08:01.005 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:08:01.008 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:08:31.152 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:08:31.155 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:09:01.299 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:09:01.302 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:09:31.454 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:09:31.458 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:10:01.604 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:10:01.608 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:10:31.760 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:10:31.765 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:11:01.887 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:11:01.890 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:11:31.989 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:11:31.993 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:12:02.136 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:12:02.139 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:12:32.191 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:12:32.195 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:13:02.347 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:13:02.350 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:13:32.478 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:13:32.482 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:14:02.574 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:14:02.577 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:14:32.640 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:14:32.644 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:15:02.726 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:15:02.729 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:15:32.864 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:15:32.867 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:16:03.000 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:16:03.003 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:16:33.058 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:16:33.061 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:17:03.187 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:17:03.190 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:17:33.250 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:17:33.253 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:18:03.415 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:18:03.418 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:18:33.739 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:18:33.745 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:19:03.801 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:19:03.804 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:19:33.931 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:19:33.935 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:20:04.070 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:20:04.074 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:20:34.186 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:20:34.189 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:21:04.325 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:21:04.329 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:21:34.411 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:21:34.414 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:22:04.542 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:22:04.545 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:22:34.693 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:22:34.696 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:23:04.833 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:23:04.836 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:23:34.972 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:23:34.976 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:24:05.036 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:24:05.039 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:24:35.184 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:24:35.187 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:25:05.315 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:25:05.318 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:25:35.454 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:25:35.461 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:26:05.602 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:26:05.605 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:26:35.732 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:26:35.735 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:27:05.814 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:27:05.818 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:27:35.953 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:27:35.956 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:28:06.100 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:28:06.103 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:28:36.199 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:28:36.202 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:29:06.336 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:29:06.340 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:29:36.443 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:29:36.447 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:30:06.574 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:30:06.577 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:30:36.705 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:30:36.710 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:31:06.832 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:31:06.836 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:31:36.987 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:31:36.989 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:32:07.126 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:32:07.129 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:32:37.287 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:32:37.292 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:33:07.433 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:33:07.436 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:33:37.583 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:33:37.586 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:34:07.721 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:34:07.724 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:34:37.867 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:34:37.871 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:35:08.115 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:35:08.118 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:35:38.186 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:35:38.189 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:36:08.399 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:36:08.403 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:36:38.457 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:36:38.460 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:37:08.594 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:37:08.636 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:37:38.774 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:37:38.777 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:38:08.903 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:38:08.906 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:38:39.026 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:38:39.029 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:39:09.172 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:39:09.175 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:39:39.393 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:39:39.396 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:40:09.508 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:40:09.511 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:40:39.647 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:40:39.650 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:41:09.702 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:41:09.705 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:41:39.848 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:41:39.851 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:42:09.903 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:42:09.906 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:42:40.045 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:42:40.048 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:43:10.180 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:43:10.183 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:43:40.502 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:43:40.505 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:44:10.582 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:44:10.586 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:44:40.709 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:44:40.712 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:45:10.816 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:45:10.819 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:45:40.872 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:45:40.876 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:46:11.019 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:46:11.023 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:46:41.092 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:46:41.096 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:47:11.231 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:47:11.234 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:47:41.361 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:47:41.364 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:48:11.509 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:48:11.512 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:48:41.564 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:48:41.567 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:49:11.696 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:49:11.699 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:49:41.765 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:49:41.769 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:50:11.849 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:50:11.852 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:50:42.180 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:50:42.184 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:51:12.327 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:51:12.330 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:51:42.386 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:51:42.390 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:52:12.539 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:52:12.542 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:52:42.630 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:52:42.634 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:53:12.768 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:53:12.771 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:53:42.833 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:53:42.836 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:54:12.962 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:54:12.965 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:54:43.094 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:54:43.098 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:55:13.224 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:55:13.227 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:55:43.372 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:55:43.375 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:56:13.428 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:56:13.431 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:56:43.558 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:56:43.561 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:57:13.681 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:57:13.685 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:57:43.742 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:57:43.745 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:58:13.805 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:58:13.808 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:58:43.860 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:58:43.863 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:59:13.913 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:59:13.917 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:59:43.979 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T12:59:43.982 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:00:14.057 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:00:14.060 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:00:44.114 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:00:44.119 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:01:14.171 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:01:14.176 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:01:44.234 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:01:44.238 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:02:14.297 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:02:14.301 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:02:44.453 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:02:44.457 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:03:14.591 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:03:14.594 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:03:44.696 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:03:44.700 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:04:14.860 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:04:14.863 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:04:45.207 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:04:45.210 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:05:15.270 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:05:15.273 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:05:45.408 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:05:45.412 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:06:15.505 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:06:15.508 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:06:45.618 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:06:45.621 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:07:15.774 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:07:15.777 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:07:45.879 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:07:45.882 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:08:16.017 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:08:16.020 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:08:46.096 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:08:46.099 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:09:16.177 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:09:16.181 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:09:46.306 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:09:46.309 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:10:16.436 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:10:16.440 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:10:46.592 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:10:46.595 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:11:16.705 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:11:16.708 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:11:46.869 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:11:46.872 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:12:16.998 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:12:17.002 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:12:47.128 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:12:47.132 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:13:17.258 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:13:17.261 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:13:47.363 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:13:47.366 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:14:17.502 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:14:17.505 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:14:47.665 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:14:47.668 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:15:17.723 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:15:17.726 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:15:47.867 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:15:47.870 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:16:17.972 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:16:17.975 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:16:48.102 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:16:48.105 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:17:18.233 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:17:18.236 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:17:48.295 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:17:48.299 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:18:18.476 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:18:18.479 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:18:48.556 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:18:48.559 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:19:18.686 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:19:18.689 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:19:48.791 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:19:48.798 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:20:18.954 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:20:18.957 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:20:49.011 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:20:49.014 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:21:19.073 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:21:19.076 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:21:49.202 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:21:49.207 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:22:19.278 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:22:19.282 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:22:49.334 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:22:49.337 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:23:19.410 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:23:19.413 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:23:49.473 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:23:49.476 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:24:19.544 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:24:19.548 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:24:49.626 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:24:49.630 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:25:19.681 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:25:19.684 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:25:49.737 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:25:49.740 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:26:19.815 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:26:19.818 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:26:49.878 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:26:49.882 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:27:19.942 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:27:19.945 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:27:50.072 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:27:50.075 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:28:20.133 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:28:20.136 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:28:50.256 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:28:50.259 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:29:20.378 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:29:20.381 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:29:50.533 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:29:50.537 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:30:20.637 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:30:20.641 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:30:50.785 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:30:50.788 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:31:20.865 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:31:20.868 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:31:51.011 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:31:51.015 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:32:21.092 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:32:21.096 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:32:51.247 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:32:51.250 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:33:21.327 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:33:21.330 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:33:51.474 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:33:51.477 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:34:21.537 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:34:21.540 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:34:51.700 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:34:51.703 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:35:21.756 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:35:21.759 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:35:51.886 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:35:51.889 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:36:21.975 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:36:21.980 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:36:52.106 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:36:52.109 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:37:22.211 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:37:22.216 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:37:52.348 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:37:52.351 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:38:22.457 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:38:22.465 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:38:52.625 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:38:52.629 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:39:22.772 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:39:22.775 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:39:52.894 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:39:52.897 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:40:23.048 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:40:23.052 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:40:53.203 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:40:53.206 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:41:23.259 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:41:23.263 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:41:53.382 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:41:53.385 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:42:23.528 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:42:23.532 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:42:53.641 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:42:53.644 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:43:23.762 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:43:23.797 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:43:53.884 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:43:53.887 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:44:24.040 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:44:24.044 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:44:54.145 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:44:54.149 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:45:24.267 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:45:24.273 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:45:54.339 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:45:54.343 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:46:24.461 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:46:24.465 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:46:54.549 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:46:54.552 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:47:24.604 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:47:24.607 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:47:54.660 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:47:54.664 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:48:24.732 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:48:24.735 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:48:54.794 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:48:54.797 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:49:24.941 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:49:24.944 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:49:55.079 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:49:55.082 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:50:25.393 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:50:25.395 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:50:55.456 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:50:55.459 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:51:25.612 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:51:25.615 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:51:55.709 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:51:55.712 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:52:25.856 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:52:25.859 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:52:55.910 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:52:55.913 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:53:26.041 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:53:26.044 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:53:56.162 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:53:56.166 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:54:26.284 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:54:26.287 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:54:56.382 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:54:56.385 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:55:26.504 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:55:26.508 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:55:56.668 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:55:56.671 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:56:26.765 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:56:26.768 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:56:56.920 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:56:56.924 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:57:27.024 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:57:27.027 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:57:57.162 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:57:57.165 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:58:27.267 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:58:27.270 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:58:57.414 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:58:57.417 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:59:27.502 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:59:27.506 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:59:57.649 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T13:59:57.652 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:00:27.779 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:00:27.782 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:00:57.919 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:00:57.922 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:01:28.008 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:01:28.011 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:01:58.172 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:01:58.176 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:02:28.307 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:02:28.310 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:02:58.424 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:02:58.427 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:03:28.776 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:03:28.779 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:03:58.916 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:03:58.919 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:04:29.026 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:04:29.030 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:04:59.140 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:04:59.143 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:05:29.289 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:05:29.292 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:05:59.355 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:05:59.359 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:06:29.508 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:06:29.512 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:06:59.622 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:06:59.626 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:07:29.685 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:07:29.688 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:07:59.817 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:07:59.820 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:08:29.922 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:08:29.925 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:09:00.054 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:09:00.057 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:09:30.192 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:09:30.196 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:10:00.331 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:10:00.334 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:10:30.521 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:10:30.524 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:11:00.628 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:11:00.637 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:11:30.701 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:11:30.704 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:12:00.814 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:12:00.817 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:12:30.876 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:12:30.879 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:13:00.939 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:13:00.943 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:13:31.019 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:13:31.022 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:14:01.095 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:14:01.098 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:14:31.242 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:14:31.246 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:15:01.365 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:15:01.369 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:15:31.478 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:15:31.482 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:16:01.553 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:16:01.560 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:16:31.699 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:16:31.702 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:17:01.762 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:17:01.765 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:17:31.910 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:17:31.913 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:18:02.040 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:18:02.043 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:18:32.155 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:18:32.158 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:19:02.285 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:19:02.288 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:19:32.383 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:19:32.386 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:20:02.531 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:20:02.534 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:20:32.618 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:20:32.622 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:21:02.959 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:21:02.962 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:21:33.016 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:21:33.020 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:22:03.137 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:22:03.140 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:22:33.210 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:22:33.213 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:23:03.315 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:23:03.318 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:23:33.472 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:23:33.475 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:24:03.608 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:24:03.611 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:24:33.740 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:24:33.744 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:25:04.087 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:25:04.091 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:25:34.159 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:25:34.164 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:26:04.318 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:26:04.322 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:26:34.420 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:26:34.424 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:27:04.544 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:27:04.547 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:27:34.675 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:27:34.678 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:28:04.734 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:28:04.737 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:28:34.869 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:28:34.872 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:29:04.999 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:29:05.003 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:29:35.123 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:29:35.126 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:30:05.189 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:30:05.253 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:30:35.367 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:30:35.370 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:31:05.490 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:31:05.493 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:31:35.621 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:31:35.624 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:32:05.724 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:32:05.727 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:32:35.841 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:32:35.845 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:33:05.930 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:33:05.933 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:33:36.186 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:33:36.189 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:34:06.241 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:34:06.244 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:34:36.324 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:34:36.328 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:35:06.386 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:35:06.390 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:35:36.451 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:35:36.455 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:36:06.524 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:36:06.527 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:36:36.652 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:36:36.655 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:37:06.706 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:37:06.709 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:37:36.871 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:37:36.874 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:38:06.975 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:38:06.978 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:38:37.029 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:38:37.032 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:39:07.084 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:39:07.088 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:39:37.148 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:39:37.151 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:40:07.204 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:40:07.207 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:40:37.299 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:40:37.302 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:41:07.357 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:41:07.361 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:41:37.417 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:41:37.420 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:42:07.486 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:42:07.489 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:42:37.600 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:42:37.603 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:43:07.657 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:43:07.660 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:43:37.796 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:43:37.799 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:44:07.934 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:44:07.937 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:44:38.050 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:44:38.053 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:45:08.177 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:45:08.180 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:45:38.303 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:45:38.306 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:46:08.446 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:46:08.449 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:46:38.505 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:46:38.515 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:47:08.572 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:47:08.576 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:47:38.628 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:47:38.631 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:48:08.682 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:48:08.685 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:48:38.830 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:48:38.834 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:49:08.959 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:49:08.962 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:49:39.114 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:49:39.117 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:50:09.202 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:50:09.205 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:50:39.349 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:50:39.353 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:51:09.422 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:51:09.425 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:51:39.551 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:51:39.554 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:52:09.681 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:52:09.684 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:52:39.803 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:52:39.806 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:53:09.874 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:53:09.877 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:53:40.030 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:53:40.033 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:54:10.092 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:54:10.096 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:54:40.165 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:54:40.168 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:55:10.285 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:55:10.289 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:55:40.414 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:55:40.417 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:56:10.570 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:56:10.574 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:56:40.692 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:56:40.696 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:57:10.822 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:57:10.825 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:57:40.935 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:57:40.939 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:58:11.082 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:58:11.085 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:58:41.220 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:58:41.223 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:59:11.316 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:59:11.319 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:59:41.463 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T14:59:41.467 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:00:11.543 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:00:11.546 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:00:41.706 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:00:41.710 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:01:11.852 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:01:11.855 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:01:41.950 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:01:41.953 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:02:12.012 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:02:12.016 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:02:42.084 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:02:42.087 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:03:12.147 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:03:12.151 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:03:42.227 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:03:42.230 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:04:12.366 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:04:12.369 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:04:42.470 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:04:42.474 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:05:12.617 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:05:12.620 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:05:42.777 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:05:42.781 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:06:12.927 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:06:12.931 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:06:43.048 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:06:43.051 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:07:13.112 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:07:13.116 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:07:43.167 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:07:43.170 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:08:13.222 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:08:13.225 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:08:43.302 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:08:43.305 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:09:13.432 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:09:13.436 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:09:43.570 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:09:43.574 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:10:13.700 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:10:13.704 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:10:43.805 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:10:43.808 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:11:13.944 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:11:13.947 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:11:44.090 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:11:44.093 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:12:14.220 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:12:14.223 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:12:44.318 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:12:44.322 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:13:14.473 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:13:14.476 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:13:44.610 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:13:44.614 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:14:14.717 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:14:14.720 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:14:44.871 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:14:44.874 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:15:14.993 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:15:14.997 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:15:45.199 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:15:45.202 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:16:15.253 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:16:15.256 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:16:45.384 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:16:45.388 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:17:15.447 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:17:15.451 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:17:45.585 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:17:45.588 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:18:15.716 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:18:15.719 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:18:46.045 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:18:46.048 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:19:16.108 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:19:16.111 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:19:46.179 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:19:46.182 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:20:16.247 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:20:16.251 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:20:46.317 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:20:46.321 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:21:16.380 DEBUG:teuthology.orchestra.run.smithi112:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:21:16.383 DEBUG:teuthology.orchestra.run.smithi142:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-01-24T15:21:44.694 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2023-01-24T15:21:44.699 DEBUG:teuthology.task.console_log:Killing console logger for smithi112 2023-01-24T15:21:44.700 DEBUG:teuthology.task.console_log:Killing console logger for smithi142 2023-01-24T15:21:44.700 DEBUG:teuthology.exit:Finished running handlers